Cannot transmit a SWA response with a large attachment (HttpEntity can't chunk multipart)

[Dan Wlodarski]

To whom it may concern:

Issue: HttpClient API does not support chunking multipart HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA) server
are failing (specifically, Apache-based Java clients are throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in the
response is large (>2 MiB). Small attachments are transmitted without
throwing any exceptions. HTTP Content-Length header is correct.

Use case: Multithreaded SOAP server capable of transmitting SWA responses of
arbitrary content length

Libraries: HttpCore v4.4.6 + HttpMime v4.5.3

Example server source for error replication is available at:
https://pastebin.com/2Hzdhpt3

Error replication:
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3 libraries
(HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary file named
"random.png.gz" with correct pathing and permissions (a readable directory
sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-party
client. Note the failure to receive the large server-generated multipart
result.

NB: SoapUI can be leveraged as an Apache-based Java client with precision
logging.

Please advise.

Thanks,

Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324

Phone: [phone] x175
Fax: [phone]
***@tdkc.com
www.tdkc.com

P.S. This issue was originally submitted on 30 May, but did not appear in the archives. Assumed dropped.


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Oleg Kalnichevski]

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart
HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA) server
are failing (specifically, Apache-based Java clients are throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in the
response is large (>2 MiB). Small attachments are transmitted without
throwing any exceptions. HTTP Content-Length header is correct.
Use case: Multithreaded SOAP server capable of transmitting SWA responses of
arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3 libraries
(HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary file named
"random.png.gz" with correct pathing and permissions (a readable directory
sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party
client. Note the failure to receive the large server-generated
multipart
result.
NB: SoapUI can be leveraged as an Apache-based Java client with precision
logging.
Please advise.

Dan,

You are mixing up non-blocking i/o transport and a blocking i/o entity
implementation. This is generally a bad idea as it cannot be done
efficiently without full or partial content buffering in memory.

If do not mind buffering the entire entity in memory, you can replace 
---
response.setEntity(responseEntity);
---

with

---
final ByteArrayOutputStream out = new ByteArrayOutputStream();
responseEntity.writeTo(out);
response.setEntity(new ByteArrayEntity(out.toByteArray()));
---

and your code will work just fine.

Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not
appear in the archives. Assumed dropped.
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Dan Wlodarski]

I've tried that solution previously. The result is a missing initial HTTP
header. The transmission begins at the MIME multipart border.

Pastebin source (https://pastebin.com/2Hzdhpt3) has been updated for further
error replication.

-----Original Message-----
From: Oleg Kalnichevski [mailto:***@apache.org]
Sent: Monday, June 5, 2017 07:14
To: HttpClient User Discussion <httpclient-***@hc.apache.org>
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart
HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA)
server are failing (specifically, Apache-based Java clients are
throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in the
response is large (>2 MiB). Small attachments are transmitted without
throwing any exceptions. HTTP Content-Length header is correct.
Use case: Multithreaded SOAP server capable of transmitting SWA
responses of arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3
libraries (HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary file
named "random.png.gz" with correct pathing and permissions (a readable
directory sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party client. Note the failure to receive the large server-generated
multipart result.
NB: SoapUI can be leveraged as an Apache-based Java client with
precision logging.
Please advise.

Dan,

You are mixing up non-blocking i/o transport and a blocking i/o entity
implementation. This is generally a bad idea as it cannot be done efficiently
without full or partial content buffering in memory.

If do not mind buffering the entire entity in memory, you can replace
---
response.setEntity(responseEntity);
---

with

---
final ByteArrayOutputStream out = new ByteArrayOutputStream();
responseEntity.writeTo(out); response.setEntity(new
ByteArrayEntity(out.toByteArray()));
---

and your code will work just fine.

Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not appear
in the archives. Assumed dropped.
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Oleg Kalnichevski]

Post by Dan Wlodarski
I've tried that solution previously. The result is a missing initial
HTTP 
header. The transmission begins at the MIME multipart border.

I see no evidence supporting this assertion.

Oleg

---
***@ok2c:~$ curl -X POST http://localhost:8080/ -v
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)

Post by Dan Wlodarski
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.52.1
Accept: */*

< HTTP/1.1 200 OK
< Date: Mon, 05 Jun 2017 14:13:23 GMT
< Server: Test/1.1
< Content-Length: 678
< Content-Type: multipart/form-data; boundary=PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
<
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="SOAP Envelope"
Content-Type: application/xml; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">;
<soapenv:Header/>
<soapenv:Body>
<AResponse>
<ResponseFile>/home/oleg/blah.txt</ResponseFile>
</AResponse>
</soapenv:Body>
</soapenv:Envelope>
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="/home/oleg/blah.txt"; filename="blah.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

blah

--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23--
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
---

Post by Dan Wlodarski
Pastebin source (https://pastebin.com/2Hzdhpt3) has been updated for
further 
error replication.
-----Original Message-----
Sent: Monday, June 5, 2017 07:14
Subject: Re: Cannot transmit a SWA response with a large attachment 
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart
HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA)
server are failing (specifically, Apache-based Java clients are
throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in the
response is large (>2 MiB). Small attachments are transmitted without
throwing any exceptions. HTTP Content-Length header is correct.
Use case: Multithreaded SOAP server capable of transmitting SWA
responses of arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3
libraries (HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary file
named "random.png.gz" with correct pathing and permissions (a readable
directory sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party client. Note the failure to receive the large server-
generated
multipart result.
NB: SoapUI can be leveraged as an Apache-based Java client with
precision logging.
Please advise.

Dan,
You are mixing up non-blocking i/o transport and a blocking i/o
entity 
implementation. This is generally a bad idea as it cannot be done
efficiently 
without full or partial content buffering in memory.
If do not mind buffering the entire entity in memory, you can replace
---
response.setEntity(responseEntity);
---
with
---
final ByteArrayOutputStream out = new ByteArrayOutputStream(); 
responseEntity.writeTo(out); response.setEntity(new 
ByteArrayEntity(out.toByteArray()));
---
and your code will work just fine.
Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not appear
in the archives. Assumed dropped.
-----------------------------------------------------------------
----
g

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Dan Wlodarski]

Running the sample server code on Netbeans v8.1 with SoapUI v5.3.0 as the
client POSTing to http://localhost:8080/, I captured the following response:

HTTP/1.1 200 OK
Date: Mon, 05 Jun 2017 13:53:45 GMT
Server: Test/1.1
Content-Length: 1346189
Connection: Keep-Alive

--5jYQHJoJgT65jloprnB-7ULf3gwvUZ-omC
Content-Disposition: form-data; name="SOAP Envelope"
Content-Type: application/xml; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header/>
<soapenv:Body>
<AResponse>
<ResponseFile>random.png.gz</ResponseFile>
</AResponse>
</soapenv:Body>
</soapenv:Envelope>
--5jYQHJoJgT65jloprnB-7ULf3gwvUZ-omC
Content-Disposition: form-data; name="random.png.gz";
filename="random.png.gz"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
<snipped data>

--5jYQHJoJgT65jloprnB-7ULf3gwvUZ-omC--

Please note the missing Content-Type HTTP header.

The Content-Type header is also missing from the response in my own curl
v7.53.0 tests.

The additional testing and verification is appreciated.

-----Original Message-----
From: Oleg Kalnichevski [mailto:***@apache.org]
Sent: Monday, June 5, 2017 10:15
To: HttpClient User Discussion <httpclient-***@hc.apache.org>
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
I've tried that solution previously. The result is a missing initial
HTTP header. The transmission begins at the MIME multipart border.

I see no evidence supporting this assertion.

Oleg

---
***@ok2c:~$ curl -X POST http://localhost:8080/ -v
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)

Post by Dan Wlodarski
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.52.1
Accept: */*

< HTTP/1.1 200 OK
< Date: Mon, 05 Jun 2017 14:13:23 GMT
< Server: Test/1.1
< Content-Length: 678
< Content-Type: multipart/form-data;
boundary=PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
<
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="SOAP Envelope"
Content-Type: application/xml; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">;
<soapenv:Header/>
<soapenv:Body>
<AResponse>
<ResponseFile>/home/oleg/blah.txt</ResponseFile>
</AResponse>
</soapenv:Body>
</soapenv:Envelope>
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="/home/oleg/blah.txt";
filename="blah.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

blah

--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23--
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
---

Post by Dan Wlodarski
Pastebin source (https://pastebin.com/2Hzdhpt3) has been updated for
further error replication.
-----Original Message-----
Sent: Monday, June 5, 2017 07:14
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart
HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA)
server are failing (specifically, Apache-based Java clients are
throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in the
response is large (>2 MiB). Small attachments are transmitted
without throwing any exceptions. HTTP Content-Length header is
correct.
Use case: Multithreaded SOAP server capable of transmitting SWA
responses of arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3
libraries (HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary file
named "random.png.gz" with correct pathing and permissions (a
readable directory sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party client. Note the failure to receive the large server-
generated multipart result.
NB: SoapUI can be leveraged as an Apache-based Java client with
precision logging.
Please advise.

Dan,
You are mixing up non-blocking i/o transport and a blocking i/o entity
implementation. This is generally a bad idea as it cannot be done
efficiently without full or partial content buffering in memory.
If do not mind buffering the entire entity in memory, you can replace
---
response.setEntity(responseEntity);
---
with
---
final ByteArrayOutputStream out = new ByteArrayOutputStream();
responseEntity.writeTo(out); response.setEntity(new
ByteArrayEntity(out.toByteArray()));
---
and your code will work just fine.
Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not
appear in the archives. Assumed dropped.
-----------------------------------------------------------------
----
g

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Oleg Kalnichevski]

Post by Dan Wlodarski
Running the sample server code on Netbeans v8.1 with SoapUI v5.3.0 as

...

Post by Dan Wlodarski
Please note the missing Content-Type HTTP header.

Did you set a Content-Type header on the response entity?

Oleg

Post by Dan Wlodarski
The Content-Type header is also missing from the response in my own
curl 
v7.53.0 tests.
The additional testing and verification is appreciated.
-----Original Message-----
Sent: Monday, June 5, 2017 10:15
Subject: Re: Cannot transmit a SWA response with a large attachment 
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
I've tried that solution previously. The result is a missing
initial
HTTP header. The transmission begins at the MIME multipart border.

I see no evidence supporting this assertion.
Oleg
---
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)

Post by Dan Wlodarski
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.52.1
Accept: */*

< HTTP/1.1 200 OK
< Date: Mon, 05 Jun 2017 14:13:23 GMT
< Server: Test/1.1
< Content-Length: 678
< Content-Type: multipart/form-data; 
boundary=PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
<
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="SOAP Envelope"
Content-Type: application/xml; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/enve
lope/">;;
   <soapenv:Header/>
   <soapenv:Body>
      <AResponse>
         <ResponseFile>/home/oleg/blah.txt</ResponseFile>
      </AResponse>
   </soapenv:Body>
</soapenv:Envelope>
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="/home/oleg/blah.txt"; 
filename="blah.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
blah
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23--
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
---

Post by Dan Wlodarski
Pastebin source (https://pastebin.com/2Hzdhpt3) has been updated for
further error replication.
-----Original Message-----
Sent: Monday, June 5, 2017 07:14
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart
HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA)
server are failing (specifically, Apache-based Java clients are
throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in the
response is large (>2 MiB). Small attachments are transmitted
without throwing any exceptions. HTTP Content-Length header is
correct.
Use case: Multithreaded SOAP server capable of transmitting SWA
responses of arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3
libraries (HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary file
named "random.png.gz" with correct pathing and permissions (a
readable directory sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party client. Note the failure to receive the large server-
generated multipart result.
NB: SoapUI can be leveraged as an Apache-based Java client with
precision logging.
Please advise.

Dan,
You are mixing up non-blocking i/o transport and a blocking i/o entity
implementation. This is generally a bad idea as it cannot be done
efficiently without full or partial content buffering in memory.
If do not mind buffering the entire entity in memory, you can replace
---
response.setEntity(responseEntity);
---
with
---
final ByteArrayOutputStream out = new ByteArrayOutputStream();
responseEntity.writeTo(out); response.setEntity(new
ByteArrayEntity(out.toByteArray()));
---
and your code will work just fine.
Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not
appear in the archives. Assumed dropped.
-----------------------------------------------------------------
----
g
or
g

-----------------------------------------------------------------
----
g

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Dan Wlodarski]

The source I ran was as provided in the updated Pastebin:
https://pastebin.com/2Hzdhpt3

How would I go about setting the content type of an HttpEntity instance? The
interface doesn't include any mutators in its contract.

The getContentType() method of the HttpEntity instance built by
MultipartEntityBuilder does indeed return a string (ex. "Content-Type:
multipart/form-data; boundary=rfNtJUy3v1e0XsLoZmxsYhXSU8k9LohLG"), but the
Content-Type header doesn't get included in the response.

-----Original Message-----
From: Oleg Kalnichevski [mailto:***@apache.org]
Sent: Monday, June 5, 2017 10:58
To: HttpClient User Discussion <httpclient-***@hc.apache.org>
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Running the sample server code on Netbeans v8.1 with SoapUI v5.3.0 as

...

Post by Dan Wlodarski
Please note the missing Content-Type HTTP header.

Did you set a Content-Type header on the response entity?

Oleg

Post by Dan Wlodarski
The Content-Type header is also missing from the response in my own curl
v7.53.0 tests.
The additional testing and verification is appreciated.
-----Original Message-----
Sent: Monday, June 5, 2017 10:15
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
I've tried that solution previously. The result is a missing initial
HTTP header. The transmission begins at the MIME multipart border.

I see no evidence supporting this assertion.
Oleg
---
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)

Post by Dan Wlodarski
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.52.1
Accept: */*

< HTTP/1.1 200 OK
< Date: Mon, 05 Jun 2017 14:13:23 GMT
< Server: Test/1.1
< Content-Length: 678
< Content-Type: multipart/form-data;
boundary=PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
<
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="SOAP Envelope"
Content-Type: application/xml; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/enve
lope/">;;
<soapenv:Header/>
<soapenv:Body>
<AResponse>
<ResponseFile>/home/oleg/blah.txt</ResponseFile>
</AResponse>
</soapenv:Body>
</soapenv:Envelope>
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="/home/oleg/blah.txt";
filename="blah.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
blah
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23--
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
---

Post by Dan Wlodarski
Pastebin source (https://pastebin.com/2Hzdhpt3) has been updated for
further error replication.
-----Original Message-----
Sent: Monday, June 5, 2017 07:14
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart
HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA)
server are failing (specifically, Apache-based Java clients are
throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in the
response is large (>2 MiB). Small attachments are transmitted
without throwing any exceptions. HTTP Content-Length header is
correct.
Use case: Multithreaded SOAP server capable of transmitting SWA
responses of arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3
libraries (HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary file
named "random.png.gz" with correct pathing and permissions (a
readable directory sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party client. Note the failure to receive the large server-
generated multipart result.
NB: SoapUI can be leveraged as an Apache-based Java client with
precision logging.
Please advise.

Dan,
You are mixing up non-blocking i/o transport and a blocking i/o entity
implementation. This is generally a bad idea as it cannot be done
efficiently without full or partial content buffering in memory.
If do not mind buffering the entire entity in memory, you can replace
---
response.setEntity(responseEntity);
---
with
---
final ByteArrayOutputStream out = new ByteArrayOutputStream();
responseEntity.writeTo(out); response.setEntity(new
ByteArrayEntity(out.toByteArray()));
---
and your code will work just fine.
Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not
appear in the archives. Assumed dropped.
-----------------------------------------------------------------
----
g
or
g

-----------------------------------------------------------------
----
g

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Oleg Kalnichevski]

The source I ran was as provided in the updated Pastebin: 
https://pastebin.com/2Hzdhpt3
How would I go about setting the content type of an HttpEntity
instance?

Something like that?

---
try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
responseEntity.writeTo(out);
response.setEntity(new ByteArrayEntity(out.toByteArray(), ContentType.get(responseEntity)));
}
---

Oleg

The 
interface doesn't include any mutators in its contract.
The getContentType() method of the HttpEntity instance built by 
MultipartEntityBuilder does indeed return a string (ex. "Content-
Type: 
multipart/form-data; boundary=rfNtJUy3v1e0XsLoZmxsYhXSU8k9LohLG"),
but the 
Content-Type header doesn't get included in the response.
-----Original Message-----
Sent: Monday, June 5, 2017 10:58
Subject: Re: Cannot transmit a SWA response with a large attachment 
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Running the sample server code on Netbeans v8.1 with SoapUI v5.3.0 as

...

Post by Dan Wlodarski
Please note the missing Content-Type HTTP header.

Did you set a Content-Type header on the response entity?
Oleg

Post by Dan Wlodarski
The Content-Type header is also missing from the response in my own curl
v7.53.0 tests.
The additional testing and verification is appreciated.
-----Original Message-----
Sent: Monday, June 5, 2017 10:15
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
I've tried that solution previously. The result is a missing initial
HTTP header. The transmission begins at the MIME multipart
border.

I see no evidence supporting this assertion.
Oleg
---
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)

Post by Dan Wlodarski
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.52.1
Accept: */*

< HTTP/1.1 200 OK
< Date: Mon, 05 Jun 2017 14:13:23 GMT
< Server: Test/1.1
< Content-Length: 678
< Content-Type: multipart/form-data;
boundary=PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
<
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="SOAP Envelope"
Content-Type: application/xml; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/en
ve
lope/">;;
   <soapenv:Header/>
   <soapenv:Body>
      <AResponse>
         <ResponseFile>/home/oleg/blah.txt</ResponseFile>
      </AResponse>
   </soapenv:Body>
</soapenv:Envelope>
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="/home/oleg/blah.txt";
filename="blah.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
blah
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23--
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
---

Post by Dan Wlodarski
Pastebin source (https://pastebin.com/2Hzdhpt3) has been updated for
further error replication.
-----Original Message-----
Sent: Monday, June 5, 2017 07:14
Subject: Re: Cannot transmit a SWA response with a large
attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA)
server are failing (specifically, Apache-based Java clients are
throwing
org.apache.http.NoHttpResponseExceptions) when the attachment
in
the
response is large (>2 MiB). Small attachments are transmitted
without throwing any exceptions. HTTP Content-Length header is
correct.
Use case: Multithreaded SOAP server capable of transmitting SWA
responses of arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3
libraries (HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary file
named "random.png.gz" with correct pathing and permissions (a
readable directory sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party client. Note the failure to receive the large server-
generated multipart result.
NB: SoapUI can be leveraged as an Apache-based Java client with
precision logging.
Please advise.

Dan,
You are mixing up non-blocking i/o transport and a blocking i/o entity
implementation. This is generally a bad idea as it cannot be done
efficiently without full or partial content buffering in memory.
If do not mind buffering the entire entity in memory, you can replace
---
response.setEntity(responseEntity);
---
with
---
final ByteArrayOutputStream out = new ByteArrayOutputStream();
responseEntity.writeTo(out); response.setEntity(new
ByteArrayEntity(out.toByteArray()));
---
and your code will work just fine.
Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not
appear in the archives. Assumed dropped.
-------------------------------------------------------------
----
----
or
g
e.
or
g

-----------------------------------------------------------------
----
g
or
g

-----------------------------------------------------------------
----
g

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Dan Wlodarski]

Adding the ContentType parameter to the setEntity(...) call does reintroduce
the Content-Type line to the primary HTTP header. SoapUI now successfully
receives the entire SWA message. Thanks for all your help, Oleg!

Question: what was the design decision that led to the Content-Type header
ever being omitted from any HTTP message?

Again, thanks,
Dan C. W.

-----Original Message-----
From: Oleg Kalnichevski [mailto:***@apache.org]
Sent: Monday, June 5, 2017 12:55
To: HttpClient User Discussion <httpclient-***@hc.apache.org>
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
https://pastebin.com/2Hzdhpt3
How would I go about setting the content type of an HttpEntity
instance?

Something like that?

---
try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
responseEntity.writeTo(out);
response.setEntity(new ByteArrayEntity(out.toByteArray(),
ContentType.get(responseEntity)));
}
---

Oleg

Post by Dan Wlodarski
The
interface doesn't include any mutators in its contract.
The getContentType() method of the HttpEntity instance built by
MultipartEntityBuilder does indeed return a string (ex. "Content-
multipart/form-data; boundary=rfNtJUy3v1e0XsLoZmxsYhXSU8k9LohLG"), but the
Content-Type header doesn't get included in the response.
-----Original Message-----
Sent: Monday, June 5, 2017 10:58
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Running the sample server code on Netbeans v8.1 with SoapUI v5.3.0 as

...

Post by Dan Wlodarski
Please note the missing Content-Type HTTP header.

Did you set a Content-Type header on the response entity?
Oleg

Post by Dan Wlodarski
The Content-Type header is also missing from the response in my own curl
v7.53.0 tests.
The additional testing and verification is appreciated.
-----Original Message-----
Sent: Monday, June 5, 2017 10:15
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
I've tried that solution previously. The result is a missing
initial HTTP header. The transmission begins at the MIME multipart
border.

I see no evidence supporting this assertion.
Oleg
---
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)

Post by Dan Wlodarski
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.52.1
Accept: */*

< HTTP/1.1 200 OK
< Date: Mon, 05 Jun 2017 14:13:23 GMT < Server: Test/1.1 <
Content-Length: 678 < Content-Type: multipart/form-data;
boundary=PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
<
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="SOAP Envelope"
Content-Type: application/xml; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/en
ve
lope/">;;
<soapenv:Header/>
<soapenv:Body>
<AResponse>
<ResponseFile>/home/oleg/blah.txt</ResponseFile>
</AResponse>
</soapenv:Body>
</soapenv:Envelope>
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="/home/oleg/blah.txt";
filename="blah.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
blah
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23--
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
---

Post by Dan Wlodarski
Pastebin source (https://pastebin.com/2Hzdhpt3) has been updated
for further error replication.
-----Original Message-----
Sent: Monday, June 5, 2017 07:14
Subject: Re: Cannot transmit a SWA response with a large
attachment (HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA)
server are failing (specifically, Apache-based Java clients are
throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in
the response is large (>2 MiB). Small attachments are
transmitted without throwing any exceptions. HTTP Content-Length
header is correct.
Use case: Multithreaded SOAP server capable of transmitting SWA
responses of arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3
libraries (HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary
file named "random.png.gz" with correct pathing and permissions
(a readable directory sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party client. Note the failure to receive the large server-
generated multipart result.
NB: SoapUI can be leveraged as an Apache-based Java client with
precision logging.
Please advise.

Dan,
You are mixing up non-blocking i/o transport and a blocking i/o
entity implementation. This is generally a bad idea as it cannot
be done efficiently without full or partial content buffering in
memory.
If do not mind buffering the entire entity in memory, you can replace
---
response.setEntity(responseEntity);
---
with
---
final ByteArrayOutputStream out = new ByteArrayOutputStream();
responseEntity.writeTo(out); response.setEntity(new
ByteArrayEntity(out.toByteArray()));
---
and your code will work just fine.
Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not
appear in the archives. Assumed dropped.
-------------------------------------------------------------
----
----
or
g
e.
or
g

-----------------------------------------------------------------
----
g
or
g

-----------------------------------------------------------------
----
g

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Oleg Kalnichevski]

Post by Dan Wlodarski
Adding the ContentType parameter to the setEntity(...) call does
reintroduce 
the Content-Type line to the primary HTTP header. SoapUI now
successfully 
receives the entire SWA message. Thanks for all your help, Oleg!
Question: what was the design decision that led to the Content-Type
header 
ever being omitted from any HTTP message?

What exactly do you mean? Some entity implementations automatically set
Content-Type when known but general purpose implementations such as
ByteArrayEntity or StringEntity expect Content-Type be set by the
caller.

Oleg

Post by Dan Wlodarski
Again, thanks,
Dan C. W.
-----Original Message-----
Sent: Monday, June 5, 2017 12:55
Subject: Re: Cannot transmit a SWA response with a large attachment 
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
https://pastebin.com/2Hzdhpt3
How would I go about setting the content type of an HttpEntity
instance?

Something like that?
---
try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
    responseEntity.writeTo(out);
    response.setEntity(new ByteArrayEntity(out.toByteArray(), 
ContentType.get(responseEntity)));
}
---
Oleg

Post by Dan Wlodarski
The
interface doesn't include any mutators in its contract.
The getContentType() method of the HttpEntity instance built by
MultipartEntityBuilder does indeed return a string (ex. "Content-
multipart/form-data; boundary=rfNtJUy3v1e0XsLoZmxsYhXSU8k9LohLG"), but the
Content-Type header doesn't get included in the response.
-----Original Message-----
Sent: Monday, June 5, 2017 10:58
Subject: Re: Cannot transmit a SWA response with a large attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Running the sample server code on Netbeans v8.1 with SoapUI
v5.3.0
as

...

Post by Dan Wlodarski
Please note the missing Content-Type HTTP header.

Did you set a Content-Type header on the response entity?
Oleg

Post by Dan Wlodarski
The Content-Type header is also missing from the response in my own
curl
v7.53.0 tests.
The additional testing and verification is appreciated.
-----Original Message-----
Sent: Monday, June 5, 2017 10:15
Subject: Re: Cannot transmit a SWA response with a large
attachment
(HttpEntity can't chunk multipart)

Post by Dan Wlodarski
I've tried that solution previously. The result is a missing
initial HTTP header. The transmission begins at the MIME
multipart
border.

I see no evidence supporting this assertion.
Oleg
---
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)

Post by Dan Wlodarski
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.52.1
Accept: */*

< HTTP/1.1 200 OK
< Date: Mon, 05 Jun 2017 14:13:23 GMT < Server: Test/1.1 <
Content-Length: 678 < Content-Type: multipart/form-data;
boundary=PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
<
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="SOAP Envelope"
Content-Type: application/xml; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/
en
ve
lope/">;;
   <soapenv:Header/>
   <soapenv:Body>
      <AResponse>
         <ResponseFile>/home/oleg/blah.txt</ResponseFile>
      </AResponse>
   </soapenv:Body>
</soapenv:Envelope>
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23
Content-Disposition: form-data; name="/home/oleg/blah.txt"; filename="blah.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
blah
--PLjXq6xrj2ONaDdnyMPLBEnpkMqZA9Wmjw23--
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
---

Post by Dan Wlodarski
Pastebin source (https://pastebin.com/2Hzdhpt3) has been
updated
for further error replication.
-----Original Message-----
Sent: Monday, June 5, 2017 07:14
Subject: Re: Cannot transmit a SWA response with a large
attachment (HttpEntity can't chunk multipart)

Post by Dan Wlodarski
Issue: HttpClient API does not support chunking multipart HttpEntities.
Clients making SOAP requests to a mock SOAP with attachments (SWA)
server are failing (specifically, Apache-based Java clients are
throwing
org.apache.http.NoHttpResponseExceptions) when the attachment in
the response is large (>2 MiB). Small attachments are
transmitted without throwing any exceptions. HTTP Content-
Length
header is correct.
Use case: Multithreaded SOAP server capable of transmitting SWA
responses of arbitrary content length
Libraries: HttpCore v4.4.6 + HttpMime v4.5.3
https://pastebin.com/2Hzdhpt3
1. Build the above source with HttpCore v4.4.6 and HttpMime v4.5.3
libraries (HttpMime is part of the HttpClient project).
2. Run the program with a sufficiently large (>2 MiB) binary
file named "random.png.gz" with correct pathing and
permissions
(a readable directory sibling of the executable).
3. Send the server an arbitrary HTTP POST request via some third-
party client. Note the failure to receive the large server-
generated multipart result.
NB: SoapUI can be leveraged as an Apache-based Java client with
precision logging.
Please advise.

Dan,
You are mixing up non-blocking i/o transport and a blocking i/o
entity implementation. This is generally a bad idea as it cannot
be done efficiently without full or partial content buffering in
memory.
If do not mind buffering the entire entity in memory, you can replace
---
response.setEntity(responseEntity);
---
with
---
final ByteArrayOutputStream out = new ByteArrayOutputStream();
responseEntity.writeTo(out); response.setEntity(new
ByteArrayEntity(out.toByteArray()));
---
and your code will work just fine.
Oleg

Post by Dan Wlodarski
Thanks,
Dan C. Wlodarski
The Design Knowledge Company
3100 Presidential Drive
Suite 103
Fairborn, Ohio 45324
Phone: [phone] x175
Fax: [phone]
www.tdkc.com
P.S. This issue was originally submitted on 30 May, but did not
appear in the archives. Assumed dropped.
-------------------------------------------------------------
----
----
e.
or
g
ch
e.
or
g

-------------------------------------------------------------
----
----
or
g
e.
or
g

-----------------------------------------------------------------
----
g
or
g

-----------------------------------------------------------------
----
g

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org