Broken pipe (Write failed) when making Unauthorized request

[Adam Retter]

I am using the Maven httpclient 4.5.6 artifact and trying to make a PUT
request to a server. The server for some URLs requires authentication, it
is impossible to know which URLs do and don't ahead of time. I only want to
send the credentials if the server replies 401 Unauthorized.

I am trying to do a HTTP PUT to the server. If the server replies HTTP 401
Unauthorized, then I plan to retry with the authorization credentials.

I have found rather a strange issue, and I am not sure if the fault is with
the Apache HTTP Client or the server. However, I get different behaviour in
the Apache HTTP Client depending on how fast I can stream the data to the
server.

What I am seeing looks like this:

1. The Client opens a HTTP connection to the server
2. The Client starts sending data to the server
3. The Server starts receiving the data from the client, and determines
that the client is not authorized.
4. The Server responds HTTP 401 Unauthorized to the client
5. The server closes the connection.

At the client end when I only have a small amount of data to PUT, or use a
fast mechanism to write the data, all is fine and Apache HTTP Client gives
me the HTTP Response 401 from the server. All is good!

However, at the client end when I have a lot of data to PUT, or use a slow
mechanism to write the data, the Apache HTTP Client is still writting data
to the server when (5) occurs. The operating system then raises a "Broken
pipe (Write failed)" error, which Java relays to the Apache HTTP Client as
a SocketException. I then see errors like the following in my application:

Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Exception in thread "main" java.net.SocketException: Broken pipe (Write
failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at
org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at
org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
at
org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
at
org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:122)
at
org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:179)
at
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
at java.io.Writer.write(Writer.java:157)
at Issue2186.copySlow(Issue2186.java:44)
at Issue2186.lambda$0(Issue2186.java:26)
at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:73)
at
org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at
org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
at
org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at Issue2186.main(Issue2186.java:55)


So what is going on here? Is there something I am doing wrong on the client
side, or should the server be waiting to receive all the data from the
client before replying with the 401 Unauthorized response?


My simple code to reproduce the issue looks like:

public class Issue2186 {

private static final Path file = Paths.get("/tmp/mediumfile.xml");

private static HttpEntity getRequestBody() {
final ContentProducer producer = os -> {
// copyFast(file, os);

// NOTE: shows the broken pipe error
copySlow(file, os);
};
final EntityTemplate entityTemplate = new EntityTemplate(producer);
entityTemplate.setContentType("application/xml");
entityTemplate.setChunked(true);
return entityTemplate;
}

private static void copyFast(final Path file, final OutputStream os)
throws IOException {
Files.copy(file, os);
}

private static void copySlow(final Path file, final OutputStream os)
throws IOException {
try(final LineNumberReader reader = new
LineNumberReader(Files.newBufferedReader(file));
final OutputStreamWriter writer = new
OutputStreamWriter(new CloseShieldOutputStream(os), UTF_8)) {

String line;
while((line = reader.readLine()) != null) {
writer.write(line + "\n");
}
}
}

public static void main(final String args[]) throws IOException {
final CloseableHttpClient httpClient = HttpClients.createDefault();
try {
final HttpPut httpPut = new HttpPut("
http://localhost:8080/exist/rest/db/mediumfile.xml");
httpPut.setEntity(getRequestBody());

final CloseableHttpResponse httpResponse =
httpClient.execute(httpPut);
try {
System.out.println(httpResponse.getStatusLine());
} finally {
httpResponse.close();
}
} finally {
httpClient.close();
}
}
}


Thanks for your time.

--Adam Retter

eXist Core Developer
{ United Kingdom / United States }
***@exist-db.org

[Oleg Kalnichevski]

Post by Adam Retter
I am using the Maven httpclient 4.5.6 artifact and trying to make a PUT
request to a server. The server for some URLs requires
authentication, it
is impossible to know which URLs do and don't ahead of time. I only want to
send the credentials if the server replies 401 Unauthorized.
I am trying to do a HTTP PUT to the server. If the server replies HTTP 401
Unauthorized, then I plan to retry with the authorization
credentials.
I have found rather a strange issue, and I am not sure if the fault is with
the Apache HTTP Client or the server. However, I get different
behaviour in
the Apache HTTP Client depending on how fast I can stream the data to the
server.
1. The Client opens a HTTP connection to the server
2. The Client starts sending data to the server
3. The Server starts receiving the data from the client, and
determines
that the client is not authorized.
4. The Server responds HTTP 401 Unauthorized to the client
5. The server closes the connection.
At the client end when I only have a small amount of data to PUT, or use a
fast mechanism to write the data, all is fine and Apache HTTP Client gives
me the HTTP Response 401 from the server. All is good!
However, at the client end when I have a lot of data to PUT, or use a slow
mechanism to write the data, the Apache HTTP Client is still writting data
to the server when (5) occurs. The operating system then raises a "Broken
pipe (Write failed)" error, which Java relays to the Apache HTTP Client as
a SocketException. I then see errors like the following in my

This issue is known as out of sequence response. Such behavior is valid
according to the HTTP spec but is difficult to accommodate for when
using the classic (blocking) i/o model as blocking connections are not
able to write data and react to input events at the same time.

You have two options

1. Turn on 'expect-continue' handshake. It is intended precisely for
such situations.

2. Migrate to a non-blocking HTTP client such as Apache HttpAsyncClient
4.1 or 5.0 that can handle out of sequence responses.

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Adam Retter]

Post by Oleg Kalnichevski
1. Turn on 'expect-continue' handshake. It is intended precisely for
such situations.

So I assume that for this I just set the HTTP Header, like so:

httpPut.setHeader("Expect", "100-continue");


If so, unfortunately that is not helping the request from the client now
looks like:

PUT /exist/rest/db/mediumfile.xml HTTP/1.1
Expect: 100-continue
Transfer-Encoding: chunked
Content-Type: application/xml
Host: localhost:8080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.6 (Java/1.8.0_162)
Accept-Encoding: gzip,deflate

The server almost immediately responds with

HTTP/1.1 100 Continue

The client then starts sending data, during which, the server responds with:

HTTP/1.1 401 Unauthorized
Date: Thu, 11 Oct 2018 10:37:39 GMT
WWW-Authenticate: Basic realm="exist"
Content-Length: 0
Server: Jetty(9.4.10.v20180503)

The server then closes the connection. I then get the same Broken pipe
(Write failed) exceptions back from the Apache HTTP Client.

Does this mean that the server is also not behaving correctly? i.e. because
it returns 100 before it checks the authorization stuff? or is that also
allowed by the spec?


2. Migrate to a non-blocking HTTP client such as Apache HttpAsyncClient

Post by Oleg Kalnichevski
4.1 or 5.0 that can handle out of sequence responses.

I will give this some thought...

[Oleg Kalnichevski]

Post by Adam Retter

Post by Oleg Kalnichevski
1. Turn on 'expect-continue' handshake. It is intended precisely for
such situations.

httpPut.setHeader("Expect", "100-continue");

There is a RequestConfig option for that.

Post by Adam Retter
If so, unfortunately that is not helping the request from the client now
PUT /exist/rest/db/mediumfile.xml HTTP/1.1
Expect: 100-continue
Transfer-Encoding: chunked
Content-Type: application/xml
Host: localhost:8080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.6 (Java/1.8.0_162)
Accept-Encoding: gzip,deflate
The server almost immediately responds with
HTTP/1.1 100 Continue
The client then starts sending data, during which, the server
HTTP/1.1 401 Unauthorized
Date: Thu, 11 Oct 2018 10:37:39 GMT
WWW-Authenticate: Basic realm="exist"
Content-Length: 0
Server: Jetty(9.4.10.v20180503)
The server then closes the connection. I then get the same Broken pipe
(Write failed) exceptions back from the Apache HTTP Client.
Does this mean that the server is also not behaving correctly? i.e. because
it returns 100 before it checks the authorization stuff? or is that also
allowed by the spec?

Unfortunately this is legal from the HTTP spec standpoint even though
this behavior is just plain silly.

I have pointed it out to the Jetty developers a long time ago but they
said they liked it that way.

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org