H2StreamResetExceptions and TimeoutExceptions seen when communicating with Apple's APNs service using HTTP/2

[Jack van Ooststroom]

Recently we switched from the Apple's legacy Binary Provider API to the
newer HTTP/2 based API in order to send our notifications using APNs. We
decided on using Apache's HttpClient 5 in order to meet the HTTP/2
requirement. However, our current support for this seems unstable as we
are seeing the occasional H2StreamResetExceptions and TimeoutExceptions.

We are currently creating the HttpAsyncClient as follows:

                    HttpAsyncClients.custom().
setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_2).
                        setConnectionManager(
PoolingAsyncClientConnectionManagerBuilder.create().
                                setTlsStrategy(
                                    new H2TlsStrategy(
                                        SSLContexts.custom().
loadKeyMaterial(_keyStore, _keyStorePassword).
                                            build(),
                                        new String[] {"TLSv1.2"},
null, null, new DefaultHostnameVerifier()
                                    )
                                ).
                                build()
                        ).
                        build()

This HttpAsyncClient is shared among a configurable amount of threads
with each thread having its own HttpClientContext instance. Each thread
is sending our HTTP/2 requests as follows:

        Future<SimpleHttpResponse> _future =
getHTTPClient().execute(_request, getHTTPContext(), null);
        try {
            SimpleHttpResponse _response = _future.get(3,
TimeUnit.MINUTES);
            ...
        } catch (final CancellationException exception) { // If the
computation was cancelled.
            ...
        } catch (final ExecutionException exception) { // If the
computation threw an exception.
            ...
        } catch (final InterruptedException exception) { // If the
current thread was interrupted while waiting.
            ...
        } catch (final TimeoutException exception) { // If the wait
timed out.
            ...
        }

On occasion we see and log the H2StreamResetException, wrapped inside an
ExecutionException, for now. The details of this exception is as follows:

java.util.concurrent.ExecutionException:
org.apache.hc.core5.http2.H2StreamResetException: Connection
terminated by the peer
        at
org.apache.hc.core5.concurrent.BasicFuture.getResult(BasicFuture.java:71)
        at
org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:102)
        ...
Caused by: org.apache.hc.core5.http2.H2StreamResetException:
Connection terminated by the peer
        at
org.apache.hc.core5.http2.impl.nio.AbstractHttp2StreamMultiplexer.consumeFrame(AbstractHttp2StreamMultiplexer.java:974)
        at
org.apache.hc.core5.http2.impl.nio.AbstractHttp2StreamMultiplexer.onInput(AbstractHttp2StreamMultiplexer.java:419)
        at
org.apache.hc.core5.http2.impl.nio.AbstractHttp2IOEventHandler.inputReady(AbstractHttp2IOEventHandler.java:63)
        at
org.apache.hc.core5.http2.impl.nio.ClientHttp2IOEventHandler.inputReady(ClientHttp2IOEventHandler.java:38)
        at
org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:117)
        at
org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:50)
        at
org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:173)
        at
org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:123)
        at
org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:80)
        at
org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
        ... 1 more

As we are using a PoolingAsyncClientConnectionManager and this exception
is indicating "Connection terminated by the peer", is there something we
need to do on our end in order to deal with this exception and to
recover from it?

Also on occassion we see and log the TimeoutException for now. The
details of this exception is as follows:

java.util.concurrent.TimeoutException
        at
org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:106)
        ...

Why would this happen on occasion? Is this something caused by the
HttpAsyncClient or possible by the APNs service?

[Oleg Kalnichevski]

Post by Jack van Ooststroom
Recently we switched from the Apple's legacy Binary Provider API to
the 
newer HTTP/2 based API in order to send our notifications using APNs.
We 
decided on using Apache's HttpClient 5 in order to meet the HTTP/2 
requirement. However, our current support for this seems unstable as
we 
are seeing the occasional H2StreamResetExceptions and
TimeoutExceptions.
                         HttpAsyncClients.custom().
    setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_2).
                             setConnectionManager(
    PoolingAsyncClientConnectionManagerBuilder.create().
                                     setTlsStrategy(
                                         new H2TlsStrategy(
                                             SSLContexts.custom().
    loadKeyMaterial(_keyStore, _keyStorePassword).
                                                 build(),
                                             new String[]
{"TLSv1.2"},
    null, null, new DefaultHostnameVerifier()
                                         )
                                     ).
                                     build()
                             ).
                             build()
This HttpAsyncClient is shared among a configurable amount of
threads 
with each thread having its own HttpClientContext instance. Each
thread 
             Future<SimpleHttpResponse> _future =
    getHTTPClient().execute(_request, getHTTPContext(), null);
             try {
                 SimpleHttpResponse _response = _future.get(3,
    TimeUnit.MINUTES);
                 ...
             } catch (final CancellationException exception) { // If
the
    computation was cancelled.
                 ...
             } catch (final ExecutionException exception) { // If the
    computation threw an exception.
                 ...
             } catch (final InterruptedException exception) { // If
the
    current thread was interrupted while waiting.
                 ...
             } catch (final TimeoutException exception) { // If the
wait
    timed out.
                 ...
             }
On occasion we see and log the H2StreamResetException, wrapped inside
an 
    org.apache.hc.core5.http2.H2StreamResetException: Connection
    terminated by the peer
             at
    org.apache.hc.core5.concurrent.BasicFuture.getResult(BasicFuture.
java:71)
             at
    org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:1
02)
             ...
    Connection terminated by the peer
             at
    org.apache.hc.core5.http2.impl.nio.AbstractHttp2StreamMultiplexer
.consumeFrame(AbstractHttp2StreamMultiplexer.java:974)
             at
    org.apache.hc.core5.http2.impl.nio.AbstractHttp2StreamMultiplexer
.onInput(AbstractHttp2StreamMultiplexer.java:419)
             at
    org.apache.hc.core5.http2.impl.nio.AbstractHttp2IOEventHandler.in
putReady(AbstractHttp2IOEventHandler.java:63)
             at
    org.apache.hc.core5.http2.impl.nio.ClientHttp2IOEventHandler.inpu
tReady(ClientHttp2IOEventHandler.java:38)
             at
    org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(Interna
lDataChannel.java:117)
             at
    org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(Interna
lChannel.java:50)
             at
    org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(Sin
gleCoreIOReactor.java:173)
             at
    org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleC
oreIOReactor.java:123)
             at
    org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(A
bstractSingleCoreIOReactor.java:80)
             at
    org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.j
ava:44)
             ... 1 more

Please note that HttpClient signals the fact that the connection has
been terminated by the opposite endpoint. By itself this does not mean
there is anything wrong with the client side. H2StreamResetException
includes an error code that would help understand the reason for
connection termination.

Post by Jack van Ooststroom
As we are using a PoolingAsyncClientConnectionManager and this
exception 
is indicating "Connection terminated by the peer", is there something
we 
need to do on our end in order to deal with this exception and to 
recover from it?

Depending on the error code and the request method it might be
sufficient to re-try the request.

Post by Jack van Ooststroom
Also on occassion we see and log the TimeoutException for now. The 
    java.util.concurrent.TimeoutException
             at
    org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:1
06)
             ...
Why would this happen on occasion? Is this something caused by the 
HttpAsyncClient or possible by the APNs service?

It is impossible to say without seeing the complete HTTP/2 session.

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org