Associating a server push with the original client request

[Alex Osborne]

If I receive a server push is it possible to determine which client request it was sent in response to? I'm not actually after the data of the request just some way to map it back. Some opaque id, object, callback or whatever I can supply with the request and later get back when a push comes in would do just fine.

I figured there might be a reference to original context in the push request's context, but unfortunately AsyncPushConsumer doesn't receive the HttpContext and even if I get at it by backtracking up the stack in a debugger it seems to be a fresh context with no reference to the original one.

Does that mean the only way to associate a server push with the original client request is to create a new HttpClient for each request? (As then there'd be only be one request for each client.) That wouldn't be so bad if they could share resources. I see a promising looking setConnectionManagerShared() but I can't see a way to get two HttpClients to use the same IOReactor and so would end up with a lot more threads than I really want.

Thanks,

Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Oleg Kalnichevski]

Post by Alex Osborne
If I receive a server push is it possible to determine which client
request it was sent in response to?

Hi Alex

Not unless the server is willing to help with that.

Why do not you add a correlation id of a sort to the promised request
from AsyncPushProducer?

Oleg

Post by Alex Osborne
I'm not actually after the data of the request just some way to map
it back. Some opaque id, object, callback or whatever I can supply
with the request and later get back when a push comes in would do
just fine.
I figured there might be a reference to original context in the push
request's context, but unfortunately AsyncPushConsumer doesn't
receive the HttpContext and even if I get at it by backtracking up
the stack in a debugger it seems to be a fresh context with no
reference to the original one.
Does that mean the only way to associate a server push with the
original client request is to create a new HttpClient for each
request? (As then there'd be only be one request for each client.)
That wouldn't be so bad if they could share resources. I see a
promising looking setConnectionManagerShared() but I can't see a way
to get two HttpClients to use the same IOReactor and so would end up
with a lot more threads than I really want.
Thanks,
Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Alex Osborne]

Post by Oleg Kalnichevski

Post by Alex Osborne
If I receive a server push is it possible to determine which client
request it was sent in response to?

Not unless the server is willing to help with that.
Why do not you add a correlation id of a sort to the promised request
from AsyncPushProducer?

Here's a simplified example of what I want to do. I want to make two requests each having its own directory and save all the corresponding responses (both the main response and any push responses) to the appropriate directory.

client.register("*", () -> new FileSavingDataConsumer("/tmp/request??/")));
client.execute(request1, new FileSavingDataConsumer("/tmp/request1/"), null);
client.execute(request2, new FileSavingDataConsumer("/tmp/request2/"), null);

While I could certainly put a counter in the AsyncPushProducer and mint ids I have no guarantee it'd be called in any particular order so I still wouldn't know which was which. As request1 and request2 are executing concurrently I believe they may produce pushes in any order, possibly even interleaved.

I'm pretty sure this is possible in the protocol as RFC 7540 says:

Pushed responses are always associated with an explicit request from
the client. The PUSH_PROMISE frames sent by the server are sent on
that explicit request's stream.

I don't think it's possible in the current implementation though as AbstractHttp2StreamMultiplexer doesn't seem to do anything with the explicit request's stream other than checking it hasn't been closed.

So I guess I need to figure out how to modify AbstractHttp2StreamMultiplexer to plumb through the information.

Cheers,

Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Oleg Kalnichevski]

Post by Alex Osborne

Post by Oleg Kalnichevski

Post by Alex Osborne
If I receive a server push is it possible to determine which client
request it was sent in response to? 

Not unless the server is willing to help with that.
Why do not you add a correlation id of a sort to the promised request
from AsyncPushProducer?

Here's a simplified example of what I want to do. I want to make two
requests each having its own directory and save all the corresponding
responses (both the main response and any push responses) to the
appropriate directory.
client.register("*", () -> new
FileSavingDataConsumer("/tmp/request??/")));
client.execute(request1, new
FileSavingDataConsumer("/tmp/request1/"), null);
client.execute(request2,  new
FileSavingDataConsumer("/tmp/request2/"), null);
While I could certainly put a counter in the AsyncPushProducer and
mint ids I have no guarantee it'd be called in any particular order
so I still wouldn't know which was which. As request1 and request2
are executing concurrently I believe they may produce pushes in any
order, possibly even interleaved.

Given the example you can clearly use the original request request URI
as a correlation id.

Post by Alex Osborne
   Pushed responses are always associated with an explicit request
from
   the client.  The PUSH_PROMISE frames sent by the server are sent
on
   that explicit request's stream.

I am sure it is possible. Please do refer me to a section of the
specification that states _how_ _exactly_ those messages are associated
other than through the promised request that server pushes to the
client.

Post by Alex Osborne
I don't think it's possible in the current implementation though as
AbstractHttp2StreamMultiplexer doesn't seem to do anything with the
explicit request's stream other than checking it hasn't been closed.
So I guess I need to figure out how to modify
AbstractHttp2StreamMultiplexer to plumb through the information.

We happily take patches. Will be more than happy to commit a change-set
that makes correlation of push response messages easier.

Oleg

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org

[Alex Osborne]

Post by Oleg Kalnichevski

Post by Alex Osborne
Here's a simplified example of what I want to do. I want to make two
requests each having its own directory and save all the corresponding
responses (both the main response and any push responses) to the
appropriate directory.
client.register("*", () -> new
FileSavingDataConsumer("/tmp/request??/")));
client.execute(request1, new
FileSavingDataConsumer("/tmp/request1/"), null);
client.execute(request2,  new
FileSavingDataConsumer("/tmp/request2/"), null);

Given the example you can clearly use the original request request URI
as a correlation id.

Sorry if I'm missing something obvious. I still don't see how to do it. :-(

Say request1 is for GET /page1.html and for that page the server pushes GET /cat.jpg.
Say request2 is for GET /page2.html and for that page the server pushes GET /dog.jpg.

I'm sending the requests to an arbitrary server I have no control over and I don't know in advance which resources its going push so I can't client.register("/cat.jpg") specifcially.

The Supplier<AsyncPushConsumer> register receives no arguments when it's called so it doesn't know the appropriate original request URI.
The AsyncPushConsumer it returns receives:

HttpRequest[GET /cat.jpg], HttpResponse[200 image/jpeg]
HttpRequest[GET /dog.jpg], HttpResponse[200 image/jpeg]

So there's still no association with the original request URIs /page1.html and /page2.html.

Post by Oleg Kalnichevski
I am sure it is possible. Please do refer me to a section of the
specification that states _how_ _exactly_ those messages are associated
other than through the promised request that server pushes to the
client.

Expacing the quote from section 8.2.1:

Pushed responses are always associated with an explicit request from
the client. The PUSH_PROMISE frames sent by the server are sent on
that explicit request's stream. The PUSH_PROMISE frame also includes
a promised stream identifier, chosen from the stream identifiers
available to the server

The PUSH_PROMISE frame is what associates them. It's sent on the stream the client sent the original request on and includes the id of a new stream the pushed response will be sent on.

[stream 1] Client: GET /page1.html, END_STREAM (half-closes)
[stream 3] Client: GET /page2.html, END_STREAM (half-closes)
[stream 1] Server: PUSH_PROMISE /cat.jpg promising stream 2
[stream 3] Server: PUSH_PROMISE /dog.jpg promising stream 4

Then interleaved:
[stream 1] Server: the response HEADERS, DATA for page1.html
[stream 2] Server: the response HEADERS, DATA for cat.jpg
[stream 3] Server: the response HEADERS, DATA for page2.html
[stream 4] Server: the response HEADERS, DATA for dog.jpg

Post by Oleg Kalnichevski
We happily take patches. Will be more than happy to commit a change-set
that makes correlation of push response messages easier.

Thanks. I'll attempt to do so and send a patch if I get it working. :-)

Alex

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-***@hc.apache.org
For additional commands, e-mail: httpclient-users-***@hc.apache.org