Bug 17682 – XHR setRequestHeader("connection", "close") is bogusly rejected

• Component: XML
• Status: RESOLVED WONTFIX

[Darin Fisher (:fishd, Google)]

XHR setRequestHeader("connection", "close") is bogusly rejected

This is used by Gmail for example to request a non-persistent connection.  This is a standard mechanism used by web apps, and the intent is to not have the XHR connection not count against the browser's limit on the maximum number of persistent connections per host that it will open at any given time.

It looks like Gmail uses this for a request that does not have an immediate response.  The server only responds to the request when it has some data to send to the client.  As a result, the connection is open for a long time, and therefore it does not make any sense to count it as a persistent connection.

Other browsers (FF2 at least) do not prevent the "connection: close" header from being set.

I see that http://www.w3.org/TR/XMLHttpRequest/ recommends blocking the "connection" request header, but I think it is a very poor recommendation.

[Alexey Proskuryakov]

Are you sure this helps to avoid the running into the two-connection limit? I would expect the network layer of any browser to close idle connections when new requests are made.

[Darin Fisher (:fishd, Google)]

I wasn't referring to idle connections.....

Imagine you have 3 requests to host A.  If all requests take a long time to complete, then one of the requests will be delayed.  However, if that third request is marked as not opening a persistent connection, then there is no reason for the browser to delay issuing the request (i.e., opening a new connection is ok).

By setting a "connection: close" request header, the request is indicating that the connection should not be made persistent.  This is documented in RFC 2616.

[Alexey Proskuryakov]

As for Firefox 3: <http://lxr.mozilla.org/seamonkey/source/content/base/src/nsXMLHttpRequest.cpp#2403>

(In reply to comment #2)
> Imagine you have 3 requests to host A.  If all requests take a long time to
> complete, then one of the requests will be delayed.  However, if that third
> request is marked as not opening a persistent connection, then there is no
> reason for the browser to delay issuing the request (i.e., opening a new
> connection is ok).

Do you have a test that shows that the third request is delayed in WebKit? I can make one myself, but if you already have a proof, we won't have to duplicate the effort.

[Mark Rowe (bdash)]

A visitor to #webkit mentioned that the Keep-Alive header is also rejected.

[Mark Rowe (bdash)]

<rdar://problem/5785951>

[Darin Fisher (:fishd, Google)]

It seems OK to reject the keep-alive header.  That is not a standard request header.  Also, it should probably not be allowed for XHR to request keep-alive semantics since the browser cannot promise that.  The browser can promise that it will not reuse the connection though ;-)

Alexey, my test case was Gmail.  I was using the web inspector and noticed that a warning appeared in the error console about "connection" being an illegal request header.

[Alexey Proskuryakov]

Yes, I understood that these warnings came from GMail. But was there any negative effect on behavior? Were connections delayed indeed?

[Darin Fisher (:fishd, Google)]

Well, from what I can tell, the XHR request in question is for the host mail.google.com.  That means that it is clogging a persistent connection, leaving one less connection free for other resources.  I can imagine that this has a negative impact on performance.  Since this XHR request is the only such long-lived request, it clearly doesn't cause a complete blockage :-)

[Maciej Stachowiak]

We can use an attempt to set "Connection: close" as a hint not to reuse the connection, but I don't think it is correct to send it along as-is, nor would sending it along necessarily make the network layer decide that the connection doesn't count towards the connection limit.

If we wanted a hint that says "this may be a long-lived connection, so don't block other loads waiting for it", then perhaps something more explicit (like a property on the XHR object) would make more sense. Or perhaps we could always treat XHR this way. IE8b1 raised their connection limit to 6, and the HTTPbis Working Group at the IETF is considering eliminating the connection limit from the spec entirely.

Note that CFNetwork's current connection limit is 4, so a single long-lived request is unlikely to block progress, but we can still make improvements here.

[Darin Fisher (:fishd, Google)]

> We can use an attempt to set "Connection: close" as a hint not to reuse the
> connection, but I don't think it is correct to send it along as-is, nor would
> sending it along necessarily make the network layer decide that the connection
> doesn't count towards the connection limit.
> 
> If we wanted a hint that says "this may be a long-lived connection, so don't
> block other loads waiting for it", then perhaps something more explicit (like a
> property on the XHR object) would make more sense. Or perhaps we could always
> treat XHR this way. 

The behavior of existing browsers is to pass it along and to respect it on the client side (in the manner I described).  I think it is a compat bug that WebKit does not work this way.  I understand that this could involve changing more than just WebKit... CFNetwork might need changes in the case of Safari.  If the UA's network stack does not recognize this request header as a "hint", then at least the origin server or proxy server would understand it and should properly respond w/ "connection: close", which should cause the UA's network stack to stop counting the connection against the max persistent connections per host limit.  So, sending this header seems like a good thing to me.  Why do you think it should not be sent?

[Maciej Stachowiak]

By the time the origin server sends the response, it's too late to stop counting the request against the connection limit, since by assumption by then it has already been open for a while. It is better to decide this up front. I am also not sure that closing the connection after the response needs to be tied to not counting towards the connection limit. If there are requests for the server queued up at the time the response comes back, it would be valid and indeed good for the UA to reuse that connection if it can.

So in conclusion, I think it may be a better design to make XHR-initiated requests always bypass the connection limit. 

[Maciej Stachowiak]

I don't think passing along the header will be a practical issue at all if we advise the network stack in whatever way is appropriate not to count any XHR-initiated requests against the connection limit. Actually sending a "Connection: close" header over the wire does not seem necessary to achieve the desired goal of bypassing the connection limit, just an accidental side effect. However, this may be the appropriate way to advise some network stacks to bypass the connection limit.

[Darin Fisher (:fishd, Google)]

(In reply to comment #11)
> By the time the origin server sends the response, it's too late to stop
> counting the request against the connection limit, since by assumption by then
> it has already been open for a while.

The origin server can send its response in chunks.  The first chunk could come immediately and only contain the response headers.  Subsequent chunks could come much later.  If the UA sees "connection: close" in that initial payload, then it could recognize that the connection will not be persistent, and therefore it can open a new connection if there are pending requests.  (This is AFAIK the typical behavior of HTTP/1.1 clients.)  Anyways, this topic is a tangent.


> So in conclusion, I think it may be a better design to make XHR-initiated
> requests always bypass the connection limit. 

This could be risky.  I have heard that some routers have trouble if they receive too many SYN packets at once.  Not having a connection limit might force web apps to do their own connection limiting, which could be problematic.  I wish I had a reference for this anecdote :-/

Again, the point of this bug is to simply match existing browser behavior.  It is a compat bug.  If there is a better long-term solution, involving nicer APIs, then that's great, but I don't think it is mutually exclusive with supporting the existing "API" for creating a XHR request that is not counted against the browser's persistent connection limit.

[Alexey Proskuryakov]

Created attachment 19629 [details]
test case

> Again, the point of this bug is to simply match existing browser behavior.  It
> is a compat bug.

I have made a test now, and I do not see "Connection: close" having the described effect on Firefox 2.0.0.9. This "non-persistent" request is still delayed until slow requests are finished.

It is possible that I got my Apache configuration (or something else) wrong, so I encourage you to check my results.

127.0.0.1 - - [10/Mar/2008:12:03:16 +0300] "GET /slow-resource.html HTTP/1.1" 200 1197
127.0.0.1 - - [10/Mar/2008:12:03:16 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:19 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:22 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:25 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:28 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:31 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:34 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:37 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:40 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:43 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5
127.0.0.1 - - [10/Mar/2008:12:03:46 +0300] "GET / HTTP/1.1" 404 309
                         ^^ That's the request with "Connection: close" set.
127.0.0.1 - - [10/Mar/2008:12:03:46 +0300] "GET /slow-resource.pl HTTP/1.1" 200 5

[Darin Fisher (:fishd, Google)]

Alexey, your test is flawed.  Try opening N different URLs.  You are seeing the requests get serialized around access to the cache.

[Alexey Proskuryakov]

Created attachment 19658 [details]
test case (tweaked for Firefox 2 compatibility)

Heh, actually the flaw was due to a bug in Firefox 2 - it was making sync requests instead of async if the third parameter to open() was omitted.

With a corrected test, I can confirm that Firefox 2 does handle "Connection: close" as described above. According to results given in a parallel public-webapi discussion, it is the only browser that does so.

[Darin Fisher (:fishd, Google)]

> Heh, actually the flaw was due to a bug in Firefox 2 - it was making sync
> requests instead of async if the third parameter to open() was omitted.

Yes, that was also a problem.

[Darin Fisher (:fishd, Google)]

based on the discussion from [e-mail], WONTFIX seems appropriate :-(