Bug 18972 – WebKit not using 304 Not Modified for Ajax requests

• Component: WebCore Misc.
• Status: RESOLVED INVALID

[scott Schmitz]

I have a web application that makes extensive use of xmlhttprequest (AJAX).  I have optimized some of my GET requests such that the server will return a 304 not modified if the ETag matches up from a prior request.  That way, if the browser cache already has the data, I just send the header back and not the data itself.

This works great for Firefox and Internet Explorer.  In the case of those browsers, I return ETag and last modified headers for the initial request.  The browser caches the data and includes

If-Modified-Since:
If-None-Match:

headers when it makes a request again.  If the server determined that the cache is good, it returns

304 not-modified response and all is good.

WebKit and Safari do not do this.  Best I can see, the browser makes its request and does not include any Etag at all.  I see this via Web Inspector - Network Panel.

The best way to test this is to use the web application that I have at:

http://www.realtyjuggler.com

Create an account and sign in.
Click on contacts icon
Create a new contact and click done
Now, click on the Contacts link in upper right corner of the screen.  It will be part of the breadcrumb.  If you click a few times you will be reloading the same ajax XML data.

This ajax call will correctly respond with 304 not modified if the appropriate headers are sent to it when making the request.

GET http://www.scottschmitz.com/subscription/AJAX/GetList.php?userID=54&databaseName=Contact&format=Contact

This URL can only be executed when you are signed in as it requires cookie authentication.

[Alexey Proskuryakov]

I cannot reproduce this issue with shipping Safari 3.1.1 on Mac OS X 10.5.2.

When I repeatedly click on "Contacts", I'm seeing the following in network trace:
vGET /subscription/AJAX/GetList.php?userID=1830&databaseName=Contact&format=Contact HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_2; ru-ru) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.1 Safari/525.18
Referer: http://www.realtyjuggler.com/main/Frameset.html
Accept: */*
Accept-Language: ru
Accept-Encoding: gzip, deflate
If-Modified-Since: Fri, 09 May 2008 18:51:34 GMT
Cookie: __utma=1.1537112792.1210358912.1210358912.1210358912.1; __utmb=1; __utmc=1; __utmz=1.1210358912.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); PHPSESSID=f5008ab44d67d2400d8e0cc0e304adfe
Connection: keep-alive
Host: www.realtyjuggler.com

Response:
HTTP/1.1 304 not modified 2008-05-09 18:51:34 GMT
Date: Fri, 09 May 2008 18:54:16 GMT
Server: Apache/2.2.6 (iTools 9.0.0i)/Mac OS X) mod_ssl/2.2.6 OpenSSL/0.9.7i DAV/2 mod_fastcgi/2.4.2 PHP/5.2.5
Connection: Keep-Alive
Keep-Alive: timeout=5, max=100
ETag: "a187247ec53d3ad2b9972e82dd0639da"
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: age=3600, private, must-revalidate
Vary: Accept-Encoding,User-Agent


As seen above, we do send If-Modified-Since, and the server responds with a 304 Not Modified response (note that its Expires header seems to be off, though).

[Alexey Proskuryakov]

But we don't do conditional requests based on ETags, that's true.

[scott Schmitz]

I just tried again with Safari 3.1 (the version I have) and the latest nightly build of WebKit and browser did not send either of these:
If-Modified-Since:
If-None-Match:
when making an ajax request that is cached.

For Safari, I looked at the Network Timeline and was able to look at the headers to see what was there.  Additionally, I have a rather long list of contacts and it took 11 seconds to send all of the data.  I owuld have expected that to be 1-2 seconds if only the header was returned.

For the latest WebKit,I was unable to view headers in the same way as the Network Timeline returns the XML, but not headers.  So, I did this in console:

sudo tcpdump -i en0 -vvv -n -s 0 -w ~/Desktop/DumpFile.dmp

and this is a bit of what I got back:

[scott Schmitz]

This is what the client sent to the server:

User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_2; en-us) AppleWebKit/526.5+ (KHTML, like Gecko) Version/3.1 Safari/525.13
Referer: http://www.scottschmitz.com/main/Frameset.html
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Cookie: PHPSESSID=2a35fc934665199723540025bc49ecd2
Connection: keep-alive
Host: www.scottschmitz.com

[scott Schmitz]

Sorry for so many posts.  The http header dump much have had a bad character and my post was truncated.

I am thinking, perhaps there's a preference setting that dictates how cache is handled and this is set to off for my machine and on for yours?  Because from my end, I don't see the cache being used here.  I seem to remember a preference setting for cache size?  And perhaps mine is set to none?  I looked in Preference panels and can't see anything that is user configurable, however.  I seem to recall that it once was, however.  I am wondering if I once set it to no cache and then you guys upgraded it so that I can no longer edit this setting?

[Alexey Proskuryakov]

> Host: www.scottschmitz.com

Maybe this server is configured differently than realtyjuggler.com? Could you try running the precise steps to reproduce from bug description, and attach output of "tcpdump -As0 -i en0"?

The only cache-related that I'm aware of is to disable WebCore cache, which shouldn't change behavior here, as XMLHttpRequest caching is done by the network layer. You may want to try testing with a freshly created user account though, to eliminate all possibilities.

I'm not sure if Network Timeline headers would include those added below WebKit.

[scott Schmitz]

I did some further testing and have come up with a test case where I can get the caching to fail and where I can get it to work.  The difference between the two situations seems to be the size of the XML data.

Here's how to repeat:
1. Go here:
http://www.scottschmitz.com/main/Frameset.html
2. Sign on as
[e-mail]
password: scott
3. The URL may change to https.  If it does, you can change back to http and reload.  I don't see a difference in behavior with or without https, however https does cause the parts of the packet trace go get garbled.
4. Click on the Contacts icon.
5. Click on the 'Contacts' link in upper left corner of screen - it is part of the breadcrumb navigation
This second load of contacts should cause 304 not modified, but does not.

Now for the case where I can get it to work.
1. Do steps 1-3, or ifyou are already there just go to the next step.
2. Click on the Buyers Icon
3. reload 'Buyers' list by clicking on it's breadcrumb link in upper-left corner.
This time, I get the 304 not modified.

I have tested several combinations:
1. Firefox returns 304 not modifies for both of these cases
2. I have deleted the preference settings for Safari, and have seen no change in behavior,
3. I get the above reported behavior with Safari 3.1 (5525.13) on Mac OS X 10.5.2)

I will attach as files two packet dump files.  One called 'Buyers Works.txt' and the other called 'Contacts fails.txt'

[scott Schmitz]

Created attachment 21058 [details]
packet trace when reloading contacts - should return 304 not modified but does not

[scott Schmitz]

Created attachment 21059 [details]
packet trace when 304 not modified is returned when I reload Buyers list

[Alexey Proskuryakov]

Created attachment 21066 [details]
fresh load and reload traces for both servers

When following these new steps to reproduce, I'm getting an empty contact list. Maybe someone else was testing with this account, and accidentally deleted all the data?

Actually, I'm not seeing conditional requests being made for contact list on scottschmitz.com. But that's correct behavior, because the response does not nave a Last-Modified header. See the attached trace, which I made after emptying NSURL cache.

Yet, your trace for the large response shows a different behavior, so there's still a mystery.

[scott Schmitz]

Sorry, I left one step out.  There's a drop-down menu in upper right corner of screen called Accessing.  Make sure that [e-mail] is the item selected.

[Alexey Proskuryakov]

I can see the problem now. Indeed, responses larger than 209715 bytes are not cached. It is a bug, the threshold should be larger than that.

This is a bug in a closed source Apple framework, currently tracked  as <rdar://problem/5674384>. Closing this report as INVALID per our process, as this is not a WebKit bug.

Thank you very much for your assistance with figuring out what is going on here!

[Alexey Proskuryakov]

*** Bug 17895 has been marked as a duplicate of this bug. ***

[Mark Rowe (bdash)]

*** Bug 19144 has been marked as a duplicate of this bug. ***

[Ryan Warnick]

If this is the same issue that we saw (Bug# 17895), then I am guessing that Safari works fine on the PC?  I will have to try it to see.  

So, you are saying that Webkit will not do anything to work around this problem?  Other browsers like Firefox work just fine on the Mac.  Do you know if Apple is working on a fix?  How would I reference this bug with Apple?  I hope that someone is working on it because the only other option that we have is to tell everyone not to use Safari on the Mac. It's hard for me to imagine that this is what Apple or Webkit want.  I mean not being able to effectively cache content over ~200K in size seems like a fairly large shortcoming with Safari on the Mac.

[Alexey Proskuryakov]

(In reply to comment #15)
> How would I reference this bug with Apple? 

Per comment 12, it is <rdar://problem/5674384>.

[scott Schmitz]

I am trying to follow-up on this bug.  Can someone tell me how to access the Apple bug?  

This URL is no good: rdar://problem/5674384

I sign into:
https://bugreport.apple.com/cgi-bin/WebObjects/RadarWeb.woa/62/wo/0ttncWsU6CokpZhOsggiXM/24.39.7

and can only see the bugs I have reported, therefore can not see item: 5674384

thanks,

Scott.

[Alexey Proskuryakov]

(In reply to comment #17)
> I am trying to follow-up on this bug.  Can someone tell me how to access the
> Apple bug?  
> 
> This URL is no good: rdar://problem/5674384

You can e-mail <[e-mail]> and ask about the bug, referencing this URL.

In this particular case, I can say that the problem with threshold being as low as 209715 bytes should be fixed as of Mac OS X 10.5.3.

[Randy Hudson]

I'm seeing this issue is well.  Responses less than 60KB are not being cached.

I have a servlet which return javascript.  If I dynamically insert a <script> tag, then both Safari/Chrome will fetch the content, cache it, and make conditional fetches upon reload of that page.

However, if I use an XMLHttpRequest to call the exact same servlet with the same query parameters, etc., neither browser will cache the content, or at least they aren't making conditional fetches if they are caching.

The response headers look like this:
Cache-Control:public, max-age=2592000
Connection:Keep-Alive
Content-Encoding:gzip
Content-Language:en-US
Content-Type:text/javascript; charset=UTF-8
Date:Wed, 24 Nov 2010 18:12:17 GMT
Etag:3JxXt_en_US
Expires:Fri, 24 Dec 2010 18:12:17 GMT
Last-Modified:Tue, 16 Nov 2010 17:28:31 GMT
Proxy-Connection:Keep-Alive
Server:WebSphere Application Server/7.0
Transfer-Encoding:Identity

Can someone reopen this?  I can reproduce the issue on Chrome and Safari, so it doesn't seem like an Apple problem.

[Alexey Proskuryakov]

Could you please file a new bug? The one discussed here has been fixed in Mac OS X 10.5.4, and the issue you are reporting is different. Discussing it here would only cause confusion for everyone.

[Alexey Proskuryakov]

> Mac OS X 10.5.4

10.5.3.