Bug 18972

Summary: WebKit not using 304 Not Modified for Ajax requests
Product: WebKit Reporter: scott Schmitz <scott>
Component: WebCore Misc.Assignee: Nobody <webkit-unassigned>
Status: RESOLVED INVALID    
Severity: Normal CC: ap, demi, hudsonr, warnickr_1
Priority: P2 Keywords: InRadar
Version: 528+ (Nightly build)   
Hardware: Mac   
OS: OS X 10.5   
URL: http://www.realtyjuggler.com
Attachments:
Description Flags
packet trace when reloading contacts - should return 304 not modified but does not
none
packet trace when 304 not modified is returned when I reload Buyers list
none
fresh load and reload traces for both servers none

Description scott Schmitz 2008-05-09 11:30:30 PDT
I have a web application that makes extensive use of xmlhttprequest (AJAX).  I have optimized some of my GET requests such that the server will return a 304 not modified if the ETag matches up from a prior request.  That way, if the browser cache already has the data, I just send the header back and not the data itself.

This works great for Firefox and Internet Explorer.  In the case of those browsers, I return ETag and last modified headers for the initial request.  The browser caches the data and includes

If-Modified-Since:
If-None-Match:

headers when it makes a request again.  If the server determined that the cache is good, it returns

304 not-modified response and all is good.

WebKit and Safari do not do this.  Best I can see, the browser makes its request and does not include any Etag at all.  I see this via Web Inspector - Network Panel.

The best way to test this is to use the web application that I have at:

http://www.realtyjuggler.com

Create an account and sign in.
Click on contacts icon
Create a new contact and click done
Now, click on the Contacts link in upper right corner of the screen.  It will be part of the breadcrumb.  If you click a few times you will be reloading the same ajax XML data.

This ajax call will correctly respond with 304 not modified if the appropriate headers are sent to it when making the request.

GET http://www.scottschmitz.com/subscription/AJAX/GetList.php?userID=54&databaseName=Contact&format=Contact

This URL can only be executed when you are signed in as it requires cookie authentication.
Comment 1 Alexey Proskuryakov 2008-05-09 11:59:59 PDT
I cannot reproduce this issue with shipping Safari 3.1.1 on Mac OS X 10.5.2.

When I repeatedly click on "Contacts", I'm seeing the following in network trace:
vGET /subscription/AJAX/GetList.php?userID=1830&databaseName=Contact&format=Contact HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_2; ru-ru) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.1 Safari/525.18
Referer: http://www.realtyjuggler.com/main/Frameset.html
Accept: */*
Accept-Language: ru
Accept-Encoding: gzip, deflate
If-Modified-Since: Fri, 09 May 2008 18:51:34 GMT
Cookie: __utma=1.1537112792.1210358912.1210358912.1210358912.1; __utmb=1; __utmc=1; __utmz=1.1210358912.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); PHPSESSID=f5008ab44d67d2400d8e0cc0e304adfe
Connection: keep-alive
Host: www.realtyjuggler.com

Response:
HTTP/1.1 304 not modified 2008-05-09 18:51:34 GMT
Date: Fri, 09 May 2008 18:54:16 GMT
Server: Apache/2.2.6 (iTools 9.0.0i)/Mac OS X) mod_ssl/2.2.6 OpenSSL/0.9.7i DAV/2 mod_fastcgi/2.4.2 PHP/5.2.5
Connection: Keep-Alive
Keep-Alive: timeout=5, max=100
ETag: "a187247ec53d3ad2b9972e82dd0639da"
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: age=3600, private, must-revalidate
Vary: Accept-Encoding,User-Agent


As seen above, we do send If-Modified-Since, and the server responds with a 304 Not Modified response (note that its Expires header seems to be off, though).
Comment 2 Alexey Proskuryakov 2008-05-09 12:00:33 PDT
But we don't do conditional requests based on ETags, that's true.
Comment 3 scott Schmitz 2008-05-09 13:08:21 PDT
I just tried again with Safari 3.1 (the version I have) and the latest nightly build of WebKit and browser did not send either of these:
If-Modified-Since:
If-None-Match:
when making an ajax request that is cached.

For Safari, I looked at the Network Timeline and was able to look at the headers to see what was there.  Additionally, I have a rather long list of contacts and it took 11 seconds to send all of the data.  I owuld have expected that to be 1-2 seconds if only the header was returned.

For the latest WebKit,I was unable to view headers in the same way as the Network Timeline returns the XML, but not headers.  So, I did this in console:

sudo tcpdump -i en0 -vvv -n -s 0 -w ~/Desktop/DumpFile.dmp

and this is a bit of what I got back:

Comment 4 scott Schmitz 2008-05-09 13:10:09 PDT
This is what the client sent to the server:

User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_2; en-us) AppleWebKit/526.5+ (KHTML, like Gecko) Version/3.1 Safari/525.13
Referer: http://www.scottschmitz.com/main/Frameset.html
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Cookie: PHPSESSID=2a35fc934665199723540025bc49ecd2
Connection: keep-alive
Host: www.scottschmitz.com
Comment 5 scott Schmitz 2008-05-09 13:20:07 PDT
Sorry for so many posts.  The http header dump much have had a bad character and my post was truncated.

I am thinking, perhaps there's a preference setting that dictates how cache is handled and this is set to off for my machine and on for yours?  Because from my end, I don't see the cache being used here.  I seem to remember a preference setting for cache size?  And perhaps mine is set to none?  I looked in Preference panels and can't see anything that is user configurable, however.  I seem to recall that it once was, however.  I am wondering if I once set it to no cache and then you guys upgraded it so that I can no longer edit this setting?
Comment 6 Alexey Proskuryakov 2008-05-10 00:25:06 PDT
> Host: www.scottschmitz.com

Maybe this server is configured differently than realtyjuggler.com? Could you try running the precise steps to reproduce from bug description, and attach output of "tcpdump -As0 -i en0"?

The only cache-related that I'm aware of is to disable WebCore cache, which shouldn't change behavior here, as XMLHttpRequest caching is done by the network layer. You may want to try testing with a freshly created user account though, to eliminate all possibilities.

I'm not sure if Network Timeline headers would include those added below WebKit.
Comment 7 scott Schmitz 2008-05-10 11:42:46 PDT
I did some further testing and have come up with a test case where I can get the caching to fail and where I can get it to work.  The difference between the two situations seems to be the size of the XML data.

Here's how to repeat:
1. Go here:
http://www.scottschmitz.com/main/Frameset.html
2. Sign on as
scott@realorganized.com
password: scott
3. The URL may change to https.  If it does, you can change back to http and reload.  I don't see a difference in behavior with or without https, however https does cause the parts of the packet trace go get garbled.
4. Click on the Contacts icon.
5. Click on the 'Contacts' link in upper left corner of screen - it is part of the breadcrumb navigation
This second load of contacts should cause 304 not modified, but does not.

Now for the case where I can get it to work.
1. Do steps 1-3, or ifyou are already there just go to the next step.
2. Click on the Buyers Icon
3. reload 'Buyers' list by clicking on it's breadcrumb link in upper-left corner.
This time, I get the 304 not modified.

I have tested several combinations:
1. Firefox returns 304 not modifies for both of these cases
2. I have deleted the preference settings for Safari, and have seen no change in behavior,
3. I get the above reported behavior with Safari 3.1 (5525.13) on Mac OS X 10.5.2)

I will attach as files two packet dump files.  One called 'Buyers Works.txt' and the other called 'Contacts fails.txt'
Comment 8 scott Schmitz 2008-05-10 11:44:42 PDT
Created attachment 21058 [details]
packet trace when reloading contacts - should return 304 not modified but does not
Comment 9 scott Schmitz 2008-05-10 11:45:10 PDT
Created attachment 21059 [details]
packet trace when 304 not modified is returned when I reload Buyers list
Comment 10 Alexey Proskuryakov 2008-05-11 02:44:43 PDT
Created attachment 21066 [details]
fresh load and reload traces for both servers

When following these new steps to reproduce, I'm getting an empty contact list. Maybe someone else was testing with this account, and accidentally deleted all the data?

Actually, I'm not seeing conditional requests being made for contact list on scottschmitz.com. But that's correct behavior, because the response does not nave a Last-Modified header. See the attached trace, which I made after emptying NSURL cache.

Yet, your trace for the large response shows a different behavior, so there's still a mystery.
Comment 11 scott Schmitz 2008-05-11 08:15:44 PDT
Sorry, I left one step out.  There's a drop-down menu in upper right corner of screen called Accessing.  Make sure that SSchmitz99@aol.com is the item selected.
Comment 12 Alexey Proskuryakov 2008-05-12 03:23:21 PDT
I can see the problem now. Indeed, responses larger than 209715 bytes are not cached. It is a bug, the threshold should be larger than that.

This is a bug in a closed source Apple framework, currently tracked  as <rdar://problem/5674384>. Closing this report as INVALID per our process, as this is not a WebKit bug.

Thank you very much for your assistance with figuring out what is going on here!
Comment 13 Alexey Proskuryakov 2008-05-12 10:19:24 PDT
*** Bug 17895 has been marked as a duplicate of this bug. ***
Comment 14 Mark Rowe (bdash) 2008-05-20 11:15:43 PDT
*** Bug 19144 has been marked as a duplicate of this bug. ***
Comment 15 Ryan Warnick 2008-05-20 11:50:30 PDT
If this is the same issue that we saw (Bug# 17895), then I am guessing that Safari works fine on the PC?  I will have to try it to see.  

So, you are saying that Webkit will not do anything to work around this problem?  Other browsers like Firefox work just fine on the Mac.  Do you know if Apple is working on a fix?  How would I reference this bug with Apple?  I hope that someone is working on it because the only other option that we have is to tell everyone not to use Safari on the Mac. It's hard for me to imagine that this is what Apple or Webkit want.  I mean not being able to effectively cache content over ~200K in size seems like a fairly large shortcoming with Safari on the Mac.
Comment 16 Alexey Proskuryakov 2008-05-20 12:11:23 PDT
(In reply to comment #15)
> How would I reference this bug with Apple? 

Per comment 12, it is <rdar://problem/5674384>.
Comment 17 scott Schmitz 2008-11-25 14:47:36 PST
I am trying to follow-up on this bug.  Can someone tell me how to access the Apple bug?  

This URL is no good: rdar://problem/5674384

I sign into:
https://bugreport.apple.com/cgi-bin/WebObjects/RadarWeb.woa/62/wo/0ttncWsU6CokpZhOsggiXM/24.39.7

and can only see the bugs I have reported, therefore can not see item: 5674384

thanks,

Scott.
Comment 18 Alexey Proskuryakov 2008-11-26 00:26:37 PST
(In reply to comment #17)
> I am trying to follow-up on this bug.  Can someone tell me how to access the
> Apple bug?  
> 
> This URL is no good: rdar://problem/5674384

You can e-mail <devbugs@apple.com> and ask about the bug, referencing this URL.

In this particular case, I can say that the problem with threshold being as low as 209715 bytes should be fixed as of Mac OS X 10.5.3.
Comment 19 Randy Hudson 2010-11-24 10:17:03 PST
I'm seeing this issue is well.  Responses less than 60KB are not being cached.

I have a servlet which return javascript.  If I dynamically insert a <script> tag, then both Safari/Chrome will fetch the content, cache it, and make conditional fetches upon reload of that page.

However, if I use an XMLHttpRequest to call the exact same servlet with the same query parameters, etc., neither browser will cache the content, or at least they aren't making conditional fetches if they are caching.

The response headers look like this:
Cache-Control:public, max-age=2592000
Connection:Keep-Alive
Content-Encoding:gzip
Content-Language:en-US
Content-Type:text/javascript; charset=UTF-8
Date:Wed, 24 Nov 2010 18:12:17 GMT
Etag:3JxXt_en_US
Expires:Fri, 24 Dec 2010 18:12:17 GMT
Last-Modified:Tue, 16 Nov 2010 17:28:31 GMT
Proxy-Connection:Keep-Alive
Server:WebSphere Application Server/7.0
Transfer-Encoding:Identity

Can someone reopen this?  I can reproduce the issue on Chrome and Safari, so it doesn't seem like an Apple problem.
Comment 20 Alexey Proskuryakov 2010-11-24 11:13:07 PST
Could you please file a new bug? The one discussed here has been fixed in Mac OS X 10.5.4, and the issue you are reporting is different. Discussing it here would only cause confusion for everyone.
Comment 21 Alexey Proskuryakov 2010-11-24 11:13:57 PST
> Mac OS X 10.5.4

10.5.3.