Shockwave unable to load GZip'd text resources when server sends Content-Length header

• Component: Plug-ins
• Status: RESOLVED FIXED

[Sulka Haro]

When a Shockwave movie loads an external text resource over HTTP and the delivery uses the gzip 
content-encoding, and the server sends the resource with the Content-Length 
header set, the download always fails for some reason. If the server doesn't send the Content-Length 
header and specifies "Transfer-Encoding: chunked", the download succeeds.

Safari seems to misreport the file size to the plugin in the case of failure - for some reason the file 
size is reported as being larger than the gzipped version (correct) but smaller than the decompressed 
resource so it seems that if the plugin asks for details of the file while the file is being decompressed, 
the decompression is somehow interrupted and the length of the already decompressed content is 
reported.

Clearly there's some functional difference in how the content is delivered to plugins depending on the 
encoding type.

Reproducible: Always

Steps to Reproduce:

See http://www.kotska.com/swtest/ for a testcase.

Actual Results:
The file being downloaded is downloaded but never delivered to the plugin.

Expected Results:  
File should have been passed onto the plugin.

[Sulka Haro]

Tried this using both Safari 2.0.1 and the latest CVS version (AppleWebKit/420+ Safari/412.5) and the test 
fails using both. Effectively this means any site using Shockwave has to disable gzip compression for 
compatibility with Safari, causing textual content to require roughly 8-10x the amount of bandwidth 
necessary with compression.

[Oliver Hunt]

Good test case

[Aldo Hoeben]

The testcase fails the same way in Firefox (1.0 and 1.5), both on OS X and Windows.

[Sulka Haro]

The Firefox bug is reported here:

https://bugzilla.mozilla.org/show_bug.cgi?id=305766

FYI: Internet Explorer (Mac and Windows) doesn't fail the test and since the plugin running within those browsers is (afaik) exactly the same code on a given platform, there has to be some differences in the browser implementation.

We have 100k+ Mac users on our site every month and as a result of this bug, most of these users have to download about 300kB of extra unnecessarily uncompressed data on every visit.

[Alexey Proskuryakov]

Created attachment 6922 [details]
proposed fix

Can we make automated tests for such bugs?

[Darin Adler]

Comment on attachment 6922 [details]
proposed fix

This patch is not quite right. The content encoding might be "identity", so just the presence of a content encoding header field is not the correct check. Also, buggy servers sometimes send strings like "identity, identity".

It's hard to code this correctly at the WebKit level and stay in sync. with the features of the NSURL level; ideally we would have a Foundation method like expectedContentLength, but one that knew to only give us the length for things it won't decode. In general we don't want code that has to second-guess what NSURL is doing for us.

Current versions of NSURLConnection support "deflate" and "gzip"; anything else is treated as "identity". The comparison is done in a case insensitive manner. NSURLConnection knows how to parse off an optional "x-" prefix. It knows how to ignore parameters that come after an optional ";" separator. It knows how to ignore whitespace. It parses a list separated by commas; if all the items in the list are the same (ignoring case, whitespace, "x-" prefixes, and parameters) then it uses it, otherwise it treats it as "identity".

Hard-coding this precise rule into WebBaseNetscapePluginStream would be kinda lame, but I don't have a better idea.

[Alexey Proskuryakov]

<rdar://4470599> for NSURLConnection.

[Alexey Proskuryakov]

Actually, I think that expectedContentLength itself should only give us the length for things it won't decode, so I'm going to just wait for the resolution of the above Radar.

That said, I still don't really understand why a simplified rule like the one used in the patch is much worse for deciding what to pass to the plugin. Also, servers SHOULD NOT send Content-Encoding: identity, according to RFC2616.

[Alexey Proskuryakov]

Created attachment 15938 [details]
updated fix

As mentioned above, I think we do not need to match NSURLConnection logic particularly well.

[Darin Adler]

Comment on attachment 15938 [details]
updated fix

OK, at this point it seems smart to work around the NSURL bug. r=me

[Alexey Proskuryakov]

Committed revision 25024.

[Mark Rowe (bdash)]

<rdar://problem/5415340>