Bug 58727 – [Qt] QWebView can not show eBay pages correctly

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Page Loading

For some eBay pages, QWebView always shows blank page. I've checked the LoadFinished signal, and the OK is false. The same pages can be rendered correctly on popular browsers, such as IE, Safari or Chrome. I can consistently reproduce the problem both on Windows and Ubuntu Linux.

Here's how to reproduce it (a 2 minute effort),

1. Create a QT Gui project
2. Drag a QWebView widget to the form
3. In the MainWindow constructor, add ui->webView->setUrl(QUrl("http://www.ebay.com"));
4. Now run your project
5. Login to eBay in your app, click on the top-right "My eBay" link, and you get a blank page.

[Benjamin Poulain]

Moving to P1, this should be investigated before QtWebKit 2.2.

I can reproduce with trunk on Linux. When clicking on "MyEbay" (or loading http://my.ebay.com/ ) after login, I only get "OK" on the page.

Changing the user-agent string to the one of Safari fixes the issue. We should have a look for the missing bits like for hotmail. If no sensible workaround can be found, we will close this bug.

We should also raise the issue to ebay.com.

(In reply to comment #1)
> Moving to P1, this should be investigated before QtWebKit 2.2.
> 
> I can reproduce with trunk on Linux. When clicking on "MyEbay" (or loading http://my.ebay.com/ ) after login, I only get "OK" on the page.
> 
> Changing the user-agent string to the one of Safari fixes the issue. We should have a look for the missing bits like for hotmail. If no sensible workaround can be found, we will close this bug.
> 
> We should also raise the issue to ebay.com.

I just did some more tests. If I put "Safari/533.20.27" or "Safari/531.22.7" in the user-agent, it will break. But if I take the Safari part out, it will work. In fact, anything other than "Safari" (such as "my browser") for user-agent would work fine.

[Luiz Agostini]

Created attachment 92487 [details]
patch

[Kenneth Rohde Christiansen]

Comment on attachment 92487 [details]
patch

could this be a security issue?

[Luiz Agostini]

(In reply to comment #4)
> (From update of attachment 92487 [details])
> could this be a security issue?

I am not sure. I think that it may be not because in this specific case QNAM is not emitting the sslErrors signal. But it would be nice to have Markus opinion on that.

[Markus Goetz]

Can you backtrace to where QNAM is setting/emitting the ProtocolFailure?

[Luiz Agostini]

(In reply to comment #6)
> Can you backtrace to where QNAM is setting/emitting the ProtocolFailure?

Sure, I will do it today.

But more then finding out what is going on in this specific case, I would like to ignore all the errors that are not related to security issues.

In this case for example the error is not SslHandshakeFailedError, no sslErrors signal is been emited (I will confirm it) and the response content seems to be good enough for WebKit to render the page and proceed the eBay navigation, meaning the the pair user/password has been accepted by eBay.

What I would like to know is how to identify the security issues with the information provided by QNetworkReply. With this information I will be able to ignore the other QNAM errors and proceed based on message headers and contents only (if they are present), on a best effort basis.

[Simon Hausmann]

Comment on attachment 92487 [details]
patch

View in context: https://bugs.webkit.org/attachment.cgi?id=92487&action=review

r- because neither the ChangeLog nor the hunk in QNetworkReplyHandler explain why this is done, i.e. why QNAM sets ProtocolFailure but we choose to ignore it.

> Source/WebCore/platform/network/qt/QNetworkReplyHandler.cpp:430
> +    if (httpStatusCode == 200 && reply->error() == QNetworkReply::ProtocolFailure)
> +        return true;
> +

Please find out why QNAM is setting ProtocolFailure as error here. If the situation is justified, we should explain with a comment why we ignore the error.

[Luiz Agostini]

Created attachment 95047 [details]
patch

[Simon Hausmann]

Comment on attachment 95047 [details]
patch

View in context: https://bugs.webkit.org/attachment.cgi?id=95047&action=review

> Source/WebCore/ChangeLog:11
> +        QNAM is having problems when uncompressing some of the contents returned by eBay and
> +        notifing error QNetworkReply::ProtocolFailure. It happens that the HTTP status code
> +        for the same message is 200 OK and QNetworkReply::ProtocolFailure does not seem to be
> +        related to any security issue.

Thanks for the analysis :). So isn't this actually a Qt bug? I mean, the fact that we get _some_ data but an error is set makes me think that either

    a) the data was successfully decompressed but Qt thinks that there's an error even though there isn't (interpreting zlib results incorrectly?)

or

    b) there was a real error decompressing the data and we deliver corrupted content to webkit, but as it turns out webkit handles the corrupted content and we don't see any visual glitches in the resulting rendering.

Do you know which one it is?

[Luiz Agostini]

(In reply to comment #10)
> (From update of attachment 95047 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=95047&action=review
> 
> > Source/WebCore/ChangeLog:11
> > +        QNAM is having problems when uncompressing some of the contents returned by eBay and
> > +        notifing error QNetworkReply::ProtocolFailure. It happens that the HTTP status code
> > +        for the same message is 200 OK and QNetworkReply::ProtocolFailure does not seem to be
> > +        related to any security issue.
> 
> Thanks for the analysis :). So isn't this actually a Qt bug? I mean, the fact that we get _some_ data but an error is set makes me think that either
> 
>     a) the data was successfully decompressed but Qt thinks that there's an error even though there isn't (interpreting zlib results incorrectly?)
> 
> or
> 
>     b) there was a real error decompressing the data and we deliver corrupted content to webkit, but as it turns out webkit handles the corrupted content and we don't see any visual glitches in the resulting rendering.
> 
> Do you know which one it is?

I have always been working with hypothesis b.

After a quick look at the QNAM's code now I think that the server is closing the connection a bit early and some bits of the message are lost. So, there were indeed a protocol failure and the content is probably corrupted.

[Luiz Agostini]

Simon, the idea was to ignore those QNAM errors that are not security issues, if there is some content to WebKit to deal with.
I think that it would not be very different from actually receiving not compressed corrupted content from the server.

It seems that QNAM aborts just after noticing the error and before uncompressing the last data chunk. It would be better, considering this best effort policy, to uncompress all the received content before aborting.

[Luiz Agostini]

How should we proceed here?

[Luiz Agostini]

Does anyone have any opinions regarding this issue?

Although I think that this change is OK, we may just close this bug and raise the issue to ebay.com.

[Kenneth Rohde Christiansen]

(In reply to comment #14)
> Does anyone have any opinions regarding this issue?
> 
> Although I think that this change is OK, we may just close this bug and raise the issue to ebay.com.

Adding christian as he might know people who can actually raise that issue.

[Robert Hogan]

*** Bug 50746 has been marked as a duplicate of this bug. ***

[Joel Parks]

(In reply to comment #16)
> *** Bug 50746 has been marked as a duplicate of this bug. ***

as part of the discussion in 50746, the Qt bug was raised:
http://bugreports.qt.nokia.com/browse/QTBUG-16022

And the associated analysis shows that the Ebay has delivered gzip-compressed content without end-of-stream marker.  


(In reply to comment #12)
> Simon, the idea was to ignore those QNAM errors that are not security issues, if there is some content to WebKit to deal with.
> I think that it would not be very different from actually receiving not compressed corrupted content from the server.
> 
> It seems that QNAM aborts just after noticing the error and before uncompressing the last data chunk. It would be better, considering this best effort policy, to uncompress all the received content before aborting.


I am in favor of that "best effort policy" attempt to uncompress all the received content before aborting.

[Thijs]

I encounter the same bug with http://sports.qq.com
At first I thought it was a generic gzip-handling issue, but now I assume this site is also missing the end-of-stream marker. 

I am also very much in favor of "best effort policy". Can I help out?

[Thijs]

I saved the gzipped reply from http://sports.qq.com to a local file and then ran zcat to decompress it. It didn't report any problems at all, so I'm not yet sure if sports.qq.com also fails because of missing end-of-stream marker, or because some other problem.

[Thomas Thrainer]

Comment #1 states that changing the user-agent string fixes the issue with eBay, maybe sports.qq.com does deliver different gzip-streams for different user-agents as well?

[Luiz Agostini]

(In reply to comment #18)
> I encounter the same bug with http://sports.qq.com
> At first I thought it was a generic gzip-handling issue, but now I assume this site is also missing the end-of-stream marker. 
> 
> I am also very much in favor of "best effort policy". Can I help out?

I could not reproduce the problem with http://sports.qq.com. 
Could you provide a sequence of steps that I could use to reproduce it? I must say that I cannot understand much of that page. :)

[Thijs]

@Thomas : I've created a separate test case and found which code causes the error (the returned data is not uncompressed, but shown inside the browser):

1) create a custom QNetworkAccessManager and override createRequest(...)
2) inside the function createRequest(...) create a copy of the request object (so I can modify it) as such :  QNetworkRequest reqCopy(req);
3) then do: reqCopy.setRawHeader("Accept-Encoding", "gzip"); //this causes the problems

Note that the value ("gzip" here) doesn't matter. Just setting the Accept-Encoding header confuses QtWebKit. 

when I disable this setRawHeader() call, it works 100% reliable (so gzipped content is handled correctly if i didn't set the "Accept-Encoding" header).


note: sports.qq.com sometimes returns uncompressed data, but usually it is compressed. You might need to try a few times and make sure caching is disabled.
I disabled cache with :
reqCopy.setAttribute(QNetworkRequest::CacheLoadControlAttribute, QNetworkRequest::AlwaysNetwork);

[Thijs]

(In reply to comment #22)

from the source code of qhttpnetworkconnection.cpp :


====================
    value = request.headerField("accept-encoding");
    if (value.isEmpty()) {
#ifndef QT_NO_COMPRESS
        request.setHeaderField("Accept-Encoding", "gzip");
        request.d->autoDecompress = true;
#else
        // if zlib is not available set this to false always
        request.d->autoDecompress = false;
#endif
    }
====================


I believe this is a bug. If I manually set the "accept-encoding" header, autoDecompress becomes false (even if I set it to "gzip").

[Thijs]

(In reply to comment #23)

> I believe this is a bug. If I manually set the "accept-encoding" header, autoDecompress becomes false (even if I set it to "gzip").

Created a new bug report this this issue with a proposed solution:
https://bugs.webkit.org/show_bug.cgi?id=63696

[Tor Arne Vestbø]

Comment on attachment 95047 [details]
patch

I believe this is related to the Qt network stack not dealing with a missing EOF marker in the gzip data, and we should fix it in Qt, not work around it here.

[Ademar Reis]

(In reply to comment #25)
> (From update of attachment 95047 [details])
> I believe this is related to the Qt network stack not dealing with a missing EOF marker in the gzip data, and we should fix it in Qt, not work around it here.

The bug in Qt is http://bugreports.qt.nokia.com/browse/QTBUG-16022, how can we make sure this bug is closed on qt-4.7 and qt-4.8 (worst case just qt-4.8)?

[Luiz Agostini]

(In reply to comment #25)
> (From update of attachment 95047 [details])
> I believe this is related to the Qt network stack not dealing with a missing EOF marker in the gzip data, and we should fix it in Qt, not work around it here.

The idea was to have a workaround for 2.2. But I agree with you.
Then I think that we should close this bug.

[Ademar Reis]

This was fixed in Qt (http://bugreports.qt.nokia.com/browse/QTBUG-16022). I've tested it (using qt-4.7) and it works fine (the fix will also be merged into 4.8 in the next days).