Bug 118232 – [Qt][Wk2] Many tests are failing with flaky on Qt 5.1.

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Ádám Kallai]

Currently there are 127 flaky tests on Qt 5.1 Wk2 bot.

You can see it:
http://build.webkit.sed.hu/results/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/r152215%20(281)/results.html

Investigation in progress.

[Allan Sandfeld Jensen]

access-control-preflight-credential-sync.html which is consistently flaky can be made to stop flaking by skipping the test before it: access-control-preflight-credential-async.html

My current theory is a problem with stopping all async requests.

[Allan Sandfeld Jensen]

The underlying problem is that the state of QNetworkAccessManager in WebKit2 is not fully cleared between pages. Probably because WebKit1 does not have a shared QNAM, and QtTestSupport::clearMemoryCaches() can not access it because it lives in WebKit2.

[Allan Sandfeld Jensen]

Created attachment 209780 [details]
Patch

[Allan Sandfeld Jensen]

The uploaded patch fixes the problem. I do however think we might need to clear the access parameters between page-loads. This is a potential security hole, though it only affects plain-text HTML forms and authentication.

[Build Bot]

Comment on attachment 209780 [details]
Patch

Attachment 209780 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.appspot.com/results/1605040

New failing tests:
http/tests/security/XFrameOptions/x-frame-options-deny-meta-tag.html
http/tests/security/XFrameOptions/x-frame-options-deny-meta-tag-parent-same-origin-deny.html

[Build Bot]

Created attachment 209798 [details]
Archive of layout-test-results from webkit-ews-09 for mac-mountainlion-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: webkit-ews-09  Port: mac-mountainlion-wk2  Platform: Mac OS X 10.8.4

[Allan Sandfeld Jensen]

(In reply to comment #4)
> The uploaded patch fixes the problem. I do however think we might need to clear the access parameters between page-loads. This is a potential security hole, though it only affects plain-text HTML forms and authentication.

Never mind. It is the same in WebKit1, and the potential security issues are solved at a different level.

[Zoltan Arvai]

(In reply to comment #7)
> (In reply to comment #4)
> > The uploaded patch fixes the problem. I do however think we might need to clear the access parameters between page-loads. This is a potential security hole, though it only affects plain-text HTML forms and authentication.
> 
> Never mind. It is the same in WebKit1, and the potential security issues are solved at a different level.

Yeah, it seems this patch wont fix flakiness, but testing time is multiplied.
On the bot 1161-1163 build was done with manually applied patch:
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1161
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1162
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1163

Build 1164 is running without the patch again.

[Allan Sandfeld Jensen]

(In reply to comment #8)
> (In reply to comment #7)
> > (In reply to comment #4)
> > > The uploaded patch fixes the problem. I do however think we might need to clear the access parameters between page-loads. This is a potential security hole, though it only affects plain-text HTML forms and authentication.
> > 
> > Never mind. It is the same in WebKit1, and the potential security issues are solved at a different level.
> 
> Yeah, it seems this patch wont fix flakiness, but testing time is multiplied.
> On the bot 1161-1163 build was done with manually applied patch:
> http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1161
> http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1162
> http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1163
> 
> Build 1164 is running without the patch again.

Right. It worked for the http/tests/xmlhttprequest tests though.

[Allan Sandfeld Jensen]

Created attachment 209862 [details]
Patch

[Jocelyn Turcotte]

Comment on attachment 209862 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=209862&action=review

This risks breaking something else, but let's see how the 5.1 bot reacts.

> Tools/ChangeLog:3
> +        [Qt][Wk2] Many tests are failing with flaky on Qt 5.1

Nit: This can be rephrased if you have a chance.

[Allan Sandfeld Jensen]

Committed r154733: <http://trac.webkit.org/changeset/154733>

[Ádám Kallai]

(In reply to comment #12)
> Committed r154733: <http://trac.webkit.org/changeset/154733>

It seems this fix didn't solve the problem. It didn't reduced number of flakiness tests. There are still many flaky tests on Qt 5.1 Wk2 bot.
For example:
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1179

http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1180

[Ádám Kallai]

I mark this bug as reopened because It needs more investigation.

[Allan Sandfeld Jensen]

(In reply to comment #14)
> I mark this bug as reopened because It needs more investigation.

Yes, that needs investigating. Oddly the same tests were also flaky on my machine but are no longer with this patch applied.

[Allan Sandfeld Jensen]

Created attachment 210468 [details]
Patch

Clearing the cache and credentials is also required, otherwise timeouts will break later tests.

[Allan Sandfeld Jensen]

Comment on attachment 210468 [details]
Patch

Unsetting review until I have the rest of the flaky tests solved.

[Allan Sandfeld Jensen]

A work-around that seems to solve all the flakiness is to in PlatformWebViewQt.cpp  PlatformWebView::viewSupportsOptions() to always return false.

This forces the TestController to create a new PlatformWebView for each test. This might be slightly slower, but I see not big difference in performance when testing it locally.

[Allan Sandfeld Jensen]

There appears to be different causes for the different flaky tests, so I have made two sub-bugs and uploaded patches for each there.

[Allan Sandfeld Jensen]

The two issues this was split into has been resolved.