Bug 118232 - [Qt][Wk2] Many tests are failing with flaky on Qt 5.1.
Summary: [Qt][Wk2] Many tests are failing with flaky on Qt 5.1.
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Allan Sandfeld Jensen
URL:
Keywords:
Depends on: 120774 120775
Blocks: 114171
  Show dependency treegraph
 
Reported: 2013-07-01 02:01 PDT by Ádám Kallai
Modified: 2013-09-11 08:13 PDT (History)
9 users (show)

See Also:


Attachments
Patch (6.99 KB, patch)
2013-08-27 09:17 PDT, Allan Sandfeld Jensen
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from webkit-ews-09 for mac-mountainlion-wk2 (471.18 KB, application/zip)
2013-08-27 14:07 PDT, Build Bot
no flags Details
Patch (1.59 KB, patch)
2013-08-28 03:45 PDT, Allan Sandfeld Jensen
no flags Details | Formatted Diff | Diff
Patch (3.78 KB, patch)
2013-09-04 09:20 PDT, Allan Sandfeld Jensen
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ádám Kallai 2013-07-01 02:01:34 PDT
Currently there are 127 flaky tests on Qt 5.1 Wk2 bot.

You can see it:
http://build.webkit.sed.hu/results/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/r152215%20(281)/results.html

Investigation in progress.
Comment 1 Allan Sandfeld Jensen 2013-08-27 02:49:32 PDT
access-control-preflight-credential-sync.html which is consistently flaky can be made to stop flaking by skipping the test before it: access-control-preflight-credential-async.html

My current theory is a problem with stopping all async requests.
Comment 2 Allan Sandfeld Jensen 2013-08-27 09:14:07 PDT
The underlying problem is that the state of QNetworkAccessManager in WebKit2 is not fully cleared between pages. Probably because WebKit1 does not have a shared QNAM, and QtTestSupport::clearMemoryCaches() can not access it because it lives in WebKit2.
Comment 3 Allan Sandfeld Jensen 2013-08-27 09:17:58 PDT
Created attachment 209780 [details]
Patch
Comment 4 Allan Sandfeld Jensen 2013-08-27 09:19:40 PDT
The uploaded patch fixes the problem. I do however think we might need to clear the access parameters between page-loads. This is a potential security hole, though it only affects plain-text HTML forms and authentication.
Comment 5 Build Bot 2013-08-27 14:07:27 PDT
Comment on attachment 209780 [details]
Patch

Attachment 209780 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.appspot.com/results/1605040

New failing tests:
http/tests/security/XFrameOptions/x-frame-options-deny-meta-tag.html
http/tests/security/XFrameOptions/x-frame-options-deny-meta-tag-parent-same-origin-deny.html
Comment 6 Build Bot 2013-08-27 14:07:29 PDT
Created attachment 209798 [details]
Archive of layout-test-results from webkit-ews-09 for mac-mountainlion-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: webkit-ews-09  Port: mac-mountainlion-wk2  Platform: Mac OS X 10.8.4
Comment 7 Allan Sandfeld Jensen 2013-08-28 01:54:51 PDT
(In reply to comment #4)
> The uploaded patch fixes the problem. I do however think we might need to clear the access parameters between page-loads. This is a potential security hole, though it only affects plain-text HTML forms and authentication.

Never mind. It is the same in WebKit1, and the potential security issues are solved at a different level.
Comment 8 Zoltan Arvai 2013-08-28 02:35:13 PDT
(In reply to comment #7)
> (In reply to comment #4)
> > The uploaded patch fixes the problem. I do however think we might need to clear the access parameters between page-loads. This is a potential security hole, though it only affects plain-text HTML forms and authentication.
> 
> Never mind. It is the same in WebKit1, and the potential security issues are solved at a different level.

Yeah, it seems this patch wont fix flakiness, but testing time is multiplied.
On the bot 1161-1163 build was done with manually applied patch:
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1161
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1162
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1163

Build 1164 is running without the patch again.
Comment 9 Allan Sandfeld Jensen 2013-08-28 02:50:48 PDT
(In reply to comment #8)
> (In reply to comment #7)
> > (In reply to comment #4)
> > > The uploaded patch fixes the problem. I do however think we might need to clear the access parameters between page-loads. This is a potential security hole, though it only affects plain-text HTML forms and authentication.
> > 
> > Never mind. It is the same in WebKit1, and the potential security issues are solved at a different level.
> 
> Yeah, it seems this patch wont fix flakiness, but testing time is multiplied.
> On the bot 1161-1163 build was done with manually applied patch:
> http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1161
> http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1162
> http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1163
> 
> Build 1164 is running without the patch again.

Right. It worked for the http/tests/xmlhttprequest tests though.
Comment 10 Allan Sandfeld Jensen 2013-08-28 03:45:19 PDT
Created attachment 209862 [details]
Patch
Comment 11 Jocelyn Turcotte 2013-08-28 05:09:45 PDT
Comment on attachment 209862 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=209862&action=review

This risks breaking something else, but let's see how the 5.1 bot reacts.

> Tools/ChangeLog:3
> +        [Qt][Wk2] Many tests are failing with flaky on Qt 5.1

Nit: This can be rephrased if you have a chance.
Comment 12 Allan Sandfeld Jensen 2013-08-28 05:12:06 PDT
Committed r154733: <http://trac.webkit.org/changeset/154733>
Comment 13 Ádám Kallai 2013-08-29 04:55:27 PDT
(In reply to comment #12)
> Committed r154733: <http://trac.webkit.org/changeset/154733>

It seems this fix didn't solve the problem. It didn't reduced number of flakiness tests. There are still many flaky tests on Qt 5.1 Wk2 bot.
For example:
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1179

http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20-%20Qt5.1%20-%20WebKit2/builds/1180
Comment 14 Ádám Kallai 2013-08-29 05:04:54 PDT
I mark this bug as reopened because It needs more investigation.
Comment 15 Allan Sandfeld Jensen 2013-08-29 13:48:41 PDT
(In reply to comment #14)
> I mark this bug as reopened because It needs more investigation.

Yes, that needs investigating. Oddly the same tests were also flaky on my machine but are no longer with this patch applied.
Comment 16 Allan Sandfeld Jensen 2013-09-04 09:20:27 PDT
Created attachment 210468 [details]
Patch

Clearing the cache and credentials is also required, otherwise timeouts will break later tests.
Comment 17 Allan Sandfeld Jensen 2013-09-04 09:27:03 PDT
Comment on attachment 210468 [details]
Patch

Unsetting review until I have the rest of the flaky tests solved.
Comment 18 Allan Sandfeld Jensen 2013-09-05 04:44:02 PDT
A work-around that seems to solve all the flakiness is to in PlatformWebViewQt.cpp  PlatformWebView::viewSupportsOptions() to always return false.

This forces the TestController to create a new PlatformWebView for each test. This might be slightly slower, but I see not big difference in performance when testing it locally.
Comment 19 Allan Sandfeld Jensen 2013-09-05 09:07:59 PDT
There appears to be different causes for the different flaky tests, so I have made two sub-bugs and uploaded patches for each there.
Comment 20 Allan Sandfeld Jensen 2013-09-11 08:13:22 PDT
The two issues this was split into has been resolved.