Bug 187658 – REGRESSION: Layout Test http/tests/security/bypassing-cors-checks-for-extension-urls.html is Flaky

• Status: RESOLVED
• Resolution: FIXED
• Component: Page Loading

[Truitt Savell]

The following layout test is failing on Sierra and High Sierra WK2

http/tests/security/bypassing-cors-checks-for-extension-urls.html

Probable cause:

Investigating further. The test just became flakey on 7/12/18 and around r233780. Investigating further but at this time I am not certain what revision started the flakiness. What I can gather is that the test is no longer reliably receiving two console messages at the start of the output. 

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2Fsecurity%2Fbypassing-cors-checks-for-extension-urls.html


diff text:

--- /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/http/tests/security/bypassing-cors-checks-for-extension-urls-expected.txt
+++ /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/http/tests/security/bypassing-cors-checks-for-extension-urls-actual.txt
@@ -1,5 +1,3 @@
-CONSOLE MESSAGE: line 19: Cross origin requests are only supported for HTTP.
-CONSOLE MESSAGE: line 19: XMLHttpRequest cannot load safari-extension://test1 due to access control checks.
 This test ensures that XHR/Fetch will bypass CORS for user extension URLS in case the page is running user scripts.

[Ryan Haddad]

We should find out what revision caused this since it happened so recently.

[Truitt Savell]

Strange thing while trying to reproduce. if you run the test with a test runner it will pass the first time then constantly fail all iterations after that. run it with 6 runners and it passes 6 times and fails all further iterations. Tested and reproduced on r233780 and r233750. going to test further back.

[Alexey Proskuryakov]

This may mean that the test is flaky because a preceding test sets some state, in the same way as this test breaks attempts to run it more than once in a single process.

Definitely worth getting a repro case.

[Radar WebKit Bug Importer]

<rdar://problem/42306442>

[Truitt Savell]

*** This bug has been marked as a duplicate of bug 187541 ***

[Truitt Savell]

marked dup accidentally.

[Truitt Savell]

It was Noticed that this test:
http/tests/security/blocked-on-redirect.html

always seems to run directly before:
http/tests/security/bypassing-cors-checks-for-extension-urls.html

on the same test runner when it fails. If they are both run together repeatedly then they both will fail continuously. 

Example:
19:17:33.285 95669 worker/5 http/tests/security/blocked-on-redirect.html passed
19:17:33.444 95661 worker/1 http/tests/navigation/replacestate-base-legal.html passed
19:17:33.623 95661 worker/1 http/tests/navigation/replacestate-updates-referrer.html passed
19:17:33.910 95669 "ruby -I /Volumes/Data/slave/highsierra-debug-tests-wk2/build/Websites/bugs.webkit.org/PrettyPatch /Volumes/Data/slave/highsierra-debug-tests-wk2/build/Websites/bugs.webkit.org/PrettyPatch/prettify.rb /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/http/tests/security/bypassing-cors-checks-for-extension-urls-diff.txt" took 0.26s
19:17:33.913 95617 [22595/50037] http/tests/security/bypassing-cors-checks-for-extension-urls.html failed unexpectedly (text diff)
19:17:33.912 95669 worker/5 http/tests/security/bypassing-cors-checks-for-extension-urls.html failed:
19:17:33.912 95669 worker/5  text diff

[Ryan Haddad]

Marked test as flaky in https://trac.webkit.org/r234101

[Truitt Savell]

Update:
After bisecting through the test runner this test ran on I found the test that triggers the failure of http/tests/security/bypassing-cors-checks-for-extension-urls.html.

http/tests/contentextensions/injected-script-callback.html will cause this failure. 

Command:
run-webkit-tests --root testbuild-235549 http/tests/contentextensions/injected-script-callback.html http/tests/security/bypassing-cors-checks-for-extension-urls.html --child-processes 1 --force --no-retry

above command will %100 cause http/tests/security/bypassing-cors-checks-for-extension-urls.html to fail. 

http/tests/contentextensions/injected-script-callback.html 
was added in https://trac.webkit.org/changeset/233782/webkit 
which lines up with when http/tests/security/bypassing-cors-checks-for-extension-urls.html became flakey.

additional validation:
I skipped running http/tests/contentextensions/injected-script-callback.html in the test runner I was bisecting and no failure occurred with http/tests/security/bypassing-cors-checks-for-extension-urls.html

[youenn fablet]

Created attachment 348696 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 348696 [details]
Patch

Clearing flags on attachment: 348696

Committed r235602: <https://trac.webkit.org/changeset/235602>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.