Bug 179256 – Multiple http/wpt/beacon/contentextensions/ test are flaky.

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Matt Lewis]

The tests:
http/wpt/beacon/contentextensions/beacon-blocked.html
http/wpt/beacon/contentextensions/beacon-redirect-blocked.html

are flaky according to the flakiness dashboard on Sierra and High Sierra.

They show signs of being flaky beyond the earliest run on the dashboard.

Example build:
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r224405%20(5492)/results.html
https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20(Tests)/builds/5492

History:
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Fwpt%2Fbeacon%2Fcontentextensions%2Fbeacon-blocked.html
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Fwpt%2Fbeacon%2Fcontentextensions%2Fbeacon-redirect-blocked.html


http/wpt/beacon/contentextensions/beacon-blocked.html diff

--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/wpt/beacon/contentextensions/beacon-blocked-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/wpt/beacon/contentextensions/beacon-blocked-actual.txt
@@ -1,5 +1,3 @@
-CONSOLE MESSAGE: line 9: Content blocker prevented frame displaying http://localhost:8800/WebKit/beacon/contentextensions/beacon-blocked.html from loading a resource from http://localhost:8800/WebKit/beacon/resources/beacon-preflight.py
-CONSOLE MESSAGE: line 9: Beacon API cannot load http://localhost:8800/WebKit/beacon/resources/beacon-preflight.py. Resource blocked by content blocker
 
-PASS Content extensions should be able to block beacons 
+FAIL Content extensions should be able to block beacons assert_false: sendBeacon should return false expected false got true
 

http/wpt/beacon/contentextensions/beacon-redirect-blocked.html diff:

--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/wpt/beacon/contentextensions/beacon-redirect-blocked-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/wpt/beacon/contentextensions/beacon-redirect-blocked-actual.txt
@@ -1,4 +1,3 @@
-CONSOLE MESSAGE: Beacon API cannot load http://127.0.0.1:8800/WebKit/beacon/resources/beacon-preflight.py?allowCors=1&cmd=put&id=f470f43c-258c-4c82-b880-ace3bcdb211c&redirect_status=307&location=http%3A%2F%2F127.0.0.1%3A8800%2FWebKit%2Fbeacon%2Fresources%2Fbeacon-preflight.py%3FallowCors%3D1%26cmd%3Dput%26id%3Df470f43c-258c-4c82-b880-ace3bcdb211c&count=1. Blocked by content extension
 
-PASS Content extensions should be able to block beacon redirects 
+FAIL Content extensions should be able to block beacon redirects assert_equals: Did not receive beacon expected 0 but got 1

[Ryan Haddad]

From stderr:
2017-11-03 13:37:34.083 WebKitTestRunner[89471:31874437] Rule list compilation failed: Unspecified error during compile.

[Radar WebKit Bug Importer]

<rdar://problem/35401978>

[Ryan Haddad]

Marked tests as flaky in https://trac.webkit.org/r224690

[Chris Dumez]

This is apparently not specific to Beacon. All content extensions tests are currently flaky and sometimes give the following error on stderr:
"Rule list compilation failed: Unspecified error during compile.".

[Chris Dumez]

(In reply to Chris Dumez from comment #4)
> This is apparently not specific to Beacon. All content extensions tests are
> currently flaky and sometimes give the following error on stderr:
> "Rule list compilation failed: Unspecified error during compile.".

Alex added more logging recently and on recent builds we see:
Content Rule List compiling failed: Moving file failed.
2018-01-09 12:15:40.338 WebKitTestRunner[91903:69125462] Rule list compilation failed: Unspecified error during compile.

[Chris Dumez]

(In reply to Chris Dumez from comment #5)
> (In reply to Chris Dumez from comment #4)
> > This is apparently not specific to Beacon. All content extensions tests are
> > currently flaky and sometimes give the following error on stderr:
> > "Rule list compilation failed: Unspecified error during compile.".
> 
> Alex added more logging recently and on recent builds we see:
> Content Rule List compiling failed: Moving file failed.
> 2018-01-09 12:15:40.338 WebKitTestRunner[91903:69125462] Rule list
> compilation failed: Unspecified error during compile.

I also see:
2018-01-08 18:34:52.233 WebKitTestRunner[40704:65555369] Error in moveFile: Error Domain=NSCocoaErrorDomain Code=4 "“ContentRuleListSTFKD0” couldn’t be moved to “ContentExtensions” because either the former doesn't exist, or the folder containing the latter doesn't exist." UserInfo={NSSourceFilePathErrorKey=/var/folders/sz/cwvl9y7149bc1xwf4f_x1_k00000gn/T/WebKitTestRunners-pkI04D/ContentRuleListSTFKD0, NSUserStringVariant=(
    Move
), NSDestinationFilePath=/Users/buildbot/Library/WebKit/WebKitTestRunner/ContentExtensions/ContentExtension-TestContentExtensions, NSFilePath=/var/folders/sz/cwvl9y7149bc1xwf4f_x1_k00000gn/T/WebKitTestRunners-pkI04D/ContentRuleListSTFKD0, NSUnderlyingError=0x7fddf7074c80 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}

[Chris Dumez]

(In reply to Chris Dumez from comment #6)
> (In reply to Chris Dumez from comment #5)
> > (In reply to Chris Dumez from comment #4)
> > > This is apparently not specific to Beacon. All content extensions tests are
> > > currently flaky and sometimes give the following error on stderr:
> > > "Rule list compilation failed: Unspecified error during compile.".
> > 
> > Alex added more logging recently and on recent builds we see:
> > Content Rule List compiling failed: Moving file failed.
> > 2018-01-09 12:15:40.338 WebKitTestRunner[91903:69125462] Rule list
> > compilation failed: Unspecified error during compile.
> 
> I also see:
> 2018-01-08 18:34:52.233 WebKitTestRunner[40704:65555369] Error in moveFile:
> Error Domain=NSCocoaErrorDomain Code=4 "“ContentRuleListSTFKD0” couldn’t be
> moved to “ContentExtensions” because either the former doesn't exist, or the
> folder containing the latter doesn't exist."
> UserInfo={NSSourceFilePathErrorKey=/var/folders/sz/
> cwvl9y7149bc1xwf4f_x1_k00000gn/T/WebKitTestRunners-pkI04D/
> ContentRuleListSTFKD0, NSUserStringVariant=(
>     Move
> ),
> NSDestinationFilePath=/Users/buildbot/Library/WebKit/WebKitTestRunner/
> ContentExtensions/ContentExtension-TestContentExtensions,
> NSFilePath=/var/folders/sz/cwvl9y7149bc1xwf4f_x1_k00000gn/T/
> WebKitTestRunners-pkI04D/ContentRuleListSTFKD0,
> NSUnderlyingError=0x7fddf7074c80 {Error Domain=NSPOSIXErrorDomain Code=2 "No
> such file or directory"}}

TestController::platformConfigureViewForTest(const TestInvocation& test)
uses
[_WKUserContentExtensionStore defaultStore]

which apparently ends up using the default store path, which ends up being shared by all WKTR instances that are running in parallel :(

[Chris Dumez]

Created attachment 330922 [details]
Patch

[EWS Watchlist]

Comment on attachment 330922 [details]
Patch

Attachment 330922 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/6023313

New failing tests:
http/tests/workers/service/postmessage-after-sw-process-crash.https.html

[EWS Watchlist]

Created attachment 330941 [details]
Archive of layout-test-results from ews106 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews106  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6

[Chris Dumez]

Comment on attachment 330941 [details]
Archive of layout-test-results from ews106 for mac-sierra-wk2

This failure cannot be related.

[WebKit Commit Bot]

Comment on attachment 330922 [details]
Patch

Clearing flags on attachment: 330922

Committed r226751: <https://trac.webkit.org/changeset/226751>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.