Bug 179256 - Multiple http/wpt/beacon/contentextensions/ test are flaky.
Summary: Multiple http/wpt/beacon/contentextensions/ test are flaky.
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Chris Dumez
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2017-11-03 12:21 PDT by Matt Lewis
Modified: 2018-01-10 18:30 PST (History)
9 users (show)

See Also:


Attachments
Patch (4.29 KB, patch)
2018-01-10 09:56 PST, Chris Dumez
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews106 for mac-sierra-wk2 (2.69 MB, application/zip)
2018-01-10 11:50 PST, EWS Watchlist
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Lewis 2017-11-03 12:21:32 PDT
The tests:
http/wpt/beacon/contentextensions/beacon-blocked.html
http/wpt/beacon/contentextensions/beacon-redirect-blocked.html

are flaky according to the flakiness dashboard on Sierra and High Sierra.

They show signs of being flaky beyond the earliest run on the dashboard.

Example build:
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r224405%20(5492)/results.html
https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20(Tests)/builds/5492

History:
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Fwpt%2Fbeacon%2Fcontentextensions%2Fbeacon-blocked.html
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Fwpt%2Fbeacon%2Fcontentextensions%2Fbeacon-redirect-blocked.html


http/wpt/beacon/contentextensions/beacon-blocked.html diff

--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/wpt/beacon/contentextensions/beacon-blocked-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/wpt/beacon/contentextensions/beacon-blocked-actual.txt
@@ -1,5 +1,3 @@
-CONSOLE MESSAGE: line 9: Content blocker prevented frame displaying http://localhost:8800/WebKit/beacon/contentextensions/beacon-blocked.html from loading a resource from http://localhost:8800/WebKit/beacon/resources/beacon-preflight.py
-CONSOLE MESSAGE: line 9: Beacon API cannot load http://localhost:8800/WebKit/beacon/resources/beacon-preflight.py. Resource blocked by content blocker
 
-PASS Content extensions should be able to block beacons 
+FAIL Content extensions should be able to block beacons assert_false: sendBeacon should return false expected false got true
 

http/wpt/beacon/contentextensions/beacon-redirect-blocked.html diff:

--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/wpt/beacon/contentextensions/beacon-redirect-blocked-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/wpt/beacon/contentextensions/beacon-redirect-blocked-actual.txt
@@ -1,4 +1,3 @@
-CONSOLE MESSAGE: Beacon API cannot load http://127.0.0.1:8800/WebKit/beacon/resources/beacon-preflight.py?allowCors=1&cmd=put&id=f470f43c-258c-4c82-b880-ace3bcdb211c&redirect_status=307&location=http%3A%2F%2F127.0.0.1%3A8800%2FWebKit%2Fbeacon%2Fresources%2Fbeacon-preflight.py%3FallowCors%3D1%26cmd%3Dput%26id%3Df470f43c-258c-4c82-b880-ace3bcdb211c&count=1. Blocked by content extension
 
-PASS Content extensions should be able to block beacon redirects 
+FAIL Content extensions should be able to block beacon redirects assert_equals: Did not receive beacon expected 0 but got 1
Comment 1 Ryan Haddad 2017-11-03 14:58:13 PDT
From stderr:
2017-11-03 13:37:34.083 WebKitTestRunner[89471:31874437] Rule list compilation failed: Unspecified error during compile.
Comment 2 Radar WebKit Bug Importer 2017-11-07 16:27:51 PST
<rdar://problem/35401978>
Comment 3 Ryan Haddad 2017-11-10 09:44:51 PST
Marked tests as flaky in https://trac.webkit.org/r224690
Comment 4 Chris Dumez 2018-01-09 15:36:00 PST
This is apparently not specific to Beacon. All content extensions tests are currently flaky and sometimes give the following error on stderr:
"Rule list compilation failed: Unspecified error during compile.".
Comment 5 Chris Dumez 2018-01-09 15:43:06 PST
(In reply to Chris Dumez from comment #4)
> This is apparently not specific to Beacon. All content extensions tests are
> currently flaky and sometimes give the following error on stderr:
> "Rule list compilation failed: Unspecified error during compile.".

Alex added more logging recently and on recent builds we see:
Content Rule List compiling failed: Moving file failed.
2018-01-09 12:15:40.338 WebKitTestRunner[91903:69125462] Rule list compilation failed: Unspecified error during compile.
Comment 6 Chris Dumez 2018-01-09 15:46:54 PST
(In reply to Chris Dumez from comment #5)
> (In reply to Chris Dumez from comment #4)
> > This is apparently not specific to Beacon. All content extensions tests are
> > currently flaky and sometimes give the following error on stderr:
> > "Rule list compilation failed: Unspecified error during compile.".
> 
> Alex added more logging recently and on recent builds we see:
> Content Rule List compiling failed: Moving file failed.
> 2018-01-09 12:15:40.338 WebKitTestRunner[91903:69125462] Rule list
> compilation failed: Unspecified error during compile.

I also see:
2018-01-08 18:34:52.233 WebKitTestRunner[40704:65555369] Error in moveFile: Error Domain=NSCocoaErrorDomain Code=4 "“ContentRuleListSTFKD0” couldn’t be moved to “ContentExtensions” because either the former doesn't exist, or the folder containing the latter doesn't exist." UserInfo={NSSourceFilePathErrorKey=/var/folders/sz/cwvl9y7149bc1xwf4f_x1_k00000gn/T/WebKitTestRunners-pkI04D/ContentRuleListSTFKD0, NSUserStringVariant=(
    Move
), NSDestinationFilePath=/Users/buildbot/Library/WebKit/WebKitTestRunner/ContentExtensions/ContentExtension-TestContentExtensions, NSFilePath=/var/folders/sz/cwvl9y7149bc1xwf4f_x1_k00000gn/T/WebKitTestRunners-pkI04D/ContentRuleListSTFKD0, NSUnderlyingError=0x7fddf7074c80 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
Comment 7 Chris Dumez 2018-01-09 15:56:51 PST
(In reply to Chris Dumez from comment #6)
> (In reply to Chris Dumez from comment #5)
> > (In reply to Chris Dumez from comment #4)
> > > This is apparently not specific to Beacon. All content extensions tests are
> > > currently flaky and sometimes give the following error on stderr:
> > > "Rule list compilation failed: Unspecified error during compile.".
> > 
> > Alex added more logging recently and on recent builds we see:
> > Content Rule List compiling failed: Moving file failed.
> > 2018-01-09 12:15:40.338 WebKitTestRunner[91903:69125462] Rule list
> > compilation failed: Unspecified error during compile.
> 
> I also see:
> 2018-01-08 18:34:52.233 WebKitTestRunner[40704:65555369] Error in moveFile:
> Error Domain=NSCocoaErrorDomain Code=4 "“ContentRuleListSTFKD0” couldn’t be
> moved to “ContentExtensions” because either the former doesn't exist, or the
> folder containing the latter doesn't exist."
> UserInfo={NSSourceFilePathErrorKey=/var/folders/sz/
> cwvl9y7149bc1xwf4f_x1_k00000gn/T/WebKitTestRunners-pkI04D/
> ContentRuleListSTFKD0, NSUserStringVariant=(
>     Move
> ),
> NSDestinationFilePath=/Users/buildbot/Library/WebKit/WebKitTestRunner/
> ContentExtensions/ContentExtension-TestContentExtensions,
> NSFilePath=/var/folders/sz/cwvl9y7149bc1xwf4f_x1_k00000gn/T/
> WebKitTestRunners-pkI04D/ContentRuleListSTFKD0,
> NSUnderlyingError=0x7fddf7074c80 {Error Domain=NSPOSIXErrorDomain Code=2 "No
> such file or directory"}}

TestController::platformConfigureViewForTest(const TestInvocation& test)
uses
[_WKUserContentExtensionStore defaultStore]

which apparently ends up using the default store path, which ends up being shared by all WKTR instances that are running in parallel :(
Comment 8 Chris Dumez 2018-01-10 09:56:05 PST
Created attachment 330922 [details]
Patch
Comment 9 EWS Watchlist 2018-01-10 11:50:25 PST
Comment on attachment 330922 [details]
Patch

Attachment 330922 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/6023313

New failing tests:
http/tests/workers/service/postmessage-after-sw-process-crash.https.html
Comment 10 EWS Watchlist 2018-01-10 11:50:26 PST
Created attachment 330941 [details]
Archive of layout-test-results from ews106 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews106  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6
Comment 11 Chris Dumez 2018-01-10 13:09:27 PST
Comment on attachment 330941 [details]
Archive of layout-test-results from ews106 for mac-sierra-wk2

This failure cannot be related.
Comment 12 WebKit Commit Bot 2018-01-10 18:30:55 PST
Comment on attachment 330922 [details]
Patch

Clearing flags on attachment: 330922

Committed r226751: <https://trac.webkit.org/changeset/226751>
Comment 13 WebKit Commit Bot 2018-01-10 18:30:57 PST
All reviewed patches have been landed.  Closing bug.