Bug 239568

Summary: [ iOS ][ Monterey Release wk2 ] imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html is a flaky failure
Product: WebKit Reporter: Karl Rackler <rackler>
Component: New BugsAssignee: Matthew Finkel <m_finkel>
Status: REOPENED    
Severity: Normal CC: m_finkel, peanutsunless, rreno, webkit-bot-watchers-bugzilla, webkit-bug-importer, wilander
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=246358

Karl Rackler
Reported 2022-04-20 15:28:54 PDT
Description: imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html The first failure that I saw on the dashboard was 3/21/2022 at r291559, but this does not appear relevant to causing this issue. This test seems to have been flaky since the start. History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Finheritance%2Fblob-url-in-main-window-self-navigate-inherits.sub.html&platform=mac&limit=50000&style=release&version_name=Monterey&flavor=wk2 Diff: --- /Volumes/Data/worker/monterey-release-applesilicon-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-expected.txt +++ /Volumes/Data/worker/monterey-release-applesilicon-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-actual.txt @@ -1,3 +1,3 @@ -PASS Violation report status OK. +FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')
Attachments
Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2022-04-20 15:29:11 PDT
<rdar://problem/92055624>
Karl Rackler
Comment 2 2022-04-20 15:38:08 PDT
I have marked this test as a flaky/constant failure/image failure/timeout/crash while this issue is investigated.
Karl Rackler
Comment 3 2022-04-20 15:38:45 PDT
I have marked this test as a flaky failure while this issue is investigated.
Karl Rackler
Comment 4 2022-04-20 15:43:31 PDT
Pull request: https://github.com/WebKit/WebKit/pull/332
EWS
Comment 5 2022-04-20 15:48:46 PDT
Committed r293128 (249830@main): <https://commits.webkit.org/249830@main> Reviewed commits have been landed. Closing PR #332 and removing active labels.
Karl Rackler
Comment 6 2022-04-20 15:49:02 PDT
Reopened because it is still being investigated.
Karl Rackler
Comment 7 2022-05-05 13:59:53 PDT
Description: imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html This is also affecting iOS. History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Finheritance%2Fblob-url-in-main-window-self-navigate-inherits.sub.html&platform=ios&limit=50000 Diff: --- /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-expected.txt +++ /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-actual.txt @@ -1,3 +1,3 @@ -PASS Violation report status OK. +FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]') I have marked this test as a flaky failure on iOS while this issue is investigated.
Karl Rackler
Comment 8 2022-05-05 14:06:23 PDT
Pull request: https://github.com/WebKit/WebKit/pull/530
EWS
Comment 9 2022-05-05 14:10:54 PDT
Test gardening commit r293859 (250324@main): <https://commits.webkit.org/250324@main> Reviewed commits have been landed. Closing PR #530 and removing active labels.
Matthew Finkel
Comment 10 2022-07-19 09:28:49 PDT
Pull request: https://github.com/webkit/WebKit/pull/2550
Matthew Finkel
Comment 11 2022-07-19 17:13:42 PDT
I have a better handle on this bug now, so I'll summarize the behavior I'm seeing. The below failure message indicates that the test expected a CSP report, but it did not arrive within a time-out period. > FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]') The testharness *should* provide an explicit assertion failure and message when this happens, but there is a problem in that code path because it evaluates `data[0]` without verifying `data[0]` is defined. With that foundation, I haven't successfully reproduced this issue locally, and AFAICT, it didn't trigger on EWS in the PR (comment #10) that deleted the flakiness expectation. Interestingly, wpt.fyi does not show failures with STP: https://wpt.fyi/results/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html?label=master&label=experimental&aligned But, I can reproduce the issue on wpt.live: http://wpt.live/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html The latter failure was due to Safari's pop-up blocker, so that's possibly unrelated to the original flakiness. The issue is also not reproducible in Minibrowser. Can we revert the test failure expectations and see if this is still flakey?
Matthew Finkel
Comment 12 2022-07-19 17:26:59 PDT
Okay, quick follow-up on comment #11, I see the problem - and we don't need to revert the TestExpectations. I did see a different failure locally, but it was not the original problem. I just looked at the failures at the link in comment #7, and the test failures are now different. Looking at the logs from , we have: > 08:12:30.123 98702 worker/3 "ruby -I /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/Websites/bugs.webkit.org/PrettyPatch /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/Websites/bugs.webkit.org/PrettyPatch/prettify.rb /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt" took 0.19s > 08:12:30.125 98702 worker/3 imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html failed: > 08:12:30.126 98702 worker/3 text diff > 08:12:30.446 98702 worker/2 worker/2 imported/w3c/web-platform-tests/content-security-policy/gen/top.meta/worker-src-wildcard/worker-import-data.https.html output stderr lines: > 08:12:30.446 98702 worker/2 CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://www1.localhost:9443/common/security-features/subresource/worker.py?redirection=keep-origin&action=purge&key=88efcd39-8d2e-47b7-83a0-210850398326&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy. > 08:12:30.447 98702 worker/2 CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://www1.localhost:9443/common/security-features/subresource/worker.py?redirection=no-redirect&action=purge&key=a9a85678-1428-429e-990f-502d23601a37&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy. > 08:12:30.447 98702 worker/2 CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://www1.localhost:9443/common/security-features/subresource/worker.py?redirection=swap-origin&action=purge&key=2418e145-e4ec-475d-8990-d739281ba09c&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy. > 08:12:30.447 98702 worker/2 CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://localhost:9443/common/security-features/subresource/worker.py?redirection=keep-origin&action=purge&key=fc54bf0f-4e9b-46c7-86b8-9c6ef34de961&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy. > 08:12:30.448 98702 worker/2 CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://localhost:9443/common/security-features/subresource/worker.py?redirection=no-redirect&action=purge&key=a190a9dc-6dfb-458c-aadd-e913b3246202&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy. > 08:12:30.448 98702 worker/2 CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://localhost:9443/common/security-features/subresource/worker.py?redirection=swap-origin&action=purge&key=6721abbc-768a-43ac-a41d-e948b94b8ce3&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy. That matches the failure I've seen locally. > CONSOLE MESSAGE: Refused to execute a script because its hash, its nonce, or 'unsafe-inline' does not appear in the script-src directive of the Content Security Policy. I'll investigate along this path.
Matthew Finkel
Comment 13 2022-07-20 16:16:43 PDT
Obviously I misread the worker number in comment #12, and most of those log entries were not relevant, but we can ignore that fact. I'm seeing three different behaviors across Mac and iOS platforms. 1) Looking at the test results from 252643@main, where we have failures on both iOS and Mac, they all show the same result: https://build.webkit.org/results/Apple-iOS-15-Simulator-Debug-WK2-Tests/252643@main%20(3194)/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/252643@main%20(3840)/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt https://build.webkit.org/results/Apple-Monterey-Release-WK2-Tests/252643@main%20(5018)/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt ================================================================ --- /Volumes/Data/worker/Apple-Monterey-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-expected.txt +++ /Volumes/Data/worker/Apple-Monterey-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-actual.txt @@ -1,3 +1,4 @@ +CONSOLE MESSAGE: Refused to execute a script because its hash, its nonce, or 'unsafe-inline' does not appear in the script-src directive of the Content Security Policy. PASS Violation report status OK. ================================================================ 2) However, there are runs like 252636@main where we see the original failure case: https://build.webkit.org/results/Apple-iOS-15-Simulator-Debug-WK2-Tests/252636@main%20(3192)/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt ================================================================ --- /Volumes/Data/worker/Apple-iOS-15-Simulator-Debug-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-expected.txt +++ /Volumes/Data/worker/Apple-iOS-15-Simulator-Debug-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-actual.txt @@ -1,3 +1,3 @@ -PASS Violation report status OK. +FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]') ================================================================ This is flakiness, but it seems relatively rare, and at this moment I need more information to continue investigating this. 3) Monterey WK2 Debug, all Monterey WK1, and all Big Sur runs are consistently passing (based on the current expectation). Next, ignoring (2) for now, we have a two competing results. (1) is caused because the default expectation sets DumpJSConsoleLogInStdErr, but the flakiness expectations added in comment #4 and comment #8 did not. I'll create a PR that changes this, so we can get a better signal on how (and when/why) the test is flaky. I'll also open a ticket and PR for the bug in the WPT test harness.
EWS
Comment 14 2022-07-20 17:04:46 PDT
Committed 252668@main (5c5cfc91f03e): <https://commits.webkit.org/252668@main> Reviewed commits have been landed. Closing PR #2550 and removing active labels.
Ryan Reno
Comment 16 2022-10-12 16:10:46 PDT
I refactored inheritance logic in these two commits which would have an impact on this test. https://commits.webkit.org/254679@main https://commits.webkit.org/255352@main The recent results from the link in the previous comment show some failures and crashes but when clicking through to the actual results page for a failure there is no mention of this test. The crashes we are seeing are crashes in the GPU process which is probably unrelated to the behavior under test.
Note You need to log in before you can comment on or make changes to this bug.