Bug 239568 – [ iOS ][ Monterey Release wk2 ] imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html is a flaky failure

• Status: REOPENED
• Component: New Bugs

[Karl Rackler]

Description:
imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html

The first failure that I saw on the dashboard was 3/21/2022 at r291559, but this does not appear relevant to causing this issue.  This test seems to have been flaky since the start.  

History:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Finheritance%2Fblob-url-in-main-window-self-navigate-inherits.sub.html&platform=mac&limit=50000&style=release&version_name=Monterey&flavor=wk2

Diff:
--- /Volumes/Data/worker/monterey-release-applesilicon-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-expected.txt
+++ /Volumes/Data/worker/monterey-release-applesilicon-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-actual.txt
@@ -1,3 +1,3 @@
 
-PASS Violation report status OK.
+FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')

[Radar WebKit Bug Importer]

<rdar://problem/92055624>

[Karl Rackler]

I have marked this test as a flaky/constant failure/image failure/timeout/crash while this issue is investigated.

[Karl Rackler]

I have marked this test as a flaky failure while this issue is investigated.

[Karl Rackler]

Pull request: https://github.com/WebKit/WebKit/pull/332

[EWS]

Committed r293128 (249830@main): <https://commits.webkit.org/249830@main>

Reviewed commits have been landed. Closing PR #332 and removing active labels.

[Karl Rackler]

Reopened because it is still being investigated.

[Karl Rackler]

Description:
imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html

This is also affecting iOS. 

History:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Finheritance%2Fblob-url-in-main-window-self-navigate-inherits.sub.html&platform=ios&limit=50000

Diff:
--- /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-expected.txt
+++ /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-actual.txt
@@ -1,3 +1,3 @@
 
-PASS Violation report status OK.
+FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')
 
I have marked this test as a flaky failure on iOS while this issue is investigated.

[Karl Rackler]

Pull request: https://github.com/WebKit/WebKit/pull/530

[EWS]

Test gardening commit r293859 (250324@main): <https://commits.webkit.org/250324@main>

Reviewed commits have been landed. Closing PR #530 and removing active labels.

[Matthew Finkel]

Pull request: https://github.com/webkit/WebKit/pull/2550

[Matthew Finkel]

I have a better handle on this bug now, so I'll summarize the behavior I'm seeing.

The below failure message indicates that the test expected a CSP report, but it did not arrive within a time-out period.

> FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')

The testharness *should* provide an explicit assertion failure and message when this happens, but there is a problem in that code path because it evaluates `data[0]` without verifying `data[0]` is defined.

With that foundation, I haven't successfully reproduced this issue locally, and AFAICT, it didn't trigger on EWS in the PR (comment #10) that deleted the flakiness expectation.

Interestingly, wpt.fyi does not show failures with STP: https://wpt.fyi/results/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html?label=master&label=experimental&aligned

But, I can reproduce the issue on wpt.live: http://wpt.live/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html

The latter failure was due to Safari's pop-up blocker, so that's possibly unrelated to the original flakiness. The issue is also not reproducible in Minibrowser.

Can we revert the test failure expectations and see if this is still flakey?

[Matthew Finkel]

Okay, quick follow-up on comment #11, I see the problem - and we don't need to revert the TestExpectations. I did see a different failure locally, but it was not the original problem. I just looked at the failures at the link in comment #7, and the test failures are now different. Looking at the logs from , we have:


> 08:12:30.123 98702 worker/3 "ruby -I /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/Websites/bugs.webkit.org/PrettyPatch /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/Websites/bugs.webkit.org/PrettyPatch/prettify.rb /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt" took 0.19s
> 08:12:30.125 98702 worker/3 imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub.html failed:
> 08:12:30.126 98702 worker/3 text diff
> 08:12:30.446 98702 worker/2 worker/2 imported/w3c/web-platform-tests/content-security-policy/gen/top.meta/worker-src-wildcard/worker-import-data.https.html output stderr lines:
> 08:12:30.446 98702 worker/2   CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://www1.localhost:9443/common/security-features/subresource/worker.py?redirection=keep-origin&action=purge&key=88efcd39-8d2e-47b7-83a0-210850398326&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy.
> 08:12:30.447 98702 worker/2   CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://www1.localhost:9443/common/security-features/subresource/worker.py?redirection=no-redirect&action=purge&key=a9a85678-1428-429e-990f-502d23601a37&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy.
> 08:12:30.447 98702 worker/2   CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://www1.localhost:9443/common/security-features/subresource/worker.py?redirection=swap-origin&action=purge&key=2418e145-e4ec-475d-8990-d739281ba09c&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy.
> 08:12:30.447 98702 worker/2   CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://localhost:9443/common/security-features/subresource/worker.py?redirection=keep-origin&action=purge&key=fc54bf0f-4e9b-46c7-86b8-9c6ef34de961&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy.
> 08:12:30.448 98702 worker/2   CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://localhost:9443/common/security-features/subresource/worker.py?redirection=no-redirect&action=purge&key=a190a9dc-6dfb-458c-aadd-e913b3246202&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy.
> 08:12:30.448 98702 worker/2   CONSOLE MESSAGE: Refused to load data:text/javascript,import 'https://localhost:9443/common/security-features/subresource/worker.py?redirection=swap-origin&action=purge&key=6721abbc-768a-43ac-a41d-e948b94b8ce3&path=%2Fmixed-content'; because it does not appear in the worker-src directive of the Content Security Policy.

That matches the failure I've seen locally.

> CONSOLE MESSAGE: Refused to execute a script because its hash, its nonce, or 'unsafe-inline' does not appear in the script-src directive of the Content Security Policy.

I'll investigate along this path.

[Matthew Finkel]

Obviously I misread the worker number in comment #12, and most of those log entries were not relevant, but we can ignore that fact.

I'm seeing three different behaviors across Mac and iOS platforms.

1) Looking at the test results from 252643@main, where we have failures on both iOS and Mac, they all show the same result:


https://build.webkit.org/results/Apple-iOS-15-Simulator-Debug-WK2-Tests/252643@main%20(3194)/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt

https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/252643@main%20(3840)/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt

https://build.webkit.org/results/Apple-Monterey-Release-WK2-Tests/252643@main%20(5018)/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt

================================================================
--- /Volumes/Data/worker/Apple-Monterey-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-expected.txt
+++ /Volumes/Data/worker/Apple-Monterey-Release-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-actual.txt
@@ -1,3 +1,4 @@
+CONSOLE MESSAGE: Refused to execute a script because its hash, its nonce, or 'unsafe-inline' does not appear in the script-src directive of the Content Security Policy.
 
 PASS Violation report status OK.
================================================================


2) However, there are runs like 252636@main where we see the original failure case:

https://build.webkit.org/results/Apple-iOS-15-Simulator-Debug-WK2-Tests/252636@main%20(3192)/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-diff.txt

================================================================
--- /Volumes/Data/worker/Apple-iOS-15-Simulator-Debug-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-expected.txt
+++ /Volumes/Data/worker/Apple-iOS-15-Simulator-Debug-WK2-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/inheritance/blob-url-in-main-window-self-navigate-inherits.sub-actual.txt
@@ -1,3 +1,3 @@
 
-PASS Violation report status OK.
+FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')
================================================================

This is flakiness, but it seems relatively rare, and at this moment I need more information to continue investigating this.

3) Monterey WK2 Debug, all Monterey WK1, and all Big Sur runs are consistently passing (based on the current expectation).

Next, ignoring (2) for now, we have a two competing results. (1) is caused because the default expectation sets DumpJSConsoleLogInStdErr, but the flakiness expectations added in comment #4 and comment #8 did not. I'll create a PR that changes this, so we can get a better signal on how (and when/why) the test is flaky.

I'll also open a ticket and PR for the bug in the WPT test harness.

[EWS]

Committed 252668@main (5c5cfc91f03e): <https://commits.webkit.org/252668@main>

Reviewed commits have been landed. Closing PR #2550 and removing active labels.

[Matthew Finkel]

Reopening for further investigation after we get some more data points:

https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Finheritance%2Fblob-url-in-main-window-self-navigate-inherits.sub.html&platform=ios&platform=mac&limit=1000

[Ryan Reno]

I refactored inheritance logic in these two commits which would have an impact on this test.

https://commits.webkit.org/254679@main
https://commits.webkit.org/255352@main

The recent results from the link in the previous comment show some failures and crashes but when clicking through to the actual results page for a failure there is no mention of this test.


The crashes we are seeing are crashes in the GPU process which is probably unrelated to the behavior under test.