Bug 246358 – Test re-sync: [macOS WK1] imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub.html is frequently failing

• Component: New Bugs
• Status: NEW

[Ryan Haddad]

imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub.html is frequently failing on macOS WK1 bots after the re-sync in https://commits.webkit.org/255124@main

--- /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub-expected.txt
+++ /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub-actual.txt
@@ -3,5 +3,5 @@
 PASS Test that image does not load
 PASS Event is fired
 FAIL Report is observable to ReportingObserver assert_equals: expected 53 but got 0
-PASS Violation report status OK.
+FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')
 
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Freporting-api%2Freporting-api-sends-reports-on-violation.https.sub.html

[Radar WebKit Bug Importer]

<rdar://problem/101045517>

[Ryan Haddad]

Seeing a similar failure with another WPT test in https://bugs.webkit.org/show_bug.cgi?id=246351, probably related?

[Ryan Haddad]

Pull request: https://github.com/WebKit/WebKit/pull/5240

[EWS]

Test gardening commit 255395@main (4900b6842429): <https://commits.webkit.org/255395@main>

Reviewed commits have been landed. Closing PR #5240 and removing active labels.

[Ryan Reno]

Unfortunate this is flaky on WK1. The part that is failing is in an async defer <script> tag. Is that feature well supported on WK1?

[Ryan Haddad]

Looks like this may also apply to imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html, which was updated in the same re-sync:

--- /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub-expected.txt
+++ /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub-actual.txt
@@ -3,5 +3,5 @@
 PASS Test that image does not load
 PASS Event is fired
 FAIL Report is observable to ReportingObserver assert_equals: expected 54 but got 0
-FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')
+PASS Violation report status OK.
 

https://results.webkit.org/?suite=layout-tests&test=imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html

[Brent Fulgham]

Yeah, both the failures are happening in the 'checkReport.sub.js' script. I don't think the 'async defer' bits are relevant, because they would only impact the timing of the page being fully parsed/constructed (whether it waits for this script to load first).

Since the script runs an XHR, and continues when that XHR receives data, it seems like it should be resilient to timing differences.

However, this test failure makes it seem like the XHR completes (without error), but contains no data.

It's like the WPT Stash thing is sometimes returning an empty value, but is not treating that as an error state.

[Brent Fulgham]

(In reply to Brent Fulgham from comment #7)
> Yeah, both the failures are happening in the 'checkReport.sub.js' script. I
> don't think the 'async defer' bits are relevant, because they would only
> impact the timing of the page being fully parsed/constructed (whether it
> waits for this script to load first).
> 
> Since the script runs an XHR, and continues when that XHR receives data, it
> seems like it should be resilient to timing differences.
> 
> However, this test failure makes it seem like the XHR completes (without
> error), but contains no data.
> 
> It's like the WPT Stash thing is sometimes returning an empty value, but is
> not treating that as an error state.

Details: https://web-platform-tests.org/tools/wptserve/docs/stash.html