246358 2022-10-11 12:34:17 -0700 Test re-sync: [macOS WK1] imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub.html is frequently failing 2022-10-12 15:58:42 -0700 1 1 1 Unclassified WebKit New Bugs Other Unspecified Unspecified NEW https://bugs.webkit.org/show_bug.cgi?id=246351 https://bugs.webkit.org/show_bug.cgi?id=239568 InRadar P2 Normal --- 1 ryanhaddad webkit-unassigned bfulgham rreno webkit-bot-watchers-bugzilla webkit-bug-importer oldest_to_newest 1904848 0 ryanhaddad 2022-10-11 12:34:17 -0700 imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub.html is frequently failing on macOS WK1 bots after the re-sync in https://commits.webkit.org/255124@main --- /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub-expected.txt +++ /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub-actual.txt @@ -3,5 +3,5 @@ PASS Test that image does not load PASS Event is fired FAIL Report is observable to ReportingObserver assert_equals: expected 53 but got 0 -PASS Violation report status OK. +FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]') https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Freporting-api%2Freporting-api-sends-reports-on-violation.https.sub.html 1904849 1 webkit-bug-importer 2022-10-11 12:35:17 -0700 <rdar://problem/101045517> 1904850 2 ryanhaddad 2022-10-11 12:35:45 -0700 Seeing a similar failure with another WPT test in https://bugs.webkit.org/show_bug.cgi?id=246351, probably related? 1904851 3 ryanhaddad 2022-10-11 12:41:33 -0700 Pull request: https://github.com/WebKit/WebKit/pull/5240 1904852 4 ews-feeder 2022-10-11 12:44:34 -0700 Test gardening commit 255395@main (4900b6842429): <https://commits.webkit.org/255395@main> Reviewed commits have been landed. Closing PR #5240 and removing active labels. 1904861 5 rreno 2022-10-11 13:01:02 -0700 Unfortunate this is flaky on WK1. The part that is failing is in an async defer <script> tag. Is that feature well supported on WK1? 1904879 6 ryanhaddad 2022-10-11 13:40:01 -0700 Looks like this may also apply to imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html, which was updated in the same re-sync: --- /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub-expected.txt +++ /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub-actual.txt @@ -3,5 +3,5 @@ PASS Test that image does not load PASS Event is fired FAIL Report is observable to ReportingObserver assert_equals: expected 54 but got 0 -FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]') +PASS Violation report status OK. https://results.webkit.org/?suite=layout-tests&test=imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html 1904951 7 bfulgham 2022-10-11 17:10:04 -0700 Yeah, both the failures are happening in the 'checkReport.sub.js' script. I don't think the 'async defer' bits are relevant, because they would only impact the timing of the page being fully parsed/constructed (whether it waits for this script to load first). Since the script runs an XHR, and continues when that XHR receives data, it seems like it should be resilient to timing differences. However, this test failure makes it seem like the XHR completes (without error), but contains no data. It's like the WPT Stash thing is sometimes returning an empty value, but is not treating that as an error state. 1904952 8 bfulgham 2022-10-11 17:10:24 -0700 (In reply to Brent Fulgham from comment #7) > Yeah, both the failures are happening in the 'checkReport.sub.js' script. I > don't think the 'async defer' bits are relevant, because they would only > impact the timing of the page being fully parsed/constructed (whether it > waits for this script to load first). > > Since the script runs an XHR, and continues when that XHR receives data, it > seems like it should be resilient to timing differences. > > However, this test failure makes it seem like the XHR completes (without > error), but contains no data. > > It's like the WPT Stash thing is sometimes returning an empty value, but is > not treating that as an error state. Details: https://web-platform-tests.org/tools/wptserve/docs/stash.html