Bug 276636 – PROGRESSION (280750@main): [ macOS wk1 ] imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-report-only-sends-reports-on-violation.https.sub.html is a consistent failure.

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Anfernee Viduya]

imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-report-only-sends-reports-on-violation.https.sub.html is consistently failing since 280750@main. 

HISTORY: 
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Freporting-api%2Freporting-api-report-only-sends-reports-on-violation.https.sub.html&platform=mac&architecture=arm64&architecture=x86_64&version_name=Sonoma&version_name=Ventura&flavor=wk1

DIFF:
 PASS Test that image does not load
 PASS Event is fired
-FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')
+PASS Violation report status OK.

LINK: 
https://build.webkit.org/results/Apple-Sonoma-Debug-AppleSilicon-WK1-Tests/280985@main%20(3593)/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-report-only-sends-reports-on-violation.https.sub-diff.txt

DESCRIPTION: 
It seems that in the commit, new WPT tests had been imported which caused this test to fail.

[Radar WebKit Bug Importer]

<rdar://problem/131793062>

[Anfernee Viduya]

REPRODUCIBILITY: 
I was able to reproduce this issue on ToT using command. 

run-webkit-tests --root 280977@main imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-report-only-sends-reports-on-violation.https.sub.html -1

[Alexey Proskuryakov]

This result is a progression (going from FAIL to PASS), and the nature of the change in 280750@main is consistent with that.

EWS didn't catch this because the test "failed" on first try and then "passed" when run singly, which suggests a problem with the test and/or test harness.

[Anfernee Viduya]

(In reply to Alexey Proskuryakov from comment #3)
> This result is a progression (going from FAIL to PASS), and the nature of
> the change in 280750@main is consistent with that.
> 
> EWS didn't catch this because the test "failed" on first try and then
> "passed" when run singly, which suggests a problem with the test and/or test
> harness.

Updating bug accordingly.

[Jonathan Bedard]

Seems like we just need a rebaseline.

[Anfernee Viduya]

(In reply to Jonathan Bedard from comment #5)
> Seems like we just need a rebaseline.

Initially I am hesitant on a re-baseline due to some queues being flakey and some being more consistent. I am currently seeking for guidance on a solution.

[EWS]

Committed 281973@main (1e5569ce34e2): <https://commits.webkit.org/281973@main>

Reviewed commits have been landed. Closing PR #31879 and removing active labels.

[Fujii Hironori]

284268@main reverted 280750@main and removed wk1 specific expectation. It is failing now again.

https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Freporting-api%2Freporting-api-report-only-sends-reports-on-violation.https.sub.html&flavor=wk1

This is the new text diff.

https://build.webkit.org/results/Apple-Sequoia-Release-AppleSilicon-WK1-Tests/284268@main%20(79)/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-report-only-sends-reports-on-violation.https.sub-diff.txt

--- /Volumes/Data/worker/Apple-Sequoia-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-report-only-sends-reports-on-violation.https.sub-expected.txt
+++ /Volumes/Data/worker/Apple-Sequoia-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-report-only-sends-reports-on-violation.https.sub-actual.txt
@@ -1,6 +1,8 @@
+CONSOLE MESSAGE: [Report Only] Refused to load https://localhost:9443/content-security-policy/support/fail.png because it does not appear in the img-src directive of the Content Security Policy.
+CONSOLE MESSAGE: [Report Only] Refused to load https://localhost:9443/content-security-policy/support/fail.png because it does not appear in the img-src directive of the Content Security Policy.
 
 
 PASS Test that image does not load
 PASS Event is fired
-PASS Violation report status OK.
+FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]')



This problem will be fixed by re-syncing wpt/tools.