Bug 232028 – Regression (r284510?): [ iOS macOS wk2 ] imported/w3c/web-platform-tests/html/cross-origin-opener-policy/reporting/navigation-reporting/report-only-four-reports.https.html is a flaky failure

• Component: New Bugs
• Status: RESOLVED DUPLICATE

imported/w3c/web-platform-tests/html/cross-origin-opener-policy/reporting/navigation-reporting/report-only-four-reports.https.html

Is a flaky failure on iOS 15 and macOS wk2.

History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fcross-origin-opener-policy%2Freporting%2Fnavigation-reporting%2Freport-only-four-reports.https.html

Result page: https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284525%20(239)/results.html

Diff:

--- /Volumes/Data/worker/ios-simulator-15-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/cross-origin-opener-policy/reporting/navigation-reporting/report-only-four-reports.https-expected.txt
+++ /Volumes/Data/worker/ios-simulator-15-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/cross-origin-opener-policy/reporting/navigation-reporting/report-only-four-reports.https-actual.txt
@@ -1,4 +1,4 @@
 
-PASS coop reporting test A test with both COOP and COOP report only setup to CROSS_ORIGIN with same-origin-allow-popups; report-to="coop-popup-report-endpoint", require-corp, same-origin; report-to="coop-popup-report-only-endpoint", require-corp
-PASS verify remaining reports
+FAIL coop reporting test A test with both COOP and COOP report only setup to CROSS_ORIGIN with same-origin-allow-popups; report-to="coop-popup-report-endpoint", require-corp, same-origin; report-to="coop-popup-report-only-endpoint", require-corp promise_test: Unhandled rejection with value: "No report matched the expected report for endpoint: coop-popup-report-endpoint, expected report: {\"body\":{\"disposition\":\"enforce\",\"effectivePolicy\":\"same-origin-allow-popups\",\"previousResponseURL\":\"\",\"referrer\":\"https://localhost:9443/\",\"type\":\"navigation-to-response\"},\"url\":\"/uuid=(uuid)$/\",\"type\":\"coop\"}, within available reports: [{\"type\":\"coop\",\"url\":\"https://127.0.0.1:9443/common/dispatcher/executor.html?pipe=|header(report-to,%7B%22group%22%3A%22coop-report-endpoint%22%5C%2C%22max_age%22%3A3600%5C%2C%22endpoints%22%3A%5B%7B%22url%22%3A%22https%3A%2F%2F127.0.0.1%3A9443%2Freporting%2Fresources%2Freport.py%3FreportID%3D(uuid)%22%7D%5D%7D%5C%2C%7B%22group%22%3A%22coop-report-only-endpoint%22%5C%2C%22max_age%22%3A3600%5C%2C%22endpoints%22%3A%5B%7B%22url%22%3A%22https%3A%2F%2F127.0.0.1%3A9443%2Freporting%2Fresources%2Freport.py%3FreportID%3D(uuid)%22%7D%5D%7D%5C%2C%7B%22group%22%3A%22coop-popup-report-endpoint%22%5C%2C%22max_age%22%3A3600%5C%2C%22endpoints%22%3A%5B%7B%22url%22%3A%22https%3A%2F%2F127.0.0.1%3A9443%2Freporting%2Fresources%2Freport.py%3FreportID%3D(uuid)%22%7D%5D%7D%5C%2C%7B%22group%22%3A%22coop-popup-report-only-endpoint%22%5C%2C%22max_age%22%3A3600%5C%2C%22endpoints%22%3A%5B%7B%22url%22%3A%22https%3A%2F%2F127.0.0.1%3A9443%2Freporting%2Fresources%2Freport.py%3FreportID%3D(uuid)%22%7D%5D%7D%5C%2C%7B%22group%22%3A%22coop-redirect-report-endpoint%22%5C%2C%22max_age%22%3A3600%5C%2C%22endpoints%22%3A%5B%7B%22url%22%3A%22https%3A%2F%2F127.0.0.1%3A9443%2Freporting%2Fresources%2Freport.py%3FreportID%3D(uuid)%22%7D%5D%7D%5C%2C%7B%22group%22%3A%22coop-redirect-report-only-endpoint%22%5C%2C%22max_age%22%3A3600%5C%2C%22endpoints%22%3A%5B%7B%22url%22%3A%22https%3A%2F%2F127.0.0.1%3A9443%2Freporting%2Fresources%2Freport.py%3FreportID%3D(uuid)%22%7D%5D%7D)|header(Cross-Origin-Opener-Policy,same-origin-allow-popups%3B%20report-to%3D%22coop-popup-report-endpoint%22)|header(Cross-Origin-Embedder-Policy,require-corp)|header(Cross-Origin-Opener-Policy-Report-Only,same-origin%3B%20report-to%3D%22coop-popup-report-only-endpoint%22)|header(Cross-Origin-Embedder-Policy-Report-Only,require-corp)&uuid=(uuid)\",\"user_agent\":\"Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148\",\"age\":0,\"body\":{\"disposition\":\"enforce\",\"effectivePolicy\":\"same-origin-allow-popups\",\"previousResponseURL\":\"\",\"type\":\"navigation-to-response\",\"referrer\":\"https://localhost:9443/\"},\"metadata\":{\"content_type\":\"application/reports+json\"}}]"
+FAIL verify remaining reports assert_equals: coop-popup-report-endpoint should be empty expected 0 but got 1

[Radar WebKit Bug Importer]

<rdar://problem/84472339>

Marked test expectations https://trac.webkit.org/changeset/284555/webkit

I was able to reproduce the failure locally using run-webkit-tests --ios-simulator --iterations 50 --exit-after-n-failures 1 imported/w3c/web-platform-tests/html/cross-origin-opener-policy/reporting/navigation-reporting/report-only-four-reports.https.html -f

The test failed at r284510. No revision available to test for r284509, but I think the failure started at https://trac.webkit.org/changeset/284510/webkit.

[Chris Dumez]

(In reply to ayumi_kojima from comment #4)
> The test failed at r284510. No revision available to test for r284509, but I
> think the failure started at https://trac.webkit.org/changeset/284510/webkit.

r284510 is when I resync'd the test from upstream. I'll see what changes got merged for this particular test.

[Chris Dumez]

Looks like I was able to reproduce as well:

--- /Volumes/Data/WebKit/OpenSource/WebKitBuild/Release-iphonesimulator/layout-test-results/imported/w3c/web-platform-tests/html/cross-origin-opener-policy/reporting/navigation-reporting/report-only-four-reports.https-expected.txt
+++ /Volumes/Data/WebKit/OpenSource/WebKitBuild/Release-iphonesimulator/layout-test-results/imported/w3c/web-platform-tests/html/cross-origin-opener-policy/reporting/navigation-reporting/report-only-four-reports.https-actual.txt
@@ -1,4 +1,4 @@
 
-PASS coop reporting test A test with both COOP and COOP report only setup to CROSS_ORIGIN with same-origin-allow-popups; report-to="coop-popup-report-endpoint", require-corp, same-origin; report-to="coop-popup-report-only-endpoint", require-corp
+FAIL coop reporting test A test with both COOP and COOP report only setup to CROSS_ORIGIN with same-origin-allow-popups; report-to="coop-popup-report-endpoint", require-corp, same-origin; report-to="coop-popup-report-only-endpoint", require-corp promise_test: Unhandled rejection with value: "No report matched the expected report for endpoint: coop-report-endpoint, expected report: {\"body\":{\"disposition\":\"enforce\",\"effectivePolicy\":\"same-origin-allow-popups\",\"nextResponseURL\":\"/uuid=(uuid)$/\",\"type\":\"navigation-from-response\"},\"url\":\"https://localhost:9443/html/cross-origin-opener-policy/reporting/navigation-reporting/report-only-four-reports.https.html\",\"type\":\"coop\"}, within available reports: []"
 PASS verify remaining reports

[Chris Dumez]

*** This bug has been marked as a duplicate of bug 234899 ***