Bug 232570 – [ iOS Debug ] http/wpt/push-api tests are flaky failing

• Status: NEW
• Component: New Bugs

http/wpt/push-api/pushEvent.any.serviceworker.html
http/wpt/push-api/pushManager.any.html
http/wpt/push-api/pushManager.any.serviceworker.html

Are flaky failing on iOS 15 Debug.

History: https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&test=http%2Fwpt%2Fpush-api%2FpushEvent.any.serviceworker.html&test=http%2Fwpt%2Fpush-api%2FpushManager.any.html&test=http%2Fwpt%2Fpush-api%2FpushManager.any.serviceworker.html

Result page: https://build.webkit.org/results/Apple-iOS-15-Simulator-Debug-WK2-Tests/r285101%20(358)/results.html

Diff:

--- /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushEvent.any.serviceworker-expected.txt
+++ /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushEvent.any.serviceworker-actual.txt
@@ -1,10 +1,4 @@
 
-PASS PushEvent without data
-PASS PushEvent with data
-PASS PushEvent arrayBuffer handling
-PASS PushEvent with bad json
-PASS Wait for activate promise
-PASS Simulating firing of a push event
-PASS Simulating firing of a push event - successful waitUntil
-PASS Simulating firing of a push event - unsuccessful waitUntil
+Harness Error (TIMEOUT), message = null
 
+

Diff:

--- /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushManager.any-expected.txt
+++ /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushManager.any-actual.txt
@@ -1,15 +1,17 @@
 
-PASS subscribe should fail if there is no active service worker
-PASS wait for active service worker
+Harness Error (TIMEOUT), message = null
+
+TIMEOUT subscribe should fail if there is no active service worker Test timed out
+NOTRUN wait for active service worker
 PASS aes128gcm should be supported
 PASS supportedContentEncodings should be frozen
-PASS pushManager should return same object
-PASS subscribe requires userVisibleOnly to be true
-PASS subscribe requires applicationServerKey
-PASS applicationServerKey string should be base64url-encoded
-PASS applicationServerKey buffer should be a valid point on the P-256 curve
-PASS applicationServerKey string should be a valid point on the P-256 curve
-PASS can subscribe with valid applicationServerKey buffer
-PASS can subscribe with valid applicationServerKey string
-PASS unregister service worker
+NOTRUN pushManager should return same object
+NOTRUN subscribe requires userVisibleOnly to be true
+NOTRUN subscribe requires applicationServerKey
+NOTRUN applicationServerKey string should be base64url-encoded
+NOTRUN applicationServerKey buffer should be a valid point on the P-256 curve
+NOTRUN applicationServerKey string should be a valid point on the P-256 curve
+NOTRUN can subscribe with valid applicationServerKey buffer
+NOTRUN can subscribe with valid applicationServerKey string
+NOTRUN unregister service worker
 

Diff:

--- /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushManager.any.serviceworker-expected.txt
+++ /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushManager.any.serviceworker-actual.txt
@@ -1,14 +1,4 @@
 
-PASS subscribe should fail if there is no active service worker
-PASS wait for active service worker
-PASS aes128gcm should be supported
-PASS supportedContentEncodings should be frozen
-PASS pushManager should return same object
-PASS subscribe requires userVisibleOnly to be true
-PASS subscribe requires applicationServerKey
-PASS applicationServerKey string should be base64url-encoded
-PASS applicationServerKey buffer should be a valid point on the P-256 curve
-PASS applicationServerKey string should be a valid point on the P-256 curve
-PASS can subscribe with valid applicationServerKey buffer
-PASS can subscribe with valid applicationServerKey string
+Harness Error (TIMEOUT), message = null
 
+

[Radar WebKit Bug Importer]

<rdar://problem/84885577>

Marked test expectations https://trac.webkit.org/changeset/285113/webkit

I was not able to reproduce the failures locally using run-webkit-tests --ios-simulator --iterations 1000 --exit-after-n-failures 1 http/wpt/push-api/pushEvent.any.serviceworker.html

Those tests were added relatively recently in https://trac.webkit.org/changeset/283916/webkit (http/wpt/push-api/pushManager.any.html & http/wpt/push-api/pushManager.any.serviceworker.html) and https://trac.webkit.org/changeset/283377/webkit (http/wpt/push-api/pushEvent.any.serviceworker.html)