Bug 232570

Summary: [ iOS Debug ] http/wpt/push-api tests are flaky failing
Product: WebKit Reporter: ayumi_kojima
Component: New BugsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: cdumez, konrad110392, nham, nicolas, olivier, webkit-bot-watchers-bugzilla, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: iPhone / iPad   
OS: Unspecified   

Description ayumi_kojima 2021-11-01 09:58:39 PDT
http/wpt/push-api/pushEvent.any.serviceworker.html
http/wpt/push-api/pushManager.any.html
http/wpt/push-api/pushManager.any.serviceworker.html

Are flaky failing on iOS 15 Debug.

History: https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&test=http%2Fwpt%2Fpush-api%2FpushEvent.any.serviceworker.html&test=http%2Fwpt%2Fpush-api%2FpushManager.any.html&test=http%2Fwpt%2Fpush-api%2FpushManager.any.serviceworker.html

Result page: https://build.webkit.org/results/Apple-iOS-15-Simulator-Debug-WK2-Tests/r285101%20(358)/results.html

Diff:

--- /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushEvent.any.serviceworker-expected.txt
+++ /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushEvent.any.serviceworker-actual.txt
@@ -1,10 +1,4 @@
 
-PASS PushEvent without data
-PASS PushEvent with data
-PASS PushEvent arrayBuffer handling
-PASS PushEvent with bad json
-PASS Wait for activate promise
-PASS Simulating firing of a push event
-PASS Simulating firing of a push event - successful waitUntil
-PASS Simulating firing of a push event - unsuccessful waitUntil
+Harness Error (TIMEOUT), message = null
 
+

Diff:

--- /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushManager.any-expected.txt
+++ /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushManager.any-actual.txt
@@ -1,15 +1,17 @@
 
-PASS subscribe should fail if there is no active service worker
-PASS wait for active service worker
+Harness Error (TIMEOUT), message = null
+
+TIMEOUT subscribe should fail if there is no active service worker Test timed out
+NOTRUN wait for active service worker
 PASS aes128gcm should be supported
 PASS supportedContentEncodings should be frozen
-PASS pushManager should return same object
-PASS subscribe requires userVisibleOnly to be true
-PASS subscribe requires applicationServerKey
-PASS applicationServerKey string should be base64url-encoded
-PASS applicationServerKey buffer should be a valid point on the P-256 curve
-PASS applicationServerKey string should be a valid point on the P-256 curve
-PASS can subscribe with valid applicationServerKey buffer
-PASS can subscribe with valid applicationServerKey string
-PASS unregister service worker
+NOTRUN pushManager should return same object
+NOTRUN subscribe requires userVisibleOnly to be true
+NOTRUN subscribe requires applicationServerKey
+NOTRUN applicationServerKey string should be base64url-encoded
+NOTRUN applicationServerKey buffer should be a valid point on the P-256 curve
+NOTRUN applicationServerKey string should be a valid point on the P-256 curve
+NOTRUN can subscribe with valid applicationServerKey buffer
+NOTRUN can subscribe with valid applicationServerKey string
+NOTRUN unregister service worker
 

Diff:

--- /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushManager.any.serviceworker-expected.txt
+++ /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/http/wpt/push-api/pushManager.any.serviceworker-actual.txt
@@ -1,14 +1,4 @@
 
-PASS subscribe should fail if there is no active service worker
-PASS wait for active service worker
-PASS aes128gcm should be supported
-PASS supportedContentEncodings should be frozen
-PASS pushManager should return same object
-PASS subscribe requires userVisibleOnly to be true
-PASS subscribe requires applicationServerKey
-PASS applicationServerKey string should be base64url-encoded
-PASS applicationServerKey buffer should be a valid point on the P-256 curve
-PASS applicationServerKey string should be a valid point on the P-256 curve
-PASS can subscribe with valid applicationServerKey buffer
-PASS can subscribe with valid applicationServerKey string
+Harness Error (TIMEOUT), message = null
 
+
Comment 1 Radar WebKit Bug Importer 2021-11-01 09:59:05 PDT
<rdar://problem/84885577>
Comment 2 ayumi_kojima 2021-11-01 10:12:51 PDT
Marked test expectations https://trac.webkit.org/changeset/285113/webkit
Comment 3 ayumi_kojima 2021-11-04 11:54:37 PDT
I was not able to reproduce the failures locally using run-webkit-tests --ios-simulator --iterations 1000 --exit-after-n-failures 1 http/wpt/push-api/pushEvent.any.serviceworker.html

Those tests were added relatively recently in https://trac.webkit.org/changeset/283916/webkit (http/wpt/push-api/pushManager.any.html & http/wpt/push-api/pushManager.any.serviceworker.html) and https://trac.webkit.org/changeset/283377/webkit (http/wpt/push-api/pushEvent.any.serviceworker.html)