Bug 234278

Summary: [ iOS ] 8 imported/w3c/web-platform-tests/fetch/api (layout-tests) are flaky text failures
Product: WebKit Reporter: Robert Jenner <jenner>
Component: New BugsAssignee: youenn fablet <youennf>
Status: RESOLVED FIXED    
Severity: Normal CC: webkit-bot-watchers-bugzilla, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Text Diffs
none
8 Text Diffs for failing tests none

Robert Jenner
Reported 2021-12-13 16:09:48 PST
The following 4 tests are flaky text failures on iOS: imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https.html imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https.html imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https.html imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https.html HISTORY: https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-no-referrer-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-frame.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-iframe.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Frequest-reset-attributes.https.html&platform=ios DIFF URLS: https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https-pretty-diff.html https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https-pretty-diff.html https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https-pretty-diff.html https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https-pretty-diff.html Each diff appears to have the following in common: +Harness Error (TIMEOUT), message = null I have attached all text diffs to this bug.
Attachments
Text Diffs (4.61 KB, application/zip)
2021-12-13 16:10 PST, Robert Jenner 		no flags
Details
8 Text Diffs for failing tests (9.45 KB, application/zip)
2021-12-13 16:59 PST, Robert Jenner 		no flags
Details
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.
Robert Jenner
Comment 1 2021-12-13 16:10:40 PST
Created attachment 447079 [details] Text Diffs
Robert Jenner
Comment 2 2021-12-13 16:11:31 PST
They are all flakey, but they all appear to have either the same failure history, or at least very similar failure points when observing their history.
Radar WebKit Bug Importer
Comment 3 2021-12-13 16:11:55 PST
<rdar://problem/86438914>
Robert Jenner
Comment 4 2021-12-13 16:35:52 PST
After looking deeper into the history, it appears there are actually 4 more associated tests that also appear to have a very similar failure history (eg. the tests fail at the same time when they fail.) The additional 4 tests are: imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-service-worker.https.html imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-when-cross-origin-service-worker.https.html imported/w3c/web-platform-tests/fetch/api/policies/referrer-unsafe-url-service-worker.https.html imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html HISTORY: https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-origin-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-origin-when-cross-origin-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-unsafe-url-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-worker.https.html&platform=ios DIFF URLS: https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https-pretty-diff.html https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https-pretty-diff.html https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https-pretty-diff.html https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https-pretty-diff.html These four additional tests all have the same thing in common in their diffs that the original 4 did posted above: +Harness Error (TIMEOUT), message = null I have attached all text diffs to this bug.
Robert Jenner
Comment 5 2021-12-13 16:43:33 PST
All of these tests had their first reported failure at r284862, and have continued since. I don't believe r284862 is what caused them to start failing, that's just what their history highlights. That revision was close to when we started testing iOS 15 on Open Source, so it appears that these are all just flaky tests on iOS 15.
Robert Jenner
Comment 6 2021-12-13 16:59:54 PST
Created attachment 447087 [details] 8 Text Diffs for failing tests
Robert Jenner
Comment 7 2021-12-13 17:09:09 PST
I was not able to reproduce any of these failures at Production or Debug iOS 15 ToT. I even took it up to 1000 iterations for each test, and they all passed. These tests randomly fail, and do not appear to have a specific regression point. I'm grouping them together because they have either the same failure history, or extremely similar failure history. They all also appear to deal with or be service worker tests. I have marked the expectations for these tests as Pass, Failure here: https://trac.webkit.org/changeset/286997/webkit
youenn fablet
Comment 8 2021-12-14 02:54:33 PST
All these tests are service worker tests that seem to fail registering/unregistering service workers on simulator.
youenn fablet
Comment 9 2022-06-19 23:11:42 PDT
imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https.html, imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https.html and imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https.html do not seem flaky anymore. imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https.html seems to crash sometimes (GPUProcess) but without any actual crash log.
youenn fablet
Comment 10 2022-06-19 23:14:50 PDT
imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-service-worker.https.html imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-when-cross-origin-service-worker.https.html imported/w3c/web-platform-tests/fetch/api/policies/referrer-unsafe-url-service-worker.https.html imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html All these tests are passing except imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html which sometimes crash. I'll fix the crash.
youenn fablet
Comment 11 2022-06-19 23:31:30 PDT
Fixing the crash at https://bugs.webkit.org/show_bug.cgi?id=241776. The test expectations should probably be unflaked.
youenn fablet
Comment 12 2022-06-19 23:34:20 PDT
Pull request: https://github.com/WebKit/WebKit/pull/1634
EWS
Comment 13 2022-06-20 05:03:38 PDT
Committed r295673 (251678@main): <https://commits.webkit.org/251678@main> Reviewed commits have been landed. Closing PR #1634 and removing active labels.
Note You need to log in before you can comment on or make changes to this bug.