Bug 234278 - [ iOS ] 8 imported/w3c/web-platform-tests/fetch/api (layout-tests) are flaky text failures
Summary: [ iOS ] 8 imported/w3c/web-platform-tests/fetch/api (layout-tests) are flaky ...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-12-13 16:09 PST by Robert Jenner
Modified: 2022-06-20 05:03 PDT (History)
3 users (show)

See Also:


Attachments
Text Diffs (4.61 KB, application/zip)
2021-12-13 16:10 PST, Robert Jenner
no flags Details
8 Text Diffs for failing tests (9.45 KB, application/zip)
2021-12-13 16:59 PST, Robert Jenner
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Jenner 2021-12-13 16:09:48 PST
The following 4 tests are flaky text failures on iOS:

imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https.html 
imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https.html 
imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https.html 
imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https.html 

HISTORY:
https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-no-referrer-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-frame.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-iframe.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Frequest-reset-attributes.https.html&platform=ios

DIFF URLS:
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https-pretty-diff.html

Each diff appears to have the following in common:

+Harness Error (TIMEOUT), message = null

I have attached all text diffs to this bug.
Comment 1 Robert Jenner 2021-12-13 16:10:40 PST
Created attachment 447079 [details]
Text Diffs
Comment 2 Robert Jenner 2021-12-13 16:11:31 PST
They are all flakey, but they all appear to have either the same failure history, or at least very similar failure points when observing their history.
Comment 3 Radar WebKit Bug Importer 2021-12-13 16:11:55 PST
<rdar://problem/86438914>
Comment 4 Robert Jenner 2021-12-13 16:35:52 PST
After looking deeper into the history, it appears there are actually 4 more associated tests that also appear to have a very similar failure history (eg. the tests fail at the same time when they fail.) The additional 4 tests are:

imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-service-worker.https.html 
imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-when-cross-origin-service-worker.https.html
imported/w3c/web-platform-tests/fetch/api/policies/referrer-unsafe-url-service-worker.https.html
imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html

HISTORY:
https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-origin-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-origin-when-cross-origin-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-unsafe-url-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-worker.https.html&platform=ios


DIFF URLS:
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https-pretty-diff.html

These four additional tests all have the same thing in common in their diffs that the original 4 did posted above:

+Harness Error (TIMEOUT), message = null

I have attached all text diffs to this bug.
Comment 5 Robert Jenner 2021-12-13 16:43:33 PST
All of these tests had their first reported failure at r284862, and have continued since. 

I don't believe r284862 is what caused them to start failing, that's just what their history highlights. That revision was close to when we started testing iOS 15 on Open Source, so it appears that these are all just flaky tests on iOS 15.
Comment 6 Robert Jenner 2021-12-13 16:59:54 PST
Created attachment 447087 [details]
8 Text Diffs for failing tests
Comment 7 Robert Jenner 2021-12-13 17:09:09 PST
I was not able to reproduce any of these failures at Production or Debug iOS 15 ToT. 

I even took it up to 1000 iterations for each test, and they all passed. These tests randomly fail, and do not appear to have a specific regression point. I'm grouping them together because they have either the same failure history, or extremely similar failure history. They all also appear to deal with or be service worker tests.

I have marked the expectations for these tests as Pass, Failure here:

https://trac.webkit.org/changeset/286997/webkit
Comment 8 youenn fablet 2021-12-14 02:54:33 PST
All these tests are service worker tests that seem to fail registering/unregistering service workers on simulator.
Comment 9 youenn fablet 2022-06-19 23:11:42 PDT
imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https.html, imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https.html and imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https.html do not seem flaky anymore.


imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https.html seems to crash sometimes (GPUProcess) but without any actual crash log.
Comment 10 youenn fablet 2022-06-19 23:14:50 PDT
imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-service-worker.https.html 
imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-when-cross-origin-service-worker.https.html
imported/w3c/web-platform-tests/fetch/api/policies/referrer-unsafe-url-service-worker.https.html
imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html

All these tests are passing except imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html which sometimes crash.
I'll fix the crash.
Comment 11 youenn fablet 2022-06-19 23:31:30 PDT
Fixing the crash at https://bugs.webkit.org/show_bug.cgi?id=241776.
The test expectations should probably be unflaked.
Comment 12 youenn fablet 2022-06-19 23:34:20 PDT
Pull request: https://github.com/WebKit/WebKit/pull/1634
Comment 13 EWS 2022-06-20 05:03:38 PDT
Committed r295673 (251678@main): <https://commits.webkit.org/251678@main>

Reviewed commits have been landed. Closing PR #1634 and removing active labels.