Bug 234278 – [ iOS ] 8 imported/w3c/web-platform-tests/fetch/api (layout-tests) are flaky text failures

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Robert Jenner]

The following 4 tests are flaky text failures on iOS:

imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https.html 
imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https.html 
imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https.html 
imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https.html 

HISTORY:
https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-no-referrer-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-frame.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-iframe.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Frequest-reset-attributes.https.html&platform=ios

DIFF URLS:
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r286958%20(727)/imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https-pretty-diff.html

Each diff appears to have the following in common:

+Harness Error (TIMEOUT), message = null

I have attached all text diffs to this bug.

[Robert Jenner]

Created attachment 447079 [details]
Text Diffs

[Robert Jenner]

They are all flakey, but they all appear to have either the same failure history, or at least very similar failure points when observing their history.

[Radar WebKit Bug Importer]

<rdar://problem/86438914>

[Robert Jenner]

After looking deeper into the history, it appears there are actually 4 more associated tests that also appear to have a very similar failure history (eg. the tests fail at the same time when they fail.) The additional 4 tests are:

imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-service-worker.https.html 
imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-when-cross-origin-service-worker.https.html
imported/w3c/web-platform-tests/fetch/api/policies/referrer-unsafe-url-service-worker.https.html
imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html

HISTORY:
https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-origin-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-origin-when-cross-origin-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Fpolicies%2Freferrer-unsafe-url-service-worker.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Ffetch%2Fapi%2Frequest%2Fdestination%2Ffetch-destination-worker.https.html&platform=ios


DIFF URLS:
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https-pretty-diff.html
https://build.webkit.org/results/Apple-iOS-15-Simulator-Release-WK2-Tests/r284862%20(318)/imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https-pretty-diff.html

These four additional tests all have the same thing in common in their diffs that the original 4 did posted above:

+Harness Error (TIMEOUT), message = null

I have attached all text diffs to this bug.

[Robert Jenner]

All of these tests had their first reported failure at r284862, and have continued since. 

I don't believe r284862 is what caused them to start failing, that's just what their history highlights. That revision was close to when we started testing iOS 15 on Open Source, so it appears that these are all just flaky tests on iOS 15.

[Robert Jenner]

Created attachment 447087 [details]
8 Text Diffs for failing tests

[Robert Jenner]

I was not able to reproduce any of these failures at Production or Debug iOS 15 ToT. 

I even took it up to 1000 iterations for each test, and they all passed. These tests randomly fail, and do not appear to have a specific regression point. I'm grouping them together because they have either the same failure history, or extremely similar failure history. They all also appear to deal with or be service worker tests.

I have marked the expectations for these tests as Pass, Failure here:

https://trac.webkit.org/changeset/286997/webkit

[youenn fablet]

All these tests are service worker tests that seem to fail registering/unregistering service workers on simulator.

[youenn fablet]

imported/w3c/web-platform-tests/fetch/api/policies/referrer-no-referrer-service-worker.https.html, imported/w3c/web-platform-tests/fetch/api/request/request-reset-attributes.https.html and imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-iframe.https.html do not seem flaky anymore.


imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https.html seems to crash sometimes (GPUProcess) but without any actual crash log.

[youenn fablet]

imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-service-worker.https.html 
imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-when-cross-origin-service-worker.https.html
imported/w3c/web-platform-tests/fetch/api/policies/referrer-unsafe-url-service-worker.https.html
imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html

All these tests are passing except imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-worker.https.html which sometimes crash.
I'll fix the crash.

[youenn fablet]

Fixing the crash at https://bugs.webkit.org/show_bug.cgi?id=241776.
The test expectations should probably be unflaked.

[youenn fablet]

Pull request: https://github.com/WebKit/WebKit/pull/1634

[EWS]

Committed r295673 (251678@main): <https://commits.webkit.org/251678@main>

Reviewed commits have been landed. Closing PR #1634 and removing active labels.