Bug 230488 – [ BigSur wk2 arm64 ] imported/w3c/web-platform-tests/infrastructure/reftest-wait.html is a flaky image failure

• Component: Tools / Tests
• Status: REOPENED

imported/w3c/web-platform-tests/infrastructure/reftest-wait.html

Is flaky image failing on BigSur wk2 arm64.

History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Finfrastructure%2Freftest-wait.html

Result page: https://build.webkit.org/results/Apple-BigSur-Release-AppleSilicon-WK2-Tests/r282745%20(4008)/results.html

Created attachment 438694 [details]
Image diff

[Radar WebKit Bug Importer]

<rdar://problem/83313533>

Marked test expectations: https://trac.webkit.org/changeset/282759/webkit

[Alexey Proskuryakov]

This test just checks if basic WPT infrastructure works. So if it's flaky, then all reftest-wait tests are.

So this is particularly important to look into.

[Sam Sneddon [:gsnedders]]

It appears like it's just some pixels (not even around the edge of the square) with a very slightly different colour. Any idea, Simon?

I was able to reproduce the image failure locally on BigSur arm64 at TOT and the oldest revision available (r268271) using run-webkit-tests --force --iterations 100 --exit-after-n-failures 1 imported/w3c/web-platform-tests/infrastructure/reftest-wait.html -f

[Simon Fraser (smfr)]

I'm not sure we even support reftest-wait.

[Sam Sneddon [:gsnedders]]

Darin added it in https://bugs.webkit.org/show_bug.cgi?id=186045

[Simon Fraser (smfr)]

The diff is pixel fuzz, so it's one of those bugs where some underlying graphics state is different, which might be something WebKit is doing, or something an underlying framework is doing.

It would be nice to know what sequence of tests reliably cause this pixel fuzz.

[Karl Rackler]

[ macOS wk2 arm64 ] imported/w3c/web-platform-tests/infrastructure/reftest-wait.html is a flaky failure/image failure/timeout/crash

Description:
imported/w3c/web-platform-tests/infrastructure/reftest-wait.html

Updating to include this is affecting Monterey wk2 arm64.

There was not one commit that I observed causing this issue.  This test seems to have been flaky from the start on Big Sur and Monterey.  It has been reproduced with ToT and oldest available spade previously. 

The first failure for Monterey that I saw on the dashboard was 3/18/2022 at r291506, but this does not appear relevant to causing this issue.

I was unable reproduce this on Monterey on Tot (r293204) or oldest available spade (r2826320, but I was able to grab the test list.

History:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Finfrastructure%2Freftest-wait.html&platform=mac&limit=50000&architecture=arm64&flavor=wk2

Link to Image Diff:
https://build.webkit.org/results/Apple-Monterey-Release-AppleSilicon-WK2-Tests/249887@main%20(2824)/imported/w3c/web-platform-tests/infrastructure/reftest-wait-diffs.html

I have marked this test as a flaky failure while this issue is investigated.

[Karl Rackler]

Created attachment 458149 [details]
test list

[Karl Rackler]

Pull request: https://github.com/WebKit/WebKit/pull/360

[Karl Rackler]

(In reply to Karl Rackler from comment #10)
> [ macOS wk2 arm64 ]
> imported/w3c/web-platform-tests/infrastructure/reftest-wait.html is a flaky
> failure/image failure/timeout/crash
> 
> Description:
> imported/w3c/web-platform-tests/infrastructure/reftest-wait.html
> 
> Updating to include this is affecting Monterey wk2 arm64.
> 
> There was not one commit that I observed causing this issue.  This test
> seems to have been flaky from the start on Big Sur and Monterey.  It has
> been reproduced with ToT and oldest available spade previously. 
> 
> The first failure for Monterey that I saw on the dashboard was 3/18/2022 at
> r291506, but this does not appear relevant to causing this issue.
> 
> I was unable reproduce this on Monterey on Tot (r293204) or oldest available
> spade (r2826320, but I was able to grab the test list.
> 
> History:
> https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-
> platform-tests%2Finfrastructure%2Freftest-wait.
> html&platform=mac&limit=50000&architecture=arm64&flavor=wk2
> 
> Link to Image Diff:
> https://build.webkit.org/results/Apple-Monterey-Release-AppleSilicon-WK2-
> Tests/249887@main%20(2824)/imported/w3c/web-platform-tests/infrastructure/
> reftest-wait-diffs.html
> 
> I have marked this test as a flaky failure while this issue is investigated.

This should read ** [ macOS wk2 arm64 ] imported/w3c/web-platform-tests/infrastructure/reftest-wait.html is a flaky failure **

[EWS]

Committed r293223 (249892@main): <https://commits.webkit.org/249892@main>

Reviewed commits have been landed. Closing PR #360 and removing active labels.

[Karl Rackler]

Reopened because the issue is still being investigated.