imported/w3c/web-platform-tests/infrastructure/reftest-wait.html Is flaky image failing on BigSur wk2 arm64. History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Finfrastructure%2Freftest-wait.html Result page: https://build.webkit.org/results/Apple-BigSur-Release-AppleSilicon-WK2-Tests/r282745%20(4008)/results.html
Created attachment 438694 [details] Image diff
<rdar://problem/83313533>
Marked test expectations: https://trac.webkit.org/changeset/282759/webkit
This test just checks if basic WPT infrastructure works. So if it's flaky, then all reftest-wait tests are. So this is particularly important to look into.
It appears like it's just some pixels (not even around the edge of the square) with a very slightly different colour. Any idea, Simon?
I was able to reproduce the image failure locally on BigSur arm64 at TOT and the oldest revision available (r268271) using run-webkit-tests --force --iterations 100 --exit-after-n-failures 1 imported/w3c/web-platform-tests/infrastructure/reftest-wait.html -f
I'm not sure we even support reftest-wait.
Darin added it in https://bugs.webkit.org/show_bug.cgi?id=186045
The diff is pixel fuzz, so it's one of those bugs where some underlying graphics state is different, which might be something WebKit is doing, or something an underlying framework is doing. It would be nice to know what sequence of tests reliably cause this pixel fuzz.
[ macOS wk2 arm64 ] imported/w3c/web-platform-tests/infrastructure/reftest-wait.html is a flaky failure/image failure/timeout/crash Description: imported/w3c/web-platform-tests/infrastructure/reftest-wait.html Updating to include this is affecting Monterey wk2 arm64. There was not one commit that I observed causing this issue. This test seems to have been flaky from the start on Big Sur and Monterey. It has been reproduced with ToT and oldest available spade previously. The first failure for Monterey that I saw on the dashboard was 3/18/2022 at r291506, but this does not appear relevant to causing this issue. I was unable reproduce this on Monterey on Tot (r293204) or oldest available spade (r2826320, but I was able to grab the test list. History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Finfrastructure%2Freftest-wait.html&platform=mac&limit=50000&architecture=arm64&flavor=wk2 Link to Image Diff: https://build.webkit.org/results/Apple-Monterey-Release-AppleSilicon-WK2-Tests/249887@main%20(2824)/imported/w3c/web-platform-tests/infrastructure/reftest-wait-diffs.html I have marked this test as a flaky failure while this issue is investigated.
Created attachment 458149 [details] test list
Pull request: https://github.com/WebKit/WebKit/pull/360
(In reply to Karl Rackler from comment #10) > [ macOS wk2 arm64 ] > imported/w3c/web-platform-tests/infrastructure/reftest-wait.html is a flaky > failure/image failure/timeout/crash > > Description: > imported/w3c/web-platform-tests/infrastructure/reftest-wait.html > > Updating to include this is affecting Monterey wk2 arm64. > > There was not one commit that I observed causing this issue. This test > seems to have been flaky from the start on Big Sur and Monterey. It has > been reproduced with ToT and oldest available spade previously. > > The first failure for Monterey that I saw on the dashboard was 3/18/2022 at > r291506, but this does not appear relevant to causing this issue. > > I was unable reproduce this on Monterey on Tot (r293204) or oldest available > spade (r2826320, but I was able to grab the test list. > > History: > https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb- > platform-tests%2Finfrastructure%2Freftest-wait. > html&platform=mac&limit=50000&architecture=arm64&flavor=wk2 > > Link to Image Diff: > https://build.webkit.org/results/Apple-Monterey-Release-AppleSilicon-WK2- > Tests/249887@main%20(2824)/imported/w3c/web-platform-tests/infrastructure/ > reftest-wait-diffs.html > > I have marked this test as a flaky failure while this issue is investigated. This should read ** [ macOS wk2 arm64 ] imported/w3c/web-platform-tests/infrastructure/reftest-wait.html is a flaky failure **
Committed r293223 (249892@main): <https://commits.webkit.org/249892@main> Reviewed commits have been landed. Closing PR #360 and removing active labels.
Reopened because the issue is still being investigated.