Bug 248544 – REGRESSION (256665@main): [ Ventura ] fast/images/avif-as-image.html is a consistent image failure

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Karl Rackler]

Description:
fast/images/avif-as-image.html

There was an AVIF tests garden at 255947@main where I first saw the history of the image failure for Ventura.  I observed later in the history at 255984@main that AVIF image decoding using libavif and dav1d is enabled, and that is where I see the test begin to pass on Monterey and BigSur, but fail for Ventura. Still, there was a test expectation in place for this.  Finally, there was work to AVIF image animates only one loop at 256665@main where the test expectation was removed, which seemed to cause the image failure on the flakiness dashboard to appear again.  

History:
https://results.webkit.org/?suite=layout-tests&test=fast%2Fimages%2Favif-as-image.html&platform=ios&platform=mac&limit=50000

Image Diff:
https://build.webkit.org/results/Apple-Ventura-Release-WK2-Tests/257176@main%20(385)/fast/images/avif-as-image-diffs.html

[Radar WebKit Bug Importer]

<rdar://problem/102823196>

[Karl Rackler]

This issue can be reproduced using the command: 
run-webkit-tests --exit-after-n-failures 1 --exit-after-n-crashes-or-timeouts 1 --iterations 50 -f --no-retry fast/images/avif-as-image.html

Result: 
Regressions: Unexpected image-only failures (1)
  fast/images/avif-as-image.html [ ImageOnlyFailure ]

[Karl Rackler]

I have marked this test as a consistent image failure while this issue is investigated.

[EWS]

Test gardening commit 257195@main (60a1d6da7b93): <https://commits.webkit.org/257195@main>

Reviewed commits have been landed. Closing PR #6985 and removing active labels.

[Said Abou-Hallawa]

Pull request: https://github.com/WebKit/WebKit/pull/7287

[EWS]

Committed 257738@main (c7d2aef57843): <https://commits.webkit.org/257738@main>

Reviewed commits have been landed. Closing PR #7287 and removing active labels.

[Karl Rackler]

Re-opening.  Looking at the history, landing 257738@main did not seem to resolve the issue.

[Said Abou-Hallawa]

The TestExpectations have these entries for this test:

LayoutTests/TestExpectations:fast/images/avif-as-image.html [ ImageOnlyFailure ]
LayoutTests/platform/win/TestExpectations:fast/images/avif-as-image.html [ Skip ]
LayoutTests/platform/mac/TestExpectations:[ Ventura+ ] fast/images/avif-as-image.html [ Pass ]

What I wanted to say by these entries is: this test is expected to fail on all platforms and on all macOS versions except on macOS Ventura.

So I am not sure why iOS says it fails even though the expectations say: "yes we expect it to fail on iOS". Also I do not know why it runs on macOS Monetary and BigSur arm64. Should I add entries in mac-ventura/TestExpectations,  mac-monterey/TestExpectations and  mac-ventura/TestExpectations for this file?

[Karl Rackler]

(In reply to Said Abou-Hallawa from comment #8)
> The TestExpectations have these entries for this test:
> 
> LayoutTests/TestExpectations:fast/images/avif-as-image.html [
> ImageOnlyFailure ]
> LayoutTests/platform/win/TestExpectations:fast/images/avif-as-image.html [
> Skip ]
> LayoutTests/platform/mac/TestExpectations:[ Ventura+ ]
> fast/images/avif-as-image.html [ Pass ]
> 
> What I wanted to say by these entries is: this test is expected to fail on
> all platforms and on all macOS versions except on macOS Ventura.
> 
> So I am not sure why iOS says it fails even though the expectations say:
> "yes we expect it to fail on iOS". Also I do not know why it runs on macOS
> Monetary and BigSur arm64. Should I add entries in
> mac-ventura/TestExpectations,  mac-monterey/TestExpectations and 
> mac-ventura/TestExpectations for this file?

You currently have the test expectations set up such that everything is passing on WebKit Results Database if you turn on "Filter expected results" except for mac-ventura.

This is because the test expectation for Mac-ventura was removed by 257738@main.

I am looking at the latest run, and the expected image is a lime square, and the actual image is a black square.  
https://build.webkit.org/results/Apple-Ventura-Release-AppleSilicon-WK2-Tests/258574@main%20(917)/fast/images/avif-as-image-diffs.html

I'm adding the test expectation back in place while this issue is investigated.

[EWS]

Test gardening commit 258590@main (eda93cc0bc7f): <https://commits.webkit.org/258590@main>

Reviewed commits have been landed. Closing PR #8342 and removing active labels.

[Said Abou-Hallawa]

Pull request: https://github.com/WebKit/WebKit/pull/8630

[EWS]

Committed 258934@main (9bffbcdd799e): <https://commits.webkit.org/258934@main>

Reviewed commits have been landed. Closing PR #8630 and removing active labels.

[Said Abou-Hallawa]

*** Bug 250045 has been marked as a duplicate of this bug. ***