Bug 284898 – REGRESSION (272231@main): [ macOS wk1 ] fonts tests following font-cache-memory-pressure-crash.html are a flaky failure

• Component: Text
• Status: NEW

[Karl Rackler]

Description:
fonts/font-weight-invalid-crash.html
fonts/ligature.html

The test fonts/font-cache-memory-pressure-crash.html , was introduced at https://commits.webkit.org/272231@main. This implementation corresponded with the onset of inconsistent test results, transitioning from consistent passes to frequent flaky failures.

This issue can be bisected to <commit> using the command: 
run-webkit-tests --debug -1 --no-retry --force --iterations=50 --clobber-old-results --child-processes 1 --test-list testlist-font-weight-invalid-crash.txt
run-webkit-tests --debug -1 --no-retry --force --iterations=50 --clobber-old-results --child-processes 1 --test-list testlist-ligature.txt

History:
https://results.webkit.org/?suite=layout-tests&suite=layout-tests&test=fonts%2Ffont-weight-invalid-crash.html&test=fonts%2Fligature.html&flavor=wk1&recent=false

Diff:
fonts/font-weight-invalid-crash.html
-PASS
-
+Test passes if it doesn't crash.
+ｕｎｄｅｆｉｎｅｄ

Image Diff:
fonts/ligature.html
On view layout test results it shows 'images diff (nan%)' and both links open to 404 Not Found

[Radar WebKit Bug Importer]

<rdar://problem/141704928>

[EWS]

Test gardening commit 288036@main (2d6e67c75819): <https://commits.webkit.org/288036@main>

Reviewed commits have been landed. Closing PR #38155 and removing active labels.

[Karl Rackler]

Created attachment 473598 [details]
testlist-font-weight-invalid-crash

[Karl Rackler]

Created attachment 473599 [details]
testlist-ligature

[Karl Rackler]

Reproduction:
This issue can be bisected to 272231@main using the command: 
run-webkit-tests --debug -1 --no-retry --force --iterations=50 --clobber-old-results --child-processes 1 --test-list testlist-font-weight-invalid-crash.txt
run-webkit-tests --debug -1 --no-retry --force --iterations=50 --clobber-old-results --child-processes 1 --test-list testlist-ligature.txt

Results:
fonts/ligature.html failed unexpectedly (reference didn't generate pixel results.)

Unexpected flakiness: text-only failures (1)
  fonts/font-weight-invalid-crash.html [ Failure Pass ]

Unexpected flakiness: image-only failures (1)
  fonts/ligature.html [ ImageOnlyFailure Pass ]

[Sam Sneddon [:gsnedders]]

(In reply to Karl Rackler from comment #0)
> Diff:
> fonts/font-weight-invalid-crash.html
> -PASS
> -
> +Test passes if it doesn't crash.
> +ｕｎｄｅｆｉｎｅｄ

This unexpected output is that of fonts/font-cache-memory-pressure-crash.html; the new test added in 272231@main, and fonts/font-weight-invalid-crash.html is the immediately following test. This feels very much like DRT is giving unexpected output and/or the driver is returning the wrong data.

> Image Diff:
> fonts/ligature.html
> On view layout test results it shows 'images diff (nan%)' and both links
> open to 404 Not Found

Especially with the 404, this seems like a no-output-image case (which was fixed in bug 273212).

To be clear, as to ordering, we're running:

Tests to skip (1)
fonts/font-fallback-prefers-pictographs.html  ['SKIP'] LayoutTests/platform/mac-wk1/TestExpectations:2913 webkit.org/b/273279 fonts/font-fallback-prefers-pictographs.html [ Skip ]

Tests to run (14)
fonts/cursive.html                            ['PASS']  
fonts/default.html                            ['PASS']  
fonts/fantasy.html                            ['PASS']  
fonts/font-cache-crash.html                   ['PASS']  
fonts/font-cache-memory-pressure-crash.html   ['PASS']  
fonts/font-weight-invalid-crash.html          ['PASS', 'FAIL'] LayoutTests/platform/mac-wk1/TestExpectations:3012 webkit.org/b/284898 [ Ventura+ ] fonts/font-weight-invalid-crash.html [ Pass Failure ]
fonts/ligature.html                           ['PASS', 'FAIL', 'IMAGE'] LayoutTests/platform/mac-wk1/TestExpectations:3013 webkit.org/b/284898 [ Ventura+ ] fonts/ligature.html [ Pass Failure ImageOnlyFailure ]
fonts/monospace.html                          ['PASS']  
fonts/sans-serif.html                         ['PASS']  
fonts/serif.html                              ['PASS']  
fonts/unicode-character-font-crash.html       ['PASS']  
fonts/use-typo-metrics-1.html                 ['PASS']  
fonts/use-typo-metrics-2.html                 ['PASS']  
fonts/valid-standard-font.html                ['PASS']  

So these are three adjacent tests.

[Sam Sneddon [:gsnedders]]

I wrote this on Slack a week ago, but:

> I think someone needs to go check what DRT is actually passing to the driver next, probably looking at the value of what test ends up with in run_test in Tools/Scripts/webkitpy/port/test.py, checking what output we're getting for what test.

[Sam Sneddon [:gsnedders]]

*** Bug 273279 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

https://github.com/WebKit/WebKit/pull/49978 (from bug 298004) shows this up again, with whatever follows fonts/font-cache-memory-pressure-crash.html crashing with "test sequence mismatch" (including itself, if run with --iterations=N).

Let's re-garden these so we skip the test causing the flakiness, not mark all those that follow as flaky.

[Sam Sneddon [:gsnedders]]

Pull request: https://github.com/WebKit/WebKit/pull/50329

[EWS]

Test gardening commit 299644@main (8e9862298f6f): <https://commits.webkit.org/299644@main>

Reviewed commits have been landed. Closing PR #50329 and removing active labels.