Bug 198195 – REGRESSION (r244353) [Mac WK2] Layout Test fast/css/sticky/sticky-left-percentage.html is a flaky ImageOnlyFailure

• Status: NEW
• Component: Tools / Tests

[Shawn Roberts]

The following layout test is flaky on Mac Release WK2

fast/css/sticky/sticky-left-percentage.html

Probable cause:

Test wad modified in https://trac.webkit.org/changeset/244353/webkit and started to become a flaky failure

Reproduced locally with:

run-webkit-tests fast/css/sticky/sticky-left-percentage.html --iter 100 -f --exit-after-n-failures=2

Usually hits 2 failures within 10 iterations. 

stdio shows:

Error: test and reference images have different sizes. Test image is 800x700, reference image is 800x600
ImageDiff crashed

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=fast%2Fcss%2Fsticky%2Fsticky-left-percentage.html

[Radar WebKit Bug Importer]

<rdar://problem/51081794>

[Shawn Roberts]

Updated expectations in https://trac.webkit.org/changeset/245718/webkit while waiting for a fix.

[Jonathan Bedard]

Created attachment 370947 [details]
Patch

[Jonathan Bedard]

I'd like to find a better approach than waiting, but I'm not really sure what that would be.

It seems that this is only a problem in the reference (I've never seen the actual test hit the race condition).

A list of approaches I've tried:
- UIScriptController, but that timed out.
- Listening for resize events on the body, but the race condition was still present.
- Modifying resizeTo in WebCore so that it blocked until the window reported the new size.

Only thing that seems to work is some variation of waiting. Not quite sure what we're waiting for, we do trigger a repaint before snapshotting, so what ever it is, it's not obvious.

[Simon Fraser (smfr)]

Comment on attachment 370947 [details]
Patch

This seems like a bad hack, and people will continue to add new tests that break. I think we need a WTR solution that is reliable.

[EWS Watchlist]

Comment on attachment 370947 [details]
Patch

Attachment 370947 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12329342

New failing tests:
fast/css/sticky/sticky-left-percentage.html

[EWS Watchlist]

Created attachment 370953 [details]
Archive of layout-test-results from ews102 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews102  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[EWS Watchlist]

Comment on attachment 370947 [details]
Patch

Attachment 370947 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/12329395

New failing tests:
fast/css/sticky/sticky-left-percentage.html

[EWS Watchlist]

Created attachment 370956 [details]
Archive of layout-test-results from ews215 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews215  Port: win-future  Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit

[EWS Watchlist]

Comment on attachment 370947 [details]
Patch

Attachment 370947 [details] did not pass mac-debug-ews (mac):
Output: https://webkit-queues.webkit.org/results/12329389

New failing tests:
fast/css/sticky/sticky-left-percentage.html

[EWS Watchlist]

Created attachment 370958 [details]
Archive of layout-test-results from ews116 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews116  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[Jonathan Bedard]

(In reply to Simon Fraser (smfr) from comment #5)
> Comment on attachment 370947 [details]
> Patch
> 
> This seems like a bad hack, and people will continue to add new tests that
> break. I think we need a WTR solution that is reliable.

What would the WTR solution look like, though?

The only thing I can think of, at this point, would be to provide a sort of WebKitTestRunner resizeTo JS function which is smart enough to the wait itself. But I still think that we will ultimately be stuck with a wait. The other option would be a wait before actually taking the snapshot, but that seems like a bad idea.

Other possible options are that we're triggering the onresize before the actual resize has occurred. But looking at the VisualViewport code, that doesn't look to be the case.  We might be able to change Mac snapshot function so that it lives in WebKitTestRunner (like it does for iOS) instead of using the CG API. But that seems like the sort of change that could break tons of layout tests.

Ultimately, I think that we're either going to end up with something like the proposed change or just futuring this bug. Other than modifications to the test, the fixes I can think of are all bigger problems than the test just being flakey.

[EWS Watchlist]

Comment on attachment 370947 [details]
Patch

Attachment 370947 [details] did not pass mac-wk2-ews (mac-wk2):
Output: https://webkit-queues.webkit.org/results/12330723

New failing tests:
http/wpt/service-workers/useragent.https.html

[EWS Watchlist]

Created attachment 370974 [details]
Archive of layout-test-results from ews107 for mac-highsierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-highsierra-wk2  Platform: Mac OS X 10.13.6

[Simon Fraser (smfr)]

fast/css-grid-layout/flex-content-sized-columns-resize.html is another ref tests that does resizeTo.

[Simon Fraser (smfr)]

(In reply to Jonathan Bedard from comment #12)
> (In reply to Simon Fraser (smfr) from comment #5)
> > Comment on attachment 370947 [details]
> > Patch
> > 
> > This seems like a bad hack, and people will continue to add new tests that
> > break. I think we need a WTR solution that is reliable.
> 
> What would the WTR solution look like, though?

UIScriptController::resizeWindow. UIScriptController knows how to wait for async things.