Bug 196096 – Many failures caused by async overflow scrolling weren't caught by EWS

• Status: NEW
• Component: Tools / Tests

[Simon Fraser (smfr)]

EWS said this patch passed:
https://webkit-queues.webkit.org/patch/365357/ios-sim-ews

but it caused lots of failures and crashes:
https://build.webkit.org/results/Apple%20iOS%2012%20Simulator%20Debug%20WK2%20(Tests)/r243288%20(2909)/results.html

WTF

[Alexey Proskuryakov]

Wow, much clickbait! Re-titling.

The crashes in post-commit are debug assertions, so those are expected to not show up in release mode.

I checked the logs on the EWS bot, and there was absolutely nothing interesting there, the tests just plain passed:

2019-03-20 12:22:25,295 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews124 build-and-test --no-clean --no-update --test --non-interactive --build-style=release --group=None --port=ios-simulator-wk2
2019-03-20 12:33:20,915 - Passed tests

Simon, what was the reason for the failures, and why was it hidden from EWS?

[Simon Fraser (smfr)]

(In reply to Alexey Proskuryakov from comment #1)
> Wow, much clickbait! Re-titling.
> 
> The crashes in post-commit are debug assertions, so those are expected to
> not show up in release mode.

The crash (assertion) is explained in bug 196123.

> I checked the logs on the EWS bot, and there was absolutely nothing
> interesting there, the tests just plain passed:
> 
> 2019-03-20 12:22:25,295 - Running: webkit-patch
> --status-host=webkit-queues.webkit.org --bot-id=ews124 build-and-test
> --no-clean --no-update --test --non-interactive --build-style=release
> --group=None --port=ios-simulator-wk2
> 2019-03-20 12:33:20,915 - Passed tests
> 
> Simon, what was the reason for the failures, and why was it hidden from EWS?

The failure were legitimate test failures caused by differences in clipping and box sizes when elements with overflow:scroll were composited. They affected iOS WK2 tests, both release and debug. The patch should have absolutely caused these failures.

[Alexey Proskuryakov]

So... why did it not cause these failures on EWS?

The only difference between EWS and post-commit bots is that EWS enables retries, thus hiding some flakiness. This is not good, but EWS can't function with the current level of flakiness, and it's better to hide some failures sometimes than to lose EWS entirely.

If these 100+ failures were hidden by retry, that's bad news. I can think of several explanations:
- A state leak between tests. If these tests only fail when run after some test that passes but leaks state (such as preferences), then they will pass on retry.
- Severe timing dependency. Retries are serial, so tests that can't run on a busy machine will pass on retry.
- Similar to #1, but persistent conditions related to iOS simulator state, not WebKit state.

In any of these cases, it sounds like there will be more work fixing things in WebKit.

Alternatively, if there is some other mysterious reason, then it is, well, mysterious.

[Simon Fraser (smfr)]

Do we know what trunk revision EWS built against? Can we try to recreate the scenario?

[Alexey Proskuryakov]

The bot updated the checkout at 2019-03-20 12:04:12, so that would be r243222.