Bug 96951 – [chromium] platform/chromium-linux/compositing/gestures are flaky on the EWS bots

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[Tony Chang]

Sometimes all the compositing image tests fail on the EWS cr-linux bots (Google Compute Engine instances).  When this happens, every compositing test is a black 800x600 png and we bail out after 100 or so failures.

jamesr skipped the platform/chromium-linux/compositing/gestures tests, this bug is to track the failures.

This is probably hard to repro, but if someone wants to investigate, we can give access to the compute engine instances.

[James Robinson]

Did the failures go away completely after that skip?

[Tony Chang]

No, other compositing tests failed instead.  I think the platform/chromium-linux/compositing/gestures test just happen to be the last set of compositing tests before hitting the 100 failure limit.

In https://bugs.webkit.org/show_bug.cgi?id=96041#c14 , I mentioned that it just moved the problem to other compositing tests.

[W. James MacLean]

zlieber@ and I have been looking at this, and so far we cannot reproduce any flaky behaviour running *only* the skipped gesture tests on a replica of the chromium-ews bot.

We are currently running multiple passes of the entire test suite to see if the flakiness shows up in conjunction with other tests.

[James Robinson]

Yeah, this smells like a general compositing flakiness not specifically related to these tests.

[W. James MacLean]

(In reply to comment #4)
> Yeah, this smells like a general compositing flakiness not specifically related to these tests.

Zeev had some luck getting the flake to reproduce by running larger test sets, and we are in the process of trying to find a small (minimal?) subset of tests that will re-produce the behaviour, in hopes of identifying the cause.

Our best guess at present is that it's some state that has survived from an earlier test that triggers this. If we can identify a minimal subset it may be possible to bisect to discover the trigger.

Findings so far: A regular run (as we do in test bot) will reproduce the flakiness in ~5% of the cases (based on 154 complete runs). If we add --run-singly, the problem goes away completely (based on 130 complete runs).

I considered the problem reproduced when it showed image-only flakiness and the actual image was filled with one solid colour (presumably black, but scripts didn't assert that).

Tests that showed this particular kind of flakiness so far are: fast/repaint, fullscreen, gestures, chromium/compositing/*, transforms/3d*, transitions.

Assuming DumpRenderTree goes into some kind of a bad state, I'll try to go deeper and pinpoint the actual moment when it happens.

[W. James MacLean]

Zeev has switched to a different team, and I currently have no one to follow up on the reason the compositor tests are failing.

That being said, I'd like to get the gesture tap highlight tests turned back on (I've fixed a number of regressions thus far, and would like to see this test coverage re-instated).

What do we need to do so I can get these tests re-enabled?

[James Robinson]

(In reply to comment #7)
> Zeev has switched to a different team, and I currently have no one to follow up on the reason the compositor tests are failing.
> 
> That being said, I'd like to get the gesture tap highlight tests turned back on (I've fixed a number of regressions thus far, and would like to see this test coverage re-instated).
> 
> What do we need to do so I can get these tests re-enabled?

Turn them on and follow up if any flakiness results.  I think that based on what we've seen it's unlikely these tests in particular are problematic.

[W. James MacLean]

(In reply to comment #8)
> (In reply to comment #7)
> > Zeev has switched to a different team, and I currently have no one to follow up on the reason the compositor tests are failing.
> > 
> > That being said, I'd like to get the gesture tap highlight tests turned back on (I've fixed a number of regressions thus far, and would like to see this test coverage re-instated).
> > 
> > What do we need to do so I can get these tests re-enabled?
> 
> Turn them on and follow up if any flakiness results.  I think that based on what we've seen it's unlikely these tests in particular are problematic.

OK, I'll verify that the tests are currently OK, then re-enable.

[Stephen Chenney]

Marked LayoutTest bugs, bugs with Chromium IDs, and some others as WontFix. Test failure bugs still are trackable via TestExpectations or disabled unit tests.