Many (most?) transitions tests are very flaky, frequently failing all at once

• Component: Tools / Tests
• Status: NEW ---

[Alexey Proskuryakov]

These tests are very very flaky, and have been for quite a while. Time to start ignoring them.

  transitions/background-transitions.html [ Failure Pass ]
  transitions/border-radius-transition.html [ Failure Pass ]
  transitions/change-values-during-transition.html [ Failure Pass ]
  transitions/clip-transition.html [ Failure Pass ]
  transitions/color-transition-all.html [ Failure Pass ]
  transitions/color-transition-premultiplied.html [ Failure Pass ]
  transitions/color-transition-rounding.html [ Failure Pass ]
  transitions/created-while-suspended.html [ Failure Pass ]

@@ -1,3 +1,5 @@
-PASS - "background-position" property for "box" element at 0.5s saw something close to: 10,10
-PASS - "-webkit-background-size" property for "box2" element at 0.5s saw something close to: 20,20
+CONSOLE MESSAGE: line 240: Failed to pause 'background-position' transition on element 'box'
+CONSOLE MESSAGE: line 240: Failed to pause '-webkit-background-size' transition on element 'box2'
+FAIL - "background-position" property for "box" element at 0.5s expected: 10,10 but saw: 20,20
+FAIL - "-webkit-background-size" property for "box2" element at 0.5s expected: 20,20 but saw: 30,30

[Alexey Proskuryakov]

Marked them as flaky in <http://trac.webkit.org/changeset/174886>.

[Simon Fraser (smfr)]

Would be nice to understand why these 8 all flake at once. Maybe some earlier test leaves bad state around?

[Alexey Proskuryakov]

> Would be nice to understand why these 8 all flake at once. 

It definitely would!

> Maybe some earlier test leaves bad state around?

I tried running tests in exactly the same order as a failing tester on a bot did, and couldn't reproduce the issue. So it must be something more involved.

[Simon Fraser (smfr)]

It's also interesting that the same bot hits this only sporadically.

Does EWS hit it more reliably?

[Alexey Proskuryakov]

EWS hits this in release builds, while regular bots only do in debug.

[Alexey Proskuryakov]

Something changed, and now it's 30 at once that fail, not counting ones that were already marked as flaky.

This happens on EWS (Mountain Lion WK2 release) and on regular bots (Yosemite WK2 release or debug).

Many of these are already marked flaky, perhaps we should mark all the rest?

[Alexey Proskuryakov]

*** Bug 113227 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

Skipped a lot more in <http://trac.webkit.org/r176125>, because having 30+ flaky failures breaks EWS. We should fix EWS (filed bug 138743), and we should fix this bug, but right now, marking the tests as flaky is the only way out.

https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/r176079%20(479)/results.html is an example of this problem occurring on a regular regression test bot. Looks like it's not only transitions tests that are affected, and this is a general timer problem.

[Alexey Proskuryakov]

This happens very rarely and is thus difficult to pinpoint. But looks like this started to happen for more tests about two weeks ago. Could it be <http://trac.webkit.org/changeset/175441>?

[Alexey Proskuryakov]

Created attachment 241628 [details]
tests.txt

This is the sequence of tests that a single worker process on a bot ran when this happened recently. But I cannot reproduce when re-running in the same order locally.

[Alexey Proskuryakov]

This still happens, so we still can't have any test coverage for transitions. How fun is that?

[Radar WebKit Bug Importer]

<rdar://problem/34342774>

[Alexey Proskuryakov]

The original issue with 8 tests failing all at once doesn't seem to be happening any more. But we have many more transition tests marked as flaky now, and a few of them still are flaky. And a few are marked as flaky, but fail 100% of time. 

This will take some effort to clean up.