Bug 137883

Summary: Many (most?) transitions tests are very flaky, frequently failing all at once
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: aestes, cdumez, dino, jonlee, rniwa, simon.fraser, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 58673, 114182, 114189, 114194, 114198, 114203, 114206    
Bug Blocks:    
Description Flags
tests.txt none

Description Alexey Proskuryakov 2014-10-20 09:37:44 PDT
These tests are very very flaky, and have been for quite a while. Time to start ignoring them.

  transitions/background-transitions.html [ Failure Pass ]
  transitions/border-radius-transition.html [ Failure Pass ]
  transitions/change-values-during-transition.html [ Failure Pass ]
  transitions/clip-transition.html [ Failure Pass ]
  transitions/color-transition-all.html [ Failure Pass ]
  transitions/color-transition-premultiplied.html [ Failure Pass ]
  transitions/color-transition-rounding.html [ Failure Pass ]
  transitions/created-while-suspended.html [ Failure Pass ]

@@ -1,3 +1,5 @@
-PASS - "background-position" property for "box" element at 0.5s saw something close to: 10,10
-PASS - "-webkit-background-size" property for "box2" element at 0.5s saw something close to: 20,20
+CONSOLE MESSAGE: line 240: Failed to pause 'background-position' transition on element 'box'
+CONSOLE MESSAGE: line 240: Failed to pause '-webkit-background-size' transition on element 'box2'
+FAIL - "background-position" property for "box" element at 0.5s expected: 10,10 but saw: 20,20
+FAIL - "-webkit-background-size" property for "box2" element at 0.5s expected: 20,20 but saw: 30,30
Comment 1 Alexey Proskuryakov 2014-10-20 09:40:58 PDT
Marked them as flaky in <http://trac.webkit.org/changeset/174886>.
Comment 2 Simon Fraser (smfr) 2014-10-20 09:46:39 PDT
Would be nice to understand why these 8 all flake at once. Maybe some earlier test leaves bad state around?
Comment 3 Alexey Proskuryakov 2014-10-20 10:06:10 PDT
> Would be nice to understand why these 8 all flake at once. 

It definitely would!

> Maybe some earlier test leaves bad state around?

I tried running tests in exactly the same order as a failing tester on a bot did, and couldn't reproduce the issue. So it must be something more involved.
Comment 4 Simon Fraser (smfr) 2014-10-20 10:17:54 PDT
It's also interesting that the same bot hits this only sporadically.

Does EWS hit it more reliably?
Comment 5 Alexey Proskuryakov 2014-10-20 10:43:04 PDT
EWS hits this in release builds, while regular bots only do in debug.
Comment 6 Alexey Proskuryakov 2014-11-13 17:07:43 PST
Something changed, and now it's 30 at once that fail, not counting ones that were already marked as flaky.

This happens on EWS (Mountain Lion WK2 release) and on regular bots (Yosemite WK2 release or debug).

Many of these are already marked flaky, perhaps we should mark all the rest?
Comment 7 Alexey Proskuryakov 2014-11-13 17:08:11 PST
*** Bug 113227 has been marked as a duplicate of this bug. ***
Comment 8 Alexey Proskuryakov 2014-11-14 10:06:12 PST
Skipped a lot more in <http://trac.webkit.org/r176125>, because having 30+ flaky failures breaks EWS. We should fix EWS (filed bug 138743), and we should fix this bug, but right now, marking the tests as flaky is the only way out.

https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/r176079%20(479)/results.html is an example of this problem occurring on a regular regression test bot. Looks like it's not only transitions tests that are affected, and this is a general timer problem.
Comment 9 Alexey Proskuryakov 2014-11-14 14:44:56 PST
This happens very rarely and is thus difficult to pinpoint. But looks like this started to happen for more tests about two weeks ago. Could it be <http://trac.webkit.org/changeset/175441>?
Comment 10 Alexey Proskuryakov 2014-11-14 14:57:04 PST
Created attachment 241628 [details]

This is the sequence of tests that a single worker process on a bot ran when this happened recently. But I cannot reproduce when re-running in the same order locally.
Comment 11 Alexey Proskuryakov 2015-07-25 19:21:19 PDT
This still happens, so we still can't have any test coverage for transitions. How fun is that?
Comment 12 Radar WebKit Bug Importer 2017-09-08 16:43:06 PDT
Comment 13 Alexey Proskuryakov 2017-09-11 09:06:10 PDT
The original issue with 8 tests failing all at once doesn't seem to be happening any more. But we have many more transition tests marked as flaky now, and a few of them still are flaky. And a few are marked as flaky, but fail 100% of time. 

This will take some effort to clean up.