Bug 192501 – [macOS WK2 Debug ] Flaky Test: animations/play-state-in-shorthand.html

• Component: Animations
• Status: RESOLVED FIXED

[Matt Lewis]

animations/play-state-in-shorthand.html is a flaky failure on macOS WK2 Debug


https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=animations%2Fplay-state-in-shorthand.html

https://build.webkit.org/results/Apple%20Mojave%20Debug%20WK2%20(Tests)/r238951%20(808)/results.html

--- /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/animations/play-state-in-shorthand-expected.txt
+++ /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/animations/play-state-in-shorthand-actual.txt
@@ -1,4 +1,4 @@
 PASS - "transform" property for "box" element at 0.5s saw something close to: 1,0,0,1,75,0
-PASS - "transform" property for "box" element at 1s saw something close to: 1,0,0,1,150,0
-PASS - "transform" property for "box" element at 2.5s saw something close to: 1,0,0,1,150,0
+FAIL - "transform" property for "box" element at 1s expected: 1,0,0,1,150,0 but saw: matrix(1, 0, 0, 1, 171.3000030517578, 0)
+FAIL - "transform" property for "box" element at 2.5s expected: 1,0,0,1,150,0 but saw: matrix(1, 0, 0, 1, 175.0500030517578, 0)
 

Based on the flakiness dashboard and the fact that the legacy version of this test is also marked as flaky this doesn't seem to have a recent regression.
The test itself usually take around 2 seconds, even when failing.

[Shawn Roberts]

Test still flaky on Mac WK2 Debug

animations/play-state-in-shorthand.html

Flakiness dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=animations%2Fplay-state-in-shorthand.html

Reproduce with: 

run-webkit-tests --root d240655 animations/play-state-in-shorthand.html --iterations 500 -f --debug

[Radar WebKit Bug Importer]

<rdar://problem/50338860>

[Shawn Roberts]

Test was added in https://trac.webkit.org/changeset/200043/webkit

Is flaky on most Mac builds, and recently started showing up on iOS Sim Release and Debug as well.

Updated test expectations for Mac Debug and iOS Simulator in https://trac.webkit.org/changeset/244780/webkit .

[Antoine Quint]

If I understand this test, it does the following (Simon, please correct me if I'm wrong):

- set a timeout for 1s in the start callback to pause the animation using the CSS `animation` property shorthand halfway through
- set various timeouts for the animated value to be tested at 500ms (before the start callback timeout has elapsed and the animation is paused), at 1000ms and 2500ms to check the animation has been paused.

So the design of this test relies exclusively on setTimeout, which is bound to be flaky, especially as both the start callback and the second value check use the same timeout.

Let's see if we can respect the design of this test while making is robust.

[Antoine Quint]

I think it would be best to rewrite this test for the new animation engine to check the play state of the animation rather than be looking at CSS values. I'm going to do that, and leave the legacy variety of this test alone.

[Antoine Quint]

Created attachment 380328 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 380328 [details]
Patch

Clearing flags on attachment: 380328

Committed r250781: <https://trac.webkit.org/changeset/250781>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.