Bug 192501 - [macOS WK2 Debug ] Flaky Test: animations/play-state-in-shorthand.html
Summary: [macOS WK2 Debug ] Flaky Test: animations/play-state-in-shorthand.html
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Animations (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Antoine Quint
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2018-12-07 10:19 PST by Matt Lewis
Modified: 2019-10-07 11:57 PDT (History)
8 users (show)

See Also:


Attachments
Patch (5.81 KB, patch)
2019-10-07 08:18 PDT, Antoine Quint
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Lewis 2018-12-07 10:19:55 PST
animations/play-state-in-shorthand.html is a flaky failure on macOS WK2 Debug


https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=animations%2Fplay-state-in-shorthand.html

https://build.webkit.org/results/Apple%20Mojave%20Debug%20WK2%20(Tests)/r238951%20(808)/results.html

--- /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/animations/play-state-in-shorthand-expected.txt
+++ /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/animations/play-state-in-shorthand-actual.txt
@@ -1,4 +1,4 @@
 PASS - "transform" property for "box" element at 0.5s saw something close to: 1,0,0,1,75,0
-PASS - "transform" property for "box" element at 1s saw something close to: 1,0,0,1,150,0
-PASS - "transform" property for "box" element at 2.5s saw something close to: 1,0,0,1,150,0
+FAIL - "transform" property for "box" element at 1s expected: 1,0,0,1,150,0 but saw: matrix(1, 0, 0, 1, 171.3000030517578, 0)
+FAIL - "transform" property for "box" element at 2.5s expected: 1,0,0,1,150,0 but saw: matrix(1, 0, 0, 1, 175.0500030517578, 0)
 

Based on the flakiness dashboard and the fact that the legacy version of this test is also marked as flaky this doesn't seem to have a recent regression.
The test itself usually take around 2 seconds, even when failing.
Comment 1 Shawn Roberts 2019-01-29 09:08:07 PST
Test still flaky on Mac WK2 Debug

animations/play-state-in-shorthand.html

Flakiness dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=animations%2Fplay-state-in-shorthand.html

Reproduce with: 

run-webkit-tests --root d240655 animations/play-state-in-shorthand.html --iterations 500 -f --debug
Comment 2 Radar WebKit Bug Importer 2019-04-30 09:58:11 PDT
<rdar://problem/50338860>
Comment 3 Shawn Roberts 2019-04-30 10:09:48 PDT
Test was added in https://trac.webkit.org/changeset/200043/webkit

Is flaky on most Mac builds, and recently started showing up on iOS Sim Release and Debug as well.

Updated test expectations for Mac Debug and iOS Simulator in https://trac.webkit.org/changeset/244780/webkit .
Comment 4 Antoine Quint 2019-10-07 07:15:40 PDT
If I understand this test, it does the following (Simon, please correct me if I'm wrong):

- set a timeout for 1s in the start callback to pause the animation using the CSS `animation` property shorthand halfway through
- set various timeouts for the animated value to be tested at 500ms (before the start callback timeout has elapsed and the animation is paused), at 1000ms and 2500ms to check the animation has been paused.

So the design of this test relies exclusively on setTimeout, which is bound to be flaky, especially as both the start callback and the second value check use the same timeout.

Let's see if we can respect the design of this test while making is robust.
Comment 5 Antoine Quint 2019-10-07 07:45:39 PDT
I think it would be best to rewrite this test for the new animation engine to check the play state of the animation rather than be looking at CSS values. I'm going to do that, and leave the legacy variety of this test alone.
Comment 6 Antoine Quint 2019-10-07 08:18:04 PDT
Created attachment 380328 [details]
Patch
Comment 7 WebKit Commit Bot 2019-10-07 11:57:35 PDT
Comment on attachment 380328 [details]
Patch

Clearing flags on attachment: 380328

Committed r250781: <https://trac.webkit.org/changeset/250781>
Comment 8 WebKit Commit Bot 2019-10-07 11:57:36 PDT
All reviewed patches have been landed.  Closing bug.