This seems to fail often enough on the commit-queue to not get reported. If a test fails twice in a row, it won't be reported as flaky, as the tree will be assumed to be red instead. :) *** Bug 51004 has been marked as a duplicate of this bug. *** I have a patch in progress to fix this class of test. I'll try to get it finished up before heading out for the holidays. The commit-queue just saw animations/suspend-resume-animation.html flake while processing attachment 76747 [details] on bug 51173. Bot: eseidel-cq-sf Port: Mac Platform: Mac OS X 10.6.4 Created attachment 76789 [details]
Failure diff from bot
The commit-queue just saw animations/suspend-resume-animation.html flake while processing attachment 77006 [details] on bug 51338. Bot: eseidel-cq-sf Port: Mac Platform: Mac OS X 10.6.4 Created attachment 77019 [details]
Failure diff from bot
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 79863 [details] on bug 52968. Bot: eseidel-cq-sl Port: Mac Platform: Mac OS X 10.6.6 Created attachment 79874 [details]
Failure diff from eseidel-cq-sl
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 79401 [details] on bug 52704. Bot: eseidel-cq-sl Port: Mac Platform: Mac OS X 10.6.6 Created attachment 79988 [details]
Failure diff from eseidel-cq-sl
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 80460 [details] on bug 53308. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 80707 [details]
Failure diff from cr-jail-8
(In reply to comment #0) > Background: > > This tests that suspendAnimations and resumeAnimations work properly. > > It starts an animation, suspends it on setTimeout(500), and resumes it on setTimeout(1000). To test that the suspend works, it checks the animation at .6 and .9 seconds, and then again at 1.5 seconds to test that resume worked: > // [animation-name, time, element-id, property, expected-value, tolerance] > ["rotate", 0.6, "box", "webkitTransform", [1,0,0,1, 200, 0], 30], > ["rotate", 0.9, "box", "webkitTransform", [1,0,0,1, 200, 0], 30], > ["rotate", 1.5, "box", "webkitTransform", [1,0,0,1, 400, 0], 30], > > Due to the timeouts involved, this can be flaky (we are seeing it fail about 2% of the time with values of 231 and 431, 1 pixel outside the tolerance). > > Rather than tweak the tolerances more, I was thinking it might make more sense to re-write the test to more explicitly test that suspend and resume work, rather than relying on the animation values. So, something like: > At 0.6 and 0.9, rather than test for being at 200, test that the value is the same at both times (aka, it is definitely suspended), and that the value is > 0 (it did indeed start), and less < the end value (it hasn't ended). > At 1.5, rather than test for being at 400, test that the value is greater than it was at 0.6/0.9 (aka, it did resume). > > Does that sound ok, or would that be losing test coverage? I generally like the concept of your idea, but I think we really need to do some range testing. Otherwise if the animation is just not working at all it would pass. So maybe your idea of repeatability coupled with a more lenient range test would give us good coverage? The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 81298 [details] on bug 53815. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 81351 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 81928 [details] on bug 54169. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 81952 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 82117 [details] on bug 54280. Bot: cr-jail-4 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 82124 [details]
Failure diff from cr-jail-4
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 82257 [details] on bug 54351. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 82270 [details]
Failure diff from cr-jail-7
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 83118 [details] on bug 54842. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 83120 [details]
Failure diff from cr-jail-7
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 83242 [details] on bug 54909. Bot: cr-jail-4 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 83268 [details]
Failure diff from cr-jail-4
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 83236 [details] on bug 54910. Bot: cr-jail-3 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 83442 [details]
Failure diff from cr-jail-3
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 83303 [details] on bug 54873. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 83443 [details]
Failure diff from cr-jail-8
https://bug-48161-attachments.webkit.org/attachment.cgi?id=83442 makes it look like there may be another race going on. The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 83482 [details] on bug 54953. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 83485 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 83875 [details] on bug 54489. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 83936 [details]
Failure diff from cr-jail-7
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 84492 [details] on bug 55632. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 84541 [details]
Failure diff from cr-jail-7
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 85538 [details] on bug 56052. Bot: cr-jail-3 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 85759 [details]
Failure diff from cr-jail-3
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 86081 [details] on bug 56580. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 86117 [details]
Failure diff from cr-jail-7
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 86919 [details] on bug 56263. Bot: cr-jail-3 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 86944 [details]
Failure diff from cr-jail-3
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 87724 [details] on bug 40239. Bot: eseidel-cq-sf Port: Mac Platform: Mac OS X 10.6.4 Created attachment 87738 [details]
Failure diff from eseidel-cq-sf
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 87925 [details] on bug 57666. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 87958 [details]
Failure diff from cr-jail-7
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 88247 [details] on bug 40261. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 88381 [details]
Failure diff from cr-jail-7
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 88510 [details] on bug 57877. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 88538 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 88734 [details] on bug 58098. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 88771 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 89475 [details] on bug 58073. Bot: cr-jail-3 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 89708 [details]
Failure diff from cr-jail-3
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 83366 [details] on bug 54833. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 89969 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 89899 [details] on bug 58719. Bot: cr-jail-3 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 90066 [details]
Failure diff from cr-jail-3
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 90452 [details] on bug 59032. Bot: cr-jail-3 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 90466 [details]
Failure diff from cr-jail-3
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 91529 [details] on bug 58365. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 91552 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 91529 [details] on bug 58365. Bot: cr-jail-3 Port: Mac Platform: Mac OS X 10.6.6 Created attachment 91554 [details]
Failure diff from cr-jail-3
This test appears to be super-flaky. Can we do something about it? Or should I just skip it? The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 91983 [details] on bug 58703. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.7 Created attachment 92111 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 92074 [details] on bug 60024. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.7 Created attachment 92146 [details]
Failure diff from cr-jail-7
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 94042 [details] on bug 61089. Bot: cr-jail-3 Port: Mac Platform: Mac OS X 10.6.7 Created attachment 94095 [details]
Failure diff from cr-jail-3
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 94069 [details] on bug 59345. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.7 Created attachment 94213 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 94583 [details] on bug 60827. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.7 Created attachment 94612 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 94617 [details] on bug 61284. Bot: cr-jail-4 Port: Mac Platform: Mac OS X 10.6.7 Created attachment 94646 [details]
Failure diff from cr-jail-4
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 94806 [details] on bug 61336. Bot: cr-jail-8 Port: Mac Platform: Mac OS X 10.6.7 Created attachment 94865 [details]
Failure diff from cr-jail-8
The commit-queue just saw animations/suspend-resume-animation.html flake (Text diff mismatch) while processing attachment 94319 [details] on bug 31721. Bot: cr-jail-7 Port: Mac Platform: Mac OS X 10.6.7 Created attachment 94914 [details]
Failure diff from cr-jail-7
|
Background: This tests that suspendAnimations and resumeAnimations work properly. It starts an animation, suspends it on setTimeout(500), and resumes it on setTimeout(1000). To test that the suspend works, it checks the animation at .6 and .9 seconds, and then again at 1.5 seconds to test that resume worked: // [animation-name, time, element-id, property, expected-value, tolerance] ["rotate", 0.6, "box", "webkitTransform", [1,0,0,1, 200, 0], 30], ["rotate", 0.9, "box", "webkitTransform", [1,0,0,1, 200, 0], 30], ["rotate", 1.5, "box", "webkitTransform", [1,0,0,1, 400, 0], 30], Due to the timeouts involved, this can be flaky (we are seeing it fail about 2% of the time with values of 231 and 431, 1 pixel outside the tolerance). Rather than tweak the tolerances more, I was thinking it might make more sense to re-write the test to more explicitly test that suspend and resume work, rather than relying on the animation values. So, something like: At 0.6 and 0.9, rather than test for being at 200, test that the value is the same at both times (aka, it is definitely suspended), and that the value is > 0 (it did indeed start), and less < the end value (it hasn't ended). At 1.5, rather than test for being at 400, test that the value is greater than it was at 0.6/0.9 (aka, it did resume). Does that sound ok, or would that be losing test coverage?