The following layout test is flaky on macOS and GTK inspector/unit-tests/throttle.html Probable cause: This test has been extremely flaky since it was added in https://trac.webkit.org/changeset/226755/webkit Flakiness Dashboard: https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=inspector%2Funit-tests%2Fthrottle.html --- /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/inspector/unit-tests/throttle-expected.txt +++ /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/inspector/unit-tests/throttle-actual.txt @@ -4,15 +4,7 @@ == Running test suite: Throttle -- Running test case: Basic throttle Call throttled function every 1 ms for 110 ms. -PASS: All calls delayed at least 50 ms. +FAIL: Delay should be at least 50 ms. Was 49 ms. +!! EXCEPTION: undefined +Stack Trace: #0: (anonymous) ((unknown)) --- Running test case: Throttle proxy uniqueness -PASS: Each call to throttle should return a new proxy. - --- Running test case: Throttled function arguments -PASS: Trailing call invoked with most recent arguments. - --- Running test case: Cancel throttle -Canceled throttled function. -PASS: Throttled function canceled. -
A quick fix would be to use expectEqualWithAccuracy, with an accuracy of 1ms. I haven't seen a delay be off by more than 1ms. That could be a last resort if the underlying problem is difficult to pin down.
I don't think we can rely on our bots to have 1ms accuracy. May be best to just skip this test / mark it as flakey since its all about timing.
(In reply to Joseph Pecoraro from comment #2) > I don't think we can rely on our bots to have 1ms accuracy. May be best to > just skip this test / mark it as flakey since its all about timing. With poor accuracy I'd still expect the throttled rate to be at least as long as the specified delay, never less. Before I just mark this flaky I'm going to investigate a little further.
Making an enhancement to Object.throttle which should fix this.
This continues to fail on the bots, so I'd like to either fix the issue or roll out the change as soon as possible.
I'd suggest marking this test as flakey rather then rolling out. The entire premise of Throttle is to test timing related things. Timing related tests are never reliable on the bots, particularly debug bots.
Marked test as flaky in https://trac.webkit.org/r227426
My justification for marking this as flakey instead of rolling out the original change is that flakiness on this test (which is timing based) does not indicate the change itself is incorrect. It may be doing what it is intended to do but missing the specific timing windows expected by the test. We've seen that is frequently the case for tests that rely on timing being run on the bots (particularly debug). The particular output is interesting: -PASS: All calls delayed at least 50 ms. +FAIL: Delay should be at least 50 ms. Was 49 ms. Getting 49 when expecting 50 probably means some round/fudge factor would be appropriate in the test. More disparate values would have been cause for concern. I suspect adding some fudge that is what Matt was alluding to improve the test. However he hasn't seemed to have gotten around to it yet.