Bug 195233 – [ Mac WK2 iOS Debug ] REGRESSION(r233667) Layout Test imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor.html is a flaky failure

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Shawn Roberts]

The following layout test is failing on Mac WK2 + iOS Debug

imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor.html

Probable cause:

Test was modified in r233667, is 100% failure before that revision and is a flaky failure since it was changed.

Reproducible with:

run-webkit-tests imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor.html --iterations 5000 -f --exit-after-n-failures=5

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Finterfaces%2FDocumentTimeline%2Fconstructor.html

Diff: 

--- /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor-actual.txt
@@ -1,5 +1,5 @@
 
-PASS An origin time of zero is used when none is supplied 
+FAIL An origin time of zero is used when none is supplied assert_approx_equals: expected 30 +/- 0.001 but got 15
 PASS A zero origin time produces a document timeline with a current time identical to the default document timeline 
 PASS A positive origin time makes the document timeline's current time lag behind the default document timeline 
 PASS A negative origin time makes the document timeline's current time run ahead of the default document timeline

[Radar WebKit Bug Importer]

<rdar://problem/48527231>

[Shawn Roberts]

Adding additional repro steps:

Test was flaky through its entire available history on dashboard and searching through Trac I could not find any recent changes. Started Bisection point at 228049 which was the oldest revision I could spade.

Once I started getting 100% failures I switched test parameters to --exit-after-n-failures=25 to limit iterations run.

For Debug failures it failed in r233667 and 233636 was the only version I could spade previous to that.

Once I found the revision it was starting to fail on, Looking on Trac for r233667 it appears test expectations were modified.

Release WK2:

242285 flaky failure
242279 flaky failure
242112 flaky failure
237300 flaky failure
235000 flaky failure
234133 flaky failure
233702 flaky failure
233676 flaky failure
233671 flaky failure
233668 flaky failure
233667 4997 tests ran as expected, 3 didn't failed unexpectedly (text diff)

233666 0 tests ran as expected, 25 didn't
233665 0 tests ran as expected, 25 didn't
233649 0 tests ran as expected, 25 didn't
233595 0 tests ran as expected, 25 didn't
233483 0 tests ran as expected, 25 didn't
233263 0 tests ran as expected, 25 didn't
231525 0 tests ran as expected, 25 didn't
228049 0 tests ran as expected, 25 didn't : Unexpected text-only failures (1) [ Failure ]


Debug WK2:

233678 733 tests ran as expected, 5 didn't (4262 didn't run) failed unexpectedly (text diff)
233636 0 tests ran as expected, 25 didn't Unexpected text-only failures (1) [ Failure ]


Since initial testing found that iOS Simulator Release WK2, and Mac Debug WK1 are now flaky failures.

iOS Simulator:

242640 722 tests ran as expected, 5 didn't (4273 didn't run) Unexpected text-only failures (1)
233667 4997 tests ran as expected, 3 didn't failed unexpectedly (text diff)
233666 0 tests ran as expected, 25 didn't Unexpected text-only failures (1) [ Failure ]


Mac Debug WK1:

233678 395 tests ran as expected, 5 didn't (4600 didn't run): failed unexpectedly (text diff)
233636 0 tests ran as expected, 25 didn't (4975 didn't run): failed unexpectedly (text diff)

[Shawn Roberts]

Correction, This is not a regression to this specific version. The code just changed in r233667. Test was likely a flaky failure before that.

[Antoine Quint]

Created attachment 366936 [details]
Patch

[Antoine Quint]

Committed r244032: <https://trac.webkit.org/changeset/244032>