195233 2019-03-01 16:08:06 -0800 [ Mac WK2 iOS Debug ] REGRESSION(r233667) Layout Test imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor.html is a flaky failure 2019-04-08 11:50:50 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=186498 InRadar P2 Normal --- 1 sroberts graouts dino graouts lforschler webkit-bot-watchers-bugzilla webkit-bug-importer oldest_to_newest 1511664 0 sroberts 2019-03-01 16:08:06 -0800 The following layout test is failing on Mac WK2 + iOS Debug imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor.html Probable cause: Test was modified in r233667, is 100% failure before that revision and is a flaky failure since it was changed. Reproducible with: run-webkit-tests imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor.html --iterations 5000 -f --exit-after-n-failures=5 Flakiness Dashboard: https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Finterfaces%2FDocumentTimeline%2Fconstructor.html Diff: --- /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor-expected.txt +++ /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/web-animations/interfaces/DocumentTimeline/constructor-actual.txt @@ -1,5 +1,5 @@ -PASS An origin time of zero is used when none is supplied +FAIL An origin time of zero is used when none is supplied assert_approx_equals: expected 30 +/- 0.001 but got 15 PASS A zero origin time produces a document timeline with a current time identical to the default document timeline PASS A positive origin time makes the document timeline's current time lag behind the default document timeline PASS A negative origin time makes the document timeline's current time run ahead of the default document timeline 1511671 1 webkit-bug-importer 2019-03-01 16:14:53 -0800 <rdar://problem/48527231> 1514102 2 sroberts 2019-03-08 11:31:57 -0800 Adding additional repro steps: Test was flaky through its entire available history on dashboard and searching through Trac I could not find any recent changes. Started Bisection point at 228049 which was the oldest revision I could spade. Once I started getting 100% failures I switched test parameters to --exit-after-n-failures=25 to limit iterations run. For Debug failures it failed in r233667 and 233636 was the only version I could spade previous to that. Once I found the revision it was starting to fail on, Looking on Trac for r233667 it appears test expectations were modified. Release WK2: 242285 flaky failure 242279 flaky failure 242112 flaky failure 237300 flaky failure 235000 flaky failure 234133 flaky failure 233702 flaky failure 233676 flaky failure 233671 flaky failure 233668 flaky failure 233667 4997 tests ran as expected, 3 didn't failed unexpectedly (text diff) 233666 0 tests ran as expected, 25 didn't 233665 0 tests ran as expected, 25 didn't 233649 0 tests ran as expected, 25 didn't 233595 0 tests ran as expected, 25 didn't 233483 0 tests ran as expected, 25 didn't 233263 0 tests ran as expected, 25 didn't 231525 0 tests ran as expected, 25 didn't 228049 0 tests ran as expected, 25 didn't : Unexpected text-only failures (1) [ Failure ] Debug WK2: 233678 733 tests ran as expected, 5 didn't (4262 didn't run) failed unexpectedly (text diff) 233636 0 tests ran as expected, 25 didn't Unexpected text-only failures (1) [ Failure ] Since initial testing found that iOS Simulator Release WK2, and Mac Debug WK1 are now flaky failures. iOS Simulator: 242640 722 tests ran as expected, 5 didn't (4273 didn't run) Unexpected text-only failures (1) 233667 4997 tests ran as expected, 3 didn't failed unexpectedly (text diff) 233666 0 tests ran as expected, 25 didn't Unexpected text-only failures (1) [ Failure ] Mac Debug WK1: 233678 395 tests ran as expected, 5 didn't (4600 didn't run): failed unexpectedly (text diff) 233636 0 tests ran as expected, 25 didn't (4975 didn't run): failed unexpectedly (text diff) 1523917 3 sroberts 2019-04-03 14:16:26 -0700 Correction, This is not a regression to this specific version. The code just changed in r233667. Test was likely a flaky failure before that. 1525183 4 366936 graouts 2019-04-08 08:07:24 -0700 Created attachment 366936 Patch 1525277 5 graouts 2019-04-08 11:50:50 -0700 Committed r244032: <https://trac.webkit.org/changeset/244032> 366936 2019-04-08 08:07:24 -0700 2019-04-08 11:39:49 -0700 Patch bug-195233-20190408170722.patch text/plain 2193 graouts U3VidmVyc2lvbiBSZXZpc2lvbjogMjQzOTI5CmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViQ29yZS9D aGFuZ2VMb2cgYi9Tb3VyY2UvV2ViQ29yZS9DaGFuZ2VMb2cKaW5kZXggZjEyMzI2MTc1NTIwNDlm ZTI2OTdjZDRmM2M0NzFlNDMwM2ZkMmQ1OS4uNDBlNTljNmFmOGNkMDAwZTRkZTM0YmJkYjI2NzM2 ZGNmNzc4YTcwYSAxMDA2NDQKLS0tIGEvU291cmNlL1dlYkNvcmUvQ2hhbmdlTG9nCisrKyBiL1Nv dXJjZS9XZWJDb3JlL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE2IEBACisyMDE5LTA0LTA4ICBBbnRv aW5lIFF1aW50ICA8Z3Jhb3V0c0BhcHBsZS5jb20+CisKKyAgICAgICAgWyBNYWMgV0syIGlPUyBE ZWJ1ZyBdIFJFR1JFU1NJT04ocjIzMzY2NykgTGF5b3V0IFRlc3QgaW1wb3J0ZWQvdzNjL3dlYi1w bGF0Zm9ybS10ZXN0cy93ZWItYW5pbWF0aW9ucy9pbnRlcmZhY2VzL0RvY3VtZW50VGltZWxpbmUv Y29uc3RydWN0b3IuaHRtbCBpcyBhIGZsYWt5IGZhaWx1cmUKKyAgICAgICAgaHR0cHM6Ly9idWdz LndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTE5NTIzMworICAgICAgICA8cmRhcjovL3Byb2Js ZW0vNDg1MjcyMzE+CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAg ICAgICAgV2UgbmVlZCB0byBjcmVhdGUgdGhlICJtYWluIiBkb2N1bWVudCB0aW1lbGluZSAoZG9j dW1lbnQudGltZWxpbmUpIGlmIGl0IGRvZXNuJ3QgYWxyZWFkeSBleGlzdCBhbmQgdXNlIGl0cyBj dXJyZW50IHRpbWUgYXMgYSBiYXNpcyBmb3IgYW55IG90aGVyIERvY3VtZW50VGltZWxpbmUgaW5z dGFuY2UuCisKKyAgICAgICAgKiBhbmltYXRpb24vRG9jdW1lbnRUaW1lbGluZS5jcHA6CisgICAg ICAgIChXZWJDb3JlOjpEb2N1bWVudFRpbWVsaW5lOjpjdXJyZW50VGltZSk6CisKIDIwMTktMDQt MDUgIFJ5b3N1a2UgTml3YSAgPHJuaXdhQHdlYmtpdC5vcmc+CiAKICAgICAgICAgTWFrZSBXZWFr UHRyPEVsZW1lbnQ+IHBvc3NpYmxlIGFuZCBkZXBsb3kgaXQgaW4gZm9ybSBhc3NvY2lhdGVkIGVs ZW1lbnRzIGNvZGUKZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJDb3JlL2FuaW1hdGlvbi9Eb2N1bWVu dFRpbWVsaW5lLmNwcCBiL1NvdXJjZS9XZWJDb3JlL2FuaW1hdGlvbi9Eb2N1bWVudFRpbWVsaW5l LmNwcAppbmRleCA3ZTJmMWM0YTEzYTE4YTE5N2IwM2I2ZGU5ZGRhMjA2ZTY2MjkzYzNmLi5jZDZk ZDBmMmY2ZDk3NjA5NzE3ZGYyZWQxYjRmMzFlNTNmODA2NzNkIDEwMDY0NAotLS0gYS9Tb3VyY2Uv V2ViQ29yZS9hbmltYXRpb24vRG9jdW1lbnRUaW1lbGluZS5jcHAKKysrIGIvU291cmNlL1dlYkNv cmUvYW5pbWF0aW9uL0RvY3VtZW50VGltZWxpbmUuY3BwCkBAIC0yNjYsMTIgKzI2NiwxMSBAQCBP cHRpb25hbDxTZWNvbmRzPiBEb2N1bWVudFRpbWVsaW5lOjpjdXJyZW50VGltZSgpCiAgICAgaWYg KCFtX2RvY3VtZW50IHx8ICFtX2RvY3VtZW50LT5kb21XaW5kb3coKSkKICAgICAgICAgcmV0dXJu IEFuaW1hdGlvblRpbWVsaW5lOjpjdXJyZW50VGltZSgpOwogCi0gICAgaWYgKGF1dG8qIG1haW5E b2N1bWVudFRpbWVsaW5lID0gbV9kb2N1bWVudC0+ZXhpc3RpbmdUaW1lbGluZSgpKSB7Ci0gICAg ICAgIGlmIChtYWluRG9jdW1lbnRUaW1lbGluZSAhPSB0aGlzKSB7Ci0gICAgICAgICAgICBpZiAo YXV0byBtYWluRG9jdW1lbnRUaW1lbGluZUN1cnJlbnRUaW1lID0gbWFpbkRvY3VtZW50VGltZWxp bmUtPmN1cnJlbnRUaW1lKCkpCi0gICAgICAgICAgICAgICAgcmV0dXJuIG1haW5Eb2N1bWVudFRp bWVsaW5lQ3VycmVudFRpbWUudmFsdWUoKSAtIG1fb3JpZ2luVGltZTsKLSAgICAgICAgICAgIHJl dHVybiBXVEY6Om51bGxvcHQ7Ci0gICAgICAgIH0KKyAgICBhdXRvJiBtYWluRG9jdW1lbnRUaW1l bGluZSA9IG1fZG9jdW1lbnQtPnRpbWVsaW5lKCk7CisgICAgaWYgKCZtYWluRG9jdW1lbnRUaW1l bGluZSAhPSB0aGlzKSB7CisgICAgICAgIGlmIChhdXRvIG1haW5Eb2N1bWVudFRpbWVsaW5lQ3Vy cmVudFRpbWUgPSBtYWluRG9jdW1lbnRUaW1lbGluZS5jdXJyZW50VGltZSgpKQorICAgICAgICAg ICAgcmV0dXJuICptYWluRG9jdW1lbnRUaW1lbGluZUN1cnJlbnRUaW1lIC0gbV9vcmlnaW5UaW1l OworICAgICAgICByZXR1cm4gV1RGOjpudWxsb3B0OwogICAgIH0KIAogICAgIGF1dG8gY3VycmVu dFRpbWUgPSBsaXZlQ3VycmVudFRpbWUoKTsK