229831 2021-09-02 14:54:46 -0700 [ MacOS EWS ] imported/w3c/web-platform-tests/html/dom/idlharness.https.html is flaky failing 2022-01-05 15:48:52 -0800 1 1 1 Unclassified WebKit New Bugs WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=230407 InRadar P2 Normal --- 1 ayumi_kojima webkit-unassigned ap cdumez darin ddkilzer heycam mark.lam ryanhaddad simon.fraser webkit-bot-watchers-bugzilla webkit-bug-importer wenson_hsieh ysuzuki oldest_to_newest 1789910 0 ayumi_kojima 2021-09-02 14:54:46 -0700 imported/w3c/web-platform-tests/html/dom/idlharness.https.html Is a flaky failure on macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS and macOS-Catalina-Release-WK2-Tests-EWS. The flaky failure is not showing up in the open source directory: https://results.webkit.org/?suite=layout-tests&test=imported/w3c/web-platform-tests/html/dom/idlharness.https.html The test started flaky at https://ews-build.webkit.org/#/builders/60/builds/6499 Diff: --- /Volumes/Data/worker/macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-expected.txt +++ /Volumes/Data/worker/macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-actual.txt @@ -3916,7 +3916,7 @@ PASS CanvasRenderingContext2D interface: operation drawFocusIfNeeded(Element) PASS CanvasRenderingContext2D interface: operation drawFocusIfNeeded(Path2D, Element) FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView() assert_own_property: interface prototype object missing non-static operation expected property "scrollPathIntoView" missing -FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView(Path2D) assert_own_property: interface prototype object missing non-static operation expected property "scrollPathIntoView" missing +FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView(Path2D) assert_own_property: interface prototype object missing non-static operation expected property "sc|ollPathIntoView" missing PASS CanvasRenderingContext2D interface: operation fillText(DOMString, unrestricted double, unrestricted double, optional unrestricted double) PASS CanvasRenderingContext2D interface: operation strokeText(DOMString, unrestricted double, unrestricted double, optional unrestricted double) PASS CanvasRenderingContext2D interface: operation measureText(DOMString) 1789912 1 webkit-bug-importer 2021-09-02 14:55:11 -0700 <rdar://problem/82694768> 1789917 2 ayumi_kojima 2021-09-02 15:01:30 -0700 Marked expectations: https://trac.webkit.org/changeset/281957/webkit 1790573 3 ap 2021-09-05 14:29:38 -0700 - scrollPathIntoView + sc|ollPathIntoView What? This can be hard to investigate, given that we don't even see the issue on post-commit bots, but super important to get to the bottom of. > Marked expectations: https://trac.webkit.org/changeset/281957/webkit This effectively removed the effect of this line, making the test show up as always failing on Mac WK2. Ayumi, please add the DumpJSConsoleLogInStdErr property to Mac-wk2 as well. LayoutTests/TestExpectations:819:imported/w3c/web-platform-tests/html/dom/idlharness.https.html [ DumpJSConsoleLogInStdErr ] 1790577 4 heycam 2021-09-05 14:43:13 -0700 "r" -> "|" is not a single bitflip otherwise I would've suspected a hardware issue. Does this appear only on a particular builder? 1790581 5 ap 2021-09-05 15:07:17 -0700 > The test started flaky at https://ews-build.webkit.org/#/builders/60/builds/6499 Hmm, I assumed that this was happening currently, but the link is for a three month old build, and I cannot find this failure on macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS in the last month. What motivated filing this bug? Was this happening somewhere else recently? 1790582 6 ap 2021-09-05 15:14:49 -0700 I see it fail on macOS-Catalina-Release-WK2-Tests-EWS recently, but with an entirely different failure: +FAIL idl_test setup promise_test: Unhandled rejection with value: object "TypeError: Load failed" 1792464 7 ayumi_kojima 2021-09-10 11:35:03 -0700 Updated test expectations to include DumpJSConsoleLogInStdErr property: https://trac.webkit.org/changeset/282283/webkit 1792523 8 ayumi_kojima 2021-09-10 14:28:52 -0700 Tried to reproduce locally on Catalina, but the test timed out instead of failing using run-webkit-tests --iterations 50 --exit-after-n-failures 1 --force imported/w3c/web-platform-tests/html/dom/idlharness.https.html 1792528 9 ayumi_kojima 2021-09-10 14:33:00 -0700 Test hanged with --no-timeout 1794132 10 ayumi_kojima 2021-09-16 11:37:56 -0700 After https://trac.webkit.org/changeset/282484/webkit, the test started failing on iOS and macOS wk2 that appears to require rebaselining. Diff: --- /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-expected.txt +++ /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-actual.txt @@ -4997,14 +4997,14 @@ PASS Worker interface: attribute onmessage FAIL Worker interface: attribute onmessageerror assert_true: The prototype object must have a property "onmessageerror" expected true got false PASS Worker interface: attribute onerror -FAIL SharedWorker interface: existence and properties of interface object assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing -FAIL SharedWorker interface object length assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing -FAIL SharedWorker interface object name assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing -FAIL SharedWorker interface: existence and properties of interface prototype object assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing -FAIL SharedWorker interface: existence and properties of interface prototype object's "constructor" property assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing -FAIL SharedWorker interface: existence and properties of interface prototype object's @@unscopables property assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing -FAIL SharedWorker interface: attribute port assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing -FAIL SharedWorker interface: attribute onerror assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing +PASS SharedWorker interface: existence and properties of interface object +PASS SharedWorker interface object length +PASS SharedWorker interface object name +PASS SharedWorker interface: existence and properties of interface prototype object +PASS SharedWorker interface: existence and properties of interface prototype object's "constructor" property +PASS SharedWorker interface: existence and properties of interface prototype object's @@unscopables property +PASS SharedWorker interface: attribute port +PASS SharedWorker interface: attribute onerror PASS WorkerNavigator interface: existence and properties of interface object PASS WorkerLocation interface: existence and properties of interface object PASS WorkletGlobalScope interface: existence and properties of interface object 1794141 11 ayumi_kojima 2021-09-16 11:53:52 -0700 Rebaselined here https://trac.webkit.org/changeset/282592/webkit. Also removed the previous test expectations. 1794264 12 ayumi_kojima 2021-09-16 16:39:02 -0700 Reverted r282592 for reason: Rebaselined wrong on macOS Committed r282618 (241776@main): <https://commits.webkit.org/241776@main> 1794273 13 ayumi_kojima 2021-09-16 16:46:07 -0700 After rebaselining both iOS and macOS, macOS started failing. (I think that I added a wrong expected.txt) Reverted my change and rebaselined again only iOS. macOS expected.txt and expectations stay same: https://trac.webkit.org/changeset/282619/webkit 1828437 14 cdumez 2022-01-05 15:48:52 -0800 Committed r287657 (245754@trunk): <https://commits.webkit.org/245754@trunk>