Bug 229831 – [ MacOS EWS ] imported/w3c/web-platform-tests/html/dom/idlharness.https.html is flaky failing

• Component: New Bugs
• Status: RESOLVED FIXED

imported/w3c/web-platform-tests/html/dom/idlharness.https.html

Is a flaky failure on macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS and macOS-Catalina-Release-WK2-Tests-EWS.

The flaky failure is not showing up in the open source directory: https://results.webkit.org/?suite=layout-tests&test=imported/w3c/web-platform-tests/html/dom/idlharness.https.html

The test started flaky at https://ews-build.webkit.org/#/builders/60/builds/6499

Diff:

--- /Volumes/Data/worker/macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-expected.txt
+++ /Volumes/Data/worker/macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-actual.txt
@@ -3916,7 +3916,7 @@
 PASS CanvasRenderingContext2D interface: operation drawFocusIfNeeded(Element)
 PASS CanvasRenderingContext2D interface: operation drawFocusIfNeeded(Path2D, Element)
 FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView() assert_own_property: interface prototype object missing non-static operation expected property "scrollPathIntoView" missing
-FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView(Path2D) assert_own_property: interface prototype object missing non-static operation expected property "scrollPathIntoView" missing
+FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView(Path2D) assert_own_property: interface prototype object missing non-static operation expected property "sc|ollPathIntoView" missing
 PASS CanvasRenderingContext2D interface: operation fillText(DOMString, unrestricted double, unrestricted double, optional unrestricted double)
 PASS CanvasRenderingContext2D interface: operation strokeText(DOMString, unrestricted double, unrestricted double, optional unrestricted double)
 PASS CanvasRenderingContext2D interface: operation measureText(DOMString)

[Radar WebKit Bug Importer]

<rdar://problem/82694768>

Marked expectations: https://trac.webkit.org/changeset/281957/webkit

[Alexey Proskuryakov]

- scrollPathIntoView
+ sc|ollPathIntoView

What?

This can be hard to investigate, given that we don't even see the issue on post-commit bots, but super important to get to the bottom of.

> Marked expectations: https://trac.webkit.org/changeset/281957/webkit

This effectively removed the effect of this line, making the test show up as always failing on Mac WK2. Ayumi, please add the DumpJSConsoleLogInStdErr property to Mac-wk2 as well.

LayoutTests/TestExpectations:819:imported/w3c/web-platform-tests/html/dom/idlharness.https.html [ DumpJSConsoleLogInStdErr ]

[Cameron McCormack (:heycam)]

"r" -> "|" is not a single bitflip otherwise I would've suspected a hardware issue. Does this appear only on a particular builder?

[Alexey Proskuryakov]

> The test started flaky at https://ews-build.webkit.org/#/builders/60/builds/6499

Hmm, I assumed that this was happening currently, but the link is for a three month old build, and I cannot find this failure on macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS in the last month.

What motivated filing this bug? Was this happening somewhere else recently?

[Alexey Proskuryakov]

I see it fail on macOS-Catalina-Release-WK2-Tests-EWS recently, but with an entirely different failure:

+FAIL idl_test setup promise_test: Unhandled rejection with value: object "TypeError: Load failed"

Updated test expectations to include DumpJSConsoleLogInStdErr property: https://trac.webkit.org/changeset/282283/webkit

Tried to reproduce locally on Catalina, but the test timed out instead of failing using run-webkit-tests --iterations 50 --exit-after-n-failures 1 --force imported/w3c/web-platform-tests/html/dom/idlharness.https.html

Test hanged with --no-timeout

After https://trac.webkit.org/changeset/282484/webkit, the test started failing on iOS and macOS wk2 that appears to require rebaselining.

Diff:

--- /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-expected.txt
+++ /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-actual.txt
@@ -4997,14 +4997,14 @@
 PASS Worker interface: attribute onmessage
 FAIL Worker interface: attribute onmessageerror assert_true: The prototype object must have a property "onmessageerror" expected true got false
 PASS Worker interface: attribute onerror
-FAIL SharedWorker interface: existence and properties of interface object assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface object length assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface object name assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: existence and properties of interface prototype object assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: existence and properties of interface prototype object's "constructor" property assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: existence and properties of interface prototype object's @@unscopables property assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: attribute port assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: attribute onerror assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
+PASS SharedWorker interface: existence and properties of interface object
+PASS SharedWorker interface object length
+PASS SharedWorker interface object name
+PASS SharedWorker interface: existence and properties of interface prototype object
+PASS SharedWorker interface: existence and properties of interface prototype object's "constructor" property
+PASS SharedWorker interface: existence and properties of interface prototype object's @@unscopables property
+PASS SharedWorker interface: attribute port
+PASS SharedWorker interface: attribute onerror
 PASS WorkerNavigator interface: existence and properties of interface object
 PASS WorkerLocation interface: existence and properties of interface object
 PASS WorkletGlobalScope interface: existence and properties of interface object

Rebaselined here https://trac.webkit.org/changeset/282592/webkit. Also removed the previous test expectations.

Reverted r282592 for reason:

Rebaselined wrong on macOS

Committed r282618 (241776@main): <https://commits.webkit.org/241776@main>

After rebaselining both iOS and macOS, macOS started failing. (I think that I added a wrong expected.txt) Reverted my change and rebaselined again only iOS. macOS expected.txt and expectations stay same: https://trac.webkit.org/changeset/282619/webkit

[Chris Dumez]

Committed r287657 (245754@trunk): <https://commits.webkit.org/245754@trunk>