Bug 229831 - [ MacOS EWS ] imported/w3c/web-platform-tests/html/dom/idlharness.https.html is flaky failing
Summary: [ MacOS EWS ] imported/w3c/web-platform-tests/html/dom/idlharness.https.html ...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-09-02 14:54 PDT by ayumi_kojima
Modified: 2021-09-17 08:41 PDT (History)
12 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description ayumi_kojima 2021-09-02 14:54:46 PDT
imported/w3c/web-platform-tests/html/dom/idlharness.https.html

Is a flaky failure on macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS and macOS-Catalina-Release-WK2-Tests-EWS.

The flaky failure is not showing up in the open source directory: https://results.webkit.org/?suite=layout-tests&test=imported/w3c/web-platform-tests/html/dom/idlharness.https.html

The test started flaky at https://ews-build.webkit.org/#/builders/60/builds/6499

Diff:

--- /Volumes/Data/worker/macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-expected.txt
+++ /Volumes/Data/worker/macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-actual.txt
@@ -3916,7 +3916,7 @@
 PASS CanvasRenderingContext2D interface: operation drawFocusIfNeeded(Element)
 PASS CanvasRenderingContext2D interface: operation drawFocusIfNeeded(Path2D, Element)
 FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView() assert_own_property: interface prototype object missing non-static operation expected property "scrollPathIntoView" missing
-FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView(Path2D) assert_own_property: interface prototype object missing non-static operation expected property "scrollPathIntoView" missing
+FAIL CanvasRenderingContext2D interface: operation scrollPathIntoView(Path2D) assert_own_property: interface prototype object missing non-static operation expected property "sc|ollPathIntoView" missing
 PASS CanvasRenderingContext2D interface: operation fillText(DOMString, unrestricted double, unrestricted double, optional unrestricted double)
 PASS CanvasRenderingContext2D interface: operation strokeText(DOMString, unrestricted double, unrestricted double, optional unrestricted double)
 PASS CanvasRenderingContext2D interface: operation measureText(DOMString)
Comment 1 Radar WebKit Bug Importer 2021-09-02 14:55:11 PDT
<rdar://problem/82694768>
Comment 2 ayumi_kojima 2021-09-02 15:01:30 PDT
Marked expectations: https://trac.webkit.org/changeset/281957/webkit
Comment 3 Alexey Proskuryakov 2021-09-05 14:29:38 PDT
- scrollPathIntoView
+ sc|ollPathIntoView

What?

This can be hard to investigate, given that we don't even see the issue on post-commit bots, but super important to get to the bottom of.

> Marked expectations: https://trac.webkit.org/changeset/281957/webkit

This effectively removed the effect of this line, making the test show up as always failing on Mac WK2. Ayumi, please add the DumpJSConsoleLogInStdErr property to Mac-wk2 as well.

LayoutTests/TestExpectations:819:imported/w3c/web-platform-tests/html/dom/idlharness.https.html [ DumpJSConsoleLogInStdErr ]
Comment 4 Cameron McCormack (:heycam) 2021-09-05 14:43:13 PDT
"r" -> "|" is not a single bitflip otherwise I would've suspected a hardware issue. Does this appear only on a particular builder?
Comment 5 Alexey Proskuryakov 2021-09-05 15:07:17 PDT
> The test started flaky at https://ews-build.webkit.org/#/builders/60/builds/6499

Hmm, I assumed that this was happening currently, but the link is for a three month old build, and I cannot find this failure on macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS in the last month.

What motivated filing this bug? Was this happening somewhere else recently?
Comment 6 Alexey Proskuryakov 2021-09-05 15:14:49 PDT
I see it fail on macOS-Catalina-Release-WK2-Tests-EWS recently, but with an entirely different failure:

+FAIL idl_test setup promise_test: Unhandled rejection with value: object "TypeError: Load failed"
Comment 7 ayumi_kojima 2021-09-10 11:35:03 PDT
Updated test expectations to include DumpJSConsoleLogInStdErr property: https://trac.webkit.org/changeset/282283/webkit
Comment 8 ayumi_kojima 2021-09-10 14:28:52 PDT
Tried to reproduce locally on Catalina, but the test timed out instead of failing using run-webkit-tests --iterations 50 --exit-after-n-failures 1 --force imported/w3c/web-platform-tests/html/dom/idlharness.https.html
Comment 9 ayumi_kojima 2021-09-10 14:33:00 PDT
Test hanged with --no-timeout
Comment 10 ayumi_kojima 2021-09-16 11:37:56 PDT
After https://trac.webkit.org/changeset/282484/webkit, the test started failing on iOS and macOS wk2 that appears to require rebaselining.

Diff:

--- /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-expected.txt
+++ /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.https-actual.txt
@@ -4997,14 +4997,14 @@
 PASS Worker interface: attribute onmessage
 FAIL Worker interface: attribute onmessageerror assert_true: The prototype object must have a property "onmessageerror" expected true got false
 PASS Worker interface: attribute onerror
-FAIL SharedWorker interface: existence and properties of interface object assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface object length assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface object name assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: existence and properties of interface prototype object assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: existence and properties of interface prototype object's "constructor" property assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: existence and properties of interface prototype object's @@unscopables property assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: attribute port assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
-FAIL SharedWorker interface: attribute onerror assert_own_property: self does not have own property "SharedWorker" expected property "SharedWorker" missing
+PASS SharedWorker interface: existence and properties of interface object
+PASS SharedWorker interface object length
+PASS SharedWorker interface object name
+PASS SharedWorker interface: existence and properties of interface prototype object
+PASS SharedWorker interface: existence and properties of interface prototype object's "constructor" property
+PASS SharedWorker interface: existence and properties of interface prototype object's @@unscopables property
+PASS SharedWorker interface: attribute port
+PASS SharedWorker interface: attribute onerror
 PASS WorkerNavigator interface: existence and properties of interface object
 PASS WorkerLocation interface: existence and properties of interface object
 PASS WorkletGlobalScope interface: existence and properties of interface object
Comment 11 ayumi_kojima 2021-09-16 11:53:52 PDT
Rebaselined here https://trac.webkit.org/changeset/282592/webkit. Also removed the previous test expectations.
Comment 12 ayumi_kojima 2021-09-16 16:39:02 PDT
Reverted r282592 for reason:

Rebaselined wrong on macOS

Committed r282618 (241776@main): <https://commits.webkit.org/241776@main>
Comment 13 ayumi_kojima 2021-09-16 16:46:07 PDT
After rebaselining both iOS and macOS, macOS started failing. (I think that I added a wrong expected.txt) Reverted my change and rebaselined again only iOS. macOS expected.txt and expectations stay same: https://trac.webkit.org/changeset/282619/webkit