Bug 231030 - [ iOS Debug ] imported/w3c/web-platform-tests/html/dom/idlharness.worker.html is a flaky failure
Summary: [ iOS Debug ] imported/w3c/web-platform-tests/html/dom/idlharness.worker.html...
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
Keywords: InRadar
Depends on:
Reported: 2021-09-30 10:41 PDT by ayumi_kojima
Modified: 2022-01-05 15:35 PST (History)
3 users (show)

See Also:

Diff (29.39 KB, text/plain)
2021-09-30 10:42 PDT, ayumi_kojima
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description ayumi_kojima 2021-09-30 10:41:38 PDT

Is flaky failing on iOS Debug & macOS Debug.

History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fdom%2Fidlharness.worker.html

Result page: https://build.webkit.org/results/Apple-BigSur-Debug-WK2-Tests/r283308%20(4012)/results.html



Catalina wk1 Debug has been failing for a while (Bug 206503)
Comment 1 ayumi_kojima 2021-09-30 10:42:39 PDT
Created attachment 439758 [details]
Comment 2 Radar WebKit Bug Importer 2021-09-30 10:43:03 PDT
Comment 3 ayumi_kojima 2021-09-30 13:30:44 PDT
I was not able to reproduce the failure locally on BigSur AS using run-webki-tests --debug imported/w3c/web-platform-tests/html/dom/idlharness.worker.html --iterations 50 --no-timeout --exit-after-n-failures 1 -f.

The test timed out instead. With --no-timeout flag, it hanged.
Comment 4 ayumi_kojima 2021-10-21 13:27:47 PDT
Marked test expectations https://trac.webkit.org/changeset/284639/webkit
Comment 5 ayumi_kojima 2021-10-22 15:33:46 PDT
It looks like the diff is different now than before on both iOS and macOS:

--- /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.worker-expected.txt
+++ /Volumes/Data/worker/ios-simulator-15-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/dom/idlharness.worker-actual.txt
@@ -1,4 +1,5 @@
 Blocked access to external URL http://invalid/
+CONSOLE MESSAGE: WebSocket connection to 'ws://invalid/' failed: A server with the specified hostname could not be found.
 PASS idl_test setup
 PASS idl_test validation

Result page: https://build.webkit.org/results/Apple-iOS-15-Simulator-Debug-WK2-Tests/r284693%20(265)/results.html

I was able to reproduce the same diff as this one locally on iOS 15 sim using run-webkit-tests --force --iterations 50 --exit-after-n-failures 1 imported/w3c/web-platform-tests/html/dom/idlharness.worker.html --ios-simulator --debug
Comment 6 ayumi_kojima 2021-10-22 16:52:20 PDT
It looks like the pass/fail expectations were overwriting "DumpJSConsoleLogInStdErr". Updated the expectations https://trac.webkit.org/changeset/284727/webkit
Comment 7 ayumi_kojima 2021-10-26 15:01:24 PDT
It seems the test passes on macOS now. Removed the expectation https://trac.webkit.org/changeset/284903/webkit
Comment 8 ayumi_kojima 2021-10-26 15:01:45 PDT
I was not able to reproduce the failure using run-webkit-tests --force --ios-simulator --debug --iterations 500 --exit-after-n-failures 1 imported/w3c/web-platform-tests/html/dom/idlharness.worker.html
Comment 9 ayumi_kojima 2021-10-27 11:25:16 PDT
The test started as flaky/pre-existing failure on macOS-Catalina-Debug-WK1-Tests-EWS.

Marked test expectations again for mac-wk1 debug:  https://trac.webkit.org/changeset/284931/webkit
Comment 10 Chris Dumez 2022-01-05 15:24:02 PST
It is a flaky timeout, the test is just slow to run. We should probably just mark it as slow or skip it in Debug.
Comment 11 Chris Dumez 2022-01-05 15:35:59 PST
Committed r287655 (245752@trunk): <https://commits.webkit.org/245752@trunk>