Bug 217931

Summary:	wpt/*/moving-between-documents/before-prepare-/move-back-* tests are flaky
Product:	WebKit	Reporter:	Hector Lopez <hector_i_lopez>
Component:	New Bugs	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	ap, cdumez, dpino, gsnedders, rackler, rniwa, ryanhaddad, tsavell, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=217908 https://bugs.webkit.org/show_bug.cgi?id=214685 https://bugs.webkit.org/show_bug.cgi?id=214734 https://bugs.webkit.org/show_bug.cgi?id=215278 https://bugs.webkit.org/show_bug.cgi?id=215283 https://bugs.webkit.org/show_bug.cgi?id=215706 https://bugs.webkit.org/show_bug.cgi?id=215783 https://bugs.webkit.org/show_bug.cgi?id=218650 https://bugs.webkit.org/show_bug.cgi?id=218651
Bug Depends on:	219655
Bug Blocks:

Description Hector Lopez 2020-10-19 16:45:10 PDT

imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/moving-between-documents/before-prepare-createHTMLDocument-success-empty-src-module.html

Test is a flaky failure according to history on macOS Debug. First occurrence of failure is at r265019.

History:

https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Fscripting-1%2Fthe-script-element%2Fmoving-between-documents%2Fbefore-prepare-createHTMLDocument-success-empty-src-module.html&style=debug&platform=mac&limit=50000

Diff:
--- /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/moving-between-documents/before-prepare-createHTMLDocument-success-empty-src-module-expected.txt
+++ /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/moving-between-documents/before-prepare-createHTMLDocument-success-empty-src-module-actual.txt
@@ -1,3 +1,4 @@
+CONSOLE MESSAGE: ReferenceError: Can't find variable: scriptOnError
 
 
 PASS Sanity check around top-level Window

Comment 1 Radar WebKit Bug Importer 2020-10-19 16:45:35 PDT

<rdar://problem/70462057>

Comment 2 Hector Lopez 2020-10-19 16:49:35 PDT

Test expectation while investigated:

https://trac.webkit.org/changeset/268698/webkit

Comment 3 Alexey Proskuryakov 2020-10-21 17:18:06 PDT

We are no up to 8 bugs for flaky tests in this one directory, failing with similar messages.

Comment 4 Ryosuke Niwa 2020-10-21 17:21:54 PDT

scriptOnError is referenced in moving-between-documents-iframe.py. Maybe moving-between-documents-helper.js isn't always loaded?

Comment 5 Alexey Proskuryakov 2020-10-21 18:57:57 PDT

I only see 200 OK responses for moving-between-documents-helper.js in WPT server log, https://build.webkit.org/results/Apple-Mojave-Debug-WK2-Tests/r268804%20(12688)/wptwk_process_log.out.txt

Comment 6 Ryosuke Niwa 2020-10-21 19:07:15 PDT

(In reply to Alexey Proskuryakov from comment #5)
> I only see 200 OK responses for moving-between-documents-helper.js in WPT
> server log,
> https://build.webkit.org/results/Apple-Mojave-Debug-WK2-Tests/
> r268804%20(12688)/wptwk_process_log.out.txt

Well, if the error happens before that helper JS is loaded, we'd still fail to find the function, right?

Comment 7 Alexey Proskuryakov 2020-10-22 09:27:24 PDT

Maybe? I hope that you found the link useful as a tiny piece of additional information.

Comment 8 Sam Sneddon [:gsnedders] 2020-12-02 15:11:04 PST

The identical failures of the various tests in this directory seems to happen on most platforms on debug but not release. They're all timing === "before-prepare" || timing === "move-back" cases in moving-between-documents-iframe.js too.

I'm suspicious of the `time.sleep(1)` (i.e., 1s) calls in html/semantics/scripting-1/the-script-element/moving-between-documents/resources/moving-between-documents-iframe.py; I suspect in debug we're just _that slow_ such that the server wins the race and the script runs before window.scriptOnError is assigned.

Comment 9 Alexey Proskuryakov 2020-12-02 18:47:55 PST

*** Bug 219464 has been marked as a duplicate of this bug. ***

Comment 10 Sam Sneddon [:gsnedders] 2020-12-08 13:43:28 PST

*** Bug 214685 has been marked as a duplicate of this bug. ***

Comment 11 Sam Sneddon [:gsnedders] 2020-12-08 13:43:38 PST

*** Bug 214734 has been marked as a duplicate of this bug. ***

Comment 12 Sam Sneddon [:gsnedders] 2020-12-08 13:43:45 PST

*** Bug 215278 has been marked as a duplicate of this bug. ***

Comment 13 Sam Sneddon [:gsnedders] 2020-12-08 13:43:51 PST

*** Bug 215283 has been marked as a duplicate of this bug. ***

Comment 14 Sam Sneddon [:gsnedders] 2020-12-08 13:43:57 PST

*** Bug 215706 has been marked as a duplicate of this bug. ***

Comment 15 Sam Sneddon [:gsnedders] 2020-12-08 13:44:02 PST

*** Bug 215783 has been marked as a duplicate of this bug. ***

Comment 16 Sam Sneddon [:gsnedders] 2020-12-08 13:44:08 PST

*** Bug 218650 has been marked as a duplicate of this bug. ***

Comment 17 Sam Sneddon [:gsnedders] 2020-12-08 13:44:13 PST

*** Bug 218651 has been marked as a duplicate of this bug. ***

Comment 18 Sam Sneddon [:gsnedders] 2020-12-08 13:44:19 PST

*** Bug 217908 has been marked as a duplicate of this bug. ***

Comment 19 Sam Sneddon [:gsnedders] 2020-12-08 13:58:07 PST

filed https://github.com/web-platform-tests/wpt/issues/26808 upstream for this flakiness