Bug 217931 – wpt/**/moving-between-documents/before-prepare-*/move-back-* tests are flaky

• Status: NEW
• Component: New Bugs

[Hector Lopez]

imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/moving-between-documents/before-prepare-createHTMLDocument-success-empty-src-module.html

Test is a flaky failure according to history on macOS Debug. First occurrence of failure is at r265019.

History:

https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Fscripting-1%2Fthe-script-element%2Fmoving-between-documents%2Fbefore-prepare-createHTMLDocument-success-empty-src-module.html&style=debug&platform=mac&limit=50000

Diff:
--- /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/moving-between-documents/before-prepare-createHTMLDocument-success-empty-src-module-expected.txt
+++ /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/moving-between-documents/before-prepare-createHTMLDocument-success-empty-src-module-actual.txt
@@ -1,3 +1,4 @@
+CONSOLE MESSAGE: ReferenceError: Can't find variable: scriptOnError
 
 
 PASS Sanity check around top-level Window

[Radar WebKit Bug Importer]

<rdar://problem/70462057>

[Hector Lopez]

Test expectation while investigated:

https://trac.webkit.org/changeset/268698/webkit

[Alexey Proskuryakov]

We are no up to 8 bugs for flaky tests in this one directory, failing with similar messages.

[Ryosuke Niwa]

scriptOnError is referenced in moving-between-documents-iframe.py. Maybe moving-between-documents-helper.js isn't always loaded?

[Alexey Proskuryakov]

I only see 200 OK responses for moving-between-documents-helper.js in WPT server log, https://build.webkit.org/results/Apple-Mojave-Debug-WK2-Tests/r268804%20(12688)/wptwk_process_log.out.txt

[Ryosuke Niwa]

(In reply to Alexey Proskuryakov from comment #5)
> I only see 200 OK responses for moving-between-documents-helper.js in WPT
> server log,
> https://build.webkit.org/results/Apple-Mojave-Debug-WK2-Tests/
> r268804%20(12688)/wptwk_process_log.out.txt

Well, if the error happens before that helper JS is loaded, we'd still fail to find the function, right?

[Alexey Proskuryakov]

Maybe? I hope that you found the link useful as a tiny piece of additional information.

[Sam Sneddon [:gsnedders]]

The identical failures of the various tests in this directory seems to happen on most platforms on debug but not release. They're all timing === "before-prepare" || timing === "move-back" cases in moving-between-documents-iframe.js too.

I'm suspicious of the `time.sleep(1)` (i.e., 1s) calls in html/semantics/scripting-1/the-script-element/moving-between-documents/resources/moving-between-documents-iframe.py; I suspect in debug we're just _that slow_ such that the server wins the race and the script runs before window.scriptOnError is assigned.

[Alexey Proskuryakov]

*** Bug 219464 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 214685 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 214734 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 215278 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 215283 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 215706 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 215783 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 218650 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 218651 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

*** Bug 217908 has been marked as a duplicate of this bug. ***

[Sam Sneddon [:gsnedders]]

filed https://github.com/web-platform-tests/wpt/issues/26808 upstream for this flakiness