Bug 209006 – REGRESSION: [ Mac iOS wk2 ] imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/execution-timing/085.html is failing

• Status: RESOLVED
• Resolution: WONTFIX
• Component: New Bugs

[Truitt Savell]

imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/execution-timing/085.html

This issue began recently but is very flaky. the diffs are the same between iOS and Mac.

Reproduce with:
run-webkit-tests --iterations 2000 --exit-after-n-failures 1 --no-retry --no-build -f imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/execution-timing/085.html

history:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Fscripting-1%2Fthe-script-element%2Fexecution-timing%2F085.html

Diff:
--- /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/execution-timing/085-expected.txt
+++ /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/scripting-1/the-script-element/execution-timing/085-actual.txt
@@ -1,4 +1,4 @@
 FAILED (This TC requires JavaScript enabled)
 
-PASS  scheduler: async script and slow-loading defer script 
+FAIL  scheduler: async script and slow-loading defer script assert_array_equals: property 0, expected "external script #2" but got "external script #1"

[Radar WebKit Bug Importer]

<rdar://problem/60379091>

[Truitt Savell]

This test was last updated in https://trac.webkit.org/changeset/249886/webkit

[Truitt Savell]

This test seems to reproduce no matter what revision I run it on.

[Truitt Savell]

History shows this started some time after 258250 but no way currently to narrow that down

[Truitt Savell]

Marked test as failing while it is investigated:
https://trac.webkit.org/changeset/258345/webkit

[Alexey Proskuryakov]

This quite certainly is a regression from r258268.

[Chris Dumez]

(In reply to Alexey Proskuryakov from comment #6)
> This quite certainly is a regression from r258268.

Flakiness is likely triggered by when the first paint actually happens since the script is delayed until first paint.

[Chris Dumez]

(In reply to Chris Dumez from comment #7)
> (In reply to Alexey Proskuryakov from comment #6)
> > This quite certainly is a regression from r258268.
> 
> Flakiness is likely triggered by when the first paint actually happens since
> the script is delayed until first paint.

I think the test is inherently flaky. It has a slow loading defer script (1 second to load) and an async script. It expects the async script to always run first.

[Chris Dumez]

(In reply to Chris Dumez from comment #8)
> (In reply to Chris Dumez from comment #7)
> > (In reply to Alexey Proskuryakov from comment #6)
> > > This quite certainly is a regression from r258268.
> > 
> > Flakiness is likely triggered by when the first paint actually happens since
> > the script is delayed until first paint.
> 
> I think the test is inherently flaky. It has a slow loading defer script (1
> second to load) and an async script. It expects the async script to always
> run first.

To be clear, there is no expectation that async scripts run before defer scripts. async script run whenever they are done loading, which may happen before or after DOMContentLoaded. defer scripts run after DOMContentLoaded always. This test is using a 1 second delay to try and make sure that the async script finishes loading before DOMContentLoaded but this is bound to be flaky. This is definitely flakier too now that we defer async scripts further in some cases.

I think marking this test as flaky is the right thing to do given that this is an upstream test.