Bug 231918

Summary: REGRESSION (r283935): [ macOS wk1 ] imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-dialog-element/dialog-autofocus-multiple-times.html is a flaky failure
Product: WebKit Reporter: ayumi_kojima
Component: New BugsAssignee: Chris Dumez <cdumez>
Status: REOPENED ---    
Severity: Normal CC: cdumez, esprehn+autocc, ews-watchlist, graouts, kangil.han, ntim, rniwa, simon.fraser, webkit-bot-watchers-bugzilla, webkit-bug-importer, wenson_hsieh
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Mac (Intel)   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=203139
Bug Depends on:    
Bug Blocks: 203139    
Attachments:
Description Flags
Patch
none
Patch none

Description ayumi_kojima 2021-10-18 16:28:12 PDT
imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-dialog-element/dialog-autofocus-multiple-times.html

Is a flaky failure on macOS wk1 Release.

History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Finteractive-elements%2Fthe-dialog-element%2Fdialog-autofocus-multiple-times.html

Result page: https://build.webkit.org/results/Apple-BigSur-Release-WK1-Tests/r284398%20(6218)/results.html

Diff:

--- /Volumes/Data/worker/bigsur-release-tests-wk1/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-dialog-element/dialog-autofocus-multiple-times-expected.txt
+++ /Volumes/Data/worker/bigsur-release-tests-wk1/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-dialog-element/dialog-autofocus-multiple-times-actual.txt
@@ -1,4 +1,6 @@
 
 
-PASS autofocus is run every time a dialog is opened
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT autofocus is run every time a dialog is opened Test timed out
+


According to the history, the test started being flaky at around r283935.
Comment 1 Radar WebKit Bug Importer 2021-10-18 16:28:35 PDT
<rdar://problem/84393055>
Comment 2 ayumi_kojima 2021-10-18 16:30:29 PDT
The failure is also seen on wk1 Debug.
Comment 3 ayumi_kojima 2021-10-18 17:04:56 PDT
I was able to reproduce the failure locally using run-webkit-tests -1 --iterations 50 --exit-after-n-failures 1 imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-dialog-element/dialog-autofocus-multiple-times.html

I was able to reproduce the failure with r283938 and the test passed with r283934. No revision available to test in between.
Comment 4 ayumi_kojima 2021-10-18 17:07:59 PDT
I think https://trac.webkit.org/changeset/283935/webkit might have caused the regression.
Comment 5 ayumi_kojima 2021-10-18 17:12:41 PDT
Marked test expectations https://trac.webkit.org/changeset/284414/webkit
Comment 6 ayumi_kojima 2021-10-19 11:26:31 PDT
*** Bug 231969 has been marked as a duplicate of this bug. ***
Comment 7 ayumi_kojima 2021-10-19 11:27:11 PDT
imported/w3c/web-platform-tests/html/semantics/selectors/pseudo-classes/focus-autofocus.html is also flaky failing on macOS wk1 Release with the same diff.
Comment 8 ayumi_kojima 2021-10-19 13:59:27 PDT
Marked expectations https://trac.webkit.org/changeset/284493/webkit
Comment 9 ayumi_kojima 2021-10-20 15:50:25 PDT
*** Bug 231969 has been marked as a duplicate of this bug. ***
Comment 10 Chris Dumez 2021-11-08 08:04:32 PST
Created attachment 443548 [details]
Patch
Comment 11 Tim Nguyen (:ntim) 2021-11-08 13:26:17 PST
Comment on attachment 443548 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=443548&action=review

> LayoutTests/platform/mac-wk1/TestExpectations:788
>  webkit.org/b/231918 imported/w3c/web-platform-tests/html/semantics/selectors/pseudo-classes/focus-autofocus.html [ Pass Failure ]
>  
>  webkit.org/b/232042 imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-dialog-element/dialog-autofocus.html [ Pass Failure ] 

Can you check if those tests stop being flaky as well and remove them if they're not?

They're part of the same series of wk1 autofocus flaky failures
Comment 13 Chris Dumez 2021-11-08 13:37:37 PST
(In reply to Tim Nguyen (:ntim) from comment #11)
> Comment on attachment 443548 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=443548&action=review
> 
> > LayoutTests/platform/mac-wk1/TestExpectations:788
> >  webkit.org/b/231918 imported/w3c/web-platform-tests/html/semantics/selectors/pseudo-classes/focus-autofocus.html [ Pass Failure ]
> >  
> >  webkit.org/b/232042 imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-dialog-element/dialog-autofocus.html [ Pass Failure ] 
> 
> Can you check if those tests stop being flaky as well and remove them if
> they're not?
> 
> They're part of the same series of wk1 autofocus flaky failures

imported/w3c/web-platform-tests/html/semantics/selectors/pseudo-classes/focus-autofocus.html is still flaky. I haven't checked the others yet.
Comment 14 Chris Dumez 2021-11-08 14:55:01 PST
imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-dialog-element/dialog-autofocus.html is fixed by this patch too. I'll unskip it.
Comment 15 Chris Dumez 2021-11-08 14:55:52 PST
*** Bug 232042 has been marked as a duplicate of this bug. ***
Comment 16 Chris Dumez 2021-11-08 14:56:38 PST
Created attachment 443611 [details]
Patch
Comment 17 Chris Dumez 2021-11-08 14:59:38 PST
(In reply to Chris Dumez from comment #14)
> imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-
> dialog-element/dialog-autofocus.html is fixed by this patch too. I'll unskip
> it.

imported/w3c/web-platform-tests/html/interaction/focus/the-autofocus-attribute/document-with-fragment-valid.html is still flaky though, so different reason.
Comment 18 Chris Dumez 2021-11-08 15:12:48 PST
(In reply to Chris Dumez from comment #17)
> (In reply to Chris Dumez from comment #14)
> > imported/w3c/web-platform-tests/html/semantics/interactive-elements/the-
> > dialog-element/dialog-autofocus.html is fixed by this patch too. I'll unskip
> > it.
> 
> imported/w3c/web-platform-tests/html/interaction/focus/the-autofocus-
> attribute/document-with-fragment-valid.html is still flaky though, so
> different reason.

Ditto for imported/w3c/web-platform-tests/html/interaction/focus/the-autofocus-attribute/spin-by-blocking-style-sheet.html. Still flaky timeout so likely caused by something else.
Comment 19 EWS 2021-11-08 17:04:53 PST
Committed r285449 (243999@main): <https://commits.webkit.org/243999@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 443611 [details].
Comment 20 Antoine Quint 2021-12-17 07:02:35 PST
I just saw this test being flaky during this test run: https://ews-build.webkit.org/#/builders/56/builds/22257.
Comment 21 Antoine Quint 2021-12-17 07:23:29 PST
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Finteractive-elements%2Fthe-dialog-element%2Fdialog-autofocus.html suggests that at least this test, which was unskipped in this patch, is not reliably passing.

Reopening based on this evidence.