Bug 196505

Summary: [ Mac ] Layout Test accessibility/mac/press-not-work-for-disabled-menu-list.html is a flaky failure
Product: WebKit Reporter: Shawn Roberts <sroberts>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: aboxhall, andresg_22, apinheiro, cfleizach, commit-queue, dmazzoni, ericliang, ews-watchlist, jcraig, jdiggs, lforschler, samuel_white, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=193878
Attachments:
Description Flags
Patch none

Description Shawn Roberts 2019-04-02 12:02:42 PDT 
The following layout test is flaky on Mac WK1, WK2 Release and Debug

accessibility/mac/press-not-work-for-disabled-menu-list.html

Probable cause:

Test was added in https://trac.webkit.org/changeset/241190/webkit and is a flaky failure on its entire available dashboard history for Mojave WK1, but through local testing I am able to get it to fail on Release and Debug WK2 as well, just not as consistently. 

Local testing for WK1 yields approximately 20 failures in 500 iterations. WK2 testing yields about 2 failures in 500 iterations. 

Reproduced with:

run-webkit-tests accessibility/mac/press-not-work-for-disabled-menu-list.html --iterations 500 -f -1

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=accessibility%2Fmac%2Fpress-not-work-for-disabled-menu-list.html

Diff:

--- /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/accessibility/mac/press-not-work-for-disabled-menu-list-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/accessibility/mac/press-not-work-for-disabled-menu-list-actual.txt
@@ -1,3 +1,4 @@
+CONSOLE MESSAGE: line 56: TypeError: undefined is not an object (evaluating 'menulist.addNotificationListener')
     
 This tests that menu lists that are disabled will not be triggered by AXPress actions.
 
@@ -7,7 +8,7 @@
 Got notification: AXPressDidFail
 Got notification: AXPressDidFail
 Got notification: AXPressDidFail
-PASS successfullyParsed is true
+FAIL successfullyParsed should be true (of type boolean). Was undefined (of type undefined).
 
 TEST COMPLETE
Comment 1 Shawn Roberts 2019-04-02 12:10:02 PDT 
Marked flaky in https://trac.webkit.org/changeset/243750/webkit while waiting for a fix.
Comment 2 Radar WebKit Bug Importer 2019-04-02 13:47:28 PDT 
<rdar://problem/49532620>
Comment 3 Andres Gonzalez 2019-08-02 13:25:31 PDT 
Created attachment 375447 [details]
Patch
Comment 4 chris fleizach 2019-08-02 13:36:59 PDT 
Comment on attachment 375447 [details]
Patch

waiting for green bots to cq
Comment 5 WebKit Commit Bot 2019-08-02 16:20:38 PDT 
Comment on attachment 375447 [details]
Patch

Clearing flags on attachment: 375447

Committed r248189: <https://trac.webkit.org/changeset/248189>
Comment 6 WebKit Commit Bot 2019-08-02 16:20:39 PDT 
All reviewed patches have been landed.  Closing bug.