Bug 196505 - [ Mac ] Layout Test accessibility/mac/press-not-work-for-disabled-menu-list.html is a flaky failure
Summary: [ Mac ] Layout Test accessibility/mac/press-not-work-for-disabled-menu-list.h...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-04-02 12:02 PDT by Shawn Roberts
Modified: 2019-08-02 16:20 PDT (History)
14 users (show)

See Also:


Attachments
Patch (2.94 KB, patch)
2019-08-02 13:25 PDT, Andres Gonzalez
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Shawn Roberts 2019-04-02 12:02:42 PDT
The following layout test is flaky on Mac WK1, WK2 Release and Debug

accessibility/mac/press-not-work-for-disabled-menu-list.html

Probable cause:

Test was added in https://trac.webkit.org/changeset/241190/webkit and is a flaky failure on its entire available dashboard history for Mojave WK1, but through local testing I am able to get it to fail on Release and Debug WK2 as well, just not as consistently. 

Local testing for WK1 yields approximately 20 failures in 500 iterations. WK2 testing yields about 2 failures in 500 iterations. 

Reproduced with:

run-webkit-tests accessibility/mac/press-not-work-for-disabled-menu-list.html --iterations 500 -f -1

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=accessibility%2Fmac%2Fpress-not-work-for-disabled-menu-list.html

Diff:

--- /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/accessibility/mac/press-not-work-for-disabled-menu-list-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/accessibility/mac/press-not-work-for-disabled-menu-list-actual.txt
@@ -1,3 +1,4 @@
+CONSOLE MESSAGE: line 56: TypeError: undefined is not an object (evaluating 'menulist.addNotificationListener')
     
 This tests that menu lists that are disabled will not be triggered by AXPress actions.
 
@@ -7,7 +8,7 @@
 Got notification: AXPressDidFail
 Got notification: AXPressDidFail
 Got notification: AXPressDidFail
-PASS successfullyParsed is true
+FAIL successfullyParsed should be true (of type boolean). Was undefined (of type undefined).
 
 TEST COMPLETE
Comment 1 Shawn Roberts 2019-04-02 12:10:02 PDT
Marked flaky in https://trac.webkit.org/changeset/243750/webkit while waiting for a fix.
Comment 2 Radar WebKit Bug Importer 2019-04-02 13:47:28 PDT
<rdar://problem/49532620>
Comment 3 Andres Gonzalez 2019-08-02 13:25:31 PDT
Created attachment 375447 [details]
Patch
Comment 4 chris fleizach 2019-08-02 13:36:59 PDT
Comment on attachment 375447 [details]
Patch

waiting for green bots to cq
Comment 5 WebKit Commit Bot 2019-08-02 16:20:38 PDT
Comment on attachment 375447 [details]
Patch

Clearing flags on attachment: 375447

Committed r248189: <https://trac.webkit.org/changeset/248189>
Comment 6 WebKit Commit Bot 2019-08-02 16:20:39 PDT
All reviewed patches have been landed.  Closing bug.