Bug 182761 – REGRESSION(r228279): [GTK] Many aria tests are now flaky

• Status: RESOLVED
• Resolution: FIXED
• Component: Accessibility

[Michael Catanzaro]

r228279 "AX: Defer attribute computation until needed." has introduced a bunch of flakiness on the GTK release bot. The following tests are all alternating between pass and fail, except for the first one which is now consistently failing. I'm going to add flakiness expectations.

--- /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-busy-changed-notification-expected.txt
+++ /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-busy-changed-notification-actual.txt
@@ -6,8 +6,6 @@
 Setting aria-busy to true on listbox.
 Setting aria-busy to false on listbox.
 AXElementBusyChanged: AXTitle:  AXRole: AXWebArea
-AXElementBusyChanged: AXTitle:  AXRole: AXListBox
-AXElementBusyChanged: AXTitle:  AXRole: AXListBox
 PASS successfullyParsed is true
 
 TEST COMPLETE

--- /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-current-changed-notification-expected.txt
+++ /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-current-changed-notification-actual.txt
@@ -5,8 +5,6 @@
 
 Setting aria-current to false on item2.
 Setting aria-current to page on item3.
-ActiveStateChanged: AXTitle: 2 AXRole: AXListItem
-ActiveStateChanged: AXTitle: 3 AXRole: AXListItem
 PASS successfullyParsed is true
 
 TEST COMPLETE

--- /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-disabled-changed-notification-expected.txt
+++ /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-disabled-changed-notification-actual.txt
@@ -5,10 +5,6 @@
 
 Setting aria-disabled to true on textbox1.
 Setting aria-disabled to false on textbox2.
-AXDisabledStateChanged true on textbox1
-AXSensitiveStateChanged false on textbox1
-AXDisabledStateChanged false on textbox2
-AXSensitiveStateChanged true on textbox2
 PASS successfullyParsed is true
 
 TEST COMPLETE

--- /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-expanded-changed-notification-expected.txt
+++ /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-expanded-changed-notification-actual.txt
@@ -7,8 +7,6 @@
 Setting aria-expanded to false on combobox2.
 Setting aria-expanded to true on menu1.
 Setting aria-expanded to false on menu2.
-AXExpandedChanged true on combobox1
-AXExpandedChanged false on combobox2
 AXExpandedChanged true on menu1
 AXExpandedChanged false on menu2
 PASS successfullyParsed is true

--- /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-pressed-changed-notification-expected.txt
+++ /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-pressed-changed-notification-actual.txt
@@ -5,8 +5,6 @@
 
 Setting aria-pressed to true on button1.
 Setting aria-pressed to false on button2.
-AXPressedStateChanged true on button1
-AXPressedStateChanged false on button2
 PASS successfullyParsed is true
 
 TEST COMPLETE

--- /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-readonly-changed-notification-expected.txt
+++ /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-readonly-changed-notification-actual.txt
@@ -5,8 +5,6 @@
 
 Setting aria-readonly to true on textbox1.
 Setting aria-readonly to false on textbox2.
-AXReadOnlyStatusChanged true on textbox1
-AXReadOnlyStatusChanged false on textbox2
 PASS successfullyParsed is true
 
 TEST COMPLETE

--- /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-required-changed-notification-expected.txt
+++ /home/slave/webkitgtk/gtk-linux-64-release-tests/build/layout-test-results/accessibility/gtk/aria-required-changed-notification-actual.txt
@@ -5,8 +5,6 @@
 
 Setting aria-required to true on textbox1.
 Setting aria-required to false on textbox2.
-AXRequiredStatusChanged true on textbox1
-AXRequiredStatusChanged false on textbox2
 PASS successfullyParsed is true
 
 TEST COMPLETE

[Radar WebKit Bug Importer]

<rdar://problem/37523716>

[Michael Catanzaro]

webkit.org/b/182761 accessibility/gtk/aria-busy-changed-notification.html [ Pass Failure ]
webkit.org/b/182761 accessibility/gtk/aria-current-changed-notification.html [ Pass Failure ]
webkit.org/b/182761 accessibility/gtk/aria-disabled-changed-notification.html [ Pass Failure ]
webkit.org/b/182761 accessibility/gtk/aria-expanded-changed-notification.html [ Pass Failure ]
webkit.org/b/182761 accessibility/gtk/aria-pressed-changed-notification.html [ Pass Failure ]
webkit.org/b/182761 accessibility/gtk/aria-readonly-changed-notification.html [ Pass Failure ]
webkit.org/b/182761 accessibility/gtk/aria-required-changed-notification.html [ Pass Failure ]

[chris fleizach]

These tests will need to be updated to account for attribute chnages being deferred. Usually that means putting the check inside a set timeout after setting the attribute.

Joanie’s expertise will Probably be needed to evaluate but it shouldn’t be too hard

[chris fleizach]

When are we going to get gtk tests in EWS?

[Michael Catanzaro]

(In reply to chris fleizach from comment #4)
> When are we going to get gtk tests in EWS?

Hopefully soon! We first need to be sure it doesn't turn red for preexisting failures as often as the Mac EWS currently does, since spurious failures on the Mac EWS are already annoying, and GTK layout tests are never in as good a state as the Mac tests. I think the current plan is to change the EWS to run failing tests several times with and without the patch to figure out if the test was already flaky or not.

[Diego Pino]

GTK post-commit bot reports these tests have been consistently failing for the last 4000 revisions. They need at least to get their baselines updated. Done in r278632.

I won't remove the tests for now. But in the future it would be worth to revisit these tests and check if their tests are consistently passing for a large number of revisions. In that case it would be safe to remove the tests from test expectations and close this bug.

[Carlos Garcia Campos]

Pull request: https://github.com/WebKit/WebKit/pull/13572

[EWS]

Committed 263855@main (4f4e045dca43): <https://commits.webkit.org/263855@main>

Reviewed commits have been landed. Closing PR #13572 and removing active labels.