Bug 214078 – REGRESSION: (r263222-263223): [ macOS wk2 debug ] accessibility/textarea-selected-text-range.html is flaky failing

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Karl Rackler]

https://results.webkit.org/?suite=layout-tests&test=accessibility%2Ftextarea-selected-text-range.html&limit=50000&platform=mac&flavor=wk2&style=debug

Description:
The test is flaky failing on [ macOS wk2 debug ] release.  It was passing consistently until 06/19/2020.

History: 
https://results.webkit.org/?suite=layout-tests&test=accessibility%2Ftextarea-selected-text-range.html&limit=50000&platform=mac&flavor=wk2&style=debug
Diff: 
--- /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/accessibility/textarea-selected-text-range-expected.txt
+++ /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/accessibility/textarea-selected-text-range-actual.txt
@@ -1,5 +1,5 @@
 {4, 0}
-{8, 2}
+{4, 0}
 {25, 0}
 PASS successfullyParsed is true

[Radar WebKit Bug Importer]

<rdar://problem/65220756>

[Karl Rackler]

Created attachment 403774 [details]
Patch

[Karl Rackler]

I have marked this is a flaky test while the issue is investigated.

[EWS]

Tools/Scripts/svn-apply failed to apply attachment 403774 [details] to trunk.
Please resolve the conflicts and upload a new patch.

[Karl Rackler]

I can reproduce this on r263223, but unable to reproduce on r263222.

Commit: r264039 

Command: run-webkit-tests --debug --iterations 500 --exit-after-n-failures 3 --debug --force -f accessibility/textarea-selected-text-range.html

Result:
[32/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[36/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[37/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
Exiting early after 3 failures. 29 tests run.


26 tests ran as expected, 3 didn't (471 didn't run):


Regressions: Unexpected text-only failures (1)
  accessibility/textarea-selected-text-range.html [ Failure ]


Commit: r263224

Command: run-webkit-tests --debug --iterations 500 --exit-after-n-failures 3 --debug --force -f accessibility/textarea-selected-text-range.html

Result: 
[47/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[52/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[53/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
Exiting early after 3 failures. 42 tests run.


39 tests ran as expected, 3 didn't (458 didn't run):


Regressions: Unexpected text-only failures (1)
  accessibility/textarea-selected-text-range.html [ Failure ]

Commit: r223222

Command: run-webkit-tests --debug --iterations 500 --exit-after-n-failures 3 --debug --force -f accessibility/textarea-selected-text-range.html

Result: 
All 500 tests ran as expected.

[Karl Rackler]

Start on Geoff because the issue looks like it may be related to the  "[Cocoa] Build callOnMainThread on WTF::RunLoop rather than on
a timer" in r263223

Commit: r263223

Command: run-webkit-tests --debug --iterations 500 --exit-after-n-failures 3 --debug --force -f accessibility/textarea-selected-text-range.html

Result: 
[47/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[52/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[53/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
Exiting early after 3 failures. 42 tests run.


39 tests ran as expected, 3 didn't (458 didn't run):


Regressions: Unexpected text-only failures (1)
  accessibility/textarea-selected-text-range.html [ Failure ]

[Jason Lawrence]

I have marked this test as flaky failing while this issue is investigated.
https://trac.webkit.org/changeset/264123/webkit

[Geoffrey Garen]

Created attachment 403802 [details]
Patch

[Darin Adler]

Comment on attachment 403802 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=403802&action=review

> LayoutTests/ChangeLog:8
> +        Make this test deterministic by waiting for the correct result.

With no time limit other than the test runner’s own limit!

[EWS]

Committed r264129: <https://trac.webkit.org/changeset/264129>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 403802 [details].