Bug 216394 - REGRESSION (r266634): fast/selectors/text-field-selection-stroke-color.html and fast/selectors/text-field-selection-window-inactive-stroke-color.html are flaky failures with pixel noise
Summary: REGRESSION (r266634): fast/selectors/text-field-selection-stroke-color.html a...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Layout and Rendering (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Sihui Liu
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-09-10 18:56 PDT by Hector Lopez
Modified: 2020-09-18 15:48 PDT (History)
8 users (show)

See Also:


Attachments
Expected Image (66.61 KB, image/png)
2020-09-10 18:58 PDT, Hector Lopez
no flags Details
Actual Image (66.61 KB, image/png)
2020-09-10 18:58 PDT, Hector Lopez
no flags Details
Diff Image (2.23 KB, image/png)
2020-09-10 18:59 PDT, Hector Lopez
no flags Details
Patch (7.61 KB, patch)
2020-09-18 10:18 PDT, Sihui Liu
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Hector Lopez 2020-09-10 18:56:57 PDT
fast/selectors/text-field-selection-stroke-color.html

Test is a flaky failure according to history on macOS wk2 Release. First occurrence of failure is at r266388 but could not determine change was cause.

History:
https://results.webkit.org/?suite=layout-tests&test=fast%2Fselectors%2Ftext-field-selection-stroke-color.html&limit=50000&platform=mac

Diff:
https://build.webkit.org/results/Apple-Catalina-Release-WK2-Tests/r266899%20(8458)/fast/selectors/text-field-selection-stroke-color-diffs.html

or see attachments

8 pixel difference
Comment 1 Hector Lopez 2020-09-10 18:58:28 PDT
Created attachment 408504 [details]
Expected Image
Comment 2 Hector Lopez 2020-09-10 18:58:59 PDT
Created attachment 408505 [details]
Actual Image
Comment 3 Hector Lopez 2020-09-10 18:59:25 PDT
Created attachment 408506 [details]
Diff Image
Comment 4 Hector Lopez 2020-09-10 19:04:27 PDT
Test expectation while investigated:

https://trac.webkit.org/changeset/266905/webkit
Comment 5 Radar WebKit Bug Importer 2020-09-10 19:04:47 PDT
<rdar://problem/68679551>
Comment 6 Alexey Proskuryakov 2020-09-13 10:27:01 PDT
There were failures with r266388 and r266392, then a string of success, then it started again with r266662. 

This makes me suspect bug 216012.

Internal bots tell a similar story, although there were some failures on macOS Mojave in early August too. Bug 216012 may not be the root cause, but it looks like it exacerbated the problem.
Comment 7 Alexey Proskuryakov 2020-09-13 12:30:53 PDT
This is very easy to reproduce, and I confirmed that this is a regression from r266634.

run-webkit-tests fast/selectors/text-field-selection-stroke-color.html --repeat 100 --child-processes=1 --no-retry

This affects more tests that this one. I only checked fast/selectors, and found these:

fast/selectors/selection-window-inactive-stroke-color.html
fast/selectors/selection-window-inactive.html
fast/selectors/text-field-selection-stroke-color.html
fast/selectors/text-field-selection-window-inactive-stroke-color.html
Comment 8 Alexey Proskuryakov 2020-09-13 12:42:12 PDT
That test was with r266634 archive. Follow-ups made a difference, it's only two tests on trunk:

fast/selectors/text-field-selection-stroke-color.html
fast/selectors/text-field-selection-window-inactive-stroke-color.html
Comment 9 Hector Lopez 2020-09-14 08:30:36 PDT
Correction for test expectation:

https://trac.webkit.org/changeset/267012/webkit
Comment 10 Sihui Liu 2020-09-14 11:50:33 PDT
(In reply to Alexey Proskuryakov from comment #8)
> That test was with r266634 archive. Follow-ups made a difference, it's only
> two tests on trunk:
> 
> fast/selectors/text-field-selection-stroke-color.html
> fast/selectors/text-field-selection-window-inactive-stroke-color.html

What config/revision did you use for reproducing the failure?
I used your command on both release and debug build with r267017 and cannot reproduce: 
run-webkit-tests fast/selectors/text-field-selection-stroke-color.html --repeat 100 --child-processes=1 --no-retry

Are you on macOS Catalina?
Comment 11 Sihui Liu 2020-09-14 11:55:49 PDT
(In reply to Sihui Liu from comment #10)
> (In reply to Alexey Proskuryakov from comment #8)
> > That test was with r266634 archive. Follow-ups made a difference, it's only
> > two tests on trunk:
> > 
> > fast/selectors/text-field-selection-stroke-color.html
> > fast/selectors/text-field-selection-window-inactive-stroke-color.html
> 
> What config/revision did you use for reproducing the failure?
> I used your command on both release and debug build with r267017 and cannot
> reproduce: 
> run-webkit-tests fast/selectors/text-field-selection-stroke-color.html
> --repeat 100 --child-processes=1 --no-retry
> 
> Are you on macOS Catalina?

Never mind. I didn't update the test expectations before running. Now I can reproduce.
Comment 12 Hector Lopez 2020-09-16 19:24:03 PDT
fast/selectors/selection-window-inactive-stroke-color.html

Test exhibits the same behavior. Adding to bug and changing test expectations

test expectation for fast/selectors/selection-window-inactive-stroke-color.html:

https://trac.webkit.org/changeset/267178/webkit
Comment 13 Sihui Liu 2020-09-18 10:18:48 PDT
Created attachment 409141 [details]
Patch
Comment 14 Alexey Proskuryakov 2020-09-18 12:54:19 PDT
Impressive analysis!

Makes me wonder if it would make sense for layout tests to dump tile size, so that we could catch such issues faster and more directly than through pixel noise.
Comment 15 EWS 2020-09-18 15:25:49 PDT
Committed r267284: <https://trac.webkit.org/changeset/267284>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 409141 [details].
Comment 16 Sihui Liu 2020-09-18 15:48:39 PDT
(In reply to Alexey Proskuryakov from comment #14)
> Impressive analysis!
> 
> Makes me wonder if it would make sense for layout tests to dump tile size,
> so that we could catch such issues faster and more directly than through
> pixel noise.

Yes, that may be useful! This kind of small flaky pixel difference could be hard to reason about. If we know the noise comes with with different tiling, at least we have some hint about what may go wrong. (It took me quite a while before noticing more PlatformCALayers were created when test failed :|)