Bug 222988

Summary: [ BigSur wk2 arm64 ] quite a few fast/forms (Layout-Tests) are text failing on Apple Silicon
Product: WebKit Reporter: Robert Jenner <jenner>
Component: New BugsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: akeerthi, ap, cdumez, pvollan, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=222619
https://bugs.webkit.org/show_bug.cgi?id=222825
Attachments:
Description Flags
File for all of the Text Diffs none

Description Robert Jenner 2021-03-09 12:40:39 PST
The tests below appear to be frequently text failing on BigSur wk2 Apple Silicon Release and Debug:

fast/forms/date/date-editable-components/date-editable-components-keyboard-events.html
fast/forms/date/date-editable-components/date-editable-components-mouse-events.html
fast/forms/date/date-editable-components/date-picker-update-on-edit.html
fast/forms/date/date-input-rendering-basic.html
fast/forms/date/date-pseudo-elements.html
fast/forms/datetimelocal/datetimelocal-editable-components/datetimelocal-editable-components-focus-and-blur-events.html
fast/forms/datetimelocal/datetimelocal-editable-components/datetimelocal-editable-components-keyboard-events.html
fast/forms/datetimelocal/datetimelocal-editable-components/datetimelocal-editable-components-mouse-events.html
fast/forms/datetimelocal/datetimelocal-editable-components/datetimelocal-editable-components-second-and-millisecond-field.html
fast/forms/datetimelocal/datetimelocal-editable-components/datetimelocal-picker-update-on-edit.html
fast/forms/time-input-svg-font.html
fast/forms/time/time-editable-components/time-editable-components-focus-and-blur-events.html
fast/forms/time/time-editable-components/time-editable-components-keyboard-events.html
fast/forms/time/time-editable-components/time-editable-components-mouse-events.html
fast/forms/time/time-editable-components/time-editable-components-second-and-millisecond-field.html
fast/forms/time/time-input-rendering-basic.html
fast/text/hyphenate-locale.html

HISTORY URL:
https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=fast%2Fforms%2Fdate%2Fdate-editable-components%2Fdate-editable-components-keyboard-events.html&test=fast%2Fforms%2Fdate%2Fdate-editable-components%2Fdate-editable-components-mouse-events.html&test=fast%2Fforms%2Fdate%2Fdate-editable-components%2Fdate-picker-update-on-edit.html&test=fast%2Fforms%2Fdate%2Fdate-input-rendering-basic.html&test=fast%2Fforms%2Fdate%2Fdate-pseudo-elements.html&test=fast%2Fforms%2Fdatetimelocal%2Fdatetimelocal-editable-components%2Fdatetimelocal-editable-components-focus-and-blur-events.html&test=fast%2Fforms%2Fdatetimelocal%2Fdatetimelocal-editable-components%2Fdatetimelocal-editable-components-keyboard-events.html&test=fast%2Fforms%2Fdatetimelocal%2Fdatetimelocal-editable-components%2Fdatetimelocal-editable-components-mouse-events.html&test=fast%2Fforms%2Fdatetimelocal%2Fdatetimelocal-editable-components%2Fdatetimelocal-editable-components-second-and-millisecond-field.html&test=fast%2Fforms%2Fdatetimelocal%2Fdatetimelocal-editable-components%2Fdatetimelocal-picker-update-on-edit.html&test=fast%2Fforms%2Ftime-input-svg-font.html&test=fast%2Fforms%2Ftime%2Ftime-editable-components%2Ftime-editable-components-focus-and-blur-events.html&test=fast%2Fforms%2Ftime%2Ftime-editable-components%2Ftime-editable-components-keyboard-events.html&test=fast%2Fforms%2Ftime%2Ftime-editable-components%2Ftime-editable-components-mouse-events.html&test=fast%2Fforms%2Ftime%2Ftime-editable-components%2Ftime-editable-components-second-and-millisecond-field.html&test=fast%2Fforms%2Ftime%2Ftime-input-rendering-basic.html&test=fast%2Ftext%2Fhyphenate-locale.html&architecture=arm64


All of tests have the same failure history. They passed the first few times since the test was introduced for Apple Silicon, but then started failing consistently.
Comment 1 Radar WebKit Bug Importer 2021-03-09 12:41:03 PST
<rdar://problem/75230416>
Comment 2 Robert Jenner 2021-03-09 12:43:06 PST
I do not have access to an Apple Silicon Mac, so I cannot reproduce the failures. 

However, I will say that they are the same test family, and they share the same failure history. These tests may just need an adjustment to run on Apple Silicon.
Comment 3 Robert Jenner 2021-03-09 13:12:22 PST
Created attachment 422750 [details]
File for all of the Text Diffs

These are all of the text DIFF files for all 17 of the failures. I put them in a folder, and also broke them down by the type of failure they were having. There are four categories of text failures these tests are having:

blur/FocusEventsFired
input.value
numberOfFields
RenderText
Comment 4 Aditya Keerthi 2021-03-09 13:39:56 PST
Based on the diffs attached it appears as though these tests are failing due to a different language/locale on the Apple Silicon bots.

The system language affects the order of fields for the date/time tests, leading to different values when editing the input.

Note that fast/text/hyphenate-locale.html doesn't belong to the same family as the other tests (date/time inputs), but is a test that is affected by locale.

The easiest way to fix this would be to update the system language/locale on the bots. A better solution would be to make these tests independent of the system language/locale. I will see if it's possible to rewrite these tests.
Comment 5 Ryan Haddad 2021-03-09 14:05:19 PST
I did find that a handful of these machines were configured with en-GB as the language, though I'm not sure how they ended up in this state.

I have now ensured that they are all configured as en-US / en_US, so lets continue to monitor the results to see what happens.
Comment 6 Ryan Haddad 2021-03-09 20:34:26 PST
The failures disappeared on the next run after I reset the language: https://build.webkit.org/results/Apple-BigSur-Release-AppleSilicon-WK2-Tests/r274171%20(114)/results.html

However, 8/10 of these bots are back on en-GB again, so the theory that this is due to a test changing the language and not returning it to the original one is back on the table

Presumably, https://bugs.webkit.org/show_bug.cgi?id=222619 would be a culprit.
Comment 7 Chris Dumez 2021-03-09 20:51:45 PST
Committed r274195 (235113@main): <https://commits.webkit.org/235113@main>
Comment 8 Chris Dumez 2021-03-09 20:53:12 PST
Please reset the language on these bots to en-US one more time. My patch should make sure it does not get modified to en-GB again.
Comment 9 Alexey Proskuryakov 2021-03-10 10:16:08 PST
Is there a root cause with potentially user facing symptoms to be worried about?
Comment 10 Chris Dumez 2021-03-10 10:23:37 PST
(In reply to Alexey Proskuryakov from comment #9)
> Is there a root cause with potentially user facing symptoms to be worried
> about?

Tracked by Bug 222825. It seems the languagechange event may not be firing on Big Sur / Apple Silicon (even though it does on Intel). This is likely not a regression, just a new find because of the newly added API test.