Bug 237372 - [iOS] editing/async-clipboard/* 20 tests are flaky timeouts on iOS
Summary: [iOS] editing/async-clipboard/* 20 tests are flaky timeouts on iOS
Status: CLOSED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks: 237559
  Show dependency treegraph
 
Reported: 2022-03-02 08:38 PST by Dawn Morningstar
Modified: 2022-03-15 10:38 PDT (History)
7 users (show)

See Also:


Attachments
Results page (2.49 MB, image/png)
2022-03-02 08:52 PST, Wenson Hsieh
no flags Details
expectations (3.06 KB, patch)
2022-03-02 11:22 PST, Dawn Morningstar
no flags Details | Formatted Diff | Diff
Apple-iOS-15-Simulator-Debug-WK2-Tests (405.37 KB, image/png)
2022-03-03 09:51 PST, Wenson Hsieh
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dawn Morningstar 2022-03-02 08:38:16 PST
A massive amount of editing/async-clipboard/ tests are flaky timing out on iOS queues:

editing/async-clipboard/clipboard-change-data-while-getting-type.html 
editing/async-clipboard/clipboard-change-data-while-reading.html 
editing/async-clipboard/clipboard-change-data-while-writing.html 
editing/async-clipboard/clipboard-do-not-read-text-from-platform-if-text-changes.html 
editing/async-clipboard/clipboard-get-type-with-old-items.html 
editing/async-clipboard/clipboard-interfaces.html
editing/async-clipboard/clipboard-item-get-type-basic.html
editing/async-clipboard/clipboard-read-basic.html
editing/async-clipboard/clipboard-read-text-from-platform.html
editing/async-clipboard/clipboard-read-text-same-origin.html
editing/async-clipboard/clipboard-read-text.html
editing/async-clipboard/clipboard-read-while-pasting.html
editing/async-clipboard/clipboard-read-write-images.html
editing/async-clipboard/clipboard-write-basic.html
editing/async-clipboard/clipboard-write-in-copy-event-handler-in-subframe.html
editing/async-clipboard/clipboard-write-in-copy-event-handler.html
editing/async-clipboard/clipboard-write-items-twice.html
editing/async-clipboard/clipboard-write-text.html
editing/async-clipboard/sanitize-when-reading-markup.html
editing/async-clipboard/sanitize-when-writing-image.html

HISTORY:
https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=editing%2Fasync-clipboard%2Fclipboard-change-data-while-getting-type.html&test=editing%2Fasync-clipboard%2Fclipboard-change-data-while-reading.html&test=editing%2Fasync-clipboard%2Fclipboard-change-data-while-writing.html&test=editing%2Fasync-clipboard%2Fclipboard-do-not-read-text-from-platform-if-text-changes.html&test=editing%2Fasync-clipboard%2Fclipboard-get-type-with-old-items.html&test=editing%2Fasync-clipboard%2Fclipboard-interfaces.html&test=editing%2Fasync-clipboard%2Fclipboard-item-get-type-basic.html&test=editing%2Fasync-clipboard%2Fclipboard-read-basic.html&test=editing%2Fasync-clipboard%2Fclipboard-read-text-from-platform.html&test=editing%2Fasync-clipboard%2Fclipboard-read-text-same-origin.html&test=editing%2Fasync-clipboard%2Fclipboard-read-text.html&test=editing%2Fasync-clipboard%2Fclipboard-read-while-pasting.html&test=editing%2Fasync-clipboard%2Fclipboard-read-write-images.html&test=editing%2Fasync-clipboard%2Fclipboard-write-basic.html&test=editing%2Fasync-clipboard%2Fclipboard-write-in-copy-event-handler-in-subframe.html&test=editing%2Fasync-clipboard%2Fclipboard-write-in-copy-event-handler.html&test=editing%2Fasync-clipboard%2Fclipboard-write-items-twice.html&test=editing%2Fasync-clipboard%2Fclipboard-write-text.html&test=editing%2Fasync-clipboard%2Fsanitize-when-reading-markup.html&test=editing%2Fasync-clipboard%2Fsanitize-when-writing-image.html
Comment 1 Radar WebKit Bug Importer 2022-03-02 08:38:48 PST
<rdar://problem/89689914>
Comment 2 Wenson Hsieh 2022-03-02 08:52:23 PST
Created attachment 453618 [details]
Results page

It looks like the majority of these failures happen in tandem, which would suggest that this happens when either the simulator or test harness gets into a bad state.

It also looks like these failures really started happening a couple of weeks ago, around 2/12..
Comment 3 Dawn Morningstar 2022-03-02 09:02:21 PST
(In reply to Wenson Hsieh from comment #2)
> Created attachment 453618 [details]
> Results page
> 
> It looks like the majority of these failures happen in tandem, which would
> suggest that this happens when either the simulator or test harness gets
> into a bad state.
> 
> It also looks like these failures really started happening a couple of weeks
> ago, around 2/12..

I am seeing the exact same thing.

Regression appears to be around r289700. But could be as you said due to the simulator being in a bad state.
Comment 4 Wenson Hsieh 2022-03-02 09:06:20 PST
(In reply to Matteo Flores from comment #3)
> (In reply to Wenson Hsieh from comment #2)
> > Created attachment 453618 [details]
> > Results page
> > 
> > It looks like the majority of these failures happen in tandem, which would
> > suggest that this happens when either the simulator or test harness gets
> > into a bad state.
> > 
> > It also looks like these failures really started happening a couple of weeks
> > ago, around 2/12..
> 
> I am seeing the exact same thing.
> 
> Regression appears to be around r289700. But could be as you said due to the
> simulator being in a bad state.

No changes near r289700 seem like they could feasibly cause these failures.

Did any major configuration changes to WebKit iOS simulator bots land around that time?
Comment 5 Dawn Morningstar 2022-03-02 09:16:35 PST
(In reply to Wenson Hsieh from comment #2)
> Created attachment 453618 [details]
> Results page
> 
> It looks like the majority of these failures happen in tandem, which would
> suggest that this happens when either the simulator or test harness gets
> into a bad state.
> 
> It also looks like these failures really started happening a couple of weeks
> ago, around 2/12..

I am seeing the exact same thing.

Regression appears to be around r289
Comment 6 Dawn Morningstar 2022-03-02 09:18:09 PST
(In reply to Matteo Flores from comment #5)
> (In reply to Wenson Hsieh from comment #2)
> > Created attachment 453618 [details]
> > Results page
> > 
> > It looks like the majority of these failures happen in tandem, which would
> > suggest that this happens when either the simulator or test harness gets
> > into a bad state.
> > 
> > It also looks like these failures really started happening a couple of weeks
> > ago, around 2/12..
> 
> I am seeing the exact same thing.
> 
> Regression appears to be around r289

Disregard, duplicate message from page refresh...

To answer your question, I have reached out to the team about possible configuration changes in that timeframe.
Comment 7 Dawn Morningstar 2022-03-02 11:05:05 PST
Marking expectations until issue is resolved.
Comment 8 Dawn Morningstar 2022-03-02 11:22:54 PST
Created attachment 453641 [details]
expectations
Comment 9 Robert Jenner 2022-03-02 13:00:04 PST
Comment on attachment 453641 [details]
expectations

Clearing flags on attachment: 453641

Committed r290750 (?): <https://commits.webkit.org/r290750>
Comment 10 Wenson Hsieh 2022-03-03 09:10:47 PST
(In reply to Matteo Flores from comment #6)
> Disregard, duplicate message from page refresh...
> 
> To answer your question, I have reached out to the team about possible
> configuration changes in that timeframe.

I'm curious if you have any new info!
Comment 11 Wenson Hsieh 2022-03-03 09:51:08 PST
Created attachment 453753 [details]
Apple-iOS-15-Simulator-Debug-WK2-Tests

From examining recent runs in the Apple-iOS-15-Simulator-Debug-WK2-Tests queue, it looks like bot653 is uniquely encountering these mass pasteboard-related timeouts, and not bot654:

bot653: https://build.webkit.org/#/workers/27
bot654: https://build.webkit.org/#/workers/75

I think there's a similar pattern on the other iOS sim testing queues.

Is there any configuration difference between 653 and 654 that might help explain this? (perhaps a difference in SDK version, for instance?)

For internal folks, this seems somewhat reminiscent of rdar://86327357, wherein the iTunes Store TCC prompt causes widespread layout test failures because the keyboard and context menus fail to appear.
Comment 12 Dawn Morningstar 2022-03-04 20:01:37 PST
Listing more layout tests that seem to suffer from this bug as well.

editing/deleting/smart-delete-paragraph-001.html
editing/deleting/smart-delete-paragraph-002.html
editing/deleting/smart-delete-paragraph-004.html
editing/deleting/smart-delete-paragraph-005.html
editing/execCommand/insert-newline-in-quoted-content-crash.html
editing/execCommand/outdent-cut-crash.html
editing/execCommand/paste-as-quotation-disconnected-paragraph-ancestor-crash.html
editing/execCommand/selectAll-copy-crash.html
editing/pasteboard/19644-1.html
editing/pasteboard/19644-2.html
editing/pasteboard/3976872.html
editing/pasteboard/4076267-2.html
editing/pasteboard/4076267-3.html 
editing/pasteboard/4076267.html 
editing/pasteboard/4242293-1.html 
editing/pasteboard/4242293.html 
editing/pasteboard/4641033.html 
editing/pasteboard/4922709.html 
editing/pasteboard/4930986-1-paste-as-quotation.html 
editing/pasteboard/4930986-2-paste-as-quotation.html 
editing/pasteboard/4930986-3-paste-as-quotation.html 
editing/pasteboard/4944770-1.html
editing/pasteboard/4944770-2.html
editing/pasteboard/4989774.html 
editing/pasteboard/5006779.html
editing/pasteboard/5028447.html
editing/pasteboard/5065605.html
editing/pasteboard/5761530-1.html
editing/spelling/editing-word-with-marker-1.html
editing/style/smoosh-styles-003.html
editing/undo-manager/undo-manager-item-labels.html 

And these have a slightly different "regression" point but also appear related:

editing/pasteboard/dom-paste/dom-paste-same-origin.html
fast/css/border-radius-invalid-pair.html
fast/events/before-input-prevent-paste.html 
fast/forms/ios/inputmode-removing-none.html 
fast/forms/plaintext-mode-2.html
fast/images/image-copy-memory-usage.html
fast/images/text-recognition/image-overlay-line-wrapping.html
fast/table/table-split-on-insert-with-section-crash.html
http/tests/security/clipboard/copy-paste-html-across-origin-sanitizes-html.html
http/tests/security/clipboard/copy-paste-html-across-origin-strips-mso-list.html
http/tests/security/clipboard/copy-paste-html-cross-origin-iframe-across-origin.html
http/tests/security/clipboard/copy-paste-html-cross-origin-iframe-in-same-origin.html 
http/tests/security/clipboard/copy-paste-url-across-origin-sanitizes-url.html 
fast/editing/insert-paragraph-with-text-in-table-crash.html
fast/editing/mutation-with-display-contents-container-crash.html
fast/editing/paste-and-indent-microtask-crash.html
fast/editing/paste-and-match-style-with-table-2-crash.html 
fast/events/before-input-prevent-cut.html
fast/events/before-input-prevent-paste.html
fast/events/input-events-paste-data.html 
fast/events/input-events-paste-rich-datatransfer.html 
editing/selection/ios/avoid-showing-callout-menu-over-controls.html 
editing/selection/ios/dispatch-mouse-events-when-modifying-selection-quirk.html 
editing/selection/ios/scrolling-with-fixed-selection-does-not-unselect-native-text-field.html 
editing/selection/ios/selection-after-changing-text-with-callout-menu.html 
editing/selection/ios/show-callout-bar-after-selecting-word.html 
editing/selection/ios/show-callout-bar-after-tap.html 
editing/selection/ios/update-selection-after-iframe-scroll.html
editing/pasteboard/paste-without-nesting.html 
editing/selection/ios/update-selection-after-overflow-scroll.html 
editing/selection/programmatic-selection-on-mac-is-directionless.html
editing/selection/replace-selection-crash-02.html 
editing/selection/select-crash-001.html
editing/selection/select-crash-002.html
editing/selection/setSelection-shadow-dom-crash.html

Also slightly different regression point but appear related:

editing/pasteboard/contenteditable-pre.html
editing/pasteboard/copy-backslash-with-euc.html
editing/pasteboard/copy-crash-with-extraneous-attribute.html
editing/pasteboard/copy-cut-paste-events-fired-when-disabled.html
editing/pasteboard/copy-display-none.html
editing/pasteboard/copy-element-with-conflicting-background-color-from-rule.html 
editing/pasteboard/copy-image-with-alt-text.html
editing/pasteboard/copy-in-password-field.html
editing/pasteboard/copy-inside-h1-preserves-h1.html
editing/pasteboard/copy-null-characters.html
editing/pasteboard/copy-paste-across-shadow-boundaries-1.html
editing/pasteboard/copy-paste-across-shadow-boundaries-2.html
editing/pasteboard/copy-paste-across-shadow-boundaries-3.html
editing/pasteboard/copy-paste-across-shadow-boundaries-4.html 
editing/pasteboard/copy-paste-across-shadow-boundaries-5.html
Comment 13 Wenson Hsieh 2022-03-05 11:06:58 PST
(In reply to Matteo Flores from comment #12)
> Listing more layout tests that seem to suffer from this bug as well.
> 
> editing/deleting/smart-delete-paragraph-001.html
> editing/deleting/smart-delete-paragraph-002.html
> editing/deleting/smart-delete-paragraph-004.html
> editing/deleting/smart-delete-paragraph-005.html
> editing/execCommand/insert-newline-in-quoted-content-crash.html
> editing/execCommand/outdent-cut-crash.html
> editing/execCommand/paste-as-quotation-disconnected-paragraph-ancestor-crash.
> html
> editing/execCommand/selectAll-copy-crash.html
> editing/pasteboard/19644-1.html
> editing/pasteboard/19644-2.html
> editing/pasteboard/3976872.html
> editing/pasteboard/4076267-2.html
> editing/pasteboard/4076267-3.html 
> editing/pasteboard/4076267.html 
> editing/pasteboard/4242293-1.html 
> editing/pasteboard/4242293.html 
> editing/pasteboard/4641033.html 
> editing/pasteboard/4922709.html 
> editing/pasteboard/4930986-1-paste-as-quotation.html 
> editing/pasteboard/4930986-2-paste-as-quotation.html 
> editing/pasteboard/4930986-3-paste-as-quotation.html 
> editing/pasteboard/4944770-1.html
> editing/pasteboard/4944770-2.html
> editing/pasteboard/4989774.html 
> editing/pasteboard/5006779.html
> editing/pasteboard/5028447.html
> editing/pasteboard/5065605.html
> editing/pasteboard/5761530-1.html
> editing/spelling/editing-word-with-marker-1.html
> editing/style/smoosh-styles-003.html
> editing/undo-manager/undo-manager-item-labels.html 
> 
> And these have a slightly different "regression" point but also appear
> related:
> 
> editing/pasteboard/dom-paste/dom-paste-same-origin.html
> fast/css/border-radius-invalid-pair.html
> fast/events/before-input-prevent-paste.html 
> fast/forms/ios/inputmode-removing-none.html 
> fast/forms/plaintext-mode-2.html
> fast/images/image-copy-memory-usage.html
> fast/images/text-recognition/image-overlay-line-wrapping.html
> fast/table/table-split-on-insert-with-section-crash.html
> http/tests/security/clipboard/copy-paste-html-across-origin-sanitizes-html.
> html
> http/tests/security/clipboard/copy-paste-html-across-origin-strips-mso-list.
> html
> http/tests/security/clipboard/copy-paste-html-cross-origin-iframe-across-
> origin.html
> http/tests/security/clipboard/copy-paste-html-cross-origin-iframe-in-same-
> origin.html 
> http/tests/security/clipboard/copy-paste-url-across-origin-sanitizes-url.
> html 
> fast/editing/insert-paragraph-with-text-in-table-crash.html
> fast/editing/mutation-with-display-contents-container-crash.html
> fast/editing/paste-and-indent-microtask-crash.html
> fast/editing/paste-and-match-style-with-table-2-crash.html 
> fast/events/before-input-prevent-cut.html
> fast/events/before-input-prevent-paste.html
> fast/events/input-events-paste-data.html 
> fast/events/input-events-paste-rich-datatransfer.html 
> editing/selection/ios/avoid-showing-callout-menu-over-controls.html 
> editing/selection/ios/dispatch-mouse-events-when-modifying-selection-quirk.
> html 
> editing/selection/ios/scrolling-with-fixed-selection-does-not-unselect-
> native-text-field.html 
> editing/selection/ios/selection-after-changing-text-with-callout-menu.html 
> editing/selection/ios/show-callout-bar-after-selecting-word.html 
> editing/selection/ios/show-callout-bar-after-tap.html 
> editing/selection/ios/update-selection-after-iframe-scroll.html
> editing/pasteboard/paste-without-nesting.html 
> editing/selection/ios/update-selection-after-overflow-scroll.html 
> editing/selection/programmatic-selection-on-mac-is-directionless.html
> editing/selection/replace-selection-crash-02.html 
> editing/selection/select-crash-001.html
> editing/selection/select-crash-002.html
> editing/selection/setSelection-shadow-dom-crash.html
> 
> Also slightly different regression point but appear related:
> 
> editing/pasteboard/contenteditable-pre.html
> editing/pasteboard/copy-backslash-with-euc.html
> editing/pasteboard/copy-crash-with-extraneous-attribute.html
> editing/pasteboard/copy-cut-paste-events-fired-when-disabled.html
> editing/pasteboard/copy-display-none.html
> editing/pasteboard/copy-element-with-conflicting-background-color-from-rule.
> html 
> editing/pasteboard/copy-image-with-alt-text.html
> editing/pasteboard/copy-in-password-field.html
> editing/pasteboard/copy-inside-h1-preserves-h1.html
> editing/pasteboard/copy-null-characters.html
> editing/pasteboard/copy-paste-across-shadow-boundaries-1.html
> editing/pasteboard/copy-paste-across-shadow-boundaries-2.html
> editing/pasteboard/copy-paste-across-shadow-boundaries-3.html
> editing/pasteboard/copy-paste-across-shadow-boundaries-4.html 
> editing/pasteboard/copy-paste-across-shadow-boundaries-5.html

We know that this is happening because the PasteboardAutomaticSync default either isn't being set properly when booting the simulator, or is being flipped on during the course of the layout test run.

Writing this default should work around the problem on affected bots:
```
defaults write com.apple.iphonesimulator PasteboardAutomaticSync -bool NO
```
Comment 14 Wenson Hsieh 2022-03-05 11:11:12 PST
I think we should revert the expectations previously set in r290750, and work around these failures for the time being by manually writing the default on all affected test runners. There are likely too many affected tests to set expectations for across the board.

To actually address this, we'll need to either figure out what is causing this default to be set after booting the simulator, or add a way to enforce this default to be set to NO when running layout tests.
Comment 15 Alexey Proskuryakov 2022-03-05 19:13:41 PST
It's passed on command line, so setting it in on-disk defaults seems unlikely to change the behavior.

Tools/Scripts/webkitpy/xcode/simulated_device.py:            SimulatedDeviceManager._managing_simulator_app = not host.executive.run_command(['open', '-g', '-b', SimulatedDeviceManager.simulator_bundle_id, '--args', '-PasteboardAutomaticSync', '0'], return_exit_code=True)
Comment 16 Wenson Hsieh 2022-03-05 19:16:29 PST
(In reply to Alexey Proskuryakov from comment #15)
> It's passed on command line, so setting it in on-disk defaults seems
> unlikely to change the behavior.
> 
> Tools/Scripts/webkitpy/xcode/simulated_device.py:           
> SimulatedDeviceManager._managing_simulator_app = not
> host.executive.run_command(['open', '-g', '-b',
> SimulatedDeviceManager.simulator_bundle_id, '--args',
> '-PasteboardAutomaticSync', '0'], return_exit_code=True)

My understanding was that `--args PasteboardAutomaticSync 0` would only affect the `Managed X` simulators that the test runner boots up. For some reason, this seems to (sometimes) be ineffective in overriding the default PasteboardAutomaticSync behavior.

Perhaps we might see different results by simply writing the default globally.
Comment 17 Alexey Proskuryakov 2022-03-05 19:25:50 PST
Seems like an easy enough check to try it out, to isolate possibilities.
Comment 18 Wenson Hsieh 2022-03-05 19:30:22 PST
(In reply to Alexey Proskuryakov from comment #17)
> Seems like an easy enough check to try it out, to isolate possibilities.

Indeed! (And even if it doesn't work, that will be an interesting datapoint in this investigation).
Comment 19 Wenson Hsieh 2022-03-05 19:55:20 PST
(In reply to Wenson Hsieh from comment #18)
> (In reply to Alexey Proskuryakov from comment #17)
> > Seems like an easy enough check to try it out, to isolate possibilities.
> 
> Indeed! (And even if it doesn't work, that will be an interesting datapoint
> in this investigation).

I ssh'd into bot651 and bot664 (both of which are consistently hitting these timeouts), and wrote the default. Let's see if it makes any difference...

<https://build.webkit.org/#/workers/98>
<https://build.webkit.org/#/workers/11>
Comment 20 Jonathan Bedard 2022-03-07 11:03:27 PST
Pull request: https://github.com/WebKit/WebKit/pull/149
Comment 21 Jonathan Bedard 2022-03-07 11:04:52 PST
Pull request: https://github.com/WebKit/WebKit/pull/191
Comment 22 Jonathan Bedard 2022-03-07 11:07:06 PST
(In reply to Jonathan Bedard from comment #20)
> Pull request: https://github.com/WebKit/WebKit/pull/149

Wrong PR, mistakingly proposed this on an existing branch. https://github.com/WebKit/WebKit/pull/191 is the right PR.
Comment 23 Jonathan Bedard 2022-03-07 13:17:33 PST
Landed https://commits.webkit.org/248136@main (r290905)!
Comment 24 Dawn Morningstar 2022-03-09 14:05:54 PST
Reverted r290750 for reason:

Bug has been fixed, expectations are no longer needed.

Committed r291066 (248238@trunk): <https://commits.webkit.org/248238@trunk>