237372 – [iOS] editing/async-clipboard/* 20 tests are flaky timeouts on iOS

Dawn Morningstar

Reported 2022-03-02 08:38:16 PST

Radar WebKit Bug Importer

Comment 1 2022-03-02 08:38:48 PST

<rdar://problem/89689914>

Wenson Hsieh

Comment 2 2022-03-02 08:52:23 PST

Created attachment 453618 [details] Results page It looks like the majority of these failures happen in tandem, which would suggest that this happens when either the simulator or test harness gets into a bad state. It also looks like these failures really started happening a couple of weeks ago, around 2/12..

Dawn Morningstar

Comment 3 2022-03-02 09:02:21 PST

(In reply to Wenson Hsieh from comment #2) > Created attachment 453618 [details] > Results page > > It looks like the majority of these failures happen in tandem, which would > suggest that this happens when either the simulator or test harness gets > into a bad state. > > It also looks like these failures really started happening a couple of weeks > ago, around 2/12.. I am seeing the exact same thing. Regression appears to be around r289700. But could be as you said due to the simulator being in a bad state.

Wenson Hsieh

Comment 4 2022-03-02 09:06:20 PST

(In reply to Matteo Flores from comment #3) > (In reply to Wenson Hsieh from comment #2) > > Created attachment 453618 [details] > > Results page > > > > It looks like the majority of these failures happen in tandem, which would > > suggest that this happens when either the simulator or test harness gets > > into a bad state. > > > > It also looks like these failures really started happening a couple of weeks > > ago, around 2/12.. > > I am seeing the exact same thing. > > Regression appears to be around r289700. But could be as you said due to the > simulator being in a bad state. No changes near r289700 seem like they could feasibly cause these failures. Did any major configuration changes to WebKit iOS simulator bots land around that time?

Dawn Morningstar

Comment 5 2022-03-02 09:16:35 PST

(In reply to Wenson Hsieh from comment #2) > Created attachment 453618 [details] > Results page > > It looks like the majority of these failures happen in tandem, which would > suggest that this happens when either the simulator or test harness gets > into a bad state. > > It also looks like these failures really started happening a couple of weeks > ago, around 2/12.. I am seeing the exact same thing. Regression appears to be around r289

Dawn Morningstar

Comment 6 2022-03-02 09:18:09 PST

(In reply to Matteo Flores from comment #5) > (In reply to Wenson Hsieh from comment #2) > > Created attachment 453618 [details] > > Results page > > > > It looks like the majority of these failures happen in tandem, which would > > suggest that this happens when either the simulator or test harness gets > > into a bad state. > > > > It also looks like these failures really started happening a couple of weeks > > ago, around 2/12.. > > I am seeing the exact same thing. > > Regression appears to be around r289 Disregard, duplicate message from page refresh... To answer your question, I have reached out to the team about possible configuration changes in that timeframe.

Dawn Morningstar

Comment 7 2022-03-02 11:05:05 PST

Marking expectations until issue is resolved.

Dawn Morningstar

Comment 8 2022-03-02 11:22:54 PST

Created attachment 453641 [details] expectations

Robert Jenner

Comment 9 2022-03-02 13:00:04 PST

Comment on attachment 453641 [details] expectations Clearing flags on attachment: 453641 Committed r290750 (?): <https://commits.webkit.org/r290750>

Wenson Hsieh

Comment 10 2022-03-03 09:10:47 PST

(In reply to Matteo Flores from comment #6) > Disregard, duplicate message from page refresh... > > To answer your question, I have reached out to the team about possible > configuration changes in that timeframe. I'm curious if you have any new info!

Wenson Hsieh

Comment 11 2022-03-03 09:51:08 PST

Created attachment 453753 [details] Apple-iOS-15-Simulator-Debug-WK2-Tests From examining recent runs in the Apple-iOS-15-Simulator-Debug-WK2-Tests queue, it looks like bot653 is uniquely encountering these mass pasteboard-related timeouts, and not bot654: bot653: https://build.webkit.org/#/workers/27 bot654: https://build.webkit.org/#/workers/75 I think there's a similar pattern on the other iOS sim testing queues. Is there any configuration difference between 653 and 654 that might help explain this? (perhaps a difference in SDK version, for instance?) For internal folks, this seems somewhat reminiscent of rdar://86327357, wherein the iTunes Store TCC prompt causes widespread layout test failures because the keyboard and context menus fail to appear.

Dawn Morningstar

Comment 12 2022-03-04 20:01:37 PST

Listing more layout tests that seem to suffer from this bug as well. editing/deleting/smart-delete-paragraph-001.html editing/deleting/smart-delete-paragraph-002.html editing/deleting/smart-delete-paragraph-004.html editing/deleting/smart-delete-paragraph-005.html editing/execCommand/insert-newline-in-quoted-content-crash.html editing/execCommand/outdent-cut-crash.html editing/execCommand/paste-as-quotation-disconnected-paragraph-ancestor-crash.html editing/execCommand/selectAll-copy-crash.html editing/pasteboard/19644-1.html editing/pasteboard/19644-2.html editing/pasteboard/3976872.html editing/pasteboard/4076267-2.html editing/pasteboard/4076267-3.html editing/pasteboard/4076267.html editing/pasteboard/4242293-1.html editing/pasteboard/4242293.html editing/pasteboard/4641033.html editing/pasteboard/4922709.html editing/pasteboard/4930986-1-paste-as-quotation.html editing/pasteboard/4930986-2-paste-as-quotation.html editing/pasteboard/4930986-3-paste-as-quotation.html editing/pasteboard/4944770-1.html editing/pasteboard/4944770-2.html editing/pasteboard/4989774.html editing/pasteboard/5006779.html editing/pasteboard/5028447.html editing/pasteboard/5065605.html editing/pasteboard/5761530-1.html editing/spelling/editing-word-with-marker-1.html editing/style/smoosh-styles-003.html editing/undo-manager/undo-manager-item-labels.html And these have a slightly different "regression" point but also appear related: editing/pasteboard/dom-paste/dom-paste-same-origin.html fast/css/border-radius-invalid-pair.html fast/events/before-input-prevent-paste.html fast/forms/ios/inputmode-removing-none.html fast/forms/plaintext-mode-2.html fast/images/image-copy-memory-usage.html fast/images/text-recognition/image-overlay-line-wrapping.html fast/table/table-split-on-insert-with-section-crash.html http/tests/security/clipboard/copy-paste-html-across-origin-sanitizes-html.html http/tests/security/clipboard/copy-paste-html-across-origin-strips-mso-list.html http/tests/security/clipboard/copy-paste-html-cross-origin-iframe-across-origin.html http/tests/security/clipboard/copy-paste-html-cross-origin-iframe-in-same-origin.html http/tests/security/clipboard/copy-paste-url-across-origin-sanitizes-url.html fast/editing/insert-paragraph-with-text-in-table-crash.html fast/editing/mutation-with-display-contents-container-crash.html fast/editing/paste-and-indent-microtask-crash.html fast/editing/paste-and-match-style-with-table-2-crash.html fast/events/before-input-prevent-cut.html fast/events/before-input-prevent-paste.html fast/events/input-events-paste-data.html fast/events/input-events-paste-rich-datatransfer.html editing/selection/ios/avoid-showing-callout-menu-over-controls.html editing/selection/ios/dispatch-mouse-events-when-modifying-selection-quirk.html editing/selection/ios/scrolling-with-fixed-selection-does-not-unselect-native-text-field.html editing/selection/ios/selection-after-changing-text-with-callout-menu.html editing/selection/ios/show-callout-bar-after-selecting-word.html editing/selection/ios/show-callout-bar-after-tap.html editing/selection/ios/update-selection-after-iframe-scroll.html editing/pasteboard/paste-without-nesting.html editing/selection/ios/update-selection-after-overflow-scroll.html editing/selection/programmatic-selection-on-mac-is-directionless.html editing/selection/replace-selection-crash-02.html editing/selection/select-crash-001.html editing/selection/select-crash-002.html editing/selection/setSelection-shadow-dom-crash.html Also slightly different regression point but appear related: editing/pasteboard/contenteditable-pre.html editing/pasteboard/copy-backslash-with-euc.html editing/pasteboard/copy-crash-with-extraneous-attribute.html editing/pasteboard/copy-cut-paste-events-fired-when-disabled.html editing/pasteboard/copy-display-none.html editing/pasteboard/copy-element-with-conflicting-background-color-from-rule.html editing/pasteboard/copy-image-with-alt-text.html editing/pasteboard/copy-in-password-field.html editing/pasteboard/copy-inside-h1-preserves-h1.html editing/pasteboard/copy-null-characters.html editing/pasteboard/copy-paste-across-shadow-boundaries-1.html editing/pasteboard/copy-paste-across-shadow-boundaries-2.html editing/pasteboard/copy-paste-across-shadow-boundaries-3.html editing/pasteboard/copy-paste-across-shadow-boundaries-4.html editing/pasteboard/copy-paste-across-shadow-boundaries-5.html

Wenson Hsieh

Comment 13 2022-03-05 11:06:58 PST

(In reply to Matteo Flores from comment #12) > Listing more layout tests that seem to suffer from this bug as well. > > editing/deleting/smart-delete-paragraph-001.html > editing/deleting/smart-delete-paragraph-002.html > editing/deleting/smart-delete-paragraph-004.html > editing/deleting/smart-delete-paragraph-005.html > editing/execCommand/insert-newline-in-quoted-content-crash.html > editing/execCommand/outdent-cut-crash.html > editing/execCommand/paste-as-quotation-disconnected-paragraph-ancestor-crash. > html > editing/execCommand/selectAll-copy-crash.html > editing/pasteboard/19644-1.html > editing/pasteboard/19644-2.html > editing/pasteboard/3976872.html > editing/pasteboard/4076267-2.html > editing/pasteboard/4076267-3.html > editing/pasteboard/4076267.html > editing/pasteboard/4242293-1.html > editing/pasteboard/4242293.html > editing/pasteboard/4641033.html > editing/pasteboard/4922709.html > editing/pasteboard/4930986-1-paste-as-quotation.html > editing/pasteboard/4930986-2-paste-as-quotation.html > editing/pasteboard/4930986-3-paste-as-quotation.html > editing/pasteboard/4944770-1.html > editing/pasteboard/4944770-2.html > editing/pasteboard/4989774.html > editing/pasteboard/5006779.html > editing/pasteboard/5028447.html > editing/pasteboard/5065605.html > editing/pasteboard/5761530-1.html > editing/spelling/editing-word-with-marker-1.html > editing/style/smoosh-styles-003.html > editing/undo-manager/undo-manager-item-labels.html > > And these have a slightly different "regression" point but also appear > related: > > editing/pasteboard/dom-paste/dom-paste-same-origin.html > fast/css/border-radius-invalid-pair.html > fast/events/before-input-prevent-paste.html > fast/forms/ios/inputmode-removing-none.html > fast/forms/plaintext-mode-2.html > fast/images/image-copy-memory-usage.html > fast/images/text-recognition/image-overlay-line-wrapping.html > fast/table/table-split-on-insert-with-section-crash.html > http/tests/security/clipboard/copy-paste-html-across-origin-sanitizes-html. > html > http/tests/security/clipboard/copy-paste-html-across-origin-strips-mso-list. > html > http/tests/security/clipboard/copy-paste-html-cross-origin-iframe-across- > origin.html > http/tests/security/clipboard/copy-paste-html-cross-origin-iframe-in-same- > origin.html > http/tests/security/clipboard/copy-paste-url-across-origin-sanitizes-url. > html > fast/editing/insert-paragraph-with-text-in-table-crash.html > fast/editing/mutation-with-display-contents-container-crash.html > fast/editing/paste-and-indent-microtask-crash.html > fast/editing/paste-and-match-style-with-table-2-crash.html > fast/events/before-input-prevent-cut.html > fast/events/before-input-prevent-paste.html > fast/events/input-events-paste-data.html > fast/events/input-events-paste-rich-datatransfer.html > editing/selection/ios/avoid-showing-callout-menu-over-controls.html > editing/selection/ios/dispatch-mouse-events-when-modifying-selection-quirk. > html > editing/selection/ios/scrolling-with-fixed-selection-does-not-unselect- > native-text-field.html > editing/selection/ios/selection-after-changing-text-with-callout-menu.html > editing/selection/ios/show-callout-bar-after-selecting-word.html > editing/selection/ios/show-callout-bar-after-tap.html > editing/selection/ios/update-selection-after-iframe-scroll.html > editing/pasteboard/paste-without-nesting.html > editing/selection/ios/update-selection-after-overflow-scroll.html > editing/selection/programmatic-selection-on-mac-is-directionless.html > editing/selection/replace-selection-crash-02.html > editing/selection/select-crash-001.html > editing/selection/select-crash-002.html > editing/selection/setSelection-shadow-dom-crash.html > > Also slightly different regression point but appear related: > > editing/pasteboard/contenteditable-pre.html > editing/pasteboard/copy-backslash-with-euc.html > editing/pasteboard/copy-crash-with-extraneous-attribute.html > editing/pasteboard/copy-cut-paste-events-fired-when-disabled.html > editing/pasteboard/copy-display-none.html > editing/pasteboard/copy-element-with-conflicting-background-color-from-rule. > html > editing/pasteboard/copy-image-with-alt-text.html > editing/pasteboard/copy-in-password-field.html > editing/pasteboard/copy-inside-h1-preserves-h1.html > editing/pasteboard/copy-null-characters.html > editing/pasteboard/copy-paste-across-shadow-boundaries-1.html > editing/pasteboard/copy-paste-across-shadow-boundaries-2.html > editing/pasteboard/copy-paste-across-shadow-boundaries-3.html > editing/pasteboard/copy-paste-across-shadow-boundaries-4.html > editing/pasteboard/copy-paste-across-shadow-boundaries-5.html We know that this is happening because the PasteboardAutomaticSync default either isn't being set properly when booting the simulator, or is being flipped on during the course of the layout test run. Writing this default should work around the problem on affected bots: ``` defaults write com.apple.iphonesimulator PasteboardAutomaticSync -bool NO ```

Wenson Hsieh

Comment 14 2022-03-05 11:11:12 PST

I think we should revert the expectations previously set in r290750, and work around these failures for the time being by manually writing the default on all affected test runners. There are likely too many affected tests to set expectations for across the board. To actually address this, we'll need to either figure out what is causing this default to be set after booting the simulator, or add a way to enforce this default to be set to NO when running layout tests.

Alexey Proskuryakov

Comment 15 2022-03-05 19:13:41 PST

It's passed on command line, so setting it in on-disk defaults seems unlikely to change the behavior. Tools/Scripts/webkitpy/xcode/simulated_device.py: SimulatedDeviceManager._managing_simulator_app = not host.executive.run_command(['open', '-g', '-b', SimulatedDeviceManager.simulator_bundle_id, '--args', '-PasteboardAutomaticSync', '0'], return_exit_code=True)

Wenson Hsieh

Comment 16 2022-03-05 19:16:29 PST

(In reply to Alexey Proskuryakov from comment #15) > It's passed on command line, so setting it in on-disk defaults seems > unlikely to change the behavior. > > Tools/Scripts/webkitpy/xcode/simulated_device.py: > SimulatedDeviceManager._managing_simulator_app = not > host.executive.run_command(['open', '-g', '-b', > SimulatedDeviceManager.simulator_bundle_id, '--args', > '-PasteboardAutomaticSync', '0'], return_exit_code=True) My understanding was that `--args PasteboardAutomaticSync 0` would only affect the `Managed X` simulators that the test runner boots up. For some reason, this seems to (sometimes) be ineffective in overriding the default PasteboardAutomaticSync behavior. Perhaps we might see different results by simply writing the default globally.

Alexey Proskuryakov

Comment 17 2022-03-05 19:25:50 PST

Seems like an easy enough check to try it out, to isolate possibilities.

Wenson Hsieh

Comment 18 2022-03-05 19:30:22 PST

(In reply to Alexey Proskuryakov from comment #17) > Seems like an easy enough check to try it out, to isolate possibilities. Indeed! (And even if it doesn't work, that will be an interesting datapoint in this investigation).

Wenson Hsieh

Comment 19 2022-03-05 19:55:20 PST

(In reply to Wenson Hsieh from comment #18) > (In reply to Alexey Proskuryakov from comment #17) > > Seems like an easy enough check to try it out, to isolate possibilities. > > Indeed! (And even if it doesn't work, that will be an interesting datapoint > in this investigation). I ssh'd into bot651 and bot664 (both of which are consistently hitting these timeouts), and wrote the default. Let's see if it makes any difference... <https://build.webkit.org/#/workers/98> <https://build.webkit.org/#/workers/11>

Jonathan Bedard

Comment 20 2022-03-07 11:03:27 PST

Pull request: https://github.com/WebKit/WebKit/pull/149

Jonathan Bedard

Comment 21 2022-03-07 11:04:52 PST

Pull request: https://github.com/WebKit/WebKit/pull/191

Jonathan Bedard

Comment 22 2022-03-07 11:07:06 PST

(In reply to Jonathan Bedard from comment #20) > Pull request: https://github.com/WebKit/WebKit/pull/149 Wrong PR, mistakingly proposed this on an existing branch. https://github.com/WebKit/WebKit/pull/191 is the right PR.

Jonathan Bedard

Comment 23 2022-03-07 13:17:33 PST

Landed https://commits.webkit.org/248136@main (r290905)!

Dawn Morningstar

Comment 24 2022-03-09 14:05:54 PST

Reverted r290750 for reason: Bug has been fixed, expectations are no longer needed. Committed r291066 (248238@trunk): <https://commits.webkit.org/248238@trunk>

Attachments
Results page (2.49 MB, image/png) 2022-03-02 08:52 PST, Wenson Hsieh	no flags	Details
expectations (3.06 KB, patch) 2022-03-02 11:22 PST, Dawn Morningstar	no flags	Details Formatted Diff Diff
Apple-iOS-15-Simulator-Debug-WK2-Tests (405.37 KB, image/png) 2022-03-03 09:51 PST, Wenson Hsieh	no flags	Details
View All Add attachment proposed patch, testcase, etc.