Bug 262489

Summary: [ iOS17 Sonoma ] Multiple tests are failing/crashing/timing out following tester bring-up.
Product: WebKit Reporter: Ben Schwartz <ben_schwartz>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED MOVED    
Severity: Normal CC: heycam, jenner, rbuis, sabouhallawa, simon.fraser, webkit-bot-watchers-bugzilla, webkit-bug-importer, zimmermann
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=262425
https://bugs.webkit.org/show_bug.cgi?id=262427
https://bugs.webkit.org/show_bug.cgi?id=262516
https://bugs.webkit.org/show_bug.cgi?id=262595
https://bugs.webkit.org/show_bug.cgi?id=262604
https://bugs.webkit.org/show_bug.cgi?id=262663
https://bugs.webkit.org/show_bug.cgi?id=262829
https://bugs.webkit.org/show_bug.cgi?id=263048
https://bugs.webkit.org/show_bug.cgi?id=263050
https://bugs.webkit.org/show_bug.cgi?id=263052
https://bugs.webkit.org/show_bug.cgi?id=263053
https://bugs.webkit.org/show_bug.cgi?id=263214
https://bugs.webkit.org/show_bug.cgi?id=263662
https://bugs.webkit.org/show_bug.cgi?id=263575
https://bugs.webkit.org/show_bug.cgi?id=262797
https://bugs.webkit.org/show_bug.cgi?id=263282
https://bugs.webkit.org/show_bug.cgi?id=262732

Ben Schwartz
Reported 2023-10-02 10:35:07 PDT
We are seeing constant failures/crashes/timeouts on macOS Sonoma and iOS 17 following tester bring-up last week. We are looking to file more specific bugs for each; however, this bug serves as a tracking issue and allows us to batch set test expectations while the issues are being investigated. The following tests are constant text failures on both macOS Sonoma (wk1 and wk2) and iOS 17: fast/block/margin-collapse/103.html fast/css/rtl-ordering.html fast/forms/input-appearance-spinbutton.html fast/forms/input-baseline.html fast/forms/listbox-bidi-align.html fast/forms/placeholder-position.html fast/forms/search/search-size-with-decorations.html fast/forms/select-visual-hebrew.html fast/forms/time/time-input-rendering-basic.html fast/text/drawBidiText.html fast/text/glyph-display-lists/glyph-display-list-colr-unshared.html fast/text/international/bidi-listbox.html fast/text/international/bidi-menulist.html fast/text/international/pop-up-button-text-alignment-and-direction.html tables/mozilla/bugs/bug2479-2.html The following tests are constant text failures on macOS Sonoma (wk1 and wk2) only: editing/input/caret-at-the-edge-of-input.html editing/inserting/before-after-input-element.html editing/inserting/multiple-lines-selected.html editing/mac/dictionary-lookup/dictionary-lookup.html editing/pasteboard/4806874.html editing/pasteboard/input-field-1.html editing/selection/3690703-2.html editing/selection/3690703.html editing/selection/3690719.html editing/selection/4895428-3.html editing/selection/4975120.html editing/selection/context-menu-text-selection-lookup.html editing/selection/drag-select-1.html editing/selection/select-from-textfield-outwards.html fast/css/line-height.html fast/forms/auto-fill-button/hide-auto-fill-strong-password-viewable-treatment-when-form-is-reset.html fast/forms/auto-fill-button/input-auto-fill-button.html fast/forms/auto-fill-button/input-contacts-auto-fill-button.html fast/forms/auto-fill-button/input-credit-card-auto-fill-button.html fast/forms/auto-fill-button/input-strong-password-viewable.html fast/forms/basic-inputs.html fast/forms/box-shadow-override.html fast/forms/button-sizes.html fast/forms/control-restrict-line-height.html fast/forms/datalist/datalist-textinput-appearance.html fast/forms/encoding-test.html fast/forms/fieldset-align.html fast/forms/form-element-geometry.html fast/forms/input-align.html fast/forms/input-appearance-bkcolor.html fast/forms/input-appearance-default-bkcolor.html fast/forms/input-appearance-focus.html fast/forms/input-appearance-height.html fast/forms/input-appearance-preventDefault.html fast/forms/input-appearance-selection.html fast/forms/input-appearance-spinbutton-up.html fast/forms/input-appearance-visibility.html fast/forms/input-appearance-width.html fast/forms/input-disabled-color.html fast/forms/input-double-click-selection-gap-bug.html fast/forms/input-placeholder-visibility-1.html fast/forms/input-placeholder-visibility-3.html fast/forms/input-spaces.html fast/forms/input-table.html fast/forms/input-text-click-inside.html fast/forms/input-text-click-outside.html fast/forms/input-text-double-click.html fast/forms/input-text-drag-down.html fast/forms/input-text-option-delete.html fast/forms/input-text-scroll-left-on-blur.html fast/forms/input-text-self-emptying-click.html fast/forms/input-text-word-wrap.html fast/forms/input-type-text-min-width.html fast/forms/input-value.html fast/forms/input-width.html fast/forms/minWidthPercent.html fast/forms/number/number-appearance-rtl.html fast/forms/number/number-appearance-spinbutton-disabled-readonly.html fast/forms/number/number-appearance-spinbutton-layer.html fast/forms/placeholder-pseudo-style.html fast/forms/search-cancel-button-style-sharing.html fast/forms/search-display-none-cancel-button.html fast/forms/search-input-rtl.html fast/forms/search-rtl.html fast/forms/search-styled.html fast/forms/search-vertical-alignment.html fast/forms/search/search-padding-cancel-results-buttons.html fast/forms/select-writing-direction-natural.html fast/forms/tabbing-input-iframe.html fast/forms/text-control-intrinsic-widths.html fast/forms/textfield-focus-ring.html fast/forms/textfield-outline.html fast/forms/textfield-overflow.html fast/forms/visual-hebrew-text-field.html fast/frames/take-focus-from-iframe.html fast/html/details-no-summary4.html fast/html/details-open-javascript.html fast/html/details-open2.html fast/html/details-open4.html fast/lists/dynamic-marker-crash.html fast/repaint/placeholder-after-caps-lock-hidden.html fast/repaint/renderer-destruction-by-invalidateSelection-crash.html fast/repaint/subtree-root-skipped.html fast/replaced/replaced-breaking-mixture.html fast/table/003.html fast/table/colspanMinWidth.html fast/table/spanOverlapRepaint.html fast/table/text-field-baseline.html fast/text/textIteratorNilRenderer.html fast/transforms/transformed-focused-text-input.html http/tests/navigation/javascriptlink-frames.html imported/w3c/web-platform-tests/fetch/range/non-matching-range-response.html imported/w3c/web-platform-tests/mathml/presentation-markup/mrow/mrow-fallback.html imported/w3c/web-platform-tests/mathml/presentation-markup/tables/table-002.html imported/w3c/web-platform-tests/mathml/presentation-markup/tables/table-003.html imported/w3c/web-platform-tests/mathml/relations/css-styling/floats/not-floating-001.html imported/w3c/web-platform-tests/mathml/relations/css-styling/ignored-properties-001.html imported/w3c/web-platform-tests/mathml/relations/css-styling/width-height-001.html svg/custom/inline-svg-in-xhtml.xml svg/hixie/mixed/003.xml tables/mozilla/bugs/bug1188.html tables/mozilla/bugs/bug12384.html tables/mozilla/bugs/bug18359.html tables/mozilla/bugs/bug24200.html tables/mozilla/bugs/bug2479-3.html tables/mozilla/bugs/bug2479-4.html tables/mozilla/bugs/bug28928.html tables/mozilla/bugs/bug4382.html tables/mozilla/bugs/bug4527.html tables/mozilla/bugs/bug46368-1.html tables/mozilla/bugs/bug46368-2.html tables/mozilla/bugs/bug51037.html tables/mozilla/bugs/bug55545.html tables/mozilla/bugs/bug59354.html tables/mozilla/bugs/bug7342.html tables/mozilla/bugs/bug96334.html tables/mozilla/bugs/bug99948.html tables/mozilla/dom/tableDom.html tables/mozilla/other/move_row.html tables/mozilla_expected_failures/bugs/bug2479-5.html tables/mozilla_expected_failures/bugs/bug92647-1.html transforms/3d/general/perspective-non-layer.html The following tests are constantly crashing on macOS Sonoma (wk1 and wk2) and iOS 17: imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/failures_X25519.https.any.html imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/failures_X25519.https.any.worker.html imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_X25519.https.any.html imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_X25519.https.any.worker.html imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey.https.any.html imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey.https.any.worker.html imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey_failures_X25519.https.any.html imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey_failures_X25519.https.any.worker.html http/wpt/mediarecorder/MediaRecorder-dataavailable.html http/wpt/mediarecorder/MediaRecorder-multiple-start-stop.html http/wpt/mediarecorder/MediaRecorder-requestData.html http/wpt/mediarecorder/pause-recording.html http/wpt/mediarecorder/record-96KHz-sources.html The following tests are constant text failures on macOS Sonoma (wk1 ONLY): editing/mac/attributed-string/anchor-element.html editing/mac/attributed-string/attrib-string-colors-with-color-filter.html editing/mac/attributed-string/attribute-string-for-copy-with-color-filter.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-1.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-2.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-3.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-4.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-5.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-with-style-1.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-with-style-2.html editing/mac/attributed-string/attributed-string-for-typing-with-color-filter.html editing/mac/attributed-string/attributed-string-for-typing.html editing/mac/attributed-string/basic.html editing/mac/attributed-string/comment-cdata-section.html editing/mac/attributed-string/font-size.html editing/mac/attributed-string/font-style-variant-effect.html editing/mac/attributed-string/font-weight.html editing/mac/attributed-string/letter-spacing.html editing/mac/attributed-string/text-decorations.html editing/mac/attributed-string/vertical-align.html The following test is a constant ImageOnlyFailure on macOS Sonoma (wk1 and wk2): fast/events/context-no-deselect.html The following test is constantly timing out on macOS Sonoma (wk1 and wk2): http/tests/media/video-webm-stall.html The following test, previously a constant text failure on both macOS Sonoma and iOS 17, is still consistently timing out on macOS x86_64 (wk1 and wk2) and flakily timing out on iOS 17 after rebaseline at 268612@main: http/tests/appcache/fail-on-update-2.html
Attachments
Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2023-10-02 10:35:18 PDT
<rdar://problem/116353628>
Ben Schwartz
Comment 2 2023-10-02 10:59:31 PDT
Correction: http/tests/appcache/fail-on-update-2.html is timing out on all versions of macOS and iOS, not just iOS17/Sonoma.
Ben Schwartz
Comment 3 2023-10-02 11:56:06 PDT
It appears the reported tests had these expectations set previously: LayoutTests/platform/ios-device/TestExpectations:27:fast/forms/input-text-word-wrap.html [ Failure ] LayoutTests/platform/ios-device/TestExpectations:28:fast/forms/search-rtl.html [ Failure ] LayoutTests/platform/ios-device/TestExpectations:29:fast/forms/search-vertical-alignment.html [ Failure ] LayoutTests/platform/ios-device/TestExpectations:31:fast/forms/visual-hebrew-text-field.html [ Failure ] LayoutTests/platform/ios-simulator-wk2/TestExpectations:169:tables/mozilla/bugs/bug2479-2.html [ Failure Pass ] LayoutTests/platform/ios-simulator/TestExpectations:14:fast/forms/input-text-word-wrap.html [ Skip ] LayoutTests/platform/ios-simulator/TestExpectations:15:fast/forms/search-vertical-alignment.html [ Skip ] LayoutTests/platform/ios-simulator/TestExpectations:17:fast/forms/visual-hebrew-text-field.html [ Skip ] LayoutTests/platform/ios-wk2/TestExpectations:1060:fast/forms/input-appearance-spinbutton.html [ Failure ] LayoutTests/platform/ios-wk2/TestExpectations:1107:webkit.org/b/225665 http/tests/appcache/fail-on-update-2.html [ DumpJSConsoleLogInStdErr Timeout Pass ] LayoutTests/platform/ios-wk2/TestExpectations:2355:webkit.org/b/252866 [ Debug ] fast/text/glyph-display-lists/glyph-display-list-colr-unshared.html [ Pass Failure ] LayoutTests/platform/ios-wk2/TestExpectations:451:http/tests/navigation/javascriptlink-frames.html LayoutTests/platform/ios-wk2/TestExpectations:49:fast/repaint/placeholder-after-caps-lock-hidden.html [ Pass ] LayoutTests/platform/ios-wk2/TestExpectations:842:fast/text/drawBidiText.html [ Failure Pass ] LayoutTests/platform/ios/TestExpectations:1666:fast/forms/input-baseline.html [ Failure ] LayoutTests/platform/ios/TestExpectations:1678:fast/forms/placeholder-position.html [ Failure ] LayoutTests/platform/ios/TestExpectations:1681:fast/forms/search/search-size-with-decorations.html [ Failure ] LayoutTests/platform/ios/TestExpectations:2608:fast/table/003.html [ Skip ] LayoutTests/platform/ios/TestExpectations:3356:http/wpt/mediarecorder/pause-recording.html [ Pass Crash ] LayoutTests/platform/ios/TestExpectations:3400:http/tests/media/video-webm-stall.html [ Skip ] LayoutTests/platform/ios/TestExpectations:419:editing/selection/context-menu-text-selection-lookup.html [ Skip ] LayoutTests/platform/ios/TestExpectations:4527:fast/repaint/placeholder-after-caps-lock-hidden.html [ Timeout ] LayoutTests/platform/ios/TestExpectations:4535:fast/css/rtl-ordering.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4538:fast/forms/basic-inputs.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4539:fast/forms/input-appearance-height.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4540:fast/forms/input-value.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4541:fast/forms/listbox-bidi-align.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4542:fast/forms/placeholder-pseudo-style.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4543:fast/forms/select-visual-hebrew.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4544:fast/forms/time/time-input-rendering-basic.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4564:imported/w3c/web-platform-tests/fetch/range/non-matching-range-response.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4575:tables/mozilla_expected_failures/bugs/bug92647-1.html [ Failure ] LayoutTests/platform/ios/TestExpectations:642:fast/forms/input-appearance-spinbutton-up.html [ Skip ] LayoutTests/platform/ios/TestExpectations:683:fast/forms/search-rtl.html [ Skip ] LayoutTests/platform/ios/TestExpectations:780:fast/text/textIteratorNilRenderer.html [ Skip ] LayoutTests/platform/mac-monterey/TestExpectations:9:http/tests/appcache/fail-on-update-2.html [ DumpJSConsoleLogInStdErr Timeout Failure ] LayoutTests/platform/mac-ventura-wk2-lbse-text/TestExpectations:251:svg/custom/inline-svg-in-xhtml.xml [ Failure ] LayoutTests/platform/mac-ventura-wk2/TestExpectations:24:imported/w3c/web-platform-tests/fetch/range/non-matching-range-response.html [ Pass Failure ] LayoutTests/platform/mac-ventura/TestExpectations:16:fast/forms/auto-fill-button/input-credit-card-auto-fill-button.html [ Failure ] LayoutTests/platform/mac-ventura/TestExpectations:21:http/tests/appcache/fail-on-update-2.html [ DumpJSConsoleLogInStdErr Timeout Failure ] LayoutTests/platform/mac-wk1/TestExpectations:1004:http/tests/media/video-webm-stall.html LayoutTests/platform/mac-wk2/TestExpectations:125:fast/repaint/placeholder-after-caps-lock-hidden.html [ Pass ] LayoutTests/platform/mac-wk2/TestExpectations:1370:webkit.org/b/225665 [ Release arm64 ] http/tests/appcache/fail-on-update-2.html [ DumpJSConsoleLogInStdErr Slow ] LayoutTests/platform/mac-wk2/TestExpectations:1695:[ Monterey+ ] http/tests/media/video-webm-stall.html [ Failure ] LayoutTests/platform/mac-wk2/TestExpectations:1700:[ Monterey Ventura x86_64 ] http/wpt/mediarecorder/pause-recording.html [ Pass Failure ] # change back to [ Monterey+ x86_64 ] when webkit.org/b/262425 is resolved. LayoutTests/platform/mac/TestExpectations:1980:[ BigSur ] http/tests/media/video-webm-stall.html [ Skip ] LayoutTests/platform/mac/TestExpectations:2746:[ Monterey ] fast/block/margin-collapse/103.html [ Pass Failure ImageOnlyFailure ] LayoutTests/platform/mac/TestExpectations:2753:[ Monterey ] imported/w3c/web-platform-tests/mathml/presentation-markup/mrow/mrow-fallback.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2754:[ Monterey ] imported/w3c/web-platform-tests/mathml/presentation-markup/tables/table-002.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2755:[ Monterey ] imported/w3c/web-platform-tests/mathml/presentation-markup/tables/table-003.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2756:[ Monterey ] imported/w3c/web-platform-tests/mathml/relations/css-styling/floats/not-floating-001.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2757:[ Monterey ] imported/w3c/web-platform-tests/mathml/relations/css-styling/ignored-properties-001.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2758:[ Monterey ] imported/w3c/web-platform-tests/mathml/relations/css-styling/width-height-001.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:3034:http/tests/appcache/fail-on-update-2.html [ Pass Timeout ] LayoutTests/platform/mac/TestExpectations:61:fast/forms/search/search-padding-cancel-results-buttons.html [ Pass ] I will work around these when I file expectations.
Ben Schwartz
Comment 4 2023-10-04 16:00:30 PDT
--- SONOMA UPDATE --- After the changes landed at 268861@main, most of the macOS Sonoma expectations and baselines should be up-to-date. RESOLVED: Below are the tests were left out of the changes in 268861@main due to already being resolved. Bug #262210 (rdar://116139561) -- rebaseline landed in 268612@main: - http/tests/appcache/fail-on-update-2.html Bug #262425 (rdar://116267156) -- fixes landed in 268775@main and 268795@main: - http/wpt/mediarecorder/MediaRecorder-dataavailable.html - http/wpt/mediarecorder/MediaRecorder-multiple-start-stop.html - http/wpt/mediarecorder/MediaRecorder-requestData.html - http/wpt/mediarecorder/pause-recording.html - http/wpt/mediarecorder/record-96KHz-sources.html Bug #262427 (rdar://116268173) -- rebaselines landed in 268823@main: - fast/forms/basic-inputs.html - fast/forms/input-disabled-color.html - fast/forms/number/number-appearance-spinbutton-disabled-readonly.html - fast/forms/placeholder-pseudo-style.html OPEN: Below are the tests that still have outstanding bug reports. Bug #262595 (rdar://116437531): - editing/mac/dictionary-lookup/dictionary-lookup.html - editing/selection/context-menu-text-selection-lookup.html Bug #262604 (rdar://116443613): - http/tests/media/video-webm-stall.html Bug #262663 (rdar://116490690): - imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/failures_X25519.https.any.html - imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/failures_X25519.https.any.worker.html - imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_X25519.https.any.html - imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_X25519.https.any.worker.html - imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey.https.any.html - imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey.https.any.worker.html - imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey_failures_X25519.https.any.html - imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey_failures_X25519.https.any.worker.html
Ben Schwartz
Comment 5 2023-10-04 16:05:54 PDT
Important note: in my previous comment, I listed a few bugs that are waiting on resolutions. These tests may require re-baselines, but we will only know after a resolution is reached.
Nikolas Zimmermann
Comment 6 2023-11-05 14:38:47 PST
Since I upgraded to macOS Sonoma, the following two SVG tests are constant text failures for me: svg/custom/inline-svg-in-xhtml.xml svg/hixie/mixed/003.xml Font related render tree text differences.. I tried to add Sonoma specific results here (https://ews-build.webkit.org/#/builders/50/builds/491), but the "macOS-AppleSilicon-Sonoma-Debug-WK2-Tests-EWS" results are actually identical to the ones obtained with Ventura/Monterey and not different, as on my machine. I was investigating the issue and came across this ticket, where you mentioned also these two tests as failing on Sonoma. Why don't we observe that on the aforementioned bot? I'm slightly confused.
Note You need to log in before you can comment on or make changes to this bug.