261306 2023-09-07 17:50:44 -0700 REGRESSION(267516@main): [ macOS ] 25 tests are constant failures following Sonoma OpenSource expectations migration. 2024-01-11 17:46:18 -0800 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=259373 https://bugs.webkit.org/show_bug.cgi?id=263982 https://bugs.webkit.org/show_bug.cgi?id=263987 https://bugs.webkit.org/show_bug.cgi?id=267439 InRadar P2 Normal --- 1 ben_schwartz webkit-unassigned Morningstar webkit-bot-watchers-bugzilla webkit-bug-importer oldest_to_newest 1976279 0 ben_schwartz 2023-09-07 17:50:44 -0700 The following tests are consistent/constant failures on macOS Monterey (both wk1 and 2): http/tests/appcache/fail-on-update-2.html The following tests are constant failures on macOS Monterey+, wk1: fast/scrolling/mac/scrollbars/scrollbar-state.html fast/scrolling/mac/scrollbars/overlay-scrollbar-state.html The following tests are constant failures on macOS Ventura, wk1: editing/caret/insert-paragraph-does-not-paint-stale-carets.html editing/input/editable-container-with-word-wrap-normal.html editing/inserting/caret-position.html editing/mac/input/caret-primary-bidi.html editing/mac/input/firstrectforcharacterrange-plain.html editing/mac/input/firstrectforcharacterrange-styled.html editing/pasteboard/interchange-newline-2.html editing/selection/5825350-2.html editing/selection/block-cursor-overtype-mode.html editing/selection/caret-in-empty-inline-1.html editing/selection/caret-in-empty-inline-2.html editing/selection/internal-caret-rect.html editing/selection/mixed-editability-10.html editing/selection/simple-line-layout-caret-is-gone.html fast/forms/auto-fill-button/input-credit-card-auto-fill-button.html imported/w3c/web-platform-tests/mathml/relations/css-styling/display-2.html media/media-can-play-mpeg4-video.html imported/w3c/web-platform-tests/xhr/open-method-case-sensitive.htm media/video-source-type-params.html The following tests are constant failures on macOS Ventura (x86_64 only), wk2. scrollingcoordinator/mac/fixed-backgrounds/fixed-background-in-overflow-in-iframe.html (x86_64 only) HISTORY: https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=editing%2Fcaret%2Finsert-paragraph-does-not-paint-stale-carets.html&test=editing%2Finput%2Feditable-container-with-word-wrap-normal.html&test=editing%2Finserting%2Fcaret-position.html&test=editing%2Fmac%2Finput%2Fcaret-primary-bidi.html&test=editing%2Fmac%2Finput%2Ffirstrectforcharacterrange-plain.html&test=editing%2Fmac%2Finput%2Ffirstrectforcharacterrange-styled.html&test=editing%2Fpasteboard%2Finterchange-newline-2.html&test=editing%2Fselection%2F5825350-2.html&test=editing%2Fselection%2Fblock-cursor-overtype-mode.html&test=editing%2Fselection%2Fcaret-in-empty-inline-1.html&test=editing%2Fselection%2Fcaret-in-empty-inline-2.html&test=editing%2Fselection%2Finternal-caret-rect.html&test=editing%2Fselection%2Fmixed-editability-10.html&test=editing%2Fselection%2Fsimple-line-layout-caret-is-gone.html&test=fast%2Fforms%2Fauto-fill-button%2Finput-credit-card-auto-fill-button.html&test=fast%2Fscrolling%2Fmac%2Fscrollbars%2Foverlay-scrollbar-state.html&test=fast%2Fscrolling%2Fmac%2Fscrollbars%2Fscrollbar-state.html&test=http%2Ftests%2Fappcache%2Ffail-on-update-2.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fmathml%2Frelations%2Fcss-styling%2Fdisplay-2.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fxhr%2Fopen-method-case-sensitive.htm&test=media%2Fmedia-can-play-mpeg4-video.html&test=media%2Fvideo-source-type-params.html&test=scrollingcoordinator%2Fmac%2Ffixed-backgrounds%2Ffixed-background-in-overflow-in-iframe.html&platform=mac REPRODUCTION: All tests can be reproduced on the reported platform by using the following command: run-webkit-tests -1 --clobber-old-results --verbose --test-list run_tests.txt ...where run_tests.txt contains the affected tests listed above. REGRESSION: According to the flakiness dashboard and bisection workflow, these tests have a clear regression point at 267516@main. 1976280 1 webkit-bug-importer 2023-09-07 17:51:04 -0700 <rdar://problem/115146287> 1976283 2 ben_schwartz 2023-09-07 18:10:09 -0700 Adding http/tests/appcache/remove-cache.html, which is a consistent/constant failure on Monterey wk1. 1976306 3 ews-feeder 2023-09-07 20:18:08 -0700 Test gardening commit 267764@main (1c1d8e1af449): <https://commits.webkit.org/267764@main> Reviewed commits have been landed. Closing PR #17569 and removing active labels. 1976456 4 ews-feeder 2023-09-08 09:57:10 -0700 Test gardening commit 267798@main (af7d03822467): <https://commits.webkit.org/267798@main> Reviewed commits have been landed. Closing PR #17596 and removing active labels. 1980627 5 ben_schwartz 2023-09-27 14:13:58 -0700 Adding imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html, which is a constant failure on macOS but is missing exceptions for Monterey wk2. 1980645 6 ews-feeder 2023-09-27 14:53:41 -0700 Test gardening commit 268540@main (0564f1d21bab): <https://commits.webkit.org/268540@main> Reviewed commits have been landed. Closing PR #18309 and removing active labels. 1986573 7 ryanhaddad 2023-10-20 15:47:52 -0700 It looks like all of these except for scrollingcoordinator/mac/fixed-backgrounds/fixed-background-in-overflow-in-iframe.html were fixed via rebaselines. http/tests/appcache/fail-on-update-2.html is a flaky timeout, but it appears that we already have at least one bug tracking this. Before closing this, we need to make sure we have removed any failure expectations added as part of this bug. 1988735 8 ben_schwartz 2023-10-30 19:14:02 -0700 Issues have been resolved on most tests, but persist on the following. This test is consistently timing out on macOS: http/tests/appcache/fail-on-update-2.html http/tests/appcache/remove-cache.html (Monterey wk1 only: when not timing out, failing) This test is constantly failing on macOS: imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html I will remove the expectations for the other tests. 1988741 9 ews-feeder 2023-10-30 19:53:56 -0700 Test gardening commit 269991@main (4b5edf812039): <https://commits.webkit.org/269991@main> Reviewed commits have been landed. Closing PR #19754 and removing active labels. 1988928 10 ben_schwartz 2023-10-31 11:45:00 -0700 Filed new bugs for the tests still experiencing issues. Closing, as verified issues with other tests resolved.