RESOLVED FIXED 261306
REGRESSION(267516@main): [ macOS ] 25 tests are constant failures following Sonoma OpenSource expectations migration. 
https://bugs.webkit.org/show_bug.cgi?id=261306
Summary REGRESSION(267516@main): [ macOS ] 25 tests are constant failures following S...
Ben Schwartz
Reported 2023-09-07 17:50:44 PDT
The following tests are consistent/constant failures on macOS Monterey (both wk1 and 2): http/tests/appcache/fail-on-update-2.html The following tests are constant failures on macOS Monterey+, wk1: fast/scrolling/mac/scrollbars/scrollbar-state.html fast/scrolling/mac/scrollbars/overlay-scrollbar-state.html The following tests are constant failures on macOS Ventura, wk1: editing/caret/insert-paragraph-does-not-paint-stale-carets.html editing/input/editable-container-with-word-wrap-normal.html editing/inserting/caret-position.html editing/mac/input/caret-primary-bidi.html editing/mac/input/firstrectforcharacterrange-plain.html editing/mac/input/firstrectforcharacterrange-styled.html editing/pasteboard/interchange-newline-2.html editing/selection/5825350-2.html editing/selection/block-cursor-overtype-mode.html editing/selection/caret-in-empty-inline-1.html editing/selection/caret-in-empty-inline-2.html editing/selection/internal-caret-rect.html editing/selection/mixed-editability-10.html editing/selection/simple-line-layout-caret-is-gone.html fast/forms/auto-fill-button/input-credit-card-auto-fill-button.html imported/w3c/web-platform-tests/mathml/relations/css-styling/display-2.html media/media-can-play-mpeg4-video.html imported/w3c/web-platform-tests/xhr/open-method-case-sensitive.htm media/video-source-type-params.html The following tests are constant failures on macOS Ventura (x86_64 only), wk2. scrollingcoordinator/mac/fixed-backgrounds/fixed-background-in-overflow-in-iframe.html (x86_64 only) HISTORY: https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=editing%2Fcaret%2Finsert-paragraph-does-not-paint-stale-carets.html&test=editing%2Finput%2Feditable-container-with-word-wrap-normal.html&test=editing%2Finserting%2Fcaret-position.html&test=editing%2Fmac%2Finput%2Fcaret-primary-bidi.html&test=editing%2Fmac%2Finput%2Ffirstrectforcharacterrange-plain.html&test=editing%2Fmac%2Finput%2Ffirstrectforcharacterrange-styled.html&test=editing%2Fpasteboard%2Finterchange-newline-2.html&test=editing%2Fselection%2F5825350-2.html&test=editing%2Fselection%2Fblock-cursor-overtype-mode.html&test=editing%2Fselection%2Fcaret-in-empty-inline-1.html&test=editing%2Fselection%2Fcaret-in-empty-inline-2.html&test=editing%2Fselection%2Finternal-caret-rect.html&test=editing%2Fselection%2Fmixed-editability-10.html&test=editing%2Fselection%2Fsimple-line-layout-caret-is-gone.html&test=fast%2Fforms%2Fauto-fill-button%2Finput-credit-card-auto-fill-button.html&test=fast%2Fscrolling%2Fmac%2Fscrollbars%2Foverlay-scrollbar-state.html&test=fast%2Fscrolling%2Fmac%2Fscrollbars%2Fscrollbar-state.html&test=http%2Ftests%2Fappcache%2Ffail-on-update-2.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fmathml%2Frelations%2Fcss-styling%2Fdisplay-2.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fxhr%2Fopen-method-case-sensitive.htm&test=media%2Fmedia-can-play-mpeg4-video.html&test=media%2Fvideo-source-type-params.html&test=scrollingcoordinator%2Fmac%2Ffixed-backgrounds%2Ffixed-background-in-overflow-in-iframe.html&platform=mac REPRODUCTION: All tests can be reproduced on the reported platform by using the following command: run-webkit-tests -1 --clobber-old-results --verbose --test-list run_tests.txt ...where run_tests.txt contains the affected tests listed above. REGRESSION: According to the flakiness dashboard and bisection workflow, these tests have a clear regression point at 267516@main.
Attachments
Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2023-09-07 17:51:04 PDT
<rdar://problem/115146287>
Ben Schwartz
Comment 2 2023-09-07 18:10:09 PDT
Adding http/tests/appcache/remove-cache.html, which is a consistent/constant failure on Monterey wk1.
EWS
Comment 3 2023-09-07 20:18:08 PDT
Test gardening commit 267764@main (1c1d8e1af449): <https://commits.webkit.org/267764@main> Reviewed commits have been landed. Closing PR #17569 and removing active labels.
EWS
Comment 4 2023-09-08 09:57:10 PDT
Test gardening commit 267798@main (af7d03822467): <https://commits.webkit.org/267798@main> Reviewed commits have been landed. Closing PR #17596 and removing active labels.
Ben Schwartz
Comment 5 2023-09-27 14:13:58 PDT
Adding imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html, which is a constant failure on macOS but is missing exceptions for Monterey wk2.
EWS
Comment 6 2023-09-27 14:53:41 PDT
Test gardening commit 268540@main (0564f1d21bab): <https://commits.webkit.org/268540@main> Reviewed commits have been landed. Closing PR #18309 and removing active labels.
Ryan Haddad
Comment 7 2023-10-20 15:47:52 PDT
It looks like all of these except for scrollingcoordinator/mac/fixed-backgrounds/fixed-background-in-overflow-in-iframe.html were fixed via rebaselines. http/tests/appcache/fail-on-update-2.html is a flaky timeout, but it appears that we already have at least one bug tracking this. Before closing this, we need to make sure we have removed any failure expectations added as part of this bug.
Ben Schwartz
Comment 8 2023-10-30 19:14:02 PDT
Issues have been resolved on most tests, but persist on the following. This test is consistently timing out on macOS: http/tests/appcache/fail-on-update-2.html http/tests/appcache/remove-cache.html (Monterey wk1 only: when not timing out, failing) This test is constantly failing on macOS: imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html I will remove the expectations for the other tests.
EWS
Comment 9 2023-10-30 19:53:56 PDT
Test gardening commit 269991@main (4b5edf812039): <https://commits.webkit.org/269991@main> Reviewed commits have been landed. Closing PR #19754 and removing active labels.
Ben Schwartz
Comment 10 2023-10-31 11:45:00 PDT
Filed new bugs for the tests still experiencing issues. Closing, as verified issues with other tests resolved.
Note You need to log in before you can comment on or make changes to this bug.