Bug 248537 – Batch mark expectations for Ventura test failures

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Robert Jenner]

We need to bring up Ventura EWS quickly. So this is going to be a master bug that highlights all of the current test failures constant and flaky for Ventura. I am going to batch mark expectations for all of them so we can bring up Ventura EWS. Then with this bug serving as the master I will divide and conquer breaking down related bugs into their own separate bugs that will be related to this bug.

[Radar WebKit Bug Importer]

<rdar://problem/102820070>

[Robert Jenner]

The first batch of test expectations to set are going to be for constant failures. They will go as follows:

Mac-wk2
[ Ventura+ ] fast/images/animated-heics-verify.html [ Timeout Crash ]
[ Ventura+ ] model-element/model-element-ready.html [ Pass Failure ]
[ Monterey+ ] model-element/model-element-graphics-layers-opacity.html [ Failure
[ Ventura+ ] http/tests/resourceLoadStatistics/website-data-removal-for-site-without-user-interaction.html [ Failure ]
[ Ventura+ ] http/tests/resourceLoadStatistics/website-data-removal-for-site-navigated-to-with-link-decoration.html [ Failure ]
[ Ventura+ ]  http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js.html [ Failure ]
[ Monterey+ ] http/tests/media/video-webm-stall.html  [ Failure ]
[ BigSur+ ] fast/images/animated-heics-draw.html [ Timeout Crash ]

Mac
[ Ventura+ ] imported/w3c/web-platform-tests/mathml/relations/css-styling/display-2.html [ Failure ]
[ Ventura+ ] fast/images/avif-as-image.html [ ImageOnlyFailure Crash ]
	Constant Image Failure on Ventura
[ BigSur+ ] fast/images/animated-avif.html [ Pass ImageOnlyFailure Timeout Crash ]

[Robert Jenner]

The second batch will be constant failures but not on every single Ventura queue. They will go as follows:


Mac-wk2
[ Ventura+ ] webrtc/canvas-to-peer-connection-2d.html [ Pass Timeout ]
[ Ventura+ ] webrtc/canvas-to-peer-connection.html [ Pass Failure Timeout ]
[ Ventura+ ] webrtc/captureCanvas-webrtc-software-h264-baseline.html [ Pass Failure Timeout ]
[ Ventura+ ] scrollbars/scrolling-backward-by-page-on-keyboard-spacebar.html [ Pass Failure ]
[ Ventura+ ] webaudio/decode-audio-data-webm-vorbis.html [ Failure ]
[ Ventura+ x86_64 ] svg/gradients/spreadMethodAlpha.svg [ ImageOnlyFailure ]
[ Ventura+ x86_64 ] scrollbars/scrolling-by-page-accounting-oversized-fixed-elements-on-keyboard-spacebar.html [ Pass Failure ]
[ Ventura+ ] imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/autoplay-allowed-by-feature-policy-attribute.https.sub.html [ Pass Failure ]
[ Ventura+ ] imported/w3c/web-platform-tests/mathml/relations/html5-tree/dynamic-childlist-002.html [ Failure ]
[ BigSur+ ] imported/w3c/web-platform-tests/webstorage/storage_local_window_open.window.html [ Pass Timeout ]
[ BigSur+ x86_64 ] imported/mozilla/svg/blend-color.svg [ ImageOnlyFailure ]
[ Monterey+ Debug arm64 ] imported/w3c/web-platform-tests/fetch/api/request/destination/fetch-destination-frame.https.html [ Crash ]
[ BigSur+ x86_64 ] imported/w3c/web-platform-tests/css/css-grid/alignment/grid-item-aspect-ratio-stretch-2.html [ ImageOnlyFailure ]
[ BigSur+ x86_64 ] imported/mozilla/svg/blend-luminosity.svg [ ImageOnlyFailure ]
[ Ventura+ Debug x86_64 ] fast/visual-viewport/viewport-dimensions-exclude-custom-scrollbars.html [ Pass Failure ]
[ Ventura+ Debug x86_64 ] fast/text/international/system-language/han-text-style.html [ ImageOnlyFailure ]
[ Ventura+ Debug x86_64 ] fast/visual-viewport/tiled-drawing/zoomed-fixed-scrolled-down-then-up.html [ Failure ]
[ Ventura+ Debug x86_64 ] fast/visual-viewport/tiled-drawing/zoomed-fixed-scrolled-down.html [ Failure ]
[ Ventura+ Debug x86_64 ] fast/visual-viewport/tiled-drawing/zoomed-fixed-scrolling-layers-state.html [ Failure ]
[ BigSur+ ] fast/speechrecognition/start-recognition-then-stop.html [ Pass Failure ]
[ BigSur+ x86_64 ] fast/scrolling/keyboard-scrolling-distance-downArrow.html [ Pass Timeout Crash ]
[ BigSur+ x86_64 ]fast/frames/message-port-postMessage-unload-event.html [ Pass Failure ]
[ BigSur+ x86_64 ] fast/mediastream/getUserMedia-to-canvas-1.html [ Pass timeout ]
[ BigSur+ x86_64 ] fast/mediastream/getUserMedia-to-canvas-2.html [ Pass timeout ]
[ Monterey+ ] fast/borders/border-painting-correctness-dashed.html [ ImageOnlyFailure ]
[ Monterey+ ] fast/borders/border-painting-correctness-dotted.html [ ImageOnlyFailure ]
[ BigSur+ ] fast/canvas/webgl/lose-context-on-timeout.html [ Timeout ]
[ BigSur+ x86_64 ] css3/blending/background-blend-mode-body-image.html [ ImageOnlyFailure ]

Mac-wk1
[ Ventura+ ] transitions/extra-transition.html [ Pass Failure ]
[ Ventura+ arm64 ] svg/transforms/transformed-text-fill-pattern.html [ ImageOnlyFailure ]
[ Ventura+ ] media/accessiblity-describes-video.html [ Crash ]
[ Ventura+ ] media/track/track-mode.html  [ Crash ]
[ Ventura+ ] media/modern-media-controls/pip-support/pip-support-click.html [ Pass Crash ]
[ Ventura+ ] media/audio-playback-restriction-removed-track-enabled.html  [ Crash ]
[ Ventura+ ] media/media-captions.html  [ Crash ]
[ Ventura+ ] media/modern-media-controls/media-controller/media-controller-auto-hide-rewind-with-mouse-enter.html  [ Crash ]
[ Ventura+ ] media/modern-media-controls/media-controls/media-controls-display-above-captions.html  [ Crash ]
[ Ventura+ ] media/track/track-delete-during-setup.html  [ Crash ]
[ Ventura+ ] media/track/track-in-band-cues-added-once.html  [ Crash ]
[ Ventura+ ] media/track/track-in-band-legacy-api.html  [ Crash ]
[ Ventura+ ] media/track/track-in-band.html  [ Crash ]
[ Ventura+ ] media/track/track-legacyapi-with-automatic-mode.html  [ Crash ]
[ Ventura+ ] media/track/track-long-captions-file.html  [ Crash ]
[ Ventura+ ] media/track/track-mode-disabled-crash.html  [ Crash ]
[ BigSur+ ] imported/w3c/web-platform-tests/secure-contexts/basic-popup-and-iframe-tests.html [ Pass Failure ]
[ BigSur+ ] imported/w3c/web-platform-tests/geolocation-API/permission.https.html [ Failure ]
[ BigSur+ ] imported/w3c/web-platform-tests/geolocation-API/PositionOptions.https.html [ Failure ]
[ Ventura+ ] http/wpt/webaudio/audioworklet-addModule-cors.sub.https.html
[ Monterey+ ] http/wpt/fetch/fetch-metadata-websocket.html [ Failure ]
[ BigSur+ ] http/tests/security/contentSecurityPolicy/1.1/securitypolicyviolation-block-image-https.html [ Pass Failure ]
[ Monterey+ ] http/tests/media/hls/hls-webvtt-seek-backwards.html [ Timeout Crash ]
[ Ventura+ ] http/tests/media/hls/hls-webvtt-default.html [ Crash ]
[ Ventura+ ] http/tests/inspector/network/resource-timing.html [ Failure ]
[ Monterey+ ] http/tests/inspector/network/resource-security-certificate.html [ Pass Failure ]
[ Ventura+ ] http/tests/inspector/network/getSerializedCertificate.html [ Failure ]
[ Ventura+ ] fast/shrink-wrap/rect-shrink-wrap.html [ ImageOnlyFailure ]
[ Ventura+ arm64 ] fast/writing-mode/english-bt-text-with-spelling-marker.html [ ImageOnlyFailure ]
[ BigSur+ ] css3/scroll-snap/scroll-padding-mainframe-paging.html [ Failure ]
[ Ventura+ ] compositing/fixed-with-main-thread-scrolling.html [ ImageOnlyFailure ]

Mac
[ BigSur+ ] imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/autoplay-disabled-by-feature-policy.https.sub.html [ Pass Failure ]
[ BigSur+ Debug ] inspector/css/pseudo-element-matches-for-pseudo-element-node.html [ Pass Crash ]
[ BigSur+ ] imported/w3c/web-platform-tests/web-animations/interfaces/Animation/onremove.html [ Pass Failure ]
[ Ventura+ x86_64 ] imported/w3c/web-platform-tests/html/canvas/element/manual/wide-gamut-canvas/canvas-display-p3-drawImage-video.html [ Pass Failure Timeout ]
[ Ventura+ x86_64 ] http/tests/media/now-playing-info.html [ Pass Failure ]
[ BigSur+ x86_64 ] animations/stop-animation-on-suspend.html [ Pass Failure ]

Mac-gpup
[ Ventura+ ] fast/scrolling/iframe-scrollable-after-back.html [ Timeout ]
[ Ventura+ ] fast/scrolling/mac/absolute-in-overflow-scroll.html [ Timeout ]
[ Ventura+ ] fast/scrolling/mac/async-overscroll-behavior-element.html [ Failure ]
[ Ventura+ ] fast/scrolling/mac/async-overscroll-behavior-iframe.html [ Failure ]
[ Ventura+ ] fast/scrolling/mac/async-overscroll-behavior-unscrollable-element.html [ Failure ]
[ Ventura+ ] fast/scrolling/mac/async-overscroll-behavior-unscrollable-iframe.html [ Failure ]
[ Ventura+ ] fast/scrolling/mac/async-scroll-overflow-hidden-on-one-axis.html [ Timeout ]
[ Ventura+ ] fast/scrolling/mac/canceled-event-with-non-zero-deltas.html [ Timeout ]
[ Ventura+ ] fast/scrolling/mac/hit-test-overflow-tiled-layer.html [ Timeout ]
[ Ventura+ ] fast/scrolling/mac/horizontal-overflow-trapping-small-deltas.html [ Timeout ]
[ Ventura+ ] fast/scrolling/mac/scrollbars/overflow-in-iframe-overlay-scrollbar-hovered.html [ Timeout ]
[ Ventura+ ] fast/scrolling/mac/scrollbars/very-wide-overlay-scrollbar.html [ Timeout ]
[ Ventura+ ] fast/scrolling/unfocusing-page-while-keyboard-scrolling.html [ Timeout ]
[ Ventura+ ] fast/scrolling/keyboard-scrolling-distance-pageDown.html [ Timeout ]
[ Ventura+ ] fast/scrolling/mac/adjust-scroll-snap-during-gesture.html [ Timeout ]
[ Ventura+ ] fast/repaint/fixed-move-after-keyboard-scroll.html [ Timeout ]
[ Ventura+ ] fast/events/autoscroll-when-zoomed.html [ Failure ]
[ Ventura+ ] fast/images/text-recognition/mac/image-overlay-data-detectors.html [ Crash ]
[ Ventura+ ] fast/gradients/conic-gradient-alpha.html [ ImageOnlyFailure ]
[ Ventura+ ] fast/gradients/conic-gradient-alpha-unpremultiplied.html [ ImageOnlyFailure ]
[ Ventura+ ] fast/events/wheel/wheel-event-in-passive-region-non-cancelable.html [ Timeout Failure ]
[ Ventura+ ] fast/canvas/webgl/lose-context-on-timeout.html [ Failure ]
[ Ventura+ ] editing/input/page-up-down-scrolls.html [ Timeout ]
[ Ventura+ ] css3/scroll-snap/scroll-padding-overflow-paging.html [ Failure Crash ]
[ Ventura+ ] css3/scroll-snap/scroll-padding-mainframe-paging.html [ Failure Crash ]
[ Ventura+ ] compositing/fixed-with-main-thread-scrolling.html [ Timeout ]

[EWS]

Test gardening commit 257223@main (d083c660067f): <https://commits.webkit.org/257223@main>

Reviewed commits have been landed. Closing PR #7011 and removing active labels.

[Robert Jenner]

The next batch of expectations will be these:

mac-wk2
[ BigSur+ ] fast/mediastream/mediastreamtrack-video-clone.html [ Pass Failure ]
[ BigSur+ ] fast/repaint/fixed-move-after-keyboard-scroll.html [ Pass Failure ]
[ BigSur+ ] fast/scrolling/mac/keyboard-scrolling-with-mouse-scroll.html [ Pass Failure ]
[ BigSur+ x86_64 ] http/tests/media/hls/track-in-band-hls-metadata-cue-duration.html [ Pass Failure ]
[ Monterey+ x86_64 http/tests/websocket/tests/hybi/handshake-ok-with-legacy-websocket-response-headers.html [ Pass Failure ]
[ Bigsur+ x86_64 ] http/wpt/mediarecorder/pause-recording.html [ Pass Failure ]
[ BigSur+ x86_64 ]http/wpt/webrtc/audiovideo-script-transform.html [ Pass Failure ]
[ Venutra+ x86_64 ] imported/w3c/web-platform-tests/html/cross-origin-embedder-policy/block-local-documents-inheriting-none.https.html [ Pass Failure ]
[ Venutra+ x86_64 ] imported/w3c/web-platform-tests/html/browsers/history/the-history-interface/history_go_no_argument.html [ Pass Failure ]
[ BigSur+ x86_64 ] imported/w3c/web-platform-tests/html/browsers/browsing-the-web/history-traversal/persisted-user-state-restoration/resume-timer-on-history-back.html [ Pass Failure ]
[ BigSur+ x86_64 ] imported/w3c/web-platform-tests/html/cross-origin-opener-policy/iframe-popup-same-origin-allow-popups-to-same-origin-allow-popups.https.html [ Pass Failure ]
[ BigSur+ x86_64 ] scrollbars/scrolling-backward-by-page-accounting-bottom-fixed-elements-on-keyboard-spacebar.html [ Pass Failure ]
[ BigSur+ x86_64 ] scrollbars/scrolling-by-page-ignoring-transparent-fixed-elements-on-keyboard-spacebar.html [ Pass Failure ]
[ BigSur+ x86_64 ] scrollbars/scrolling-by-page-on-keyboard-spacebar.html [ Pass Failure ]
[ Venutra+ ] platform/mac-wk2/plugins/asynchronous-destroy-before-initialization.html [ Pass Failure ]
[ Monterey+ x86_64 ] media/restricted-audio-playback-with-multiple-settimeouts.html [ Pass Failure ]
[ BigSur+ ] imported/w3c/web-platform-tests/workers/SharedWorker_dataUrl.html [ Pass Failure ]
[ Bigsur+ x86_64 ] imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https.html [ Pass Failure ]
[ Monterey+ Debug x86_64 ] imported/w3c/web-platform-tests/html/cross-origin-opener-policy/tentative/restrict-properties/iframe-popup-to-so.https.html [ Pass Failure ]
[ Monterey+ Debug x86_64 ] imported/w3c/web-platform-tests/html/cross-origin-opener-policy/tentative/restrict-properties/iframe-popup.https.html [ Pass Failure ]
[ Ventura+ Debug x86_64 ] webrtc/datachannel/bufferedAmountLowThreshold-default.html [ Pass Failure ]
[ Ventura Debug x86_64 ] editing/caret/color-span-inside-editable-background.html [ Pass ImageOnlyFailure ]
[ Ventura+ x86_64 ] fast/css/text-overflow-input-focus-placeholder.html [ Pass ImageOnlyFailure ]
[ BigSur+ x86_64 ] webrtc/captureCanvas-webrtc-software-h264-high.html [ Pass Failure Timeout ]
[ Ventura+ x86_64 ] webgl/pending/conformance/glsl/misc/swizzle-as-lvalue.html [ Pass Timeout ]
[ Ventura+ x86_64 Debug ] webgl/2.0.0/conformance2/textures/canvas_sub_rectangle/tex-3d-rg32f-rg-float.html [ Pass Timeout ]
[ Ventura+ Debug x86_64 ] webgl/2.0.0/conformance/state/gl-object-get-calls.html [ Pass Timeout ]
[ Monterey+ Debug x86_64 ] webgl/2.0.0/conformance/context/context-release-with-workers.html [ Pass Timeout ]
[ BigSur+ x86_64 ] resize-observer/resize-observer-with-zoom.html [ Pass Timeout ]
[ BigSur+ Debug ] imported/w3c/web-platform-tests/fetch/api/redirect/redirect-back-to-original-origin.any.html [ Pass Crash ]
[ Monterey+ Debug x86_64 ] storage/indexeddb/request-with-null-open-db-request.html [ Pass Crash ]
[ BigSur+ ] imported/w3c/web-platform-tests/service-workers/cache-storage/cross-partition.https.tentative.html [ Pass Failure Crash ]

mac
[ Ventura+ ] animations/animation-events-not-cancelable.html [ Pass Failure ]
[ Bigsur+ ] animations/remove-syncing-animation.html [ Pass Failure ]
[ BigSur+ ] css3/scroll-snap/scroll-padding-mainframe-paging.html [ Pass Failure ]
[ BigSur+ ] css3/scroll-snap/scroll-padding-overflow-paging.html [ Pass Failure ]
[ Ventura+ ] fast/events/before-unload-navigate-different-window.html [ Pass Failure ]
[ Bigsur+ ] http/tests/media/fairplay/fps-mse-unmuxed-key-rotation.html [ Pass Failure ]
[ Monterey+ x86_64 ] imported/w3c/web-platform-tests/html/canvas/element/manual/wide-gamut-canvas/canvas-display-p3-drawImage-ImageBitmap-video.html [ Pass Failure Timeout ]
[ Venutra+ x86_64 ] imported/w3c/web-platform-tests/html/browsers/history/the-history-interface/history_go_undefined.html [ Pass Failure ]
[ BigSur+ x86_64 ] imported/w3c/web-platform-tests/html/browsers/browsing-the-web/navigating-across-documents/javascript-url-security-check-failure.sub.html [ Pass Failure ]
[ Monterey+ x86_64 ] media/media-fragments/TC0086.html [ Pass Failure ]
[ Monterey+ x86_64 ] media/media-fragments/TC0088.html [ Pass Failure ]
[ Monterey+ x86_64 ] media/media-fragments/TC0089.html [ Pass Failure ]
[ Monterey+ x86_64 ] media/media-fragments/TC0091.html [ Pass Failure ]
[ BigSur+ x86_64 ] media/track/track-cues-enter-exit.html [ Pass Failure ]
[ Monterey+ x86_64 ] inspector/animation/lifecycle-css-transition.html [ Pass Failure ]
[ BigSur+ ] imported/w3c/web-platform-tests/html/semantics/embedded-content/media-elements/autoplay-allowed-by-feature-policy-attribute-redirect-on-load.https.sub.html [ Pass Failure ]
[ Ventura+ ] fast/text/synthetic-bold-transformed.html [ Pass ImageOnlyFailure ]
[ Monterey+ x86_64 ] media/video-object-fit.html [ Pass ImageOnlyFailure ]
[ Monterey+ x86_64 ] media/track/track-cues-cuechange.html [ Pass Timeout Failure ]
[ BigSur+ x86_64 ] js/promises-tests/promises-tests-2-3-3.html [ Pass Timeout ]

mac-wk1
[ BigSur+ Debug x86_64 ] editing/execCommand/delete-non-editable-range-crash.html [ Pass Timeout ]


and that should take care of almost everything occurring on Ventura so we can bring up EWS.

[EWS]

Test gardening commit 257459@main (e87eb5586000): <https://commits.webkit.org/257459@main>

Reviewed commits have been landed. Closing PR #7249 and removing active labels.

[Karl Rackler]

Update test expectation:
fast/repaint/fixed-move-after-keyboard-scroll.html

[EWS]

Test gardening commit 267206@main (969f99e0f53b): <https://commits.webkit.org/267206@main>

Reviewed commits have been landed. Closing PR #16994 and removing active labels.

[Karl Rackler]

Update test expectations for editing/execCommand/delete-non-editable-range-crash.html

[EWS]

Test gardening commit 286640@main (c769d7d435e0): <https://commits.webkit.org/286640@main>

Reviewed commits have been landed. Closing PR #36701 and removing active labels.