Bug 215459 - Migrate some Apple internal Mac expectations here into WebKit sources
Summary: Migrate some Apple internal Mac expectations here into WebKit sources
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Darin Adler
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-08-13 12:32 PDT by Darin Adler
Modified: 2020-08-31 13:15 PDT (History)
15 users (show)

See Also:


Attachments
Patch (28 bytes, patch)
2020-08-13 12:48 PDT, Darin Adler
no flags Details | Formatted Diff | Diff
Patch (524.89 KB, patch)
2020-08-13 12:50 PDT, Darin Adler
jbedard: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Darin Adler 2020-08-13 12:32:33 PDT
Migrate some Apple internal Mac expectations here into WebKit sources
Comment 1 Darin Adler 2020-08-13 12:48:39 PDT Comment hidden (obsolete)
Comment 2 Darin Adler 2020-08-13 12:48:42 PDT Comment hidden (obsolete)
Comment 3 Darin Adler 2020-08-13 12:50:58 PDT
Created attachment 406532 [details]
Patch
Comment 4 Darin Adler 2020-08-13 13:06:04 PDT
Comment on attachment 406532 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=406532&action=review

> LayoutTests/platform/mac-wk2/TestExpectations:1211
> +[ arm64 ] http/tests/security/storage-blocking-loosened-plugin.html [ Skip ]

As suggested by Alexey, can and will break this into two lines to get rid of the run-webkit-tests lint warning:

[ arm64 Release ] http/tests/security/storage-blocking-loosened-plugin.html [ Skip ]
[ arm64 Debug ] http/tests/security/storage-blocking-loosened-plugin.html [ Skip ]
Comment 5 Darin Adler 2020-08-13 15:07:20 PDT
This is associated with the wrong bug in Radar. How do I fix that?
Comment 6 Ryan Haddad 2020-08-13 15:43:33 PDT
(In reply to Darin Adler from comment #5)
> This is associated with the wrong bug in Radar. How do I fix that?
Paste the rdar URL in a new comment and I believe the importer should take care of it.
Comment 7 Darin Adler 2020-08-13 16:38:11 PDT
rdar://67026638
Comment 8 Darin Adler 2020-08-13 16:39:03 PDT
Looking for review on this; it will be easier to merge if it’s done before people make a lot of other changes to test expectations.
Comment 9 Jonathan Bedard 2020-08-13 16:56:46 PDT
Comment on attachment 406532 [details]
Patch

Looks like our flakey test is not one we had an expectation change for, so let's land!
Comment 10 Darin Adler 2020-08-13 17:26:02 PDT
Committed r265639: <https://trac.webkit.org/changeset/265639>
Comment 11 Alexey Proskuryakov 2020-08-15 20:40:53 PDT
Comment on attachment 406532 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=406532&action=review

This is a very large patch that mixes different kinds of changes. I think that some of these are desirable, but not all.

Only did a spot check, as this is so large, and already landed.

> LayoutTests/platform/mac-wk1/TestExpectations:1087
> +# <rdar://problem/61461935> [ macOS wk2 ] fast/scrolling/rtl-point-in-iframe.html is flaky failing
> +[ Mojave ] fast/scrolling/rtl-point-in-iframe.html [ Pass Failure ]

This isn't failing anywhere, open source or internal.

> LayoutTests/platform/mac/TestExpectations:1944
> +# <rdar://problem/31017631> Stress GC bot failing LayoutTest step with timeout while running WebCryptoAPI/derive_bits_keys tests
> +[ Debug ] imported/w3c/web-platform-tests/WebCryptoAPI/derive_bits_keys [ Skip ]

How is this a reason to skip the test on all Mac bots?

> LayoutTests/platform/mac/TestExpectations:1948
> +# The navigator.language tests rely on functionality only available in recent releases of macOS Catalina and onward.
> +# <rdar://problem/59461792> REGRESSION: [ Mac iOS ] ( r254389 - r254411 ) fast/text/international/system-language/declarative-language.html is failing internally
> +fast/text/international/system-language/declarative-language.html [ Pass Failure ]

The expectation doesn't match the comment, and neither matches the actual behavior.

> LayoutTests/platform/mac/TestExpectations:1955
> +# <rdar://problem/59636115> REGRESSION: [ iOS & macOS ] two imported/w3c/web-platform-tests/WebCryptoAPI are failing
> +imported/w3c/web-platform-tests/WebCryptoAPI/import_export/rsa_importKey.https.worker.html [ Pass Failure ]
> +imported/w3c/web-platform-tests/WebCryptoAPI/import_export/test_rsa_importKey.https.html [ Pass Failure ]

This is an OS specific regression, so marking the tests flaky unconditionally regresses test coverage.

> LayoutTests/platform/mac/TestExpectations:2041
> +# <rdar://problem/61114827> [ Stress GC ] http/tests/navigation/page-cache-fontfaceset.html is flaky crashing.
> +[ Mojave ] http/tests/navigation/page-cache-fontfaceset.html [ Pass Crash ]

All of these are a very internal hack, enabled by running Stress GC on Mojave only. This is seems confusing and misplaced in open source expectations.

> LayoutTests/platform/mac/TestExpectations:2203
> +# rdar://66802945 (REGRESSION (20A2323e-20A234b): [ Layout Test] 12 canvas layout tests are a constant failure)
> +[ arm64 ] canvas/philip/tests/2d.line.miter.obtuse.html [ Failure ]
> +[ arm64 ] canvas/philip/tests/2d.path.arc.selfintersect.1.html [ Failure ]
> +[ arm64 ] canvas/philip/tests/2d.path.arc.shape.3.html [ Failure ]
> +[ arm64 ] fast/canvas/canvas-ellipse-zero-lineto.html [ Failure ]
> +[ arm64 ] imported/w3c/canvas/2d.line.miter.obtuse.html [ Failure ]
> +[ arm64 ] imported/w3c/canvas/2d.path.arc.selfintersect.1.html [ Failure ]
> +[ arm64 ] imported/w3c/canvas/2d.path.arc.shape.3.html [ Failure ]
> +[ arm64 ] imported/w3c/web-platform-tests/html/canvas/element/line-styles/2d.line.miter.obtuse.html [ Failure ]
> +[ arm64 ] imported/w3c/web-platform-tests/html/canvas/element/path-objects/2d.path.arc.selfintersect.1.html [ Failure ]
> +[ arm64 ] imported/w3c/web-platform-tests/html/canvas/element/path-objects/2d.path.arc.shape.3.html [ Failure ]

As these are due to an issue below WebKit, moving the expectations to open source makes them harder to keep track of, I think.
Comment 12 Darin Adler 2020-08-16 12:32:49 PDT
I can see that you and I have very different ideas about open source vs. internal. As far as I am concerned, only secret changes should be internal.
Comment 13 Alexey Proskuryakov 2020-08-16 13:33:31 PDT
I agree with this as a basic philosophy! What I don't believe in is that mechanically moving everything into open source is the right first step. This is an essentially non-reversible move that loses information.

Tracking issues that are below WebKit and are only affecting future OSes, possibly for a short period of time that isn't aligned with public beta schedule, is somewhat of a gray area for me.
Comment 14 Alexey Proskuryakov 2020-08-31 13:15:36 PDT
Comment on attachment 406532 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=406532&action=review

> LayoutTests/ChangeLog:23
> +        * platform/mac-bigsur-wk1/editing/pasteboard/drop-text-without-selection-expected.txt: Added.

We all missed the big issue. Expectations for latest macOS live in platform/mac (because that's the present and also the likely future), and only older OSes get named directories.

Will get these updated.