Bug 149768 – Import some Blink tests

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Jon Honeycutt]

We should import a subset of Blink layout tests whose expected results match WebKit’s actual results.

[Jon Honeycutt]

Created attachment 262360 [details]
Patch

[Andy Estes]

Comment on attachment 262360 [details]
Patch

Did you consider placing these in LayoutTests/imported/blink/ so that it's clear these are imported from another project?

[Jon Honeycutt]

(In reply to comment #2)
> Comment on attachment 262360 [details]
> Patch
> 
> Did you consider placing these in LayoutTests/imported/blink/ so that it's
> clear these are imported from another project?

I'll do that. Thanks for the review!

[Build Bot]

Comment on attachment 262360 [details]
Patch

Attachment 262360 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/238121

Number of test failures exceeded the failure limit.

[Build Bot]

Created attachment 262366 [details]
Archive of layout-test-results from ews102 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews102  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[Build Bot]

Comment on attachment 262360 [details]
Patch

Attachment 262360 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/238174

Number of test failures exceeded the failure limit.

[Build Bot]

Created attachment 262370 [details]
Archive of layout-test-results from ews107 for mac-mavericks-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-mavericks-wk2  Platform: Mac OS X 10.9.5

[Jon Honeycutt]

Created attachment 262422 [details]
Patch v2

Try removing some tests that pass locally but fail on the bots.

[WebKit Commit Bot]

Attachment 262422 [details] did not pass style-queue:


ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2802:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2803:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2804:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2805:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2806:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2807:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2808:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2809:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2810:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2811:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2812:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2813:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2814:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2815:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2816:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2817:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2818:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2819:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2820:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2821:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2822:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2823:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2824:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2825:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2826:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2827:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2828:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2829:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2830:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2831:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2832:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2833:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2835:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2836:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2837:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2838:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2841:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2842:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2843:  Path does not exist.  [test/expectations] [5]
ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2844:  Path does not exist.  [test/expectations] [5]
Total errors found: 40 in 2636 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Build Bot]

Comment on attachment 262422 [details]
Patch v2

Attachment 262422 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/246003

New failing tests:
imported/blink/compositing/canvas-with-object-fit-contain-in-composited-layer.html
imported/blink/fast/canvas/canvas-partial-invalidation-zoomed.html
svg/custom/use-href-change-local-to-invalid-remote.html

[Build Bot]

Created attachment 262424 [details]
Archive of layout-test-results from ews100 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[Build Bot]

Comment on attachment 262422 [details]
Patch v2

Attachment 262422 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/246015

New failing tests:
svg/custom/use-href-change-local-to-invalid-remote.html
imported/blink/printing/ellipsis-printing-style.html

[Build Bot]

Created attachment 262425 [details]
Archive of layout-test-results from ews105 for mac-mavericks-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105  Port: mac-mavericks-wk2  Platform: Mac OS X 10.9.5

[Jon Honeycutt]

Created attachment 262457 [details]
Patch v3

[WebKit Commit Bot]

Attachment 262457 [details] did not pass style-queue:


ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2833:  Path does not exist.  [test/expectations] [5]
Total errors found: 1 in 2630 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Jon Honeycutt]

Created attachment 262483 [details]
Patch v4

Corrects my mistaken move of platform-specific iOS results to imported/blink/platform/. Removes another flakey test.

[Jon Honeycutt]

Landed in <http://trac.webkit.org/changeset/190629>.

[Simon Fraser (smfr)]

I am not convinced that bulk-landing tests like this is useful.

[Jon Honeycutt]

Mac test gardening in <http://trac.webkit.org/changeset/190654>.

[Jon Honeycutt]

Windows test gardening in <http://trac.webkit.org/changeset/190656>.

[Jon Honeycutt]

More Windows test gardening in <http://trac.webkit.org/changeset/190657>.

[Jon Honeycutt]

iOS test gardening in <http://trac.webkit.org/changeset/190659>.

[youenn fablet]

If you plan to keep up with that work, and continue syncing these tests, would it make sense to integrate with current test importer (like Tools/Scripts/import-w3c-tests script)?

If so, with a few additional edits, the main edit would be  to add blink git repo in LayoutTests/imported/w3c/resources/TestRepositories.

[Jon Honeycutt]

(In reply to comment #23)
> If you plan to keep up with that work, and continue syncing these tests,
> would it make sense to integrate with current test importer (like
> Tools/Scripts/import-w3c-tests script)?
> 
> If so, with a few additional edits, the main edit would be  to add blink git
> repo in LayoutTests/imported/w3c/resources/TestRepositories.

Thanks! I wasn't aware of this.

I'm not sure whether we'll reach the point of automatically importing tests. The tests that have been landed so far were carefully curated, because there are thousands of tests in Blink that time out, flakily time out, or flakily fail when run in WebKit.

[youenn fablet]

> I'm not sure whether we'll reach the point of automatically importing tests.

Me neither, but getting closer and closer is good.

> The tests that have been landed so far were carefully curated, because there
> are thousands of tests in Blink that time out, flakily time out, or flakily
> fail when run in WebKit.

LayoutTests/imported/w3c/resources/ImportExpectations allows controlling tests imported by import-w3c-scripts, which could serve your purpose here.

I guess test_downloader.py should be upgraded to use a "subpath" parameter to locate blink repository test subfolder, import-w3c-tests should be renamed import-tests and so on...

Another point on blink tests: LayoutTests/imported/blink/imported tests should probably be moved to LayoutTests/imported/w3c.

Ideally these tests should be reimported by updating ImportExpectations and running "Tools/Scripts/import-w3c-tests". This could probably be done once I finished resyncing LayoutTests/imported/w3c.

Or they can reimported using something like "Tools/Scripts/import-w3c-tests -t web-platform-tests/shadow-dom -t csswg-test/css-writing-modes-3"