We should import a subset of Blink layout tests whose expected results match WebKit’s actual results.
Created attachment 262360 [details] Patch
Comment on attachment 262360 [details] Patch Did you consider placing these in LayoutTests/imported/blink/ so that it's clear these are imported from another project?
(In reply to comment #2) > Comment on attachment 262360 [details] > Patch > > Did you consider placing these in LayoutTests/imported/blink/ so that it's > clear these are imported from another project? I'll do that. Thanks for the review!
Comment on attachment 262360 [details] Patch Attachment 262360 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/238121 Number of test failures exceeded the failure limit.
Created attachment 262366 [details] Archive of layout-test-results from ews102 for mac-mavericks The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews102 Port: mac-mavericks Platform: Mac OS X 10.9.5
Comment on attachment 262360 [details] Patch Attachment 262360 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/238174 Number of test failures exceeded the failure limit.
Created attachment 262370 [details] Archive of layout-test-results from ews107 for mac-mavericks-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews107 Port: mac-mavericks-wk2 Platform: Mac OS X 10.9.5
Created attachment 262422 [details] Patch v2 Try removing some tests that pass locally but fail on the bots.
Attachment 262422 [details] did not pass style-queue: ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2802: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2803: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2804: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2805: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2806: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2807: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2808: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2809: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2810: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2811: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2812: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2813: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2814: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2815: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2816: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2817: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2818: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2819: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2820: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2821: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2822: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2823: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2824: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2825: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2826: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2827: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2828: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2829: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2830: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2831: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2832: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2833: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2835: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2836: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2837: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2838: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2841: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2842: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2843: Path does not exist. [test/expectations] [5] ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2844: Path does not exist. [test/expectations] [5] Total errors found: 40 in 2636 files If any of these errors are false positives, please file a bug against check-webkit-style.
Comment on attachment 262422 [details] Patch v2 Attachment 262422 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/246003 New failing tests: imported/blink/compositing/canvas-with-object-fit-contain-in-composited-layer.html imported/blink/fast/canvas/canvas-partial-invalidation-zoomed.html svg/custom/use-href-change-local-to-invalid-remote.html
Created attachment 262424 [details] Archive of layout-test-results from ews100 for mac-mavericks The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews100 Port: mac-mavericks Platform: Mac OS X 10.9.5
Comment on attachment 262422 [details] Patch v2 Attachment 262422 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/246015 New failing tests: svg/custom/use-href-change-local-to-invalid-remote.html imported/blink/printing/ellipsis-printing-style.html
Created attachment 262425 [details] Archive of layout-test-results from ews105 for mac-mavericks-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews105 Port: mac-mavericks-wk2 Platform: Mac OS X 10.9.5
Created attachment 262457 [details] Patch v3
Attachment 262457 [details] did not pass style-queue: ERROR: LayoutTests/platform/ios-simulator/TestExpectations:2833: Path does not exist. [test/expectations] [5] Total errors found: 1 in 2630 files If any of these errors are false positives, please file a bug against check-webkit-style.
Created attachment 262483 [details] Patch v4 Corrects my mistaken move of platform-specific iOS results to imported/blink/platform/. Removes another flakey test.
Landed in <http://trac.webkit.org/changeset/190629>.
I am not convinced that bulk-landing tests like this is useful.
Mac test gardening in <http://trac.webkit.org/changeset/190654>.
Windows test gardening in <http://trac.webkit.org/changeset/190656>.
More Windows test gardening in <http://trac.webkit.org/changeset/190657>.
iOS test gardening in <http://trac.webkit.org/changeset/190659>.
If you plan to keep up with that work, and continue syncing these tests, would it make sense to integrate with current test importer (like Tools/Scripts/import-w3c-tests script)? If so, with a few additional edits, the main edit would be to add blink git repo in LayoutTests/imported/w3c/resources/TestRepositories.
(In reply to comment #23) > If you plan to keep up with that work, and continue syncing these tests, > would it make sense to integrate with current test importer (like > Tools/Scripts/import-w3c-tests script)? > > If so, with a few additional edits, the main edit would be to add blink git > repo in LayoutTests/imported/w3c/resources/TestRepositories. Thanks! I wasn't aware of this. I'm not sure whether we'll reach the point of automatically importing tests. The tests that have been landed so far were carefully curated, because there are thousands of tests in Blink that time out, flakily time out, or flakily fail when run in WebKit.
> I'm not sure whether we'll reach the point of automatically importing tests. Me neither, but getting closer and closer is good. > The tests that have been landed so far were carefully curated, because there > are thousands of tests in Blink that time out, flakily time out, or flakily > fail when run in WebKit. LayoutTests/imported/w3c/resources/ImportExpectations allows controlling tests imported by import-w3c-scripts, which could serve your purpose here. I guess test_downloader.py should be upgraded to use a "subpath" parameter to locate blink repository test subfolder, import-w3c-tests should be renamed import-tests and so on... Another point on blink tests: LayoutTests/imported/blink/imported tests should probably be moved to LayoutTests/imported/w3c. Ideally these tests should be reimported by updating ImportExpectations and running "Tools/Scripts/import-w3c-tests". This could probably be done once I finished resyncing LayoutTests/imported/w3c. Or they can reimported using something like "Tools/Scripts/import-w3c-tests -t web-platform-tests/shadow-dom -t csswg-test/css-writing-modes-3"