Summary: | run-webkit-tests should detect w3c test resource files | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | youenn fablet <youennf> | ||||||||
Component: | Tools / Tests | Assignee: | youenn fablet <youennf> | ||||||||
Status: | RESOLVED FIXED | ||||||||||
Severity: | Normal | CC: | ap, cdumez, commit-queue, glenn, lforschler, rniwa | ||||||||
Priority: | P2 | ||||||||||
Version: | WebKit Nightly Build | ||||||||||
Hardware: | Unspecified | ||||||||||
OS: | Unspecified | ||||||||||
Attachments: |
|
Description
youenn fablet
2016-08-29 02:27:03 PDT
Created attachment 287262 [details]
Patch
Created attachment 287263 [details]
Patch
Comment on attachment 287263 [details]
Patch
On my setup when doing --print-expectations before and after this patch, the list of runned test is the same.
Comment on attachment 287263 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=287263&action=review > Tools/ChangeLog:4 > + run-webkit-tests should detect w3c test resource files > + https://bugs.webkit.org/show_bug.cgi?id=161307 Would it be feasible to make W3C tests adhere to an automation friendly layout, such as using a resources folder? Seems like everyone would have this problem. > Would it be feasible to make W3C tests adhere to an automation friendly
> layout, such as using a resources folder? Seems like everyone would have
> this problem.
Given the number of files that are skipped here, the refactoring time would be quite important.
The wpt test infrastructure allows generating a manifest file containing all tests (using a python script, see web-platform-tests/tools, probably in manifest.py file).
We could reuse that tool and that file.
The downside is that whenever a WebKit patch is adding a test or a test-suite to LayoutTests/imported/w3c, this file would need to be regenerated/updated so that the bots do not skip the new tests.
It might be easier to only update which files are test/not-test when doing a full wpt resync, typically doing the following:
- generate the manifest file using wpt script
- generate our own skip-list using the manifest
w3c test importer could do that easily I think.
Ping review? Comment on attachment 287263 [details]
Patch
We should really fix this in the upstream repository though. Having to track random source files like this isn't a scalable solution in the long term.
(In reply to comment #7) > Comment on attachment 287263 [details] > Patch > > We should really fix this in the upstream repository though. Having to > track random source files like this isn't a scalable solution in the long > term. The w3c importer could handle the generation of this file based on wpt test manifest. Comment on attachment 287263 [details] Patch Rejecting attachment 287263 [details] from commit-queue. Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-02', 'apply-attachment', '--no-update', '--non-interactive', 287263, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit Last 500 characters of output: eLog Hunk #1 succeeded at 1 with fuzz 3. patching file LayoutTests/imported/w3c/ChangeLog Hunk #1 succeeded at 1 with fuzz 3. patching file LayoutTests/TestExpectations Hunk #2 FAILED at 259. 1 out of 2 hunks FAILED -- saving rejects to file LayoutTests/TestExpectations.rej patching file LayoutTests/imported/w3c/resources/resource-files.json Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force', '--reviewer', u'Ryosuke Niwa']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit Full output: http://webkit-queues.webkit.org/results/1999349 Created attachment 287857 [details]
Patch for landing
Comment on attachment 287857 [details] Patch for landing Clearing flags on attachment: 287857 Committed r205399: <http://trac.webkit.org/changeset/205399> All reviewed patches have been landed. Closing bug. (In reply to comment #8) > (In reply to comment #7) > > Comment on attachment 287263 [details] > > Patch > > > > We should really fix this in the upstream repository though. Having to > > track random source files like this isn't a scalable solution in the long > > term. > > The w3c importer could handle the generation of this file based on wpt test > manifest. But that's really silly. We shouldn't have dozens of exceptions like this. WPT should follow the convention to always put files under "resources" directory instead. (In reply to comment #13) > (In reply to comment #8) > WPT should follow the convention to always put files under "resources" > directory instead. This issue could be brought up, but I think it is a lost cause. At this point, I don't know who can commit time to update the tests, find reviewers, update the lint tool, update the manifest generator script. |