Summary: | run-webkit-tests should have an option to run new tests | ||||||
---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | youenn fablet <youennf> | ||||
Component: | Tools / Tests | Assignee: | youenn fablet <youennf> | ||||
Status: | RESOLVED WONTFIX | ||||||
Severity: | Normal | CC: | ap, commit-queue, glenn, ossy | ||||
Priority: | P2 | ||||||
Version: | 528+ (Nightly build) | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Bug Depends on: | |||||||
Bug Blocks: | 118294 | ||||||
Attachments: |
|
Description
youenn fablet
2015-03-26 03:52:02 PDT
Created attachment 249482 [details]
Patch
Uploaded a patch at bug 143093 to show a potential use of --new-tests and --pass-if-no-test options. Comment on attachment 249482 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=249482&action=review > Tools/ChangeLog:8 > + Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository. Can't we figure that out outside run-webkit-tests? I'm not sure if this is a mode that engineers need to have access to. EWS can just pass the list of new tests if any. (In reply to comment #3) > Comment on attachment 249482 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=249482&action=review > > > Tools/ChangeLog:8 > > + Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository. > > Can't we figure that out outside run-webkit-tests? I'm not sure if this is a > mode that engineers need to have access to. > > EWS can just pass the list of new tests if any. EWS can pass list of the tests to run-webkit-tests as command line arguments directly and in a file with --test-list option. But I'm not sure how are test expectations handled in this cases. AFAIR skipped tests are run too, but I'm not sure. (In reply to comment #4) > (In reply to comment #3) > > Comment on attachment 249482 [details] > > Patch > > > > View in context: > > https://bugs.webkit.org/attachment.cgi?id=249482&action=review > > > > > Tools/ChangeLog:8 > > > + Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository. > > > > Can't we figure that out outside run-webkit-tests? I'm not sure if this is a > > mode that engineers need to have access to. > > > > EWS can just pass the list of new tests if any. Combined with git, this option runs all tests added/modified by the set of commits compared to the remote origin master. Handy for branch development, rebasing -i... It might also be useful when reimporting W3C tests. > EWS can pass list of the tests to run-webkit-tests as command > line arguments directly and in a file with --test-list option. > > But I'm not sure how are test expectations handled in this > cases. AFAIR skipped tests are run too, but I'm not sure. Good point. It should be checked as part of integration/unit tests that should be added. > > EWS can pass list of the tests to run-webkit-tests as command
> > line arguments directly and in a file with --test-list option.
> >
> > But I'm not sure how are test expectations handled in this
> > cases. AFAIR skipped tests are run too, but I'm not sure.
>
> Good point.
> It should be checked as part of integration/unit tests that should be added.
There is the --skipped=always/default option which enables to run/skip the tests here.
It all ends up whether we want to expose this option in rwt or not...
I see some value to it as exposed previously.
The main downsides I see with that option is that there is the need to handle within rwt the case of runs with no tests. This patch introduces the --pass-if-no-option for that purpose. But this case would probably be best handled within EWS.
Thoughts?
Another complication is layering. Currently, run-webkit-tests knows nothing about version control, and it would be confusing if it started to use that just for one command. If enough people want a special mode to run just the new/modified tests, webkit-patch is unclean enough to have that added too, I think, but probably not run-webkit-tests. (In reply to comment #7) > Another complication is layering. Currently, run-webkit-tests knows nothing > about version control, and it would be confusing if it started to use that > just for one command. > > If enough people want a special mode to run just the new/modified tests, > webkit-patch is unclean enough to have that added too, I think, but probably > not run-webkit-tests. Then we should directly go to bug 143093... |