It would be nice to run all tests modified/created through run-webkit-tests. That may help developers as well as provide limited EWS bots the ability to check a small but critical amount of tests.
Created attachment 249482 [details] Patch
Uploaded a patch at bug 143093 to show a potential use of --new-tests and --pass-if-no-test options.
Comment on attachment 249482 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=249482&action=review > Tools/ChangeLog:8 > + Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository. Can't we figure that out outside run-webkit-tests? I'm not sure if this is a mode that engineers need to have access to. EWS can just pass the list of new tests if any.
(In reply to comment #3) > Comment on attachment 249482 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=249482&action=review > > > Tools/ChangeLog:8 > > + Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository. > > Can't we figure that out outside run-webkit-tests? I'm not sure if this is a > mode that engineers need to have access to. > > EWS can just pass the list of new tests if any. EWS can pass list of the tests to run-webkit-tests as command line arguments directly and in a file with --test-list option. But I'm not sure how are test expectations handled in this cases. AFAIR skipped tests are run too, but I'm not sure.
(In reply to comment #4) > (In reply to comment #3) > > Comment on attachment 249482 [details] > > Patch > > > > View in context: > > https://bugs.webkit.org/attachment.cgi?id=249482&action=review > > > > > Tools/ChangeLog:8 > > > + Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository. > > > > Can't we figure that out outside run-webkit-tests? I'm not sure if this is a > > mode that engineers need to have access to. > > > > EWS can just pass the list of new tests if any. Combined with git, this option runs all tests added/modified by the set of commits compared to the remote origin master. Handy for branch development, rebasing -i... It might also be useful when reimporting W3C tests. > EWS can pass list of the tests to run-webkit-tests as command > line arguments directly and in a file with --test-list option. > > But I'm not sure how are test expectations handled in this > cases. AFAIR skipped tests are run too, but I'm not sure. Good point. It should be checked as part of integration/unit tests that should be added.
> > EWS can pass list of the tests to run-webkit-tests as command > > line arguments directly and in a file with --test-list option. > > > > But I'm not sure how are test expectations handled in this > > cases. AFAIR skipped tests are run too, but I'm not sure. > > Good point. > It should be checked as part of integration/unit tests that should be added. There is the --skipped=always/default option which enables to run/skip the tests here. It all ends up whether we want to expose this option in rwt or not... I see some value to it as exposed previously. The main downsides I see with that option is that there is the need to handle within rwt the case of runs with no tests. This patch introduces the --pass-if-no-option for that purpose. But this case would probably be best handled within EWS. Thoughts?
Another complication is layering. Currently, run-webkit-tests knows nothing about version control, and it would be confusing if it started to use that just for one command. If enough people want a special mode to run just the new/modified tests, webkit-patch is unclean enough to have that added too, I think, but probably not run-webkit-tests.
(In reply to comment #7) > Another complication is layering. Currently, run-webkit-tests knows nothing > about version control, and it would be confusing if it started to use that > just for one command. > > If enough people want a special mode to run just the new/modified tests, > webkit-patch is unclean enough to have that added too, I think, but probably > not run-webkit-tests. Then we should directly go to bug 143093...