Bug 143090 – run-webkit-tests should have an option to run new tests

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[youenn fablet]

It would be nice to run all tests modified/created through run-webkit-tests.
That may help developers as well as provide limited EWS bots the ability to check a small but critical amount of tests.

[youenn fablet]

Created attachment 249482 [details]
Patch

[youenn fablet]

Uploaded a patch at bug 143093 to show a potential use of --new-tests and --pass-if-no-test options.

[Alexey Proskuryakov]

Comment on attachment 249482 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=249482&action=review

> Tools/ChangeLog:8
> +        Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository.

Can't we figure that out outside run-webkit-tests? I'm not sure if this is a mode that engineers need to have access to.

EWS can just pass the list of new tests if any.

[Csaba Osztrogonác]

(In reply to comment #3)
> Comment on attachment 249482 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=249482&action=review
> 
> > Tools/ChangeLog:8
> > +        Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository.
> 
> Can't we figure that out outside run-webkit-tests? I'm not sure if this is a
> mode that engineers need to have access to.
> 
> EWS can just pass the list of new tests if any.


EWS can pass list of the tests to run-webkit-tests as command 
line arguments directly and in a file with --test-list option.

But I'm not sure how are test expectations handled in this 
cases. AFAIR skipped tests are run too, but I'm not sure.

[youenn fablet]

(In reply to comment #4)
> (In reply to comment #3)
> > Comment on attachment 249482 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=249482&action=review
> > 
> > > Tools/ChangeLog:8
> > > +        Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository.
> > 
> > Can't we figure that out outside run-webkit-tests? I'm not sure if this is a
> > mode that engineers need to have access to.
> > 
> > EWS can just pass the list of new tests if any.

Combined with git, this option runs all tests added/modified by the set of commits compared to the remote origin master. Handy for branch development, rebasing -i...

It might also be useful when reimporting W3C tests.

> EWS can pass list of the tests to run-webkit-tests as command 
> line arguments directly and in a file with --test-list option.
> 
> But I'm not sure how are test expectations handled in this 
> cases. AFAIR skipped tests are run too, but I'm not sure.

Good point.
It should be checked as part of integration/unit tests that should be added.

[youenn fablet]

> > EWS can pass list of the tests to run-webkit-tests as command 
> > line arguments directly and in a file with --test-list option.
> > 
> > But I'm not sure how are test expectations handled in this 
> > cases. AFAIR skipped tests are run too, but I'm not sure.
> 
> Good point.
> It should be checked as part of integration/unit tests that should be added.

There is the --skipped=always/default option which enables to run/skip the tests here.

It all ends up whether we want to expose this option in rwt or not...

I see some value to it as exposed previously.
The main downsides I see with that option is that there is the need to handle within rwt the case of runs with no tests. This patch introduces the --pass-if-no-option for that purpose. But this case would probably be best handled within EWS.

Thoughts?

[Alexey Proskuryakov]

Another complication is layering. Currently, run-webkit-tests knows nothing about version control, and it would be confusing if it started to use that just for one command.

If enough people want a special mode to run just the new/modified tests, webkit-patch is unclean enough to have that added too, I think, but probably not run-webkit-tests.

[youenn fablet]

(In reply to comment #7)
> Another complication is layering. Currently, run-webkit-tests knows nothing
> about version control, and it would be confusing if it started to use that
> just for one command.
> 
> If enough people want a special mode to run just the new/modified tests,
> webkit-patch is unclean enough to have that added too, I think, but probably
> not run-webkit-tests.

Then we should directly go to bug 143093...