Bug 143090

Summary: run-webkit-tests should have an option to run new tests
Product: WebKit Reporter: youenn fablet <youennf>
Component: Tools / TestsAssignee: youenn fablet <youennf>
Status: RESOLVED WONTFIX    
Severity: Normal CC: ap, commit-queue, glenn, ossy
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on:    
Bug Blocks: 118294    
Attachments:
Description Flags
Patch none

Description youenn fablet 2015-03-26 03:52:02 PDT
It would be nice to run all tests modified/created through run-webkit-tests.
That may help developers as well as provide limited EWS bots the ability to check a small but critical amount of tests.
Comment 1 youenn fablet 2015-03-26 04:28:18 PDT
Created attachment 249482 [details]
Patch
Comment 2 youenn fablet 2015-03-26 04:41:17 PDT
Uploaded a patch at bug 143093 to show a potential use of --new-tests and --pass-if-no-test options.
Comment 3 Alexey Proskuryakov 2015-03-26 10:00:46 PDT
Comment on attachment 249482 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=249482&action=review

> Tools/ChangeLog:8
> +        Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository.

Can't we figure that out outside run-webkit-tests? I'm not sure if this is a mode that engineers need to have access to.

EWS can just pass the list of new tests if any.
Comment 4 Csaba Osztrogonác 2015-03-26 11:45:06 PDT
(In reply to comment #3)
> Comment on attachment 249482 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=249482&action=review
> 
> > Tools/ChangeLog:8
> > +        Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository.
> 
> Can't we figure that out outside run-webkit-tests? I'm not sure if this is a
> mode that engineers need to have access to.
> 
> EWS can just pass the list of new tests if any.


EWS can pass list of the tests to run-webkit-tests as command 
line arguments directly and in a file with --test-list option.

But I'm not sure how are test expectations handled in this 
cases. AFAIR skipped tests are run too, but I'm not sure.
Comment 5 youenn fablet 2015-03-26 12:01:01 PDT
(In reply to comment #4)
> (In reply to comment #3)
> > Comment on attachment 249482 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=249482&action=review
> > 
> > > Tools/ChangeLog:8
> > > +        Add a --new-tests option to run-webkit-tests which adds tests that have been changed/created compared to the repository.
> > 
> > Can't we figure that out outside run-webkit-tests? I'm not sure if this is a
> > mode that engineers need to have access to.
> > 
> > EWS can just pass the list of new tests if any.

Combined with git, this option runs all tests added/modified by the set of commits compared to the remote origin master. Handy for branch development, rebasing -i...

It might also be useful when reimporting W3C tests.

> EWS can pass list of the tests to run-webkit-tests as command 
> line arguments directly and in a file with --test-list option.
> 
> But I'm not sure how are test expectations handled in this 
> cases. AFAIR skipped tests are run too, but I'm not sure.

Good point.
It should be checked as part of integration/unit tests that should be added.
Comment 6 youenn fablet 2015-03-26 13:08:25 PDT
> > EWS can pass list of the tests to run-webkit-tests as command 
> > line arguments directly and in a file with --test-list option.
> > 
> > But I'm not sure how are test expectations handled in this 
> > cases. AFAIR skipped tests are run too, but I'm not sure.
> 
> Good point.
> It should be checked as part of integration/unit tests that should be added.

There is the --skipped=always/default option which enables to run/skip the tests here.

It all ends up whether we want to expose this option in rwt or not...

I see some value to it as exposed previously.
The main downsides I see with that option is that there is the need to handle within rwt the case of runs with no tests. This patch introduces the --pass-if-no-option for that purpose. But this case would probably be best handled within EWS.

Thoughts?
Comment 7 Alexey Proskuryakov 2015-03-26 14:47:58 PDT
Another complication is layering. Currently, run-webkit-tests knows nothing about version control, and it would be confusing if it started to use that just for one command.

If enough people want a special mode to run just the new/modified tests, webkit-patch is unclean enough to have that added too, I think, but probably not run-webkit-tests.
Comment 8 youenn fablet 2015-03-27 10:27:45 PDT
(In reply to comment #7)
> Another complication is layering. Currently, run-webkit-tests knows nothing
> about version control, and it would be confusing if it started to use that
> just for one command.
> 
> If enough people want a special mode to run just the new/modified tests,
> webkit-patch is unclean enough to have that added too, I think, but probably
> not run-webkit-tests.

Then we should directly go to bug 143093...