Bug 118294 - Let EWS bots run only touched tests
Summary: Let EWS bots run only touched tests
Status: REOPENED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on: 143090
Blocks: 117831 118293
  Show dependency treegraph
 
Reported: 2013-07-02 05:16 PDT by Csaba Osztrogonác
Modified: 2015-03-26 03:52 PDT (History)
7 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Csaba Osztrogonác 2013-07-02 05:16:46 PDT
Now only Apple Mac WK1, Apple Mac WK2 and Apple Windows EWS bots run layout tests. 
Other EWS bots  (GTK-WK1, GTK-WK2, EFL-WK1, EFL-WK2, Qt-WK1, Qt-WK2) don't run tests.
Running all tests on them isn't possible now because of some reason:
- it would extremely slow down patch the processing time
- we needed more and more hardware
- parallel test running is unstable/flakey on Qt (runtime is 25 mins on WK1, 35 mins on WK2)

The idea is that Apple Mac EWS bots could guarantee the general test coverage,
and we could run only touched tests on the other EWS bots to help rebasing
ahead of time. 

https://bugs.webkit.org/show_bug.cgi?id=118293 is going to add a new run-tests-touched-by-attachment
command for webkit-patch which download the given attachment from bugzilla, apply the patch,
build with the patch, determine which tests or expected files are touched by the attachment 
on any platform, run only these tests.

See the main bug for more information: https://bugs.webkit.org/show_bug.cgi?id=117831
Comment 1 Csaba Osztrogonác 2014-12-03 03:54:46 PST
no interest in it anymore
Comment 2 youenn fablet 2015-03-26 03:47:55 PDT
Reopening this bug.
A potential implementation plan may be:
- Add a --new-tests option to run-webkit-tests so that it runs only modified tests. (may be handy for developpers as well as EWS)
- Add an option to webkit-patch build-and-test to iterate a number of times on new tests and make a report in case of failures.
- Add an option for limited EWS bots to make reports on new tests
- Add an option for full testing EWS bots to iterate over the new tests and make reports(flakky test reporting)