Bug 118294 – Let EWS bots run only touched tests

• Status: REOPENED
• Component: Tools / Tests

[Csaba Osztrogonác]

Now only Apple Mac WK1, Apple Mac WK2 and Apple Windows EWS bots run layout tests. 
Other EWS bots  (GTK-WK1, GTK-WK2, EFL-WK1, EFL-WK2, Qt-WK1, Qt-WK2) don't run tests.
Running all tests on them isn't possible now because of some reason:
- it would extremely slow down patch the processing time
- we needed more and more hardware
- parallel test running is unstable/flakey on Qt (runtime is 25 mins on WK1, 35 mins on WK2)

The idea is that Apple Mac EWS bots could guarantee the general test coverage,
and we could run only touched tests on the other EWS bots to help rebasing
ahead of time. 

https://bugs.webkit.org/show_bug.cgi?id=118293 is going to add a new run-tests-touched-by-attachment
command for webkit-patch which download the given attachment from bugzilla, apply the patch,
build with the patch, determine which tests or expected files are touched by the attachment 
on any platform, run only these tests.

See the main bug for more information: https://bugs.webkit.org/show_bug.cgi?id=117831

[Csaba Osztrogonác]

no interest in it anymore

[youenn fablet]

Reopening this bug.
A potential implementation plan may be:
- Add a --new-tests option to run-webkit-tests so that it runs only modified tests. (may be handy for developpers as well as EWS)
- Add an option to webkit-patch build-and-test to iterate a number of times on new tests and make a report in case of failures.
- Add an option for limited EWS bots to make reports on new tests
- Add an option for full testing EWS bots to iterate over the new tests and make reports(flakky test reporting)