Bug 29220 – run-webkit-tests needs a --repeat-each=N option (AAABBBCCC instead of ABCABCABC)

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Eric Seidel (no email)]

run-webkit-tests needs a --repeat-each=N option which will repeat each test N times.

so:

run-webkit-tests foo.html bar.html --repeat-each=N
would do:

foo -> passed
foo -> passed
bar -> passed
bar -> passed

This is useful for finding flakey tests (like https://bugs.webkit.org/show_bug.cgi?id=28845#c35) or tests which are causing later tests to fail.  It also could be useful for finding tests which fail if their resources are already cached, etc.

[Alexey Proskuryakov]

*** This bug has been marked as a duplicate of bug 29263 ***

[Simon Fraser (smfr)]

Actually in 29263 I took the easy way out and just duplicated the @tests array N times, so that the order will be:

ABCABCABC

not

AAABBBCCC

I'd be fine either way.

[Eric Seidel (no email)]

After discussion in bug 29263, unduping...

bug 29263 added support for running tests ABC as ABCABCABC
this bug is about running test ABC as AAABBBCCC

[Eric Seidel (no email)]

Created attachment 39941 [details]
Patch v1

[Darin Adler]

Comment on attachment 39941 [details]
Patch v1

> -  --iterations n                  Number of times to run the tests
> +  --iterations n                  Number of times to run the set of tests (e.g. ABCABCABC)
> +  --repeat-each n                 Number of times to run each test (e.g. AAABBBCCC)

These are alphabetical, so --repeat-each should go further down in the list.

[Eric Seidel (no email)]

Committed r48649: <http://trac.webkit.org/changeset/48649>

[Eric Seidel (no email)]

Comment on attachment 39941 [details]
Patch v1

Thanks!  Fixed.

[Eric Seidel (no email)]

Committed r48655: <http://trac.webkit.org/changeset/48655>