Bug 16869 – run-webkit-tests should have a --random switch to run tests in random order

• Component: Tools / Tests
• Status: RESOLVED FIXED

[David Kilzer (:ddkilzer)]

The run-webkit-tests script shoudl have a --random switch to run tests in a random order to find inter-dependencies on tests that should not exit.  (Perhaps even running the tests backwards from Z to A would also find issues.)

[Holger Freyther]

Created attachment 18439 [details]
Use a perl 5.8.0  function to shuffle the tests array

Shuffle the array

[Darin Adler]

Comment on attachment 18439 [details]
Use a perl 5.8.0  function to shuffle the tests array

r=me

+    'random' => sub { $randomizeTests = 1; },

Why not just => \$randomizeTests? Is there a reason for this one option to be different than the others? Also, could you preserve the alphabetical order by putting this before "root"?

+if ($randomizeTests == 1) {
+    @tests = shuffle(@tests);
+}

I'd write in more-idiomatic perl as:

    @tests = shuffle(@tests) if $randomizeTests;

[Holger Freyther]

(In reply to comment #2)
> (From update of attachment 18439 [details] [edit])
> r=me
> 
> +    'random' => sub { $randomizeTests = 1; },
> 
> Why not just => \$randomizeTests? Is there a reason for this one option to be
> different than the others? Also, could you preserve the alphabetical order by
> putting this before "root"?
> 
> +if ($randomizeTests == 1) {
> +    @tests = shuffle(@tests);
> +}
> 
> I'd write in more-idiomatic perl as:
> 
>     @tests = shuffle(@tests) if $randomizeTests;
> 

I have no perl skills at all. Will post a revised patch.

[Holger Freyther]

Created attachment 18440 [details]
Randomize the test array

Take Darin's comments into account.

[David Kilzer (:ddkilzer)]

Comment on attachment 18439 [details]
Use a perl 5.8.0  function to shuffle the tests array

Great!

>+    'random' => sub { $randomizeTests = 1; },

This should be declared as follows since it's boolean and to allow for "--no-random":

>+    'random!' => \$randomizeTests,

Needs a ChangeLog.

r=me assuming the above two issues are fixed.

Once this lands, we need a way to make a given randomization repeatable, e.g., by writing out the order into a text file.  We should also be able to reduce the failure to the minimum set of test cases needed reproduce it.  I'll file a follow-up bug.

[Darin Adler]

Comment on attachment 18440 [details]
Randomize the test array

Thanks, looks nicer.

[Holger Freyther]

Created attachment 18441 [details]
Randomize the test array

Take Darin's comments into account.

[Holger Freyther]

Comment on attachment 18441 [details]
Randomize the test array

Oops, QtLauncher bug :)

[Holger Freyther]

Created attachment 18442 [details]
Add a --reverse  option as well

Add an --reverse option as well.

[Darin Adler]

Comment on attachment 18442 [details]
Add a --reverse  option as well

Sorting the tests with $b cmp $a is not really "reversing" them. The normal sort order uses pathcmp.

Perl has a "reverse" function, which you could use.

[Holger Freyther]

Landed patches in revision r29473 and r29472.