Bug 16869

Summary: run-webkit-tests should have a --random switch to run tests in random order
Product: WebKit Reporter: David Kilzer (:ddkilzer) <ddkilzer>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Enhancement    
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: All   
OS: All   
Bug Depends on:    
Bug Blocks: 16872, 16873    
Attachments:
Description Flags
Use a perl 5.8.0 function to shuffle the tests array
darin: review+
Randomize the test array
darin: review+
Randomize the test array
none
Add a --reverse option as well darin: review-

Description David Kilzer (:ddkilzer) 2008-01-13 22:02:11 PST
The run-webkit-tests script shoudl have a --random switch to run tests in a random order to find inter-dependencies on tests that should not exit.  (Perhaps even running the tests backwards from Z to A would also find issues.)
Comment 1 Holger Freyther 2008-01-14 09:54:48 PST
Created attachment 18439 [details]
Use a perl 5.8.0  function to shuffle the tests array

Shuffle the array
Comment 2 Darin Adler 2008-01-14 10:00:04 PST
Comment on attachment 18439 [details]
Use a perl 5.8.0  function to shuffle the tests array

r=me

+    'random' => sub { $randomizeTests = 1; },

Why not just => \$randomizeTests? Is there a reason for this one option to be different than the others? Also, could you preserve the alphabetical order by putting this before "root"?

+if ($randomizeTests == 1) {
+    @tests = shuffle(@tests);
+}

I'd write in more-idiomatic perl as:

    @tests = shuffle(@tests) if $randomizeTests;
Comment 3 Holger Freyther 2008-01-14 10:03:47 PST
(In reply to comment #2)
> (From update of attachment 18439 [details] [edit])
> r=me
> 
> +    'random' => sub { $randomizeTests = 1; },
> 
> Why not just => \$randomizeTests? Is there a reason for this one option to be
> different than the others? Also, could you preserve the alphabetical order by
> putting this before "root"?
> 
> +if ($randomizeTests == 1) {
> +    @tests = shuffle(@tests);
> +}
> 
> I'd write in more-idiomatic perl as:
> 
>     @tests = shuffle(@tests) if $randomizeTests;
> 

I have no perl skills at all. Will post a revised patch.
Comment 4 Holger Freyther 2008-01-14 10:08:36 PST
Created attachment 18440 [details]
Randomize the test array

Take Darin's comments into account.
Comment 5 David Kilzer (:ddkilzer) 2008-01-14 10:10:57 PST
Comment on attachment 18439 [details]
Use a perl 5.8.0  function to shuffle the tests array

Great!

>+    'random' => sub { $randomizeTests = 1; },

This should be declared as follows since it's boolean and to allow for "--no-random":

>+    'random!' => \$randomizeTests,

Needs a ChangeLog.

r=me assuming the above two issues are fixed.

Once this lands, we need a way to make a given randomization repeatable, e.g., by writing out the order into a text file.  We should also be able to reduce the failure to the minimum set of test cases needed reproduce it.  I'll file a follow-up bug.
Comment 6 Darin Adler 2008-01-14 10:14:23 PST
Comment on attachment 18440 [details]
Randomize the test array

Thanks, looks nicer.
Comment 7 Holger Freyther 2008-01-14 10:14:33 PST
Created attachment 18441 [details]
Randomize the test array

Take Darin's comments into account.
Comment 8 Holger Freyther 2008-01-14 10:15:09 PST
Comment on attachment 18441 [details]
Randomize the test array

Oops, QtLauncher bug :)
Comment 9 Holger Freyther 2008-01-14 10:16:29 PST
Created attachment 18442 [details]
Add a --reverse  option as well

Add an --reverse option as well.
Comment 10 Darin Adler 2008-01-14 10:19:14 PST
Comment on attachment 18442 [details]
Add a --reverse  option as well

Sorting the tests with $b cmp $a is not really "reversing" them. The normal sort order uses pathcmp.

Perl has a "reverse" function, which you could use.
Comment 11 Holger Freyther 2008-01-14 10:42:44 PST
Landed patches in revision r29473 and r29472.