The run-webkit-tests script shoudl have a --random switch to run tests in a random order to find inter-dependencies on tests that should not exit. (Perhaps even running the tests backwards from Z to A would also find issues.)
Created attachment 18439 [details] Use a perl 5.8.0 function to shuffle the tests array Shuffle the array
Comment on attachment 18439 [details] Use a perl 5.8.0 function to shuffle the tests array r=me + 'random' => sub { $randomizeTests = 1; }, Why not just => \$randomizeTests? Is there a reason for this one option to be different than the others? Also, could you preserve the alphabetical order by putting this before "root"? +if ($randomizeTests == 1) { + @tests = shuffle(@tests); +} I'd write in more-idiomatic perl as: @tests = shuffle(@tests) if $randomizeTests;
(In reply to comment #2) > (From update of attachment 18439 [details] [edit]) > r=me > > + 'random' => sub { $randomizeTests = 1; }, > > Why not just => \$randomizeTests? Is there a reason for this one option to be > different than the others? Also, could you preserve the alphabetical order by > putting this before "root"? > > +if ($randomizeTests == 1) { > + @tests = shuffle(@tests); > +} > > I'd write in more-idiomatic perl as: > > @tests = shuffle(@tests) if $randomizeTests; > I have no perl skills at all. Will post a revised patch.
Created attachment 18440 [details] Randomize the test array Take Darin's comments into account.
Comment on attachment 18439 [details] Use a perl 5.8.0 function to shuffle the tests array Great! >+ 'random' => sub { $randomizeTests = 1; }, This should be declared as follows since it's boolean and to allow for "--no-random": >+ 'random!' => \$randomizeTests, Needs a ChangeLog. r=me assuming the above two issues are fixed. Once this lands, we need a way to make a given randomization repeatable, e.g., by writing out the order into a text file. We should also be able to reduce the failure to the minimum set of test cases needed reproduce it. I'll file a follow-up bug.
Comment on attachment 18440 [details] Randomize the test array Thanks, looks nicer.
Created attachment 18441 [details] Randomize the test array Take Darin's comments into account.
Comment on attachment 18441 [details] Randomize the test array Oops, QtLauncher bug :)
Created attachment 18442 [details] Add a --reverse option as well Add an --reverse option as well.
Comment on attachment 18442 [details] Add a --reverse option as well Sorting the tests with $b cmp $a is not really "reversing" them. The normal sort order uses pathcmp. Perl has a "reverse" function, which you could use.
Landed patches in revision r29473 and r29472.