run-webkit-tests should respect argument order

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Eric Seidel (no email)]

REGRESSION(48516?): run-webkit-tests does not respect argument order

At some point, in the not-so-distant past, it used to be possible to pass a list of tests to run-webkit-tests and it would run them in the order specified.

This no longer seems to be the case.

This is very useful for debugging test ordering problems.  Currently there is no way besides --random and hoping that you happen to get the same order again. :)

http://trac.webkit.org/changeset/48516 looks suspicious, but I feel like I've used this ability more recently than that.

[Alexey Proskuryakov]

I think this regressed with r52976 or one of the related changes. This was supposed to be fixed in r53015.

[Darin Adler]

As far as I know, run-webkit-tests has always sorted tests in "pathcmp" order, and not paid attention to the order tests were passed on the command line, back multiple years.

[Csaba Osztrogonác]

(In reply to comment #2)
> As far as I know, run-webkit-tests has always sorted tests in "pathcmp" order,
> and not paid attention to the order tests were passed on the command line, back
> multiple years.

I can confirm it, "pathcmp" order is a very old thing.

But I should be glad if we make a new option, for example 
--fixed-test-order and modify sorting method, because it
can be very useful for debugging:

-@testsToRun = sort pathcmp @testsToRun; 
+@testsToRun = sort pathcmp @testsToRun unless $fixedTestOrder;

But it is a more complex and we should think it over.
For example what do you want to do, if you call
run-webkit-tests css svg ?
1.)
  - all tests in css directory (in semi-random order)
  - all tests in svg directory (in semi-random order)
2.)
  - all tests in css directory (in alphabarical order)
  - all tests in svg directory (in alphabetical order)

As for me I prefer the 2nd one, we can do a thing like this:

for my $test (@ARGV) { 
   ...
   @tests = sort pathcmp @tests;  
   push @testsToRun, @tests; 
}

What do you think? Should we start implementation?

[Darin Adler]

For individual tests passed on the command line, no sorting would be fine.

If a directory is passed on the command line, the files will be discovered in a random order depending on how the file system enumerates them. The pathcmp is used so that tests run in a predictable order.

The tool could sort within the sets of files discovered by iterating the file system based on each indivdual command line argument, but respect of the order of the command line arguments.

[Darin Adler]

I think Csaba and I are saying the same thing.

But why are we enhancing run-webkit-tests if it's about to be replaced by a Python script? How soon we can all switch over to the new script?

[Darin Adler]

It doesn't matter really whether this is a regression or not. Alexey came over and we decided to fix it.

[Darin Adler]

http://trac.webkit.org/changeset/57306

[Eric Seidel (no email)]

(In reply to comment #5)
> But why are we enhancing run-webkit-tests if it's about to be replaced by a
> Python script? How soon we can all switch over to the new script?

We're very close.  One of the problems is that tests fail when run out of order, which this bug made difficult to test.

run-webkit-tests (soon to be old-run-webkit-tests) will be with us for a long time, even if not the default..  new-run-webkit-tests may never work on platforms w/o Python 2.5 (like Tiger) for instance.

[Darin Adler]

(In reply to comment #8)
> run-webkit-tests (soon to be old-run-webkit-tests) will be with us for a long
> time, even if not the default.

That's bad news. I really don't want us to maintain two of these.

[Eric Seidel (no email)]

OK.  Well, one way to kill old-run-webkit-tests is to install python 2.5 everywhere.  If keeping old-run-webkit-tests around is a concern, we can easily kill it when the time comes.  Right now I'm focused on making new-run-webkit-tests ready for general consumption.  Once it's there, we can deal with the issues of transition.

[Alexey Proskuryakov]

My recollection from a webkit-dev discussion is that only "non-essential" scripts were going to be inoperable on Tiger.