Bug 37257 - run-webkit-tests should respect argument order
: run-webkit-tests should respect argument order
Status: RESOLVED FIXED
: WebKit
Tools / Tests
: 528+ (Nightly build)
: PC Mac OS X 10.5
: P2 Normal
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2010-04-08 01:47 PST by
Modified: 2010-04-08 19:45 PST (History)


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description From 2010-04-08 01:47:00 PST
REGRESSION(48516?): run-webkit-tests does not respect argument order

At some point, in the not-so-distant past, it used to be possible to pass a list of tests to run-webkit-tests and it would run them in the order specified.

This no longer seems to be the case.

This is very useful for debugging test ordering problems.  Currently there is no way besides --random and hoping that you happen to get the same order again. :)

http://trac.webkit.org/changeset/48516 looks suspicious, but I feel like I've used this ability more recently than that.
------- Comment #1 From 2010-04-08 11:10:45 PST -------
I think this regressed with r52976 or one of the related changes. This was supposed to be fixed in r53015.
------- Comment #2 From 2010-04-08 14:50:16 PST -------
As far as I know, run-webkit-tests has always sorted tests in "pathcmp" order, and not paid attention to the order tests were passed on the command line, back multiple years.
------- Comment #3 From 2010-04-08 15:21:47 PST -------
(In reply to comment #2)
> As far as I know, run-webkit-tests has always sorted tests in "pathcmp" order,
> and not paid attention to the order tests were passed on the command line, back
> multiple years.

I can confirm it, "pathcmp" order is a very old thing.

But I should be glad if we make a new option, for example 
--fixed-test-order and modify sorting method, because it
can be very useful for debugging:

-@testsToRun = sort pathcmp @testsToRun; 
+@testsToRun = sort pathcmp @testsToRun unless $fixedTestOrder;

But it is a more complex and we should think it over.
For example what do you want to do, if you call
run-webkit-tests css svg ?
1.)
  - all tests in css directory (in semi-random order)
  - all tests in svg directory (in semi-random order)
2.)
  - all tests in css directory (in alphabarical order)
  - all tests in svg directory (in alphabetical order)

As for me I prefer the 2nd one, we can do a thing like this:

for my $test (@ARGV) { 
   ...
   @tests = sort pathcmp @tests;  
   push @testsToRun, @tests; 
}

What do you think? Should we start implementation?
------- Comment #4 From 2010-04-08 17:21:16 PST -------
For individual tests passed on the command line, no sorting would be fine.

If a directory is passed on the command line, the files will be discovered in a random order depending on how the file system enumerates them. The pathcmp is used so that tests run in a predictable order.

The tool could sort within the sets of files discovered by iterating the file system based on each indivdual command line argument, but respect of the order of the command line arguments.
------- Comment #5 From 2010-04-08 17:22:30 PST -------
I think Csaba and I are saying the same thing.

But why are we enhancing run-webkit-tests if it's about to be replaced by a Python script? How soon we can all switch over to the new script?
------- Comment #6 From 2010-04-08 17:36:17 PST -------
It doesn't matter really whether this is a regression or not. Alexey came over and we decided to fix it.
------- Comment #7 From 2010-04-08 17:37:45 PST -------
http://trac.webkit.org/changeset/57306
------- Comment #8 From 2010-04-08 18:36:01 PST -------
(In reply to comment #5)
> But why are we enhancing run-webkit-tests if it's about to be replaced by a
> Python script? How soon we can all switch over to the new script?

We're very close.  One of the problems is that tests fail when run out of order, which this bug made difficult to test.

run-webkit-tests (soon to be old-run-webkit-tests) will be with us for a long time, even if not the default..  new-run-webkit-tests may never work on platforms w/o Python 2.5 (like Tiger) for instance.
------- Comment #9 From 2010-04-08 18:46:46 PST -------
(In reply to comment #8)
> run-webkit-tests (soon to be old-run-webkit-tests) will be with us for a long
> time, even if not the default.

That's bad news. I really don't want us to maintain two of these.
------- Comment #10 From 2010-04-08 18:49:21 PST -------
OK.  Well, one way to kill old-run-webkit-tests is to install python 2.5 everywhere.  If keeping old-run-webkit-tests around is a concern, we can easily kill it when the time comes.  Right now I'm focused on making new-run-webkit-tests ready for general consumption.  Once it's there, we can deal with the issues of transition.
------- Comment #11 From 2010-04-08 19:45:10 PST -------
My recollection from a webkit-dev discussion is that only "non-essential" scripts were going to be inoperable on Tiger.