Bug 62753 - new-run-webkit-tests: order shards by name, not number of tests
Summary: new-run-webkit-tests: order shards by name, not number of tests
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: 528+ (Nightly build)
Hardware: PC All
: P2 Normal
Assignee: Dirk Pranke
Depends on:
Blocks: 58625
  Show dependency treegraph
Reported: 2011-06-15 13:35 PDT by Dirk Pranke
Modified: 2011-06-20 18:19 PDT (History)
5 users (show)

See Also:

Patch (2.04 KB, patch)
2011-06-15 13:35 PDT, Dirk Pranke
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Pranke 2011-06-15 13:35:01 PDT
new-run-webkit-tests: order shards by name, not number of tests
Comment 1 Dirk Pranke 2011-06-15 13:35:40 PDT
Created attachment 97356 [details]
Comment 2 Dirk Pranke 2011-06-20 12:24:17 PDT
I don't know if we even care about this, but I'll file it just to track things.
Comment 3 Ojan Vafai 2011-06-20 12:52:09 PDT
What's the rationale behind this? Is the theory that tests will be less flaky because we're more likely to run them in the same order as running them serially?

I guess since the http tests are by far the gating factor on cycle time, this would only slow down test runs that aren't running the http tests.
Comment 4 Dirk Pranke 2011-06-20 12:56:29 PDT
See the blocker bug ... I was attempting to ensure that NRWT ran the same tests in the same order as ORWT. In theory, running tests in a different order could reveal inter-test dependencies and cause failures with NRWT that don't occur with ORWT.

That said, I don't actually care personally if we do make NRWT match ORWT test-for-test, I just wanted to satisfy to myself that it could and there weren't other bugs (which I did find, and I am now satisfied).

I'm happy to close this as WONTFIX, or land it; it doesn't much matter to me.

Given that we now shard tests by directory, rather than only top-level directory, there are usually far more shards than workers, and I don't think any ordering matters much for determining the total test run time.
Comment 5 Ojan Vafai 2011-06-20 13:08:07 PDT
Comment on attachment 97356 [details]

Seems fine to me. If it ends up reducing flakiness, it's certainly worth it. If it doesn't it's not a big deal. We can try it and look at the before/after cycle time on the chromium bots to make sure it doesn't have a significant impact.
Comment 6 WebKit Review Bot 2011-06-20 16:05:53 PDT
Comment on attachment 97356 [details]

Clearing flags on attachment: 97356

Committed r89302: <http://trac.webkit.org/changeset/89302>
Comment 7 WebKit Review Bot 2011-06-20 16:05:57 PDT
All reviewed patches have been landed.  Closing bug.
Comment 8 Ojan Vafai 2011-06-20 18:19:55 PDT
Looking at http://build.chromium.org/p/chromium.webkit/stats, the doesn't seem to be any cycle time regression on any of the bots.