Bug 62753

Summary: new-run-webkit-tests: order shards by name, not number of tests
Product: WebKit Reporter: Dirk Pranke <dpranke>
Component: New BugsAssignee: Dirk Pranke <dpranke>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth, eric, ojan, tony, webkit.review.bot
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: PC   
OS: All   
Bug Depends on:    
Bug Blocks: 58625    
Attachments:
Description Flags
Patch none

Dirk Pranke
Reported 2011-06-15 13:35:01 PDT
new-run-webkit-tests: order shards by name, not number of tests
Attachments
Patch (2.04 KB, patch)
2011-06-15 13:35 PDT, Dirk Pranke 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Dirk Pranke
Comment 1 2011-06-15 13:35:40 PDT
Created attachment 97356 [details] Patch
Dirk Pranke
Comment 2 2011-06-20 12:24:17 PDT
I don't know if we even care about this, but I'll file it just to track things.
Ojan Vafai
Comment 3 2011-06-20 12:52:09 PDT
What's the rationale behind this? Is the theory that tests will be less flaky because we're more likely to run them in the same order as running them serially? I guess since the http tests are by far the gating factor on cycle time, this would only slow down test runs that aren't running the http tests.
Dirk Pranke
Comment 4 2011-06-20 12:56:29 PDT
See the blocker bug ... I was attempting to ensure that NRWT ran the same tests in the same order as ORWT. In theory, running tests in a different order could reveal inter-test dependencies and cause failures with NRWT that don't occur with ORWT. That said, I don't actually care personally if we do make NRWT match ORWT test-for-test, I just wanted to satisfy to myself that it could and there weren't other bugs (which I did find, and I am now satisfied). I'm happy to close this as WONTFIX, or land it; it doesn't much matter to me. Given that we now shard tests by directory, rather than only top-level directory, there are usually far more shards than workers, and I don't think any ordering matters much for determining the total test run time.
Ojan Vafai
Comment 5 2011-06-20 13:08:07 PDT
Comment on attachment 97356 [details] Patch Seems fine to me. If it ends up reducing flakiness, it's certainly worth it. If it doesn't it's not a big deal. We can try it and look at the before/after cycle time on the chromium bots to make sure it doesn't have a significant impact.
WebKit Review Bot
Comment 6 2011-06-20 16:05:53 PDT
Comment on attachment 97356 [details] Patch Clearing flags on attachment: 97356 Committed r89302: <http://trac.webkit.org/changeset/89302>
WebKit Review Bot
Comment 7 2011-06-20 16:05:57 PDT
All reviewed patches have been landed. Closing bug.
Ojan Vafai
Comment 8 2011-06-20 18:19:55 PDT
Looking at http://build.chromium.org/p/chromium.webkit/stats, the doesn't seem to be any cycle time regression on any of the bots.
Note You need to log in before you can comment on or make changes to this bug.