| Summary: | [meta] Run http tests parallel | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Csaba Osztrogonác <ossy> | ||||||
| Component: | New Bugs | Assignee: | Alexey Proskuryakov <ap> | ||||||
| Status: | RESOLVED FIXED | ||||||||
| Severity: | Normal | CC: | ap, bfulgham, commit-queue, dbates, ddkilzer, glenn, ossy | ||||||
| Priority: | P2 | ||||||||
| Version: | 528+ (Nightly build) | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| See Also: |
https://bugs.webkit.org/show_bug.cgi?id=139225 https://bugs.webkit.org/show_bug.cgi?id=139226 https://bugs.webkit.org/show_bug.cgi?id=139279 |
||||||||
| Bug Depends on: | 136722, 138583, 138894, 138959, 139135, 139149, 139243, 139347 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
|
Description
Csaba Osztrogonác
2014-11-21 03:30:00 PST
I think that we are ready to flip the switch (at least on Mac), but I can't figure out which switch to flip. There are still some failures when running with --fully-parallel, but they should not affect normal mode, as far as I can tell. (In reply to comment #1) > I think that we are ready to flip the switch (at least on Mac), but I can't > figure out which switch to flip. > > There are still some failures when running with --fully-parallel, but they > should not affect normal mode, as far as I can tell. "-f, --fully-parallel - run all tests in parallel" Ah, this help message is a little bit confusing. Tests are sharded by subdirectories, leaf subdirectories are atomic and can't be distributed to different threads by default. Threads pick up shards from the queue. This fully parallel option disables sharding and each threads can pick up tests from the queue instead of shards. I think we aren't ready for it yet, but it can reveal many flakiness. It would be a good expetiment. Back to parallel http test running. Now http and perf tests are considered as locked tests and locked tests run on 1 thread by default. You can easily override it with: --max-locked-shards command line option or WEBKIT_TEST_MAX_ LOCKED_SHARDS. (cmdline is the stronger one) The default option is defined to 1 in base.py in default_max_locked_shards. First we can override it for ports one by one. And then in base.py once it is stable everywhere. The ideal number would be the same as default_child_processes, but it depends on how much concurrent request can be served by the http server. The question is how do I make HTTP tests parallel while keeping perf tests non-parallel. (In reply to comment #3) > The question is how do I make HTTP tests parallel while keeping perf tests > non-parallel. I think the easiest way is what I suggested in bug136722 - run them in a different buildstep on the bots. One more thing. perf tests can run parallel now with non perf tests. If we increasw max_locked_shards, they will run parallel with more http test thread instead of random non-http test. But 2 perf test won't run parallel, because they are in the same shard. Created attachment 242595 [details]
proposed patch
This makes debug tests run in 8 minutes and 12 seconds on my Mac Pro. Without the patch, they used to take over 15 minutes.
Comment on attachment 242595 [details] proposed patch View in context: https://bugs.webkit.org/attachment.cgi?id=242595&action=review Looks sane to me. > Tools/ChangeLog:17 > + think that these parameters do what we wanted them to do; Apache handles the load Nit: do => did Or wanted => want Created attachment 242599 [details]
patch for landing
Comment on attachment 242599 [details] patch for landing Clearing flags on attachment: 242599 Committed r176830: <http://trac.webkit.org/changeset/176830> All reviewed patches have been landed. Closing bug. |