Bug 78185

Summary:

test-webkitpy should support parallel execution of tests

Product:

WebKit

Reporter:

Dirk Pranke <dpranke>

Component:

Tools / Tests

Assignee:

Dirk Pranke <dpranke>

Status:

RESOLVED WONTFIX

Severity:

Normal

CC:

abarth, eric, levin, ojan, rniwa, tony

Priority:

Version:

528+ (Nightly build)

Hardware:

Unspecified

OS:

Unspecified

Bug Depends on:

76765, 77807, 78171, 78181, 78187

Bug Blocks:

76022

Attachments:

Description	Flags
work in progress	none
dramatically cleaned up patch	none
work in progress	none
Work in progress (includes rollup of work in 77687)	none

Dirk Pranke

Reported 2012-02-08 17:06:14 PST

assuming it's relatively easy. Currently the tests run in ~53 seconds or thereabouts on a mac pro (and we aren't running some of the integration tests even then); running them in parallel should give us a near-linear speedup.

Attachments
work in progress (12.52 KB, patch) 2012-02-08 20:19 PST, Dirk Pranke	no flags	Details Formatted Diff Diff
dramatically cleaned up patch (5.33 KB, patch) 2012-02-09 20:49 PST, Dirk Pranke	no flags	Details Formatted Diff Diff
work in progress (15.23 KB, patch) 2012-02-23 11:23 PST, Dirk Pranke	no flags	Details Formatted Diff Diff
Work in progress (includes rollup of work in 77687) (41.08 KB, patch) 2012-02-24 20:10 PST, Dirk Pranke	no flags	Details Formatted Diff Diff
Show Obsolete (4) View All Add attachment proposed patch, testcase, etc.

Dirk Pranke

Comment 1 2012-02-08 20:19:31 PST

Created attachment 126231 [details] work in progress

Dirk Pranke

Comment 2 2012-02-08 20:25:32 PST

Okay, I have a preliminary version of the patch that adds roughly ~100 lines of code and shards tests by the individual test_method. I suspect once I clean things up a bit I can probably get this down to 50-75 LOC. Running with 8 or 16 tests in parallel, total test time is about ~10 seconds. It looks like there's ~3 seconds of purely serial work to actually get all of the tests (this is expensive since we have to load all of the test files and parse them in order to shard by method). There are also several tests that take multiple seconds to complete, so we're not getting it down much further w/o speeding up individual tests. That said, we can probably get this down to ~5-6 seconds without very much work. That's a pretty good speedup from ~53 seconds. Running tests individually, we have at least two tests that are flaky and depend on previous tests being run; that should be easy to fix. Also, perhaps unsurprisingly, it looks like tests that actually use multiprocessing get confused and fail. I would need to look into that further. If we shard tests by file (instead of method), the speedups are only about 2x, since run_webkit_tests_integrationtest.py is by far the long pole. Clearly we could speed up the individual tests in this file regardless of whether we want to run things in parallel or not, and that would be good.

Dirk Pranke

Comment 3 2012-02-08 20:26:36 PST

(I would say that the results are encouraging but not conclusive as to whether this is worth it).

Dirk Pranke

Comment 4 2012-02-09 20:49:00 PST

Created attachment 126445 [details] dramatically cleaned up patch

Dirk Pranke

Comment 5 2012-02-09 20:53:44 PST

Comment on attachment 126445 [details] dramatically cleaned up patch here's a cleaned up patch adding parallel support, as a delta from attachment 126446 [details] on bug 77687

Dirk Pranke

Comment 6 2012-02-23 11:23:07 PST

Created attachment 128514 [details] work in progress

Dirk Pranke

Comment 7 2012-02-24 20:10:06 PST

Created attachment 128842 [details] Work in progress (includes rollup of work in 77687)

Dirk Pranke

Comment 8 2012-06-26 19:53:00 PDT

closing ... I will file new bugs as I have patches for review. The rollup in bug 76022 contains the real code, so we're not losing anything here.

Note You need to log in before you can comment on or make changes to this bug.