Bug 10906 - Parallelize run-webkit-tests
Summary: Parallelize run-webkit-tests
Status: RESOLVED WONTFIX
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 420+
Hardware: Macintosh OS X 10.4
: P4 Enhancement
Assignee: Nobody
URL:
Keywords:
: 24460 (view as bug list)
Depends on: 10912 24704 25101 25128
Blocks:
  Show dependency treegraph
 
Reported: 2006-09-17 13:40 PDT by mitz
Modified: 2010-04-08 00:46 PDT (History)
14 users (show)

See Also:


Attachments
Just my work in progress on this bug. (19.96 KB, patch)
2009-03-20 16:53 PDT, David Levin
no flags Details | Formatted Diff | Diff
New work in progress. (66.13 KB, patch)
2009-04-08 10:47 PDT, David Levin
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description mitz 2006-09-17 13:40:03 PDT
Make run-webkit-tests run one DumpRenderTree instance per processor. Make this the default, but add an option to disable it or limit the number of instances.
Comment 1 Eric Seidel (no email) 2006-09-18 00:29:19 PDT
This will require us to fix all the inter-test dependancies.  (which is a good thing)
Comment 2 Eric Seidel (no email) 2006-09-18 01:01:50 PDT
This will likely require using threading in the run-webkit-tests perl script:
http://www.mathematik.uni-ulm.de/help/perl5/doc/perlthrtut.html
Comment 3 Alexey Proskuryakov 2006-10-31 09:14:42 PST
From an IRC discussion, some of difficulties that needs to be fixed:
- several Apache instances;
- sharing of icon database, cookies and disk cache;
- changing the display profile.

Anything else?
Comment 4 Eric Seidel (no email) 2008-07-10 21:52:28 PDT
(In reply to comment #3)
> From an IRC discussion, some of difficulties that needs to be fixed:
> - several Apache instances;
> - sharing of icon database, cookies and disk cache;
> - changing the display profile.
> 
> Anything else?

I expect that we would implement this by running multiple instances of DRT.  Instead of say multiple threads in one DRT.  I assume we're all on the same page about that part. :)

One running Apache should be enough to serve for all of these DRT instances, no?

Icon DB, Cookies, Cache, etc. need to find some way to not be shared, yes.

The changing the display profile should ideally be broken out into a separate tool, which knows how to set and reset the display profile.  run-webkit-tests could then launch that tool.  (Or just launch DRT with a specific set/reset arg).

This was probably all said over IRC... but I've said it here just in case my comments are helpful. :)
Comment 5 Jacob Refstrup 2008-07-10 22:35:54 PDT
My initial thoughts for a design were as follows:

- The main test loop starts with doing an openDumpTool(); just prior to that we would 
    - if total number of outstanding tests was greater than threshold then wait for an one to finish
    - then fork another test (essentially just forking the perl program)
    - when test is done don't update the $tests{result} hash but rather exit with an error code
       and have the parent use that error code to update $tests{result}.
 
Pros:
- relatively simple

Cons:
- extra (perl) process per test

Other ideas:
- perhaps use real threads (rather than processes) to avoid overhead;  not sure how the SIGPIPE handler would work then.
Comment 6 Mark Rowe (bdash) 2008-07-10 22:38:15 PDT
An extra perl process per test sounds like a large amount of overhead.
Comment 7 Jacob Refstrup 2008-07-10 22:55:38 PDT
(In reply to comment #6)
> An extra perl process per test sounds like a large amount of overhead.
> 

It does... Hm. 

Looking more closely at the current run-webkit-tests I realize that it attempts to keep the current DRT open by default for 1000 tests. So perhaps the best way of doing it is to keep N instances of DRT open for 1000 tests. That will leave one perl process coordinating everything (which is fine) but there's some logic needed to figure out which pipe was broken (in case of one of the DRTs crashing) and we need some select logic to wait for the output to be available.
Comment 8 Jacob Refstrup 2008-07-11 00:36:17 PDT
(In reply to comment #1)
> This will require us to fix all the inter-test dependancies.  (which is a good
> thing)
> 

Are the some examples of inter-test dependencies that I could check out?
Comment 9 Jacob Refstrup 2008-07-11 00:40:48 PDT
(In reply to comment #7)
> Looking more closely at the current run-webkit-tests I realize that it attempts
> to keep the current DRT open by default for 1000 tests. So perhaps the best way
> of doing it is to keep N instances of DRT open for 1000 tests. That will leave
> one perl process coordinating everything (which is fine) but there's some logic
> needed to figure out which pipe was broken (in case of one of the DRTs
> crashing) and we need some select logic to wait for the output to be available.
> 

It looks like if we simply keep track of all the DRT instances and their file-handles and PIDs then we can use waitpid(-1,WNOHANG) in the SIGPIPE handler to figure out which DRT instance failed.

The prologue and epilogue stuff will require some serialization -- but right now it looks like it's only windows that use them; and it'd be a great start if just MacOSX and linux could use parallel DRTs.   

I'll start putting something together...
- Jacob
Comment 10 Alexey Proskuryakov 2008-07-11 01:03:25 PDT
(In reply to comment #8)
> Are the some examples of inter-test dependencies that I could check out?

You can try running the tests with --singly and/or --reverse options, which results in a bunch of failures. I am not aware of any interdependencies that are not caught by this.
Comment 11 mitz 2008-07-11 01:14:58 PDT
(In reply to comment #10)
> (In reply to comment #8)
> > Are the some examples of inter-test dependencies that I could check out?
> 
> You can try running the tests with --singly and/or --reverse options, which
> results in a bunch of failures. I am not aware of any interdependencies that
> are not caught by this.
> 

Or --random, if you are very adventurous.
Comment 12 Alexey Proskuryakov 2009-03-09 10:15:24 PDT
*** Bug 24460 has been marked as a duplicate of this bug. ***
Comment 13 David Levin 2009-03-20 16:53:39 PDT
Created attachment 28811 [details]
Just my work in progress on this bug.

This attachment isn't ready for review.

It has several issues that I need to fix but I think it is headed in a nice direction and gives a feel for my current direction on this.
Comment 14 David Levin 2009-04-08 10:47:11 PDT
Created attachment 29339 [details]
New work in progress.
Comment 15 David Levin 2009-11-19 15:53:15 PST
Unassigning from myself as I won't have time to work on this in the foreseeable future.
Comment 16 Dirk Pranke 2010-03-26 19:04:06 PDT
We might want to consider marking this as WONTFIX and just switching to new-run-webkit-tests, since that does support running in parallel.
Comment 17 Eric Seidel (no email) 2010-04-08 00:46:01 PDT
Yup.  We're very close to having new-run-webkit-tests ready for prime-time.  Closing this as WONTFIX.