Bug 89725 – restructure manager/broker/worker code

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Dirk Pranke]

restructure manager/broker/worker code

[Dirk Pranke]

Created attachment 148931 [details]
Patch

[Dirk Pranke]

Created attachment 148956 [details]
Patch

[Dirk Pranke]

Created attachment 149665 [details]
roll-up patch based off of r121304; not for review or landing

attaching a patch that basically shows what manager.py, manager_worker_broker.py, and worker.py will look like after I've cleaned up the multiprocessing code. This patch needs to be broken up into reviewable chunks and some of the unit tests need to be redone, so don't bother to review it.

[Dirk Pranke]

Also, note that "manager_worker_broker", although never a good name, is now a really bad name.

I am trying to structure the code like the multiprocessing.Pool interface and the concurrency.futures.ProcessExecutor interface. The interfaces is much simpler though, as all we provide is a single run([list of things]) call. We could provide more flexibility easily, but it isn't needed at this time.

I suggest we use something like MessagePool or MessagePoolExecutor or something like that ...

To recap why we're not just using one of those two interfaces ...

The workers are stateful, and those two interfaces assume each task is stateless. We could probably work around this using globals and restarting as necessary, but that would make things kinda ugly.

There isn't a 1:1 mapping between tasks and messages, since we group tests into shards: running one shard generates N results for tests and we want those results as they occur

As an aside using messages instead of request/reply tasks allows us to handle errors better, log messages, better, and generate better stats.

[Adam Barth]

I'm slightly unsure where to start looking at these patches.  The rollup patches are pretty big.  :)

Maybe I should wait for you to mark one of the patches for review.

[Dirk Pranke]

(In reply to comment #5)
> I'm slightly unsure where to start looking at these patches.  The rollup patches are pretty big.  :)
> 
> Maybe I should wait for you to mark one of the patches for review.

absolutely, I don't want/expect you to review these. this stuff is just in case you wanted to apply the patches and take a look at the end state(s).

[Dirk Pranke]

all patches have been landed.