restructure manager/broker/worker code
Created attachment 148931 [details] Patch
Created attachment 148956 [details] Patch
Created attachment 149665 [details] roll-up patch based off of r121304; not for review or landing attaching a patch that basically shows what manager.py, manager_worker_broker.py, and worker.py will look like after I've cleaned up the multiprocessing code. This patch needs to be broken up into reviewable chunks and some of the unit tests need to be redone, so don't bother to review it.
Also, note that "manager_worker_broker", although never a good name, is now a really bad name. I am trying to structure the code like the multiprocessing.Pool interface and the concurrency.futures.ProcessExecutor interface. The interfaces is much simpler though, as all we provide is a single run([list of things]) call. We could provide more flexibility easily, but it isn't needed at this time. I suggest we use something like MessagePool or MessagePoolExecutor or something like that ... To recap why we're not just using one of those two interfaces ... The workers are stateful, and those two interfaces assume each task is stateless. We could probably work around this using globals and restarting as necessary, but that would make things kinda ugly. There isn't a 1:1 mapping between tasks and messages, since we group tests into shards: running one shard generates N results for tests and we want those results as they occur As an aside using messages instead of request/reply tasks allows us to handle errors better, log messages, better, and generate better stats.
I'm slightly unsure where to start looking at these patches. The rollup patches are pretty big. :) Maybe I should wait for you to mark one of the patches for review.
(In reply to comment #5) > I'm slightly unsure where to start looking at these patches. The rollup patches are pretty big. :) > > Maybe I should wait for you to mark one of the patches for review. absolutely, I don't want/expect you to review these. this stuff is just in case you wanted to apply the patches and take a look at the end state(s).
all patches have been landed.