Now that it does the rebaselines for the baselines of an individual test in parallel, it's many many times faster, but when I'm doing hundreds of tests, it still takes ~20 minutes to catch up. When the server sees that there are multiple tests enqueued, it should batch them together so that they can all be done in parallel.
For reference, rebaselining 700 windows only failures took ~8 minutes from San Francisco. That's good enough, but we can do better. Also, in the process, we could share code better with "webkit-patch rebaseline-expectations" so that gets the parallelism benefits as well (right now the latter doesn't do anything in parallel, where the garden-o-matic does the expectations for each individual test in parallel).
I think garden-o-matic is actually rebaselining all of the test in a given cluster in parallel, but it then only optimizes each test serially. We can probably do the latter in parallel as well by separating out the scm operations from the hashing and de-duping, but I suspect that will still be only a 5-10% speedup. I need to look at the logs to see if there are more opportunities for parallelism.