We're seeing "failed download-built-product" a lot. I'm not sure why, but have a hunch that it's related to buildbot's mergeRequests and Jon's out-of-order building/testing and maybe the buildbot upgrade. I'd like to try disabling mergeRequests and see if it's related. I don't think we want it anyway.
Created attachment 145415 [details] speculative patch
Attachment 145415 [details] did not pass style-queue: Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/BuildSlaveSupport/build.webkit.org-c..." exit_code: 1 Tools/BuildSlaveSupport/build.webkit.org-config/config.json:74: Expecting object: line 74 column 40 (char 4421) [json/syntax] [5] Total errors found: 1 in 2 files If any of these errors are false positives, please file a bug against check-webkit-style.
Comment on attachment 145415 [details] speculative patch You may want to file a bug about the style checker
Leaving this open after landing it in http://trac.webkit.org/changeset/119305 until we can confirm it's fixed.
Unsuccessful, buildbot is still merging builds: http://build.webkit.org/builders/Lion%20Release%20%28Build%29/builds/10221
The master buildbot got kicked this morning at around 9 am, and it's looking a lot better now.
If we make it through today without a) getting super backed up and b) any "failed download-built-product", we are probably OK for now. That said, we should probably look into fixing the problem properly instead of hacking around it like this, since this is going to make our queue longer for sure.
Created attachment 145860 [details] Patch
Posting a patch that would revert building bots to their old behavior (pick oldest request, merge any later requests), and have testing, performance, and leaks bots choose the latest build.
Jon and I determined that this is due to an underlying Buildbot bug. I wrote it up as <http://trac.buildbot.net/ticket/2309>.
Committed r119519: <http://trac.webkit.org/changeset/119519>
Reopening, to monitor the bots for a while with this change.