As discussed at the WebKit Developers Conference, the webkit.org waterfall and associated pages should have all the same cool features that have been added to Chromium's waterfall. In addition, it needs to be made more hackable (source in the tree, auto deployment etc).
I did have a chat with some of our buildbot guys after that talk ... as you would expect, it's largely a mixture of we are (or already have) upstreamed that, we could upstream that pretty easy, and you wouldn't want to upstream that :) Another thing to consider, of course, is that there may be cases where time spent merging functionality for X would be better spent coming up with a better way of doing X. Not that any of these things leaps to mind at the moment ...
Also, copying sfalken on this; we didn't get to talk in person, but I think I think I talked to someone (darin?) about this, at least ... the topic of being able to shard the layout tests across multiple bots came up at the meeting. I've spent a pretty good amount of time thinking about this, and there's an easy way (which we do today for some of the debug bots) and a much harder way. The easy way is to just manually run parts of the tests as a step on each bot (e.g., split the tests in half, and one run half on one bot and one half on the other). For example: http://build.chromium.org/p/chromium.webkit/waterfall?builder=Webkit+Win+Builder+(dbg)&builder=Webkit+Win+(dbg)(1)&builder=Webkit+Win+(dbg)(2) We don't make any attempt to merge the results of the two test bots (except that the flakiness dashboard deals with this somehow). Doing some sort of automatic sharding and execution would require us to synchronize the filesystem across the two machines, or do a remote copy of the binary onto the targets plus converting the local-file tests to http tests (either directly or indirectly). It's all quite doable but it would probably take a while to kick out the bugs and make sure it's stable. The alternative is to just use bigger iron and spend a little time tweaking the tests to run stably in parallel. On my Mac Pro I can run the tests in 3 minutes with 12 chromium release DRTs with minimal flakiness if I spend a little time stubbing out flaky tests. Get things much faster than that and you start not seeing a real benefit because other steps like update and archive are still slow. If we think there's still interest in going down this road it deserves its own bug, though ...
What specifically are the features on the Chromium waterfall that aren't on the WebKit one? The only features I can think of are: 1. The bar at the top with the status message. This would maybe be useful for WebKit, although it wouldn't be a huge improvement. 2. The compacted view of all the builders. This is useful for a high-level view of whether the bots are green. I don't think this view is useful without having a project-wide commitment to closing the tree to commits if any bot is red, which I would vote against. If most of the bots are regularly red with 1 or 2 failures, then the compacted view doesn't give you any useful information. 3. The merged view of the console page. This is definitely useful and we should merge into the WebKit buildbot.
Things I'd like to see on the b.w.o/waterfall page 1. trac link for entries in the Changes column 2. Header shows the svn rev for the build that build/test are shown for, and an indicator of how old it is
(In reply to comment #3) > What specifically are the features on the Chromium waterfall that aren't on the WebKit one? > > The only features I can think of are: > 1. The bar at the top with the status message. This would maybe be useful for WebKit, although it wouldn't be a huge improvement. > > 2. The compacted view of all the builders. This is useful for a high-level view of whether the bots are green. I don't think this view is useful without having a project-wide commitment to closing the tree to commits if any bot is red, which I would vote against. If most of the bots are regularly red with 1 or 2 failures, then the compacted view doesn't give you any useful information. > I'm not even sure which thing you're referring to here .. > 3. The merged view of the console page. This is definitely useful and we should merge into the WebKit buildbot. Yup, this is definitely one. Others: * the per-bot stats pages * the JSON builder output URLs * the ability to "clobber" a bot without logging in * possibly the list of failing tests in a waterfall entry rather than just "4 tests failed" * the ability to trigger a build even if nothing has been checked in Note that there are a couple things that the webkit.org bots have that the chromium.org bots don't, namely: * the ability to see > 5 most recent builds on a bot page * the ability to auto-restart when the master's configuration changes (this probably wouldn't ever fly in chromium land unless we could ensure that no active slaves were affected and we only restarted in-between builds). There's probably other features but these are the ones I recall discussing (and can think of at the moment).
(In reply to comment #5) > * the ability to "clobber" a bot without logging in I would really like us calling it "Force clean build" instead.
(In reply to comment #6) > (In reply to comment #5) > > * the ability to "clobber" a bot without logging in > > I would really like us calling it "Force clean build" instead. Not that I'm a fan of clobber, but "force clean build" is a bit ambiguous as well. There's a distinction between "do an incremental update & build", "do an incremental update but a full build" (what chromium calls a clobber), and "do a full checkout and a full build" (sometimes necessary and requires someone to log into the bot, even on chromium).
(In reply to comment #7) > (In reply to comment #6) > > (In reply to comment #5) > > > * the ability to "clobber" a bot without logging in > > > > I would really like us calling it "Force clean build" instead. > > Not that I'm a fan of clobber, but "force clean build" is a bit ambiguous as well. There's a distinction between "do an incremental update & build", "do an incremental update but a full build" (what chromium calls a clobber), and "do a full checkout and a full build" (sometimes necessary and requires someone to log into the bot, even on chromium). I referred to "build" as in building webkit but I guess it's confusing since each task triggered by a commit is referred to as "build" in the buildbot world :( Maybe "Delete build files and build" or "Clean compile"? I don't think we need to add "clean checkout" option since buildbot already does that when "svn up" step fails.