Bug 29062 – Consider integrating commit-queue with try servers

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: Tools / Tests

[Adam Barth]

Integrating commit-queue with the chromium try servers might improve pre-submit coverage.  I'm not sure how much work it will be or how reliable the try bots are.

[Mark Rowe (bdash)]

This is WebKit functionality so it should work with WebKit servers.  It doesn't make any sense to use a third-party server for that.

[Adam Barth]

Great!  Where are the WebKit try servers?

[Mark Rowe (bdash)]

The buildbot configuration is in SVN.

[Adam Barth]

(In reply to comment #3)
> The buildbot configuration is in SVN.

I don't understand.  Can we use this information to check whether a patch builds and passes tests on, say, the GTK port without actually committing to svn.webkit.org?

The advantage of sending patches to the chromium try servers is that they will build and test a patch on Windows, Mac, and Linux without disrupting other contributors to WebKit or to Chromium.

It would be super useful to have similar servers for WebKit proper.  Selfishly, I'd really like to see Windows, Qt, and GTK try servers because I can already building and test Mac myself.

[Mark Rowe (bdash)]

(In reply to comment #4)
> (In reply to comment #3)
> > The buildbot configuration is in SVN.
> 
> I don't understand.  Can we use this information to check whether a patch
> builds and passes tests on, say, the GTK port without actually committing to
> svn.webkit.org?

I'm saying that if people want try servers set up, they're welcome to do so.  The buildbot configuration is all in SVN.

> The advantage of sending patches to the chromium try servers is that they will
> build and test a patch on Windows, Mac, and Linux without disrupting other
> contributors to WebKit or to Chromium.

And yet it won't actually test any of the Mac, Windows and Linux ports that build on build.webkit.org, which makes it much less useful for telling you whether you're going to break any of the builds.

> It would be super useful to have similar servers for WebKit proper.  Selfishly,
> I'd really like to see Windows, Qt, and GTK try servers because I can already
> building and test Mac myself.

[Marc-Antoine Ruel]

Setup:
I highly recommend to use a separate master for try jobs. Since slaves are more frequently added, removed and configuration changes occurs more frequently, it's better to not constantly break the continuous build, especially during the initial setup. The try server config could be in http://trac.webkit.org/browser/trunk/WebKitTools/BuildSlaveSupport/try.webkit.org-config or something similar, depending on the host name you choose.


Maintenance:
Note that the current chromium's try server is not maintenance free so the slaves need a way to be fixed manually from time to time. Since webkit's tests aren't as messy as chromium's ones, it shouldn't be too frequent. Having slaves dispersed across the global doesn't help.


Triggering:
You guys need to decide what method is used to triggers try jobs. Example ways are:
- a patch checked-in in a specific svn repository that is polled by buildbot (implemented)
- a direct http PUSH to the master (implemented, fast but unsafe)
- bugzilla poll on the master (probably best with webkit workflow but not implemented, probably similar to commit queue)
- buildbot try (requires users to have buildbot+twisted code locally, not preferred, at least not to me)

I don't think anyone wants to open a http port directly to the buildbot master for try jobs. I think having a trigger directly from bugzilla would be interesting. I'm not used to the webkit workflow so I can't tell which method is better.


Access control:
I'd say you'd probably want to limit try job trigger to webkit committers. That's what we do. A committer can send a try job on behalf of a contributor.


Hosting:
Google will provide slaves for multiple platforms in a DMZ. We may not have full coverage but we definitely can help. Dimitry may tell more about his plans.


Try server stuffing and supported platforms:
If try job scheduling is manual, 2 per platform would be sufficient. If the try jobs are automatically triggered by bugzilla patch updates, I'd say around 8 per platform depending on speed and real usage.

In any case, it's better to start with manual trigger for a few week to see rough usage and fix issues and make it automatic IIF concluding. It's easier to start with a small amount of slaves and scale accordingly. I'd propose to start with 8 platforms:
(apple) mac leopard intel debug
(apple) windows debug
gtk linux release
qt linux release
chromium win debug
chromium mac debug
chromium linux debug

[Adam Barth]

Adding Mark in case he missed the above.

[Mark Rowe (bdash)]

A few thoughts:
* Polling Bugzilla from the master seems like an approach that lacks flexibility.
* I'm not sure that I see the issue with pushing patches to the master, so long as they are authenticated and access is limited to committers.
* A SnowLeopard builder would be more useful than a Leopard builder.
* It seems backwards to have try servers for platforms that we don't have build bots for.

[Jonathan Bedard]

Closing this bug, we migrated all of EWS (including Commit-Queue, now Merge-Queue) to buildbot.