Upgrade Buildbot instance on build.webkit.org to latest Buildbot

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

We are running Buildbot version 0.8.6p1 on build.webkit.org (https://build.webkit.org/about). This is more than 5 years old version.

Buildbot 0.9 was released last year and has a lot of new features, including brand-new UI (See https://nine.buildbot.net).

We should upgrade the buildbot instance to the latest version.

[Carlos Alberto Lopez Perez]

I find the waterfall view on 0.9 to be really less useful than the one from 0.8 because it will not longer be possible to see the details of what step failed without hovering the mouse over each red rectangle.

Check for example: http://buildbot.scons.org/#/waterfall (you need to move the mouse over each red build to see what went wrong). Compare that with our current waterfall https://build.webkit.org/waterfall where the details of what exactly failed are wrote directly as text on the red rectangles, so you can see exactly what went wrong much easier.

This is a known limitation of 0.9 as documented here http://docs.buildbot.net/current/manual/cfg-www.html

[Aakash Jain]

Also see: <rdar://problem/26245354>

[Simon Fraser (smfr)]

Agreed, the new waterfall is horrible.

[Fujii Hironori]

[webkit-dev] Buildbot upgrade on build.webkit.org
https://lists.webkit.org/pipermail/webkit-dev/2017-December/029804.html

Waterfall view should display steps information without hovering over · Issue #3884 · buildbot/buildbot
https://github.com/buildbot/buildbot/issues/3884

[Don Olmstead]

Is there any status for this? We'd like to test out a buildbot for playstation locally before turning it on in trunk. Would be great to just target the latest buildbot. The latest release at this time is 2.0.1, http://docs.buildbot.net/current/relnotes/index.html .

[Aakash Jain]

(In reply to Don Olmstead from comment #5)
> Is there any status for this? 
Haven't started work on this. One of the main blocker is the concern people have about waterfall/console view of new Buildbot.

[Michael Catanzaro]

FWIW I think waterfall view is not very important, because we have https://build.webkit.org/dashboard/, which is an elaborate workaround to avoid the need to ever look at the waterfall. Is this dashboard good enough for people used to looking at the waterfall?

For GTK/WPE, all our bots should be on the dashboard.

[Ryosuke Niwa]

(In reply to Michael Catanzaro from comment #7)
> FWIW I think waterfall view is not very important, because we have
> https://build.webkit.org/dashboard/, which is an elaborate workaround to
> avoid the need to ever look at the waterfall. Is this dashboard good enough
> for people used to looking at the waterfall?

I never use the dashboard because I can never reason what's happening.

[Don Olmstead]

(In reply to Ryosuke Niwa from comment #8)
> (In reply to Michael Catanzaro from comment #7)
> > FWIW I think waterfall view is not very important, because we have
> > https://build.webkit.org/dashboard/, which is an elaborate workaround to
> > avoid the need to ever look at the waterfall. Is this dashboard good enough
> > for people used to looking at the waterfall?
> 
> I never use the dashboard because I can never reason what's happening.

For Sony we're fine with just having the dashboard and plan on adding our bots to it after landing. Having a single buildbot version to target for per commit builds as well as EWS would reduce the burden for running infrastructure for us.

Its my understanding that we can customize our own dashboards using buildbot so is there something we could expand on the dashboard that would make the waterfall users happy? We can always expand the dashboard over time with additional functionality. I'm not sure realistically how long we can keep up with an old buildbot especially with the maintenance burden and potential security risks.

[Aakash Jain]

I am going to start working on this Buildbot upgrade soon.

As we are running an ancient version of Buildbot on build.webkit.org:
- we risk Security
- maintenance cost is higher (e.g.: many scripts (like dashboard) needs to talk to both old and new Buildbot API, we can't easily share code between build.webkit.org and ews-build.webkit.org)
- users don't have a consistent experience on build.webkit.org and EWS buildbot (ews-build.webkit.org)

Many users are eagerly waiting for build.webkit.org upgrade. There might still be concerns about waterfall view (and I do agree with waterfall view being not so useful in newer Buildbot). At the same time, we need to move forward, we can't stick to ancient version of Buildbot forever. I talked to some of the people who had concerns previously, and they don't object to upgrade anymore. Also, Bot watchers in Apple (whom I would have expected the most frequent users of waterfall/console views) don't actually use waterfall/console views, they use the dashboard instead.

[Aakash Jain]

Also, the plan is to upgrade to latest Buildbot, and make the Buildbot configuration Python 3 compatible.

[Aakash Jain]

We have upgraded build.webkit.org to new Buildbot (v2.10.0, released on 2021-01-02).

[Aakash Jain]

We have now upgraded build.webkit.org to latest Buildbot (v2.10.1, released on 2021-01-29).

[Aakash Jain]

This is a brand new server. Buildbot doesn't support migrating data from old version. 

Old instance is accessible at https://build-old.webkit.org

[Aakash Jain]

I just upgraded build.webkit.org to Buildbot v2.10.2 (released on 2021-03-07).