Bug 73484 – Mac slaves take 7+ minutes to upload or download a build

• Component: Tools / Tests
• Status: RESOLVED CONFIGURATION CHANGED

[Adam Roben (:aroben)]

It takes our Mac slaves 7+ minutes to upload or download a build. Testing with curl shows that it can be done in under 1 minute. We should figure out how to make this faster!

[Adam Roben (:aroben)]

We could use curl for downloading. But we need some better solution for uploading. Mark suggests that we figure out what it is about Buildbot that is going so slowly, fix it, and upstream our fix.

[Adam Roben (:aroben)]

Looks like buildbot transfers files in 16kb blocks. Maybe a different block size would be faster?

[Adam Roben (:aroben)]

I set up a master and slave on the same machine and was able to transfer one of the archives in 24 seconds with the 16KB block size.

[Adam Roben (:aroben)]

cp for the same file takes 2.8 seconds, while scp takes 7 seconds.

[Adam Roben (:aroben)]

Mark noticed that the buildbot process on build.webkit.org is taking up >50% CPU pretty much constantly. He also thinks buildbot is single-threaded, so if we're spending lots of time servicing web requests (e.g.) then we won't have time to service upload requests very frequently.

[Adam Roben (:aroben)]

According to <http://trac.buildbot.net/ticket/2055#comment:8>, "Buildbot makes heavier use of threads in 0.8.4 and above".

[Adam Roben (:aroben)]

…though looking at the changes since 0.8.3 (the version we're currently using), the only additional use of threads seems to be when interacting with the database.

[Adam Roben (:aroben)]

If Mark is right, it seems like it's going to be hard to make buildbot do the transfers any faster. So maybe we need to do them some other way.

Using curl for downloads is super easy. We could use it for uploads, too, if we created a URL to which we could POST builds. Then we'd just have to handle authentication.

I'm working on teaching buildbot how to use htdigest files for bug 73353. Maybe we could change passwords.json to an htdigest file and use that for authentication with the upload URL?

[Adam Roben (:aroben)]

Just using curl for downloads would speed up the test bots quite a bit. And presumably it would lessen the CPU load on the master too. We should try that first and see if the uploads get any faster for free!

[Adam Roben (:aroben)]

Created attachment 117285 [details]
Use curl for downloading builds to test slaves

[Adam Roben (:aroben)]

Comment on attachment 117285 [details]
Use curl for downloading builds to test slaves

I'll land this tomorrow morning.

[Adam Roben (:aroben)]

Comment on attachment 117285 [details]
Use curl for downloading builds to test slaves

Landed as r101662 with a followup in r101663.

[Adam Roben (:aroben)]

Hooray, down to 40 seconds using curl: http://build.webkit.org/builders/Lion%20Intel%20Debug%20%28Tests%29/builds/1537/steps/download-built-product

[Adam Roben (:aroben)]

It isn't clear this has had a huge effect on CPU usage on the master though.

[Eric Seidel (no email)]

Comment on attachment 117285 [details]
Use curl for downloading builds to test slaves

Cleared Mark Rowe's review+ from obsolete attachment 117285 [details] so that this bug does not appear in http://webkit.org/pending-commit.

[Ahmad Saleem]

@Aakash & @Ryan - anything to do here, or we can close this?

[Aakash Jain]

We don't download build product from buildbot server anymore (we upload/download it to/from S3).