Bug 73484

Summary: Mac slaves take 7+ minutes to upload or download a build
Product: WebKit Reporter: Adam Roben (:aroben) <aroben>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: lforschler, mrowe
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Description Flags
Use curl for downloading builds to test slaves none

Description Adam Roben (:aroben) 2011-11-30 13:41:13 PST
It takes our Mac slaves 7+ minutes to upload or download a build. Testing with curl shows that it can be done in under 1 minute. We should figure out how to make this faster!
Comment 1 Adam Roben (:aroben) 2011-11-30 13:41:48 PST
We could use curl for downloading. But we need some better solution for uploading. Mark suggests that we figure out what it is about Buildbot that is going so slowly, fix it, and upstream our fix.
Comment 2 Adam Roben (:aroben) 2011-11-30 13:45:21 PST
Looks like buildbot transfers files in 16kb blocks. Maybe a different block size would be faster?
Comment 3 Adam Roben (:aroben) 2011-11-30 14:48:45 PST
I set up a master and slave on the same machine and was able to transfer one of the archives in 24 seconds with the 16KB block size.
Comment 4 Adam Roben (:aroben) 2011-11-30 14:51:25 PST
cp for the same file takes 2.8 seconds, while scp takes 7 seconds.
Comment 5 Adam Roben (:aroben) 2011-11-30 14:52:18 PST
Mark noticed that the buildbot process on build.webkit.org is taking up >50% CPU pretty much constantly. He also thinks buildbot is single-threaded, so if we're spending lots of time servicing web requests (e.g.) then we won't have time to service upload requests very frequently.
Comment 6 Adam Roben (:aroben) 2011-11-30 14:55:12 PST
According to <http://trac.buildbot.net/ticket/2055#comment:8>, "Buildbot makes heavier use of threads in 0.8.4 and above".
Comment 7 Adam Roben (:aroben) 2011-11-30 14:59:15 PST
…though looking at the changes since 0.8.3 (the version we're currently using), the only additional use of threads seems to be when interacting with the database.
Comment 8 Adam Roben (:aroben) 2011-11-30 15:07:42 PST
If Mark is right, it seems like it's going to be hard to make buildbot do the transfers any faster. So maybe we need to do them some other way.

Using curl for downloads is super easy. We could use it for uploads, too, if we created a URL to which we could POST builds. Then we'd just have to handle authentication.

I'm working on teaching buildbot how to use htdigest files for bug 73353. Maybe we could change passwords.json to an htdigest file and use that for authentication with the upload URL?
Comment 9 Adam Roben (:aroben) 2011-11-30 15:09:06 PST
Just using curl for downloads would speed up the test bots quite a bit. And presumably it would lessen the CPU load on the master too. We should try that first and see if the uploads get any faster for free!
Comment 10 Adam Roben (:aroben) 2011-11-30 15:24:49 PST
Created attachment 117285 [details]
Use curl for downloading builds to test slaves
Comment 11 Adam Roben (:aroben) 2011-11-30 15:27:38 PST
Comment on attachment 117285 [details]
Use curl for downloading builds to test slaves

I'll land this tomorrow morning.
Comment 12 Adam Roben (:aroben) 2011-12-01 05:51:55 PST
Comment on attachment 117285 [details]
Use curl for downloading builds to test slaves

Landed as r101662 with a followup in r101663.
Comment 13 Adam Roben (:aroben) 2011-12-01 05:52:00 PST
Hooray, down to 40 seconds using curl: http://build.webkit.org/builders/Lion%20Intel%20Debug%20%28Tests%29/builds/1537/steps/download-built-product
Comment 14 Adam Roben (:aroben) 2011-12-01 05:53:30 PST
It isn't clear this has had a huge effect on CPU usage on the master though.
Comment 15 Eric Seidel (no email) 2011-12-19 10:49:47 PST
Comment on attachment 117285 [details]
Use curl for downloading builds to test slaves

Cleared Mark Rowe's review+ from obsolete attachment 117285 [details] so that this bug does not appear in http://webkit.org/pending-commit.