73484 2011-11-30 13:41:13 -0800 Mac slaves take 7+ minutes to upload or download a build 2024-07-08 07:39:24 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED CONFIGURATION CHANGED P2 Normal --- 1 aroben webkit-unassigned aakash_jain ahmad.saleem792 lforschler mrowe ryanhaddad oldest_to_newest 512082 0 aroben 2011-11-30 13:41:13 -0800 It takes our Mac slaves 7+ minutes to upload or download a build. Testing with curl shows that it can be done in under 1 minute. We should figure out how to make this faster! 512083 1 aroben 2011-11-30 13:41:48 -0800 We could use curl for downloading. But we need some better solution for uploading. Mark suggests that we figure out what it is about Buildbot that is going so slowly, fix it, and upstream our fix. 512090 2 aroben 2011-11-30 13:45:21 -0800 Looks like buildbot transfers files in 16kb blocks. Maybe a different block size would be faster? 512150 3 aroben 2011-11-30 14:48:45 -0800 I set up a master and slave on the same machine and was able to transfer one of the archives in 24 seconds with the 16KB block size. 512155 4 aroben 2011-11-30 14:51:25 -0800 cp for the same file takes 2.8 seconds, while scp takes 7 seconds. 512158 5 aroben 2011-11-30 14:52:18 -0800 Mark noticed that the buildbot process on build.webkit.org is taking up >50% CPU pretty much constantly. He also thinks buildbot is single-threaded, so if we're spending lots of time servicing web requests (e.g.) then we won't have time to service upload requests very frequently. 512163 6 aroben 2011-11-30 14:55:12 -0800 According to <http://trac.buildbot.net/ticket/2055#comment:8>, "Buildbot makes heavier use of threads in 0.8.4 and above". 512165 7 aroben 2011-11-30 14:59:15 -0800 …though looking at the changes since 0.8.3 (the version we're currently using), the only additional use of threads seems to be when interacting with the database. 512170 8 aroben 2011-11-30 15:07:42 -0800 If Mark is right, it seems like it's going to be hard to make buildbot do the transfers any faster. So maybe we need to do them some other way. Using curl for downloads is super easy. We could use it for uploads, too, if we created a URL to which we could POST builds. Then we'd just have to handle authentication. I'm working on teaching buildbot how to use htdigest files for bug 73353. Maybe we could change passwords.json to an htdigest file and use that for authentication with the upload URL? 512171 9 aroben 2011-11-30 15:09:06 -0800 Just using curl for downloads would speed up the test bots quite a bit. And presumably it would lessen the CPU load on the master too. We should try that first and see if the uploads get any faster for free! 512180 10 117285 aroben 2011-11-30 15:24:49 -0800 Created attachment 117285 Use curl for downloading builds to test slaves 512185 11 117285 aroben 2011-11-30 15:27:38 -0800 Comment on attachment 117285 Use curl for downloading builds to test slaves I'll land this tomorrow morning. 512710 12 117285 aroben 2011-12-01 05:51:55 -0800 Comment on attachment 117285 Use curl for downloading builds to test slaves Landed as r101662 with a followup in r101663. 512711 13 aroben 2011-12-01 05:52:00 -0800 Hooray, down to 40 seconds using curl: http://build.webkit.org/builders/Lion%20Intel%20Debug%20%28Tests%29/builds/1537/steps/download-built-product 512712 14 aroben 2011-12-01 05:53:30 -0800 It isn't clear this has had a huge effect on CPU usage on the master though. 524007 15 117285 eric 2011-12-19 10:49:47 -0800 Comment on attachment 117285 Use curl for downloading builds to test slaves Cleared Mark Rowe's review+ from obsolete attachment 117285 so that this bug does not appear in http://webkit.org/pending-commit. 2044949 16 ahmad.saleem792 2024-07-06 13:58:01 -0700 @Aakash & @Ryan - anything to do here, or we can close this? 2045073 17 aakash_jain 2024-07-08 07:39:24 -0700 We don't download build product from buildbot server anymore (we upload/download it to/from S3). 117285 2011-11-30 15:24:49 -0800 2011-12-19 10:49:47 -0800 Use curl for downloading builds to test slaves bug-73484-20111130182448.patch text/plain 3597 aroben U3VidmVyc2lvbiBSZXZpc2lvbjogMTAxNTM0CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggN2QyNjIxNmM4NzllMmZlOTA2M2JhOGFhOTkwYzAxODNl MzBiZWViNi4uYWM4Nzg3NGZjNGQ3ZTBmNzM2ODUyYmEyZDk2ZjlhMDFhNTc2ZDE2NSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSw1ICsxLDMw IEBACiAyMDExLTExLTMwICBBZGFtIFJvYmVuICA8YXJvYmVuQGFwcGxlLmNvbT4KIAorICAgICAg ICBVc2UgY3VybCBmb3IgZG93bmxvYWRpbmcgYnVpbGRzIHRvIHRlc3Qgc2xhdmVzCisKKyAgICAg ICAgQnVpbGRib3QncyBidWlsdC1pbiBmaWxlIHRyYW5zZmVyIGNhcGFiaWxpdGllcyBhcmUgdHJh bnNmZXJyaW5nIGZpbGVzIG11Y2ggbW9yZSBzbG93bHkgdGhhbgorICAgICAgICBjdXJsIGRvZXMu IFRoaXMgc2VlbXMgdG8gYmUgZHVlIHRvIGEgY29tYmluYXRpb24gb2YgYnVpbGRib3QgYmVpbmcg c2luZ2xlLXRocmVhZGVkIChhbmQgdGh1cworICAgICAgICBvZnRlbiBidXN5IHNlcnZpY2luZyB3 ZWIgcmVxdWVzdHMsIGV0Yy4pIGFuZCB1c2luZyBhbiBJUEMgbWVjaGFuaXNtIHdpdGggZmFpcmx5 IGhpZ2gKKyAgICAgICAgb3ZlcmhlYWQgKGNvdXJ0ZXN5IG9mIHRoZSBUd2lzdGVkIGZyYW1ld29y aykuCisKKyAgICAgICAgRXZlbnR1YWxseSB3ZSdkIGxpa2UgdG8gbWFrZSBCdWlsZGJvdCdzIGJ1 aWx0LWluIHRyYW5zZmVycmluZyBtb3JlIGVmZmljaWVudC4gSW4gdGhlCisgICAgICAgIG1lYW50 aW1lIHdlJ2xsIHRyeSB1c2luZyBjdXJsIGZvciBkb3dubG9hZHMgYW5kIGtlZXAgdGhpbmtpbmcg YWJvdXQgYSBzb2x1dGlvbiBmb3IgdXBsb2Fkcy4KKyAgICAgICAgVGhpcyBzaG91bGQgbWFrZSB0 aGUgdGVzdCBzbGF2ZXMgcXVpdGUgYSBiaXQgZmFzdGVyLCBhbmQgc2hvdWxkIGVhc2UgdGhlIENQ VSBsb2FkIG9uCisgICAgICAgIGJ1aWxkLndlYmtpdC5vcmcgYSBiaXQuCisKKyAgICAgICAgUGFy dCBvZiA8aHR0cDovL3dlYmtpdC5vcmcvYi83MzQ4ND4gTWFjIHNsYXZlcyB0YWtlIDcrIG1pbnV0 ZXMgdG8gdXBsb2FkIG9yIGRvd25sb2FkIGEgYnVpbGQKKworICAgICAgICBSZXZpZXdlZCBieSBO T0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIEJ1aWxkU2xhdmVTdXBwb3J0L2J1aWxkLndlYmtp dC5vcmctY29uZmlnL21hc3Rlci5jZmc6IEFkZGVkIHRoZSBDcmVhdGVXZWJLaXRCdWlsZERpcmVj dG9yeQorICAgICAgICB0byBUZXN0IHNsYXZlcy4gSXQncyBydW4ganVzdCBiZWZvcmUgRG93bmxv YWRCdWlsdFByb2R1Y3QuCisgICAgICAgIChEb3dubG9hZEJ1aWx0UHJvZHVjdCk6IENoYW5nZWQg dG8gYmUgYSBub3JtYWwgU2hlbGxDb21tYW5kIHRoYXQgaW52b2tlcyBjdXJsIHRvIHBlcmZvcm0g dGhlCisgICAgICAgIGRvd25sb2FkLgorICAgICAgICAoQ3JlYXRlV2ViS2l0QnVpbGREaXJlY3Rv cnkpOiBBZGRlZC4gQ3JlYXRlcyB0aGUgV2ViS2l0QnVpbGQgZGlyZWN0b3J5IG9uIGEgc2xhdmUu CisgICAgICAgIEZpbGVEb3dubG9hZCB1c2VkIHRvIGRvIHRoaXMgZm9yIHVzIGF1dG9tYXRpY2Fs bHkuCisKKzIwMTEtMTEtMzAgIEFkYW0gUm9iZW4gIDxhcm9iZW5AYXBwbGUuY29tPgorCiAgICAg ICAgIE1ha2UgTlJXVCBzaG93IERSVC9XVFIgYnVpbGQgbG9ncyB3aGVuIHRoZSBidWlsZCBmYWls cyBvciAtLXZlcmJvc2UgaXMgcGFzc2VkCiAKICAgICAgICAgVGhpcyB3aWxsIG1ha2UgaXQgYSBs b3QgZWFzaWVyIHRvIGludmVzdGlnYXRlIGJ1aWxkIGZhaWx1cmVzIG9uIHRoZSBib3RzLgpkaWZm IC0tZ2l0IGEvVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcv bWFzdGVyLmNmZyBiL1Rvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2J1aWxkLndlYmtpdC5vcmctY29u ZmlnL21hc3Rlci5jZmcKaW5kZXggOGYzMTgxNGU5NjU5MGVmMmFmODAxZjcwNzA0ZmVmNmY2NTZj NTljZi4uYzlhM2NhMWJhOTcyZjcyMjU4YTBkODJmYTQzN2M1ZjFkODhhZTA2YSAxMDA2NDQKLS0t IGEvVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvbWFzdGVy LmNmZworKysgYi9Ub29scy9CdWlsZFNsYXZlU3VwcG9ydC9idWlsZC53ZWJraXQub3JnLWNvbmZp Zy9tYXN0ZXIuY2ZnCkBAIC0xNjgsMTQgKzE2OCwxNiBAQCBjbGFzcyBVcGxvYWRCdWlsdFByb2R1 Y3QodHJhbnNmZXIuRmlsZVVwbG9hZCk6CiAgICAgICAgIHJldHVybiByZXN1bHQKIAogCi1jbGFz cyBEb3dubG9hZEJ1aWx0UHJvZHVjdCh0cmFuc2Zlci5GaWxlRG93bmxvYWQpOgotICAgIHNsYXZl ZGVzdCA9IFdpdGhQcm9wZXJ0aWVzKCJXZWJLaXRCdWlsZC8lKGNvbmZpZ3VyYXRpb24pcy56aXAi KQotICAgIG1hc3RlcnNyYyA9IFdpdGhQcm9wZXJ0aWVzKCJhcmNoaXZlcy8lKGZ1bGxQbGF0Zm9y bSlzLSUoYXJjaGl0ZWN0dXJlKXMtJShjb25maWd1cmF0aW9uKXMvJShnb3RfcmV2aXNpb24pcy56 aXAiKQorY2xhc3MgRG93bmxvYWRCdWlsdFByb2R1Y3Qoc2hlbGwuU2hlbGxDb21tYW5kKToKKyAg ICBjb21tYW5kID0gWyJjdXJsIiwgIi0tb3V0cHV0IiwgV2l0aFByb3BlcnRpZXMoIldlYktpdEJ1 aWxkLyUoY29uZmlndXJhdGlvbilzLnppcCIpLCBXaXRoUHJvcGVydGllcyhjWyJidWlsZGJvdFVS TCJdICsgImFyY2hpdmVzLyUoZnVsbFBsYXRmb3JtKXMtJShhcmNoaXRlY3R1cmUpcy0lKGNvbmZp Z3VyYXRpb24pcy8lKGdvdF9yZXZpc2lvbilzLnppcCIpXQogICAgIGhhbHRPbkZhaWx1cmUgPSBU cnVlCiAgICAgZmx1bmtPbkZhaWx1cmUgPSBUcnVlCiAKLSAgICBkZWYgX19pbml0X18oc2VsZik6 Ci0gICAgICAgIHRyYW5zZmVyLkZpbGVEb3dubG9hZC5fX2luaXRfXyhzZWxmLCBzZWxmLm1hc3Rl cnNyYywgc2VsZi5zbGF2ZWRlc3QpCisKK2NsYXNzIENyZWF0ZVdlYktpdEJ1aWxkRGlyZWN0b3J5 KHNoZWxsLlNoZWxsQ29tbWFuZCk6CisgICAgY29tbWFuZCA9IFsibWtkaXIiLCAiLXAiLCAiV2Vi S2l0QnVpbGQiXQorICAgIGhhbHRPbkZhaWx1cmUgPSBUcnVlCisgICAgZmx1bmtPbkZhaWx1cmUg PSBUcnVlCiAKIAogY2xhc3MgUnVuSmF2YVNjcmlwdENvcmVUZXN0cyhzaGVsbC5UZXN0KToKQEAg LTY0Miw2ICs2NDQsNyBAQCBjbGFzcyBUZXN0RmFjdG9yeShGYWN0b3J5KToKICAgICBFeHRyYWN0 VGVzdFJlc3VsdHNDbGFzcyA9IEV4dHJhY3RUZXN0UmVzdWx0cwogICAgIGRlZiBfX2luaXRfXyhz ZWxmLCBwbGF0Zm9ybSwgY29uZmlndXJhdGlvbiwgYXJjaGl0ZWN0dXJlcywgKiprd2FyZ3MpOgog ICAgICAgICBGYWN0b3J5Ll9faW5pdF9fKHNlbGYsIHBsYXRmb3JtLCBjb25maWd1cmF0aW9uLCBh cmNoaXRlY3R1cmVzLCBGYWxzZSwgKiprd2FyZ3MpCisgICAgICAgIHNlbGYuYWRkU3RlcChDcmVh dGVXZWJLaXRCdWlsZERpcmVjdG9yeSkKICAgICAgICAgc2VsZi5hZGRTdGVwKERvd25sb2FkQnVp bHRQcm9kdWN0KQogICAgICAgICBzZWxmLmFkZFN0ZXAoRXh0cmFjdEJ1aWx0UHJvZHVjdCkKICAg ICAgICAgc2VsZi5hZGRTdGVwKFJ1bkphdmFTY3JpcHRDb3JlVGVzdHMsIGJ1aWxkSlNDVG9vbD1G YWxzZSkK