RESOLVED CONFIGURATION CHANGED 174261
Teach buildbot testers to download archives from s3 
https://bugs.webkit.org/show_bug.cgi?id=174261
Summary Teach buildbot testers to download archives from s3
Lucas Forschler
Reported 2017-07-07 10:33:27 PDT
We are now storing built product archives on s3. We need to teach our build workers to use these archives. This will involve teaching the download/extract tool about the location of the s3 object. I see two options: 1. Retrieve the archive via the aws/s3 tool this is probably the easiest, but will require installing the AWS tools on the bot. I'd prefer not to introduce another dependency. 2. Retrieve the archive via https this is likely the cleanest solution, since the files on s3 are public and served via https.
Attachments
Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2017-07-07 10:34:04 PDT
<rdar://problem/33184218>
Lucas Forschler
Comment 2 2017-07-07 10:37:18 PDT
note: The builders currently upload to the buildbot master. The master then transfers the file to S3. After this is working, we need to find a good way to remove the archive from the master... This could possibly be done in the same script where we transfer to S3. If the archive is uploaded successfully, simply delete the local copy.
Alexey Proskuryakov
Comment 3 2017-07-07 11:30:50 PDT
Are the S3 versions minified? What exactly is the impact on c ash log quality?
Lucas Forschler
Comment 4 2017-07-11 13:47:27 PDT
*** Bug 174384 has been marked as a duplicate of this bug. ***
Lucas Forschler
Comment 5 2017-07-11 13:48:29 PDT
We keep the full AND Minified version in S3. Full will stick around for 14 days Minified will stick around for 2.5 years. This change will have no impact on what is tested, as the bits are still the same as before. Eventually, we can opt to use the minified versions.
Lucas Forschler
Comment 6 2017-07-11 13:48:41 PDT
<rdar://problem/33184218>
Lucas Forschler
Comment 7 2017-07-11 13:49:09 PDT
Committed revision 219358.
Lucas Forschler
Comment 8 2017-07-11 13:49:52 PDT
This will require a master restart. But, I'm going to wait to give the queues a chance to catch up.... since some non-Mac test queues may not have builds on S3 yet. (that change was landed earlier this morning)
Lucas Forschler
Comment 9 2017-07-11 16:59:58 PDT
oops. Our nodes cannot reach s3 due to networking permissions. Will need to fix that before restarting the master. example fail: https://build.webkit.org/builders/Apple%20El%20Capitan%20Debug%20Test262%20%28Tests%29/builds/2613/steps/download-built-product/logs/stdio
Note You need to log in before you can comment on or make changes to this bug.