174261 2017-07-07 10:33:27 -0700 Teach buildbot testers to download archives from s3 2017-07-11 16:59:58 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED CONFIGURATION CHANGED InRadar P2 Normal --- 1 lforschler lforschler ap lforschler webkit-bug-importer oldest_to_newest 1326514 0 lforschler 2017-07-07 10:33:27 -0700 We are now storing built product archives on s3. We need to teach our build workers to use these archives. This will involve teaching the download/extract tool about the location of the s3 object. I see two options: 1. Retrieve the archive via the aws/s3 tool this is probably the easiest, but will require installing the AWS tools on the bot. I'd prefer not to introduce another dependency. 2. Retrieve the archive via https this is likely the cleanest solution, since the files on s3 are public and served via https. 1326516 1 webkit-bug-importer 2017-07-07 10:34:04 -0700 <rdar://problem/33184218> 1326518 2 lforschler 2017-07-07 10:37:18 -0700 note: The builders currently upload to the buildbot master. The master then transfers the file to S3. After this is working, we need to find a good way to remove the archive from the master... This could possibly be done in the same script where we transfer to S3. If the archive is uploaded successfully, simply delete the local copy. 1326536 3 ap 2017-07-07 11:30:50 -0700 Are the S3 versions minified? What exactly is the impact on c ash log quality? 1327711 4 lforschler 2017-07-11 13:47:27 -0700 *** Bug 174384 has been marked as a duplicate of this bug. *** 1327712 5 lforschler 2017-07-11 13:48:29 -0700 We keep the full AND Minified version in S3. Full will stick around for 14 days Minified will stick around for 2.5 years. This change will have no impact on what is tested, as the bits are still the same as before. Eventually, we can opt to use the minified versions. 1327714 6 lforschler 2017-07-11 13:48:41 -0700 <rdar://problem/33184218> 1327716 7 lforschler 2017-07-11 13:49:09 -0700 Committed revision 219358. 1327718 8 lforschler 2017-07-11 13:49:52 -0700 This will require a master restart. But, I'm going to wait to give the queues a chance to catch up.... since some non-Mac test queues may not have builds on S3 yet. (that change was landed earlier this morning) 1327845 9 lforschler 2017-07-11 16:59:58 -0700 oops. Our nodes cannot reach s3 due to networking permissions. Will need to fix that before restarting the master. example fail: https://build.webkit.org/builders/Apple%20El%20Capitan%20Debug%20Test262%20%28Tests%29/builds/2613/steps/download-built-product/logs/stdio