Bug 216662

Summary: results.webkit.org: Use s3 for cold storage
Product: WebKit Reporter: Jonathan Bedard <jbedard>
Component: Tools / TestsAssignee: Jonathan Bedard <jbedard>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, ap, lingho, ryanhaddad, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: All   
OS: All   
See Also: https://bugs.webkit.org/show_bug.cgi?id=207277
Attachments:
Description Flags
Patch
none
Patch
none
Patch
none
Patch
none
Patch
none
Patch
none
Patch none

Jonathan Bedard
Reported 2020-09-17 13:26:42 PDT
Cassandra is a decent solution for caching results archives, but it's a pretty terrible solution for storing them permanently. We should us s3 for cold storage of result archives.
Attachments
Patch (14.21 KB, patch)
2020-09-17 13:37 PDT, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch (14.32 KB, patch)
2020-09-17 16:36 PDT, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch (14.86 KB, patch)
2020-09-21 17:11 PDT, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch (14.87 KB, patch)
2020-09-22 11:06 PDT, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch (14.88 KB, patch)
2020-09-22 16:41 PDT, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch (14.89 KB, patch)
2020-09-24 14:40 PDT, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch (15.15 KB, patch)
2020-09-25 10:02 PDT, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Show Obsolete (6) View All Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2020-09-17 13:26:54 PDT
<rdar://problem/69092010>
Jonathan Bedard
Comment 2 2020-09-17 13:37:24 PDT
Created attachment 409063 [details] Patch
Jonathan Bedard
Comment 3 2020-09-17 16:36:56 PDT
Created attachment 409085 [details] Patch
Jonathan Bedard
Comment 4 2020-09-21 17:11:23 PDT
Created attachment 409334 [details] Patch
Jonathan Bedard
Comment 5 2020-09-22 11:06:06 PDT
Created attachment 409381 [details] Patch
Aakash Jain
Comment 6 2020-09-22 15:16:15 PDT
Comment on attachment 409381 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=409381&action=review rs=me is this tested? > Tools/ChangeLog:24 > + (S3Archiver.save): Save an archive to S3 by it's hash. Nit: it's => its > Tools/Scripts/libraries/resultsdbpy/resultsdbpy/model/s3_archiver.py:59 > + ttl_seconds = ttl_seconds or 60 * 60 * 24 * 365 might be a good idea to store this time limit (1 year) in a separate variable, and use that variable here.
Jonathan Bedard
Comment 7 2020-09-22 16:19:33 PDT
(In reply to Aakash Jain from comment #6) > Comment on attachment 409381 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=409381&action=review > > rs=me > > is this tested? > > > Tools/ChangeLog:24 > > + (S3Archiver.save): Save an archive to S3 by it's hash. > > Nit: it's => its > > > Tools/Scripts/libraries/resultsdbpy/resultsdbpy/model/s3_archiver.py:59 > > + ttl_seconds = ttl_seconds or 60 * 60 * 24 * 365 > > might be a good idea to store this time limit (1 year) in a separate > variable, and use that variable here. Not tested in an automated way, because that would entail mocking S3's API, and I didn't think it was worth the trouble. I did deploy this to our staging instances, though.
Jonathan Bedard
Comment 8 2020-09-22 16:41:33 PDT
Created attachment 409422 [details] Patch
Jonathan Bedard
Comment 9 2020-09-24 14:40:52 PDT
Created attachment 409621 [details] Patch
Jonathan Bedard
Comment 10 2020-09-25 10:02:01 PDT
Created attachment 409702 [details] Patch
EWS
Comment 11 2020-09-25 10:55:08 PDT
Committed r267579: <https://trac.webkit.org/changeset/267579> All reviewed patches have been landed. Closing bug and clearing flags on attachment 409702 [details].
Note You need to log in before you can comment on or make changes to this bug.