Bug 216662 - results.webkit.org: Use s3 for cold storage
Summary: results.webkit.org: Use s3 for cold storage
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: All All
: P2 Normal
Assignee: Jonathan Bedard
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-09-17 13:26 PDT by Jonathan Bedard
Modified: 2020-09-25 10:55 PDT (History)
5 users (show)

See Also:


Attachments
Patch (14.21 KB, patch)
2020-09-17 13:37 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (14.32 KB, patch)
2020-09-17 16:36 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (14.86 KB, patch)
2020-09-21 17:11 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (14.87 KB, patch)
2020-09-22 11:06 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (14.88 KB, patch)
2020-09-22 16:41 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (14.89 KB, patch)
2020-09-24 14:40 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (15.15 KB, patch)
2020-09-25 10:02 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Bedard 2020-09-17 13:26:42 PDT
Cassandra is a decent solution for caching results archives, but it's a pretty terrible solution for storing them permanently. We should us s3 for cold storage of result archives.
Comment 1 Radar WebKit Bug Importer 2020-09-17 13:26:54 PDT
<rdar://problem/69092010>
Comment 2 Jonathan Bedard 2020-09-17 13:37:24 PDT
Created attachment 409063 [details]
Patch
Comment 3 Jonathan Bedard 2020-09-17 16:36:56 PDT
Created attachment 409085 [details]
Patch
Comment 4 Jonathan Bedard 2020-09-21 17:11:23 PDT
Created attachment 409334 [details]
Patch
Comment 5 Jonathan Bedard 2020-09-22 11:06:06 PDT
Created attachment 409381 [details]
Patch
Comment 6 Aakash Jain 2020-09-22 15:16:15 PDT
Comment on attachment 409381 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=409381&action=review

rs=me

is this tested?

> Tools/ChangeLog:24
> +        (S3Archiver.save): Save an archive to S3 by it's hash.

Nit: it's => its

> Tools/Scripts/libraries/resultsdbpy/resultsdbpy/model/s3_archiver.py:59
> +        ttl_seconds = ttl_seconds or 60 * 60 * 24 * 365

might be a good idea to store this time limit (1 year) in a separate variable, and use that variable here.
Comment 7 Jonathan Bedard 2020-09-22 16:19:33 PDT
(In reply to Aakash Jain from comment #6)
> Comment on attachment 409381 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=409381&action=review
> 
> rs=me
> 
> is this tested?
> 
> > Tools/ChangeLog:24
> > +        (S3Archiver.save): Save an archive to S3 by it's hash.
> 
> Nit: it's => its
> 
> > Tools/Scripts/libraries/resultsdbpy/resultsdbpy/model/s3_archiver.py:59
> > +        ttl_seconds = ttl_seconds or 60 * 60 * 24 * 365
> 
> might be a good idea to store this time limit (1 year) in a separate
> variable, and use that variable here.

Not tested in an automated way, because that would entail mocking S3's API, and I didn't think it was worth the trouble. I did deploy this to our staging instances, though.
Comment 8 Jonathan Bedard 2020-09-22 16:41:33 PDT
Created attachment 409422 [details]
Patch
Comment 9 Jonathan Bedard 2020-09-24 14:40:52 PDT
Created attachment 409621 [details]
Patch
Comment 10 Jonathan Bedard 2020-09-25 10:02:01 PDT
Created attachment 409702 [details]
Patch
Comment 11 EWS 2020-09-25 10:55:08 PDT
Committed r267579: <https://trac.webkit.org/changeset/267579>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 409702 [details].