This is a step towards storing archives on S3 instead of in Cassandra. Apart from the drive-by bug fix of adding ttl to Cassandra storage, this change will be functionally identical to the existing code.
Created attachment 389831 [details] Patch
Created attachment 389876 [details] Patch
Created attachment 390022 [details] Patch
Comment on attachment 390022 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=390022&action=review > Tools/resultsdbpy/resultsdbpy/model/archiver.py:27 > +class Archiver(object): I guess a delete method is also needed. Besides, I think a timestamp for the archive will be also very useful. We then can have something auto clean the archives. > Tools/resultsdbpy/resultsdbpy/model/cassandra_archiver.py:31 > + MAX_ARCHIVE = 500 * 1024 * 1024 # Archives should be smaller than 500 MB I think I saw some debug archives have 600~ mb
(In reply to Zhifei Fang from comment #4) > Comment on attachment 390022 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=390022&action=review > > > Tools/resultsdbpy/resultsdbpy/model/archiver.py:27 > > +class Archiver(object): > > I guess a delete method is also needed. Besides, I think a timestamp for the > archive will be also very useful. We then can have something auto clean the > archives. The retain_for parameter is intended to provide automatic clean up. The timestamp is actually less useful because we're sorting them by a primary key hash, meaning you'd need to actually request the specific archive you want deleted. (you could do that through the management table). Generally, I've avoided putting any specific delete functions into the results database because while we occasionally need to do it, it's not something that the API generally supports. It should be a task that requires management access to the database. Even if we did want a delete function, I don't think it should be part of this patch since this is intended as a no-op and the existing code didn't support deletion either. > > > Tools/resultsdbpy/resultsdbpy/model/cassandra_archiver.py:31 > > + MAX_ARCHIVE = 500 * 1024 * 1024 # Archives should be smaller than 500 MB > > I think I saw some debug archives have 600~ mb It's not for builds, only for test result archives, which should all be in the ~50 MB range. We could revisit this technique for builds later.
unofficially r=me
rs=me
Created attachment 390251 [details] Patch for landing
Comment on attachment 390251 [details] Patch for landing Clearing flags on attachment: 390251 Committed r256172: <https://trac.webkit.org/changeset/256172>
All reviewed patches have been landed. Closing bug.
<rdar://problem/59311107>