Bug 207277 – results.webkit.org: Abstract archive storage solutions

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Jonathan Bedard]

This is a step towards storing archives on S3 instead of in Cassandra. Apart from the drive-by bug fix of adding ttl to Cassandra storage, this change will be functionally identical to the existing code.

[Jonathan Bedard]

Created attachment 389831 [details]
Patch

[Jonathan Bedard]

Created attachment 389876 [details]
Patch

[Jonathan Bedard]

Created attachment 390022 [details]
Patch

[Zhifei Fang]

Comment on attachment 390022 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=390022&action=review

> Tools/resultsdbpy/resultsdbpy/model/archiver.py:27
> +class Archiver(object):

I guess a delete method is also needed. Besides, I think a timestamp for the archive will be also very useful. We then can have something auto clean the archives.

> Tools/resultsdbpy/resultsdbpy/model/cassandra_archiver.py:31
> +    MAX_ARCHIVE = 500 * 1024 * 1024  # Archives should be smaller than 500 MB

I think I saw some debug archives have 600~ mb

[Jonathan Bedard]

(In reply to Zhifei Fang from comment #4)
> Comment on attachment 390022 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=390022&action=review
> 
> > Tools/resultsdbpy/resultsdbpy/model/archiver.py:27
> > +class Archiver(object):
> 
> I guess a delete method is also needed. Besides, I think a timestamp for the
> archive will be also very useful. We then can have something auto clean the
> archives.

The retain_for parameter is intended to provide automatic clean up.

The timestamp is actually less useful because we're sorting them by a primary key hash, meaning you'd need to actually request the specific archive you want deleted. (you could do that through the management table).

Generally, I've avoided putting any specific delete functions into the results database because while we occasionally need to do it, it's not something that the API generally supports. It should be a task that requires management access to the database. Even if we did want a delete function, I don't think it should be part of this patch since this is intended as a no-op and the existing code didn't support deletion either.

> 
> > Tools/resultsdbpy/resultsdbpy/model/cassandra_archiver.py:31
> > +    MAX_ARCHIVE = 500 * 1024 * 1024  # Archives should be smaller than 500 MB
> 
> I think I saw some debug archives have 600~ mb

It's not for builds, only for test result archives, which should all be in the ~50 MB range.

We could revisit this technique for builds later.

[Zhifei Fang]

unofficially r=me

[Aakash Jain]

rs=me

[Jonathan Bedard]

Created attachment 390251 [details]
Patch for landing

[WebKit Commit Bot]

Comment on attachment 390251 [details]
Patch for landing

Clearing flags on attachment: 390251

Committed r256172: <https://trac.webkit.org/changeset/256172>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/59311107>