RESOLVED CONFIGURATION CHANGED 176634
bisect-builds : some archives not found when using --full 
https://bugs.webkit.org/show_bug.cgi?id=176634
Summary bisect-builds : some archives not found when using --full
Lucas Forschler
Reported 2017-09-08 15:11:51 PDT
It appears that expired archives (older than 14 days) are not being removed from the dynamoDB. I see no evidence that the lambda function which removes items from dynamoDB is being called when the S3 object expires.
Attachments
Add attachment proposed patch, testcase, etc.
Lucas Forschler
Comment 1 2017-09-08 15:14:37 PDT
note: manually deleting an archive does properly trigger the lambda. So, it appears this is simply not connected to the AWS lifecycle.
Radar WebKit Bug Importer
Comment 2 2017-09-11 10:47:55 PDT
<rdar://problem/34370549>
Lucas Forschler
Comment 3 2017-09-11 17:19:19 PDT
I believe we may need to do some smart cleanup on our side. It does not look like Amazon is sending out events when items expire from S3. This is sad, and I feel is a bug. I want to make sure this is maintenance free, so I’m considering adding a field to the database to track when archives are posted. Then, I can update the queries to simply not return anything outside of the retention time. This would involve storing the retention time somewhere, and could be hard to find if we need to update it. The other option is to write a lambda function which would clean up the database automatically… but we’d need to find an event to trigger it. It may be possible to schedule lambda functions by date/time, I haven't looked into that yet.
Lucas Forschler
Comment 4 2017-09-11 17:20:39 PDT
This looks promising: http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html
Lucas Forschler
Comment 5 2017-09-28 14:46:05 PDT
We are now adding a creationTime and expirationTime to all entries in the Database. DynamoDB will use the expirationTime as a TTL (time to live) field, and automatically purge entries. This should be fixed moving forward.
Note You need to log in before you can comment on or make changes to this bug.