Attachment 343260 Details for Bug 185790 – Teach bisect-builds to use the LastEvaluatedKey option.

[patch] Teach bisect-builds to use the LastEvaluatedKey option.

185790.patch (text/plain), 3.32 KB, created by Lucas Forschler on 2018-06-21 13:14:33 PDT

(hide)

Description:

Filename:

MIME Type:

Creator: Lucas Forschler

Created: 2018-06-21 13:14:33 PDT

Size: 3.32 KB

patch

obsolete

>Index: Tools/ChangeLog
>===================================================================
>--- Tools/ChangeLog	(revision 233053)
>+++ Tools/ChangeLog	(working copy)
>@@ -1,3 +1,22 @@
>+2018-06-21  Lucas Forschler  <lforschler@apple.com>
>+
>+        bisect-builds --list not showing all builds
>+        https://bugs.webkit.org/show_bug.cgi?id=185790
>+
>+        The AWS Gateway API call into dynamoDB is limited to returning 1MB of query data. Our archive set has overgrown this limit,
>+        causing us to retrieve a truncated list of results. Fortunately, when this limit is reached, dynamoDB returns a 
>+        "LastEvaluatedKey", which can be used to continue the query in another request. 
>+
>+        This patch teaches bisect-build to pass along this key when needed. The backend api was updated from v2 to v2_1 to support
>+        this new querystring option. Any existing code using v2 will not be affected.
>+        
>+        Reviewed by NOBODY (OOPS!).
>+
>+        * Scripts/bisect-builds:
>+        (get_api_url): added optional LastEvaluatedKey argument, which will append the query string to the base api url.
>+        (fetch_revision_list): Added to allow a recursive call which will query until we retrieve every item requested.
>+        (main): teach how to use fetch_revision_list.
>+
> 2018-06-21  Aakash Jain  <aakash_jain@apple.com>
> 
>         [ews-build] unit-tests fail when run from another directory
>Index: Tools/Scripts/bisect-builds
>===================================================================
>--- Tools/Scripts/bisect-builds	(revision 233053)
>+++ Tools/Scripts/bisect-builds	(working copy)
>@@ -38,7 +38,7 @@ import tempfile
> import urllib2
> import urlparse
> 
>-REST_API_URL = 'https://q1tzqfy48e.execute-api.us-west-2.amazonaws.com/v2/'
>+REST_API_URL = 'https://q1tzqfy48e.execute-api.us-west-2.amazonaws.com/v2_1/'
> REST_API_ENDPOINT = 'archives/'
> REST_API_MINIFIED_ENDPOINT = 'minified-archives/'
> 
>@@ -93,13 +93,17 @@ def find_ge(a, x):
> # ---- end bisect helpers ----
> 
> 
>-def get_api_url(options):
>+def get_api_url(options, LastEvaluatedKey=None):
>     if options.full:
>         base_url = urlparse.urljoin(REST_API_URL, REST_API_ENDPOINT)
>     else:
>         base_url = urlparse.urljoin(REST_API_URL, REST_API_MINIFIED_ENDPOINT)
> 
>     api_url = urlparse.urljoin(base_url, '-'.join([options.platform, options.architecture, options.configuration]))
>+    if LastEvaluatedKey:
>+        querystring = urllib2.quote(json.dumps(LastEvaluatedKey))
>+        api_url += '?ExclusiveStartKey=' + querystring
>+        
>     return api_url
> 
> 
>@@ -223,15 +227,23 @@ def print_list_and_exit(revision_list, o
>         print(revision_list)
>         exit(0)
> 
>-def main(options):
>-    validate_options(options)
>-
>-    url = get_api_url(options)
>+def fetch_revision_list(options, LastEvaluatedKey=None):
>+    url = get_api_url(options, LastEvaluatedKey)
>     r = urllib2.urlopen(url)
>     data = json.load(r)
>-    
>     revision_list = get_sorted_revisions(data)
> 
>+    if 'LastEvaluatedKey' in data['revisions']:
>+        LastEvaluatedKey = data['revisions']['LastEvaluatedKey']
>+        revision_list += fetch_revision_list(options, LastEvaluatedKey)
>+        
>+    return revision_list
>+    
>+def main(options):
>+    validate_options(options)
>+
>+    revision_list = fetch_revision_list(options)
>+    
>     if options.list:
>         print_list_and_exit(revision_list, options)
>

Flags:

aakash_jain: review+
lforschler: commit-queue-

Actions: View | Formatted Diff | Diff

Attachments on bug 185790: 343260