Bug 57891 - Rebaseline queue server
Summary: Rebaseline queue server
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-04-05 15:13 PDT by Mihai Parparita
Modified: 2011-04-13 15:54 PDT (History)
6 users (show)

See Also:


Attachments
Patch (22.25 KB, patch)
2011-04-05 18:56 PDT, Mihai Parparita
no flags Details | Formatted Diff | Diff
Patch (27.58 KB, patch)
2011-04-13 14:33 PDT, Mihai Parparita
eric: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Mihai Parparita 2011-04-05 15:13:01 PDT
For the Chromium GPU bots we'd like to have a friendly web UI for triggering rebaselines (the bots will have per-machine expectations, so updating them by hand would be too tedious). This generally seems like a useful concept for gardening (and for all bots).

Something like:
- http://rebaseline-queue.appspot.com/add gives a GUI that lists the bots and for a bot, the currently (unexpected) failing tests
- User checks the boxes for the tests they want rebaselined
- Form submission POSTs to http://rebaseline-queue.appspot.com/<bot-name>/queue/add, that adds them to the datastore
- http://rebaseline-queue.appspot.com/<bot-name>/queue dumps a JSON list of the tests that were selected for rebaseline
- http://rebaseline-queue.appspot.com/<bot-name>/queue/remove removes passed in tests from that bot's rebaseline list

The GPU bots would run a script that pulls from their queue, runs with --reset-results and then removes the rebaseline tests.

If we had something like Dimitri's LaTER, that could also use the queue to make a gardener's job easier.

 (this could also be done as part of the existing test results server, but I don't think it would share too much functionality with it)
Comment 1 Adam Barth 2011-04-05 16:59:31 PDT
Yep.  We want something similar for the normal gardening bots.  It would be nice if there was an "record as failure" button too that updated test_expectations.txt.
Comment 2 Mihai Parparita 2011-04-05 18:56:58 PDT
Created attachment 88349 [details]
Patch
Comment 3 Mihai Parparita 2011-04-05 19:00:09 PDT
Attached is the initial skeleton for this. It has a picker for letting you choose which builder group and builder to act on: http://rebaseline-queue.appspot.com/add

As well as a URL for listing the tests for a bot directly: http://rebaseline-queue.appspot.com/builder/GPU%20XP%20Tests%20-%20GPU/queue/add

You can see what's already been queued for a bot: http://rebaseline-queue.appspot.com/builder/GPU%20XP%20Tests%20-%20GPU/queue (also as JSON: http://rebaseline-queue.appspot.com/builder/GPU%20XP%20Tests%20-%20GPU/queue/json)

There's also the add/remove POST handlers too.

(In reply to comment #1)
> It would be nice if there was an "record as failure" button too that updated test_expectations.txt.

Perhaps the best way to accomodate something like this is to add an "action" field to QueueEntry. For the GPU case, it would be "rebaseline", for your case it would be "recordfailure." Different bots can then pull different actions from a bot's queue.
Comment 4 Mihai Parparita 2011-04-05 19:27:54 PDT
Now that I think about it, the generic add/remove/list queue functionality may already be covered by QueueStatusServer. I'll look into whether that can just be used tomorrow.
Comment 5 Mihai Parparita 2011-04-13 14:33:35 PDT
Created attachment 89462 [details]
Patch
Comment 6 Mihai Parparita 2011-04-13 14:35:20 PDT
Still running at http://rebaseline-queue.appspot.com/ if you're like to play around with it.

Sorry if the patch is on the large side. If it's easier to review, I can break it up (though it's all pretty straightforward code, and a bunch of App Engine boilerplate).
Comment 7 Eric Seidel (no email) 2011-04-13 14:50:46 PDT
Comment on attachment 89462 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=89462&action=review

Seems reasonable from a high level.  I'm hapy to rs=me, this.  If you were looking for a nitty-gritty review, I'm not very useful as I dont' understand the full architecture behind the rebaseline queue server (or what it's really for).

> Tools/RebaselineQueueServer/app.yaml:4
> +application: rebaseline-queue
> +version: 1
> +runtime: python
> +api_version: 1

Do you want to request a newer version of django here?

> Tools/RebaselineQueueServer/handlers/builderqueue.py:66
> +        for test in tests:
> +            if test not in current_tests:

Isn't this just a set intersection?

> Tools/RebaselineQueueServer/handlers/builderqueue.py:69
> +        self.redirect('/builder/%s/queue' % builder_name)

You might want something to generate these URLs.

> Tools/RebaselineQueueServer/handlers/pages.py:37
> +        builder_names = QueueEntry.get_builder_names()

get_ isn't really webkit style.

> Tools/RebaselineQueueServer/main.py:31
> +use_library('django', '1.2')

I thought it was possible to do this in app.yaml these days.  But I may be wrong.
Comment 8 Mihai Parparita 2011-04-13 15:50:45 PDT
Comment on attachment 89462 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=89462&action=review

>> Tools/RebaselineQueueServer/handlers/builderqueue.py:66
>> +            if test not in current_tests:
> 
> Isn't this just a set intersection?

Difference, but yes. Changed.

>> Tools/RebaselineQueueServer/handlers/builderqueue.py:69
>> +        self.redirect('/builder/%s/queue' % builder_name)
> 
> You might want something to generate these URLs.

Added _queue_list_url to QueueHandler.

>> Tools/RebaselineQueueServer/handlers/pages.py:37
>> +        builder_names = QueueEntry.get_builder_names()
> 
> get_ isn't really webkit style.

Renamed to builder_names (and get_for_builder_name to entries_for_builder).

>> Tools/RebaselineQueueServer/main.py:31
>> +use_library('django', '1.2')
> 
> I thought it was possible to do this in app.yaml these days.  But I may be wrong.

http://code.google.com/appengine/docs/python/tools/libraries.html#Django only mentions doing it from main.py.
Comment 9 Mihai Parparita 2011-04-13 15:54:35 PDT
Committed r83781: <http://trac.webkit.org/changeset/83781>