Bug 57891 – Rebaseline queue server

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Mihai Parparita]

For the Chromium GPU bots we'd like to have a friendly web UI for triggering rebaselines (the bots will have per-machine expectations, so updating them by hand would be too tedious). This generally seems like a useful concept for gardening (and for all bots).

Something like:
- http://rebaseline-queue.appspot.com/add gives a GUI that lists the bots and for a bot, the currently (unexpected) failing tests
- User checks the boxes for the tests they want rebaselined
- Form submission POSTs to http://rebaseline-queue.appspot.com/<bot-name>/queue/add, that adds them to the datastore
- http://rebaseline-queue.appspot.com/<bot-name>/queue dumps a JSON list of the tests that were selected for rebaseline
- http://rebaseline-queue.appspot.com/<bot-name>/queue/remove removes passed in tests from that bot's rebaseline list

The GPU bots would run a script that pulls from their queue, runs with --reset-results and then removes the rebaseline tests.

If we had something like Dimitri's LaTER, that could also use the queue to make a gardener's job easier.

 (this could also be done as part of the existing test results server, but I don't think it would share too much functionality with it)

[Adam Barth]

Yep.  We want something similar for the normal gardening bots.  It would be nice if there was an "record as failure" button too that updated test_expectations.txt.

[Mihai Parparita]

Created attachment 88349 [details]
Patch

[Mihai Parparita]

Attached is the initial skeleton for this. It has a picker for letting you choose which builder group and builder to act on: http://rebaseline-queue.appspot.com/add

As well as a URL for listing the tests for a bot directly: http://rebaseline-queue.appspot.com/builder/GPU%20XP%20Tests%20-%20GPU/queue/add

You can see what's already been queued for a bot: http://rebaseline-queue.appspot.com/builder/GPU%20XP%20Tests%20-%20GPU/queue (also as JSON: http://rebaseline-queue.appspot.com/builder/GPU%20XP%20Tests%20-%20GPU/queue/json)

There's also the add/remove POST handlers too.

(In reply to comment #1)
> It would be nice if there was an "record as failure" button too that updated test_expectations.txt.

Perhaps the best way to accomodate something like this is to add an "action" field to QueueEntry. For the GPU case, it would be "rebaseline", for your case it would be "recordfailure." Different bots can then pull different actions from a bot's queue.

[Mihai Parparita]

Now that I think about it, the generic add/remove/list queue functionality may already be covered by QueueStatusServer. I'll look into whether that can just be used tomorrow.

[Mihai Parparita]

Created attachment 89462 [details]
Patch

[Mihai Parparita]

Still running at http://rebaseline-queue.appspot.com/ if you're like to play around with it.

Sorry if the patch is on the large side. If it's easier to review, I can break it up (though it's all pretty straightforward code, and a bunch of App Engine boilerplate).

[Eric Seidel (no email)]

Comment on attachment 89462 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=89462&action=review

Seems reasonable from a high level.  I'm hapy to rs=me, this.  If you were looking for a nitty-gritty review, I'm not very useful as I dont' understand the full architecture behind the rebaseline queue server (or what it's really for).

> Tools/RebaselineQueueServer/app.yaml:4
> +application: rebaseline-queue
> +version: 1
> +runtime: python
> +api_version: 1

Do you want to request a newer version of django here?

> Tools/RebaselineQueueServer/handlers/builderqueue.py:66
> +        for test in tests:
> +            if test not in current_tests:

Isn't this just a set intersection?

> Tools/RebaselineQueueServer/handlers/builderqueue.py:69
> +        self.redirect('/builder/%s/queue' % builder_name)

You might want something to generate these URLs.

> Tools/RebaselineQueueServer/handlers/pages.py:37
> +        builder_names = QueueEntry.get_builder_names()

get_ isn't really webkit style.

> Tools/RebaselineQueueServer/main.py:31
> +use_library('django', '1.2')

I thought it was possible to do this in app.yaml these days.  But I may be wrong.

[Mihai Parparita]

Comment on attachment 89462 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=89462&action=review

>> Tools/RebaselineQueueServer/handlers/builderqueue.py:66
>> +            if test not in current_tests:
> 
> Isn't this just a set intersection?

Difference, but yes. Changed.

>> Tools/RebaselineQueueServer/handlers/builderqueue.py:69
>> +        self.redirect('/builder/%s/queue' % builder_name)
> 
> You might want something to generate these URLs.

Added _queue_list_url to QueueHandler.

>> Tools/RebaselineQueueServer/handlers/pages.py:37
>> +        builder_names = QueueEntry.get_builder_names()
> 
> get_ isn't really webkit style.

Renamed to builder_names (and get_for_builder_name to entries_for_builder).

>> Tools/RebaselineQueueServer/main.py:31
>> +use_library('django', '1.2')
> 
> I thought it was possible to do this in app.yaml these days.  But I may be wrong.

http://code.google.com/appengine/docs/python/tools/libraries.html#Django only mentions doing it from main.py.

[Mihai Parparita]

Committed r83781: <http://trac.webkit.org/changeset/83781>