Bug 47761 – Rebaseline server tool

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Mihai Parparita]

For mass rebaselines (Leopard -> Snow Leopard, SVG changes, Skia changes), etc. it would be nice to be able to update baselines locally en-masse.

[Mihai Parparita]

Created attachment 70938 [details]
Patch

[Dirk Pranke]

nice! I've often thought about writing a script that did exactly this ...

[Tony Chang]

I like the use of a local httpd to handle the file copies.  I'm happy to review this once you're happy with it.

[Mihai Parparita]

Created attachment 71568 [details]
Patch

Progress

[Mihai Parparita]

Created attachment 71619 [details]
Patch

More progress

[Mihai Parparita]

Created attachment 71621 [details]
Patch

[Mihai Parparita]

(notes for anyone interested in trying this)

Current usage:
1. Make a change that requires lots of (pixel) rebaselines
2. Run new-run-webkit-tests --tolerance 0 --pixel-tests [path/to/tests]
3. Run webkit-patch local-rebaseline /path/to/tests/results (usually WebKitBuild/{Debug|Release}/layout-test-results
4. Go to http://localhost:8127/

In the future, it probably also makes sense to allow a results zip file to be pulled from a bot, so that you can do mass rebaselines for other platforms too.

In the web UI (which requires a 30 inch display), you can use the menus at the top to select tests by failure type and directory. Once you have a set of tests selected, you can navigate between them with the « and » buttons, or by using the left and right arrow keys.

For a particular test, the current and expected outputs will be displayed, along with a diff. For image diffs, you can click anywhere in the images to get a "loupe" that lets you see pixel values up close.

You can use the two drop-downs at the bottom of the screen to choose what you want done with the baselines. The baseline target is where the test outputs should be copied to. If moving between platforms, you may wish to move current baselines to another directory (e.g. if updating current Mac baselines for Snow Leopard, you would set the target to "mac" and move current baselines to "mac-leopard").

Once you've determined that baselines should be updated, you can use the "Add to rebaseline queue" button (or the q key) to add it to the list. Once you've added enough tests to the queue (commits probably shouldn't get any bigger than ~250 baselines), you show the queue with the "Queue" link and use the "Rebaseline tests" button (or the r key). You can watch the progress in the log area (toggled by the link in the upper right).

The tool uses scm.py to move baselines around and mark them for addition/move, but I've only tried it in git checkouts so far.

[Mihai Parparita]

+Ojan since he's interested in making rebaselining less painful too.

[Adam Barth]

Hum...  I'm having trouble using the tool because I don't have a results.json.  It doesn't seem to be included in the zip from the bot.

[Mihai Parparita]

Created attachment 71799 [details]
Patch

[Mihai Parparita]

For anyone wanting to use this to help with rebaselining of pixel tests for Snow Leopard:
1. webkit-patch apply-attachment 71799 [details]
2. Claim a directory from https://spreadsheets.google.com/ccc?key=0AkV6W8Td7jGwdFpFdzJLNzZXNFZkVmhXYmlNVS1PZVE&hl=en&authkey=COyB-LwN
3. new-run-webkit-tests --tolerance=0.0 --pixel-tests directory/you/picked
4. webkit-patch local-rebaseline WebKitBuild/Debug/layout-test-results
5. Go to http://localhost:8127
6. Set the baseline target as "mac" and move current baselines to "mac-leopard"

For now,  rebaseline only tests that have text anti-aliasing differences or HTML color name differences. Have no more than 100 tests per patch. See comment 7 for details on how to use the tool UI.

[Adam Barth]

Yay!  I got it to work.  Pretty.

[James Robinson]

I updated the fast/table Snow Leopard expectations with this: http://trac.webkit.org/changeset/70835.  Worked beautifully!

[Mihai Parparita]

Created attachment 72384 [details]
Patch

[Mihai Parparita]

Created attachment 72614 [details]
Patch

[Mihai Parparita]

Created attachment 72778 [details]
Patch

[Mihai Parparita]

Created attachment 72996 [details]
Patch

[Mihai Parparita]

Comment on attachment 72996 [details]
Patch

Oops, this ended up on the wrong bug.

[Mihai Parparita]

The tool is now checked in (thanks to Tony doing lots of reviews). I've also written up https://trac.webkit.org/wiki/RebaselineServer with instructions on how to use it. Feel free to file bugs and cc me on them if you notice any bugs or limitations.