Bug 86797 – results.html should let you rebaseline a test and modify it's listing in test_expectations.txt/Skipped files from the html page

• Status: NEW
• Component: Tools / Tests

[Ojan Vafai]

Most of the infrastructure for rebaselining tests and modifying test_expectations.txt already exists for garden-o-matic via webkit-patch commands. We just need to start garden-o-matic localhost server that results.html talks to in order to execute the webkit-patch commands. I suppose the webkit-patch rebaseline-test command also needs to be able to take a local path to a layout test results directory instead of a buildbot name.

In addition to being super convenient, this would also make for less manual editing of test_expectations.txt files, alleviating some of Ryosuke's frustrations with the file format.

[Adam Barth]

One possibility is to pass resbaseline-test the base URL to the results directory, with the default being the buildbot base URL.

[Ojan Vafai]

(In reply to comment #1)
> One possibility is to pass resbaseline-test the base URL to the results directory, with the default being the buildbot base URL.

Well, normally we pass it a builder name, but we don't need that in this case since you're clearly rebaselining the platform you're on. (i.e. however run-webkit-tests figures out the platform your on, webkit-patch rebaseline-test should do the same). So I think we can pass the base URL instead of the builder name. But...I don't feel strongly about how it's implemented, that's just how I would do it.

[Adam Barth]

Automatically detecting the port can be tricky, especially in cases where more than one port runs on a given machine.  For example, rebaseline-test would need to distinguish between apple-mac and chromium-mac when running on OS X.

[Dirk Pranke]

(In reply to comment #3)
> Automatically detecting the port can be tricky, especially in cases where more than one port runs on a given machine.  For example, rebaseline-test would need to distinguish between apple-mac and chromium-mac when running on OS X.

If you're starting from results.html, it would be easy to know which port generated the results, no?

[Adam Barth]

> If you're starting from results.html, it would be easy to know which port generated the results, no?

Yeah, we'll probably need to pass that information to rebaseline-test.

[Tony Chang]

Rather than having to have garden-o-matic running on localhost, you could provide a single webkit-patch command line for rebaselining a test that the user would copy/paste.  Maybe we can encode all the necessary information in the command line (port, server to download from, tests to rebase, etc).

[Ryosuke Niwa]

(In reply to comment #6)
> Rather than having to have garden-o-matic running on localhost, you could provide a single webkit-patch command line for rebaselining a test that the user would copy/paste.  Maybe we can encode all the necessary information in the command line (port, server to download from, tests to rebase, etc).

Not that I'm opposed to adding such a command, but having to pass all those information in command line seems annoying.

[Tony Chang]

(In reply to comment #7)
> (In reply to comment #6)
> > Rather than having to have garden-o-matic running on localhost, you could provide a single webkit-patch command line for rebaselining a test that the user would copy/paste.  Maybe we can encode all the necessary information in the command line (port, server to download from, tests to rebase, etc).
> 
> Not that I'm opposed to adding such a command, but having to pass all those information in command line seems annoying.

You could base64 encode a binary format so the command is short.  E.g., webkit-patch rebaseline-tests ja90134jkld914-jsf90a8a=.  Or did you mean annoying to implement?

[Ryosuke Niwa]

(In reply to comment #8)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > Rather than having to have garden-o-matic running on localhost, you could provide a single webkit-patch command line for rebaselining a test that the user would copy/paste.  Maybe we can encode all the necessary information in the command line (port, server to download from, tests to rebase, etc).
> > 
> > Not that I'm opposed to adding such a command, but having to pass all those information in command line seems annoying.
> 
> You could base64 encode a binary format so the command is short.  E.g., webkit-patch rebaseline-tests ja90134jkld914-jsf90a8a=.  Or did you mean annoying to implement?

Would that work when we have 150+ tests to rebaseline like I had to yesterday?

[Tony Chang]

(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #7)
> > > (In reply to comment #6)
> > > > Rather than having to have garden-o-matic running on localhost, you could provide a single webkit-patch command line for rebaselining a test that the user would copy/paste.  Maybe we can encode all the necessary information in the command line (port, server to download from, tests to rebase, etc).
> > > 
> > > Not that I'm opposed to adding such a command, but having to pass all those information in command line seems annoying.
> > 
> > You could base64 encode a binary format so the command is short.  E.g., webkit-patch rebaseline-tests ja90134jkld914-jsf90a8a=.  Or did you mean annoying to implement?
> 
> Would that work when we have 150+ tests to rebaseline like I had to yesterday?

Maybe.  Depends on how you encode the data :)

[Dirk Pranke]

I'm confused by how this bug has developed ... we're talking about doing local rebaselines in a checkout, right, not pulling results down from a bot?

can't you already get most of the way there with copying and pasting suppressions into test_expectations.txt and/or re-running NRWT with --reset-results and a list of tests?

Certainly we can further automate this, but I just want to get on the same page ...

[Tony Chang]

(In reply to comment #11)
> I'm confused by how this bug has developed ... we're talking about doing local rebaselines in a checkout, right, not pulling results down from a bot?

Yes.

> can't you already get most of the way there with copying and pasting suppressions into test_expectations.txt and/or re-running NRWT with --reset-results and a list of tests?

Yes, but it's error prone and slow.  More automation here would be better.

[Dirk Pranke]

(In reply to comment #12)
> (In reply to comment #11)
> > I'm confused by how this bug has developed ... we're talking about doing local rebaselines in a checkout, right, not pulling results down from a bot?
> 
> Yes.
> 
> > can't you already get most of the way there with copying and pasting suppressions into test_expectations.txt and/or re-running NRWT with --reset-results and a list of tests?
> 
> Yes, but it's error prone and slow.  More automation here would be better.

Agreed. Isn't all this basically what Mihai's rebaseline server did? Are we talking about rebuilding that functionality on top of the garden-o-matic code base (I have no objection to doing so, it makes sense, just making sure).

[Ojan Vafai]

(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #11)
> > > I'm confused by how this bug has developed ... we're talking about doing local rebaselines in a checkout, right, not pulling results down from a bot?
> > 
> > Yes.
> > 
> > > can't you already get most of the way there with copying and pasting suppressions into test_expectations.txt and/or re-running NRWT with --reset-results and a list of tests?
> > 
> > Yes, but it's error prone and slow.  More automation here would be better.
> 
> Agreed. Isn't all this basically what Mihai's rebaseline server did? Are we talking about rebuilding that functionality on top of the garden-o-matic code base (I have no objection to doing so, it makes sense, just making sure).

More or less, but with the results.html UI. IMO, this results.html UI plus the functionality provided by garden-o-matic obsoletes the rebaseline server. Would be nice to have one fewer tool to maintain, especially as the rebaseline server is only really useful for really large rebaselines (e.g. bringing up a new port or making a skia change) and learning a new tool for that one use-case just doesn't happen much in practice.