Bug 47232 – webkit-patch rebaseline should cover more cases

• Status: NEW
• Component: Tools / Tests

[Ojan Vafai]

Some thoughts:
1. Take a --platforms argument and put the new results in the appropriate platform directory
2. Run deduplicate-tests for each test where results were added
3. Make sure svn-eol-style is set to LF and that there aren't CRLFs from grabbing the results off the windows bot

Ideally it would be possible to also specify platforms to rebaseline on a per-test basis. Then something like the results.html page or the chromium flakiness dashboard could give a rich GUI for rebaselining tests.

[Ojan Vafai]

Also, when it's done it should pop up a page similar to the result.html page that shows the diff of what changed, ideally including the image diffs.

[Ojan Vafai]

Tony pointed out that deduplicate-tests only works with git and only if all the changes are committed locally. :(

[Adam Barth]

> 3. Make sure svn-eol-style is set to LF and that there aren't CRLFs from grabbing the results off the windows bot

This requires an SVN working copy, right?  We run all the bots out of git working copies because SVN working copies accumulate junk over time (and are slow).

[Dimitri Glazkov (Google)]

Also: detect when existing platform results are split to sub-platform folders (i.e. chromium-win-xp and chromium-win-vista) and do the right thing.

[Ojan Vafai]

> > 3. Make sure svn-eol-style is set to LF and that there aren't CRLFs from grabbing the results off the windows bot
> 
> This requires an SVN working copy, right?  We run all the bots out of git working copies because SVN working copies accumulate junk over time (and are slow).

That's sucktastic. We can at least convert CRLFs to LFs.

> Also: detect when existing platform results are split to sub-platform folders (i.e. chromium-win-xp and chromium-win-vista) and do the right thing.

I don't see what the correct behavior here is. Given that we don't have bots for both of these, don't we always just want to put the results in the chromium-win directory? Alternately, XP falls back to Vista, so we could always put them in the chromium-win-vista directory.

[Dimitri Glazkov (Google)]

(In reply to comment #5)
> > > 3. Make sure svn-eol-style is set to LF and that there aren't CRLFs from grabbing the results off the windows bot
> > 
> > This requires an SVN working copy, right?  We run all the bots out of git working copies because SVN working copies accumulate junk over time (and are slow).
> 
> That's sucktastic. We can at least convert CRLFs to LFs.
> 
> > Also: detect when existing platform results are split to sub-platform folders (i.e. chromium-win-xp and chromium-win-vista) and do the right thing.
> 
> I don't see what the correct behavior here is. Given that we don't have bots for both of these, don't we always just want to put the results in the chromium-win directory? Alternately, XP falls back to Vista, so we could always put them in the chromium-win-vista directory.

The correct behavior should be:
* Detect if there's existing sub-port baseline (file exists in chromium-[precise platform of the build bot])
*  If it does, put baseline there
* otherwise, put in genertic-port baseline.

[Ojan Vafai]

> The correct behavior should be:
> * Detect if there's existing sub-port baseline (file exists in chromium-[precise platform of the build bot])
> *  If it does, put baseline there
> * otherwise, put in genertic-port baseline.

But we only run an XP bot and the Vista bot doesn't fallback to the XP bot. I could see how this would work if we only ran a Vista bot and put the results in the chromium-win-vista directory. Although, really, we should just jump straight to win7.

[Dimitri Glazkov (Google)]

(In reply to comment #7)
> > The correct behavior should be:
> > * Detect if there's existing sub-port baseline (file exists in chromium-[precise platform of the build bot])
> > *  If it does, put baseline there
> > * otherwise, put in genertic-port baseline.
> 
> But we only run an XP bot and the Vista bot doesn't fallback to the XP bot. I could see how this would work if we only ran a Vista bot and put the results in the chromium-win-vista directory. Although, really, we should just jump straight to win7.

Yup, it's not a perfect solution, but still better than the current behavior, where I have to _know_ to move the files manually after rebaselining is done.

[Ojan Vafai]

> Yup, it's not a perfect solution, but still better than the current behavior, where I have to _know_ to move the files manually after rebaselining is done.

99%, putting them in the chromium-win directory should work fine, no? I guess, I still don't follow what exactly you're proposing. We run an XP bot, so are you saying we should always put the results in chromium-win-xp?

[Dimitri Glazkov (Google)]

(In reply to comment #9)
> > Yup, it's not a perfect solution, but still better than the current behavior, where I have to _know_ to move the files manually after rebaselining is done.
> 
> 99%, putting them in the chromium-win directory should work fine, no? I guess, I still don't follow what exactly you're proposing. We run an XP bot, so are you saying we should always put the results in chromium-win-xp?

Let me illustrate. I did this with the tool: http://trac.webkit.org/changeset/68990. Then I had to tweak it by hand here: http://trac.webkit.org/changeset/68991

The problem here is that putting things into chromium-win is a no-op when there's a platform-specific baseline. Thus, rebaselining as it works currently doesn't produce results you expect it to produce (in the example above, one test was still failing). Based on my experience with gardeners, this is a well-known and poorly understood source of confusion. 

BTW, we run both vista and xp at the moment (Vista on build.webkit.org, XP downstream).