webkit-patch rebaseline-expectations incorrectly rebaselines calendar-picker test results

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Adrienne Walker]

schenney rebaselined a number of tests in http://trac.webkit.org/changeset/143336 using tools.  After this rev, a bunch calendar-picker tests started failing.  The Linux (dbg) results all look fine, but that's it.  In some cases, previously passing tests are now failing.

Flakiness dashboard link:
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=@ToT - chromium.org&tests=platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-required-ar.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-required.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ru.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance.html,platform/chromium/fast/forms/calendar-picker/month-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/month-picker-appearance.html,platform/chromium/fast/forms/calendar-picker/week-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/week-picker-appearance.html

I attempted to rebaseline these tests locally using webkit-patch rebaseline-expectations.  However, no images were updated as a result of this, so it appears as though the tool thinks the right images are being used.

[Stephen Chenney]

I'm rebaselining using garden-o-matic, which seems to be working. http://trac.webkit.org/changeset/143364

Maybe you had some issues using rebaseline-expectations directly.

[Glenn Adams]

If you still can't get it working, you might want to retry the rebaseline after doing a local working copy revert of 

http://trac.webkit.org/changeset/142941

If the rebaseline works after going this local revert, then that revision may have caused a regression, in which case I will need to reopen or open a new bug.

[Adrienne Walker]

Created attachment 189175 [details]
stderr from rebaseline-expectations

I still think there's also a bug somewhere with rebaseline-expectations.  If I try to rebaseline all or one of the calendar-picker tests from that flakiness dashboard link (with and without r142941 reverted) it just removes the Rebaseline lines but updates no images, e.g. a line like this:

webkit.org/b/110240 platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html [ Rebaseline ]

(As for the tests not really passing post-rebaseline, maybe they're all just flaky and they've been failing for so long it wasn't obvious?)

[Dirk Pranke]

(In reply to comment #3)
> Created an attachment (id=189175) [details]
> stderr from rebaseline-expectations
> 
> I still think there's also a bug somewhere with rebaseline-expectations.  If I try to rebaseline all or one of the calendar-picker tests from that flakiness dashboard link (with and without r142941 reverted) it just removes the Rebaseline lines but updates no images, e.g. a line like this:
> 
> webkit.org/b/110240 platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html [ Rebaseline ]
> 
> (As for the tests not really passing post-rebaseline, maybe they're all just flaky and they've been failing for so long it wasn't obvious?)

Ah, I think you need to put either "[ Rebaseline Failure ]" or "[ Rebaseline ImageOnlyFailure ]" so that the tool knows which type of failures to try and rebaseline. I think "[ Rebaseline ]" is equivalent to "[ Rebaseline Skip ]".

This could probably be improved :).

[Adrienne Walker]

Oops.  Thanks for the explanation.