Bug 110240 - webkit-patch rebaseline-expectations incorrectly rebaselines calendar-picker test results
Summary: webkit-patch rebaseline-expectations incorrectly rebaselines calendar-picker ...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-02-19 11:18 PST by Adrienne Walker
Modified: 2013-02-19 16:09 PST (History)
4 users (show)

See Also:


Attachments
stderr from rebaseline-expectations (157.17 KB, text/x-log)
2013-02-19 15:09 PST, Adrienne Walker
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Adrienne Walker 2013-02-19 11:18:37 PST
schenney rebaselined a number of tests in http://trac.webkit.org/changeset/143336 using tools.  After this rev, a bunch calendar-picker tests started failing.  The Linux (dbg) results all look fine, but that's it.  In some cases, previously passing tests are now failing.

Flakiness dashboard link:
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=@ToT - chromium.org&tests=platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-required-ar.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-required.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ru.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance.html,platform/chromium/fast/forms/calendar-picker/month-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/month-picker-appearance.html,platform/chromium/fast/forms/calendar-picker/week-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/week-picker-appearance.html

I attempted to rebaseline these tests locally using webkit-patch rebaseline-expectations.  However, no images were updated as a result of this, so it appears as though the tool thinks the right images are being used.
Comment 1 Stephen Chenney 2013-02-19 11:32:09 PST
I'm rebaselining using garden-o-matic, which seems to be working. http://trac.webkit.org/changeset/143364

Maybe you had some issues using rebaseline-expectations directly.
Comment 2 Glenn Adams 2013-02-19 11:37:13 PST
If you still can't get it working, you might want to retry the rebaseline after doing a local working copy revert of 

http://trac.webkit.org/changeset/142941

If the rebaseline works after going this local revert, then that revision may have caused a regression, in which case I will need to reopen or open a new bug.
Comment 3 Adrienne Walker 2013-02-19 15:09:22 PST
Created attachment 189175 [details]
stderr from rebaseline-expectations

I still think there's also a bug somewhere with rebaseline-expectations.  If I try to rebaseline all or one of the calendar-picker tests from that flakiness dashboard link (with and without r142941 reverted) it just removes the Rebaseline lines but updates no images, e.g. a line like this:

webkit.org/b/110240 platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html [ Rebaseline ]

(As for the tests not really passing post-rebaseline, maybe they're all just flaky and they've been failing for so long it wasn't obvious?)
Comment 4 Dirk Pranke 2013-02-19 15:20:21 PST
(In reply to comment #3)
> Created an attachment (id=189175) [details]
> stderr from rebaseline-expectations
> 
> I still think there's also a bug somewhere with rebaseline-expectations.  If I try to rebaseline all or one of the calendar-picker tests from that flakiness dashboard link (with and without r142941 reverted) it just removes the Rebaseline lines but updates no images, e.g. a line like this:
> 
> webkit.org/b/110240 platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html [ Rebaseline ]
> 
> (As for the tests not really passing post-rebaseline, maybe they're all just flaky and they've been failing for so long it wasn't obvious?)

Ah, I think you need to put either "[ Rebaseline Failure ]" or "[ Rebaseline ImageOnlyFailure ]" so that the tool knows which type of failures to try and rebaseline. I think "[ Rebaseline ]" is equivalent to "[ Rebaseline Skip ]".

This could probably be improved :).
Comment 5 Adrienne Walker 2013-02-19 16:09:49 PST
Oops.  Thanks for the explanation.