|Summary:||webkit-patch rebaseline-expectations incorrectly rebaselines calendar-picker test results|
|Product:||WebKit||Reporter:||Adrienne Walker <enne>|
|Component:||Tools / Tests||Assignee:||Nobody <webkit-unassigned>|
|Severity:||Normal||CC:||dpranke, enne, glenn, schenney|
|Version:||528+ (Nightly build)|
Description Adrienne Walker 2013-02-19 11:18:37 PST
schenney rebaselined a number of tests in http://trac.webkit.org/changeset/143336 using tools. After this rev, a bunch calendar-picker tests started failing. The Linux (dbg) results all look fine, but that's it. In some cases, previously passing tests are now failing. Flakiness dashboard link: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=@ToT - chromium.org&tests=platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-required-ar.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-required.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ru.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance.html,platform/chromium/fast/forms/calendar-picker/month-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/month-picker-appearance.html,platform/chromium/fast/forms/calendar-picker/week-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/week-picker-appearance.html I attempted to rebaseline these tests locally using webkit-patch rebaseline-expectations. However, no images were updated as a result of this, so it appears as though the tool thinks the right images are being used.
Comment 1 Stephen Chenney 2013-02-19 11:32:09 PST
I'm rebaselining using garden-o-matic, which seems to be working. http://trac.webkit.org/changeset/143364 Maybe you had some issues using rebaseline-expectations directly.
Comment 2 Glenn Adams 2013-02-19 11:37:13 PST
If you still can't get it working, you might want to retry the rebaseline after doing a local working copy revert of http://trac.webkit.org/changeset/142941 If the rebaseline works after going this local revert, then that revision may have caused a regression, in which case I will need to reopen or open a new bug.
Comment 3 Adrienne Walker 2013-02-19 15:09:22 PST
Created attachment 189175 [details] stderr from rebaseline-expectations I still think there's also a bug somewhere with rebaseline-expectations. If I try to rebaseline all or one of the calendar-picker tests from that flakiness dashboard link (with and without r142941 reverted) it just removes the Rebaseline lines but updates no images, e.g. a line like this: webkit.org/b/110240 platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html [ Rebaseline ] (As for the tests not really passing post-rebaseline, maybe they're all just flaky and they've been failing for so long it wasn't obvious?)
Comment 4 Dirk Pranke 2013-02-19 15:20:21 PST
(In reply to comment #3) > Created an attachment (id=189175) [details] > stderr from rebaseline-expectations > > I still think there's also a bug somewhere with rebaseline-expectations. If I try to rebaseline all or one of the calendar-picker tests from that flakiness dashboard link (with and without r142941 reverted) it just removes the Rebaseline lines but updates no images, e.g. a line like this: > > webkit.org/b/110240 platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html [ Rebaseline ] > > (As for the tests not really passing post-rebaseline, maybe they're all just flaky and they've been failing for so long it wasn't obvious?) Ah, I think you need to put either "[ Rebaseline Failure ]" or "[ Rebaseline ImageOnlyFailure ]" so that the tool knows which type of failures to try and rebaseline. I think "[ Rebaseline ]" is equivalent to "[ Rebaseline Skip ]". This could probably be improved :).
Comment 5 Adrienne Walker 2013-02-19 16:09:49 PST
Oops. Thanks for the explanation.