schenney rebaselined a number of tests in http://trac.webkit.org/changeset/143336 using tools. After this rev, a bunch calendar-picker tests started failing. The Linux (dbg) results all look fine, but that's it. In some cases, previously passing tests are now failing.
Flakiness dashboard link:
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=@ToT - chromium.org&tests=platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-required-ar.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-required.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ru.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance.html,platform/chromium/fast/forms/calendar-picker/month-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/month-picker-appearance.html,platform/chromium/fast/forms/calendar-picker/week-picker-appearance-step.html,platform/chromium/fast/forms/calendar-picker/week-picker-appearance.html
I attempted to rebaseline these tests locally using webkit-patch rebaseline-expectations. However, no images were updated as a result of this, so it appears as though the tool thinks the right images are being used.
I'm rebaselining using garden-o-matic, which seems to be working. http://trac.webkit.org/changeset/143364
Maybe you had some issues using rebaseline-expectations directly.
If you still can't get it working, you might want to retry the rebaseline after doing a local working copy revert of
If the rebaseline works after going this local revert, then that revision may have caused a regression, in which case I will need to reopen or open a new bug.
Created attachment 189175 [details]
stderr from rebaseline-expectations
I still think there's also a bug somewhere with rebaseline-expectations. If I try to rebaseline all or one of the calendar-picker tests from that flakiness dashboard link (with and without r142941 reverted) it just removes the Rebaseline lines but updates no images, e.g. a line like this:
webkit.org/b/110240 platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html [ Rebaseline ]
(As for the tests not really passing post-rebaseline, maybe they're all just flaky and they've been failing for so long it wasn't obvious?)
(In reply to comment #3)
> Created an attachment (id=189175) [details]
> stderr from rebaseline-expectations
> I still think there's also a bug somewhere with rebaseline-expectations. If I try to rebaseline all or one of the calendar-picker tests from that flakiness dashboard link (with and without r142941 reverted) it just removes the Rebaseline lines but updates no images, e.g. a line like this:
> webkit.org/b/110240 platform/chromium/fast/forms/calendar-picker/calendar-picker-appearance-ar.html [ Rebaseline ]
> (As for the tests not really passing post-rebaseline, maybe they're all just flaky and they've been failing for so long it wasn't obvious?)
Ah, I think you need to put either "[ Rebaseline Failure ]" or "[ Rebaseline ImageOnlyFailure ]" so that the tool knows which type of failures to try and rebaseline. I think "[ Rebaseline ]" is equivalent to "[ Rebaseline Skip ]".
This could probably be improved :).
Oops. Thanks for the explanation.