Bug 138070 - Test regressions are not detected when image result is missing
Summary: Test regressions are not detected when image result is missing
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alexey Proskuryakov
Depends on:
Reported: 2014-10-25 13:54 PDT by Alexey Proskuryakov
Modified: 2014-10-25 16:29 PDT (History)
7 users (show)

See Also:

proposed fix (5.72 KB, patch)
2014-10-25 13:58 PDT, Alexey Proskuryakov
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2014-10-25 13:54:11 PDT
When a test fails on first try, some bots retry it with pixel testing enabled. For tests that don't have pixel results checked in, this results in a "TEXT MISSING" failure (TEXT on first try, MISSING on second try). These are not reported as true regressions by run-webkit-tests, buildbot, or dashboard.

See e.g. https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK2%20(Tests)/builds/173 - there are 6 editing/selection tests that fail, but the test run is not red.
Comment 1 Alexey Proskuryakov 2014-10-25 13:58:37 PDT
Created attachment 240457 [details]
proposed fix
Comment 2 Simon Fraser (smfr) 2014-10-25 14:11:51 PDT
Comment on attachment 240457 [details]
proposed fix

Comment 3 WebKit Commit Bot 2014-10-25 16:29:50 PDT
Comment on attachment 240457 [details]
proposed fix

Clearing flags on attachment: 240457

Committed r175204: <http://trac.webkit.org/changeset/175204>
Comment 4 WebKit Commit Bot 2014-10-25 16:29:53 PDT
All reviewed patches have been landed.  Closing bug.