Bug 138070 - Test regressions are not detected when image result is missing
Summary: Test regressions are not detected when image result is missing
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alexey Proskuryakov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-10-25 13:54 PDT by Alexey Proskuryakov
Modified: 2014-10-25 16:29 PDT (History)
7 users (show)

See Also:

Attachments
proposed fix (5.72 KB, patch)
2014-10-25 13:58 PDT, Alexey Proskuryakov 		no flags Details | Formatted Diff | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2014-10-25 13:54:11 PDT 
When a test fails on first try, some bots retry it with pixel testing enabled. For tests that don't have pixel results checked in, this results in a "TEXT MISSING" failure (TEXT on first try, MISSING on second try). These are not reported as true regressions by run-webkit-tests, buildbot, or dashboard.

See e.g. https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK2%20(Tests)/builds/173 - there are 6 editing/selection tests that fail, but the test run is not red.
Comment 1 Alexey Proskuryakov 2014-10-25 13:58:37 PDT 
Created attachment 240457 [details]
proposed fix
Comment 2 Simon Fraser (smfr) 2014-10-25 14:11:51 PDT 
Comment on attachment 240457 [details]
proposed fix

Nice.
Comment 3 WebKit Commit Bot 2014-10-25 16:29:50 PDT 
Comment on attachment 240457 [details]
proposed fix

Clearing flags on attachment: 240457

Committed r175204: <http://trac.webkit.org/changeset/175204>
Comment 4 WebKit Commit Bot 2014-10-25 16:29:53 PDT 
All reviewed patches have been landed.  Closing bug.