Bug 138070

Summary: Test regressions are not detected when image result is missing
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: Alexey Proskuryakov <ap>
Status: RESOLVED FIXED    
Severity: Normal CC: commit-queue, dbates, glenn, jake.nielsen.webkit, rniwa, simon.fraser, thorton
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
proposed fix none

Description Alexey Proskuryakov 2014-10-25 13:54:11 PDT 
When a test fails on first try, some bots retry it with pixel testing enabled. For tests that don't have pixel results checked in, this results in a "TEXT MISSING" failure (TEXT on first try, MISSING on second try). These are not reported as true regressions by run-webkit-tests, buildbot, or dashboard.

See e.g. https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK2%20(Tests)/builds/173 - there are 6 editing/selection tests that fail, but the test run is not red.
Comment 1 Alexey Proskuryakov 2014-10-25 13:58:37 PDT 
Created attachment 240457 [details]
proposed fix
Comment 2 Simon Fraser (smfr) 2014-10-25 14:11:51 PDT 
Comment on attachment 240457 [details]
proposed fix

Nice.
Comment 3 WebKit Commit Bot 2014-10-25 16:29:50 PDT 
Comment on attachment 240457 [details]
proposed fix

Clearing flags on attachment: 240457

Committed r175204: <http://trac.webkit.org/changeset/175204>
Comment 4 WebKit Commit Bot 2014-10-25 16:29:53 PDT 
All reviewed patches have been landed.  Closing bug.