Bug 138070

Summary: Test regressions are not detected when image result is missing
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: Alexey Proskuryakov <ap>
Status: RESOLVED FIXED    
Severity: Normal CC: commit-queue, dbates, glenn, jake.nielsen.webkit, rniwa, simon.fraser, thorton
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
proposed fix none

Alexey Proskuryakov
Reported 2014-10-25 13:54:11 PDT
When a test fails on first try, some bots retry it with pixel testing enabled. For tests that don't have pixel results checked in, this results in a "TEXT MISSING" failure (TEXT on first try, MISSING on second try). These are not reported as true regressions by run-webkit-tests, buildbot, or dashboard. See e.g. https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK2%20(Tests)/builds/173 - there are 6 editing/selection tests that fail, but the test run is not red.
Attachments
proposed fix (5.72 KB, patch)
2014-10-25 13:58 PDT, Alexey Proskuryakov 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Alexey Proskuryakov
Comment 1 2014-10-25 13:58:37 PDT
Created attachment 240457 [details] proposed fix
Simon Fraser (smfr)
Comment 2 2014-10-25 14:11:51 PDT
Comment on attachment 240457 [details] proposed fix Nice.
WebKit Commit Bot
Comment 3 2014-10-25 16:29:50 PDT
Comment on attachment 240457 [details] proposed fix Clearing flags on attachment: 240457 Committed r175204: <http://trac.webkit.org/changeset/175204>
WebKit Commit Bot
Comment 4 2014-10-25 16:29:53 PDT
All reviewed patches have been landed. Closing bug.
Note You need to log in before you can comment on or make changes to this bug.