Bug 126300

Summary: full_results.json should distinguish unexpected failures from expected ones
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: Alexey Proskuryakov <ap>
Status: RESOLVED FIXED    
Severity: Normal CC: commit-queue, glenn, rniwa, timothy
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
proposed patch timothy: review+

Alexey Proskuryakov
Reported 2013-12-30 12:30:44 PST
full_results.json contains expected/actual results for tests, and it contains total counts for regressions, flaky tests and missing results. But it's hard to tell which of the many results are regressions, because the logic for deciding that is convoluted. For example, here is an entry from full_results.json that doesn't count as a regression: "4242293.html":{"expected":"PASS","actual":"TEXT IMAGE","image_diff_percent":0.16} I'm guessing that this is because we are not running image tests, so an image only failure on retry is as good as a pass. But I really don't want to copy this logic in JavaScript - it's already duplicated in at least two places in webkitpy, which is bad enough. My plan is to use full_results.json for results in popovers at <http://build.webkit.org/dashboard>.
Attachments
proposed patch (9.44 KB, patch)
2013-12-30 12:34 PST, Alexey Proskuryakov 		timothy: review+
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Alexey Proskuryakov
Comment 1 2013-12-30 12:34:41 PST
Created attachment 220119 [details] proposed patch
Alexey Proskuryakov
Comment 2 2013-12-30 22:29:48 PST
Committed <http://trac.webkit.org/r161171>.
Note You need to log in before you can comment on or make changes to this bug.