Bug 126300 - full_results.json should distinguish unexpected failures from expected ones
Summary: full_results.json should distinguish unexpected failures from expected ones
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alexey Proskuryakov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-12-30 12:30 PST by Alexey Proskuryakov
Modified: 2013-12-30 22:29 PST (History)
4 users (show)

See Also:


Attachments
proposed patch (9.44 KB, patch)
2013-12-30 12:34 PST, Alexey Proskuryakov
timothy: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2013-12-30 12:30:44 PST
full_results.json contains expected/actual results for tests, and it contains total counts for regressions, flaky tests and missing results. But it's hard to tell which of the many results are regressions, because the logic for deciding that is convoluted. For example, here is an entry from full_results.json that doesn't count as a regression:

"4242293.html":{"expected":"PASS","actual":"TEXT IMAGE","image_diff_percent":0.16}

I'm guessing that this is because we are not running image tests, so an image only failure on retry is as good as a pass. But I really don't want to copy this logic in JavaScript - it's already duplicated in at least two places in webkitpy, which is bad enough.

My plan is to use full_results.json for results in popovers at <http://build.webkit.org/dashboard>.
Comment 1 Alexey Proskuryakov 2013-12-30 12:34:41 PST
Created attachment 220119 [details]
proposed patch
Comment 2 Alexey Proskuryakov 2013-12-30 22:29:48 PST
Committed <http://trac.webkit.org/r161171>.