| Summary: | full_results.json should distinguish unexpected failures from expected ones | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Alexey Proskuryakov <ap> | ||||
| Component: | Tools / Tests | Assignee: | Alexey Proskuryakov <ap> | ||||
| Status: | RESOLVED FIXED | ||||||
| Severity: | Normal | CC: | commit-queue, glenn, rniwa, timothy | ||||
| Priority: | P2 | ||||||
| Version: | 528+ (Nightly build) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Attachments: |
|
||||||
Created attachment 220119 [details]
proposed patch
Committed <http://trac.webkit.org/r161171>. |
full_results.json contains expected/actual results for tests, and it contains total counts for regressions, flaky tests and missing results. But it's hard to tell which of the many results are regressions, because the logic for deciding that is convoluted. For example, here is an entry from full_results.json that doesn't count as a regression: "4242293.html":{"expected":"PASS","actual":"TEXT IMAGE","image_diff_percent":0.16} I'm guessing that this is because we are not running image tests, so an image only failure on retry is as good as a pass. But I really don't want to copy this logic in JavaScript - it's already duplicated in at least two places in webkitpy, which is bad enough. My plan is to use full_results.json for results in popovers at <http://build.webkit.org/dashboard>.