| Summary: | Shouldn't Failure expectation work for image-only failures? | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Alexey Proskuryakov <ap> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | REOPENED --- | ||
| Severity: | Normal | CC: | dbates, dburkart, dean_johnson, dpranke, matthew_hanson, rniwa, zalan |
| Priority: | P2 | ||
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
|
Description
Alexey Proskuryakov
2014-01-20 20:15:40 PST
Yeah, I always disliked this. Failure should include ImageOnlyFailure. There were discussions on this when we were talking about changing the syntax back in 6/12 (and then in 9/12 when the changes were landing). The general theory (I think) was that if a test is failing the pixel results and starts failing the text results as well, that almost certainly is a regression that should be noticed (and, conversely, if a test is now only failing the pixel tests, that's an improvement that should be noticed). There was also the belief that this mattered more to the Chromium devs than others :). Further, there were also debates as to whether reftests should be ImageOnlyFailure or just Failure. I don't think this one matters much one way or another. You are, of course, free to change it to however you'd like it to work these days. Won't fixing this. Please re-open this bug if you feel strongly about it. Yes, people still get confused about this. In fact, the latest I heard someone complain was last week. (In reply to comment #5) > Yes, people still get confused about this. In fact, the latest I heard > someone complain was last week. Okay. This should be easy enough for someone in the tools' team to fix. I can give some guidance as needed. |