Bug 127332
| Summary: | Shouldn't Failure expectation work for image-only failures? | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Alexey Proskuryakov <ap> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | REOPENED | ||
| Severity: | Normal | CC: | dbates, dburkart, dean_johnson, dpranke, matthew_hanson, rniwa, zalan |
| Priority: | P2 | ||
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
Alexey Proskuryakov
Failure allows for most kinds of failures, namely TEXT, IMAGE_PLUS_TEXT, and AUDIO. But why not image-only failures? Notably, one would expect that it works for reftests, I think.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Ryosuke Niwa
Yeah, I always disliked this. Failure should include ImageOnlyFailure.
Dirk Pranke
There were discussions on this when we were talking about changing the syntax back in 6/12 (and then in 9/12 when the changes were landing).
The general theory (I think) was that if a test is failing the pixel results and starts failing the text results as well, that almost certainly is a regression that should be noticed (and, conversely, if a test is now only failing the pixel tests, that's an improvement that should be noticed). There was also the belief that this mattered more to the Chromium devs than others :).
Further, there were also debates as to whether reftests should be ImageOnlyFailure or just Failure. I don't think this one matters much one way or another.
Dirk Pranke
You are, of course, free to change it to however you'd like it to work these days.
Ryosuke Niwa
Won't fixing this. Please re-open this bug if you feel strongly about it.
Alexey Proskuryakov
Yes, people still get confused about this. In fact, the latest I heard someone complain was last week.
Ryosuke Niwa
(In reply to comment #5)
> Yes, people still get confused about this. In fact, the latest I heard
> someone complain was last week.
Okay. This should be easy enough for someone in the tools' team to fix. I can give some guidance as needed.