See bug 63057 for the initial bug report ... A problem occurs when a text-only test fails before it gets a chance to call layoutTestController.dumpAsText() ... DRT will output an image block, and NRWT will then look for (and not find) an -expected.png file. Older versions of NRWT used to print out what kind of image test failure was occuring (missing hash, missing text, missing checksum), but we've lumped them all together now into the one bucket. In addition, because the result is MISSING, the results.html file doesn't pull in the (actually existing) -expected.txt, and we can't see in the results that there is a text diff. I think this is causing unnecessary (and preventable) confusion, because it will cause people to think that the -expected.txt file couldn't be found. I'm not quite sure what to do about this, though, since there's no way to tell from the test itself whether it's text only or not. Possible options: 1) reserve MISSING for only tests that really have no expected text output, and possibly add a MISSING_IMAGE result type. 2) lump missing image files into an IMAGE failure? 3) modify the results.json file to know what kind of missing results we actually got. 4) ?? As another example, I recently did a test run and got output that included: Unexpected flakiness: no expected results found (3) fast/forms/form-associated-element-crash.html = MISSING PASS fast/forms/form-attribute-elements-order.html = MISSING PASS fast/forms/form-attribute-elements-order2.html = MISSING PASS Which, from a normal user's point of view, shouldn't be possible.
We should stop outputting MISSING entirely. Only output the more specific type of MISSING: MISSING_IMAGE, MISSING_TEXT, MISSING_IMAGE+TEXT. In the text_expectations.txt file we accept MISSING (syntactic sugar for the union of all three missing cases) or any of the above.
(In reply to comment #1) > We should stop outputting MISSING entirely. Only output the more specific type of MISSING: > MISSING_IMAGE, MISSING_TEXT, MISSING_IMAGE+TEXT. > > In the text_expectations.txt file we accept MISSING (syntactic sugar for the union of all three missing cases) or any of the above. Hm. This is a pretty good suggestion (at least as good as any of the others I made), but I don't think calling MISSING syntactic sugar is quite right. This is an example of the more general problem that we don't have a 1:1 mapping between possible combinations of failures and expectation types (and I don't know that we should). For example, how would we differentiate "text failure + missing image" from "text passed + missing image"? Presumably at least one of them is MISSING_IMAGE; what's the other?
(In reply to comment #2) > For example, how would we differentiate "text failure + missing image" from "text passed + missing image"? Presumably at least one of them is MISSING_IMAGE; what's the other? These cases are rare enough that I'm comfortable with just calling both of those MISSING_IMAGE.
I'm unclear as to if this is still an issue or what needs to be done here?
If it "blocks" conversion of the bots, it should be related to bug 34984. If it's a user polish issue, it should relate to bug 64491. If it's a chromium-only issue it can stay as-is.
I think we should probably distinguish MISSING_IMAGE from MISSING_TEXT somehow in the output. I'm not sure if we need to introduce new keywords, or not.
*** Bug 94532 has been marked as a duplicate of this bug. ***