Bug 60381
| Summary: | NRWT has unexpected failures for MISSING tests when run with --no-pixel | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Tony Chang <tony> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | RESOLVED WONTFIX | ||
| Severity: | Normal | CC: | abarth, dpranke, eric, ojan |
| Priority: | P2 | ||
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Bug Depends on: | |||
| Bug Blocks: | 64367 | ||
Tony Chang
abarth is trying to get the ews chromium linux bots to run layout tests. He's using --no-pixel to start. There are two tests that are marked as MISSING because there are no png files checked in. When running running with --no-pixel, these tests fail text diff (i.e., TEXT failure) and show up as unexpected failures.
Maybe when running --no-pixel, if a test is marked as MISSING we still need to check for a png file? Another possibility is to have MISSING imply other failures, but that reduces the incentives to remove MISSING from test_expectations.txt.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Dirk Pranke
Hm. Well, a lame workaround would be to mark the test as TEXT MISSING. I will mull this over further.
Ojan Vafai
I've actually wanted us to get rid of MISSING and instead having MISSING-TEXT, MISSING-IMAGE, MISSING-IMAGE+TEXT, MISSING-AUDIO. This would be a step in the direction of having the JSON be a 1:1 map to the TestFailure types. I'd still be OK with allowing people to put MISSING in the test_expectations file and it would just be syntactic sugar for any of the above.
We'd be able to remove is_missing_audio, is_missing_text and is_missing_image from the json we output at http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py#L164
Ojan Vafai
I guess that doesn't actually solve this problem. Ugh. Sorry for the noise.
Dirk Pranke
Having thought about this further, I think the lame workaround should be good enough. MISSING is an unusual expectation that is trivially fixed, and I don't think it makes sense to alter the code to do anything about it.
What do others think?
Ojan Vafai
I agree. We need to work to simplify the test_expectations syntax as much as possible, not complicate it.
I'm OK with Tony's first solution of still checking for a png file when running --no-pixel if it's not a dumpAsText test.
Tony Chang
Updating test_expectations.txt is fine with me.
I think platform/gtk/fonts/fontconfig-synthetic-bold.html is the only one left now. I poked the other test author and I think a png got checked in for fast/forms/date-input-visible-strings.html.
Dirk Pranke
(In reply to comment #5)
> I agree. We need to work to simplify the test_expectations syntax as much as possible, not complicate it.
>
> I'm OK with Tony's first solution of still checking for a png file when running --no-pixel if it's not a dumpAsText test.
Unfortunately NRWT doesn't know if a test is a dumpAsText()/dumpAsMarkup() test. It can't distinguish a test that will never generate PNGs from a test with missing PNGs except insofar as it'll see that there are audio or reftest expectations instead.
Eric Seidel (no email)
This seems related to bug 64367?
Eric Seidel (no email)
I assume this is still an issue?
Dirk Pranke
nothing changed in the code to address this bug. However, I'm not clear if this is still considered a bug worth doing anything about.
Dirk Pranke
I'm going to close this as WONTFIX; please re-open or file a new bug if anyone would like to see something different happen here.