RESOLVED WONTFIX60381
NRWT has unexpected failures for MISSING tests when run with --no-pixel 
https://bugs.webkit.org/show_bug.cgi?id=60381
Summary NRWT has unexpected failures for MISSING tests when run with --no-pixel
Tony Chang
Reported 2011-05-06 10:34:18 PDT
abarth is trying to get the ews chromium linux bots to run layout tests. He's using --no-pixel to start. There are two tests that are marked as MISSING because there are no png files checked in. When running running with --no-pixel, these tests fail text diff (i.e., TEXT failure) and show up as unexpected failures. Maybe when running --no-pixel, if a test is marked as MISSING we still need to check for a png file? Another possibility is to have MISSING imply other failures, but that reduces the incentives to remove MISSING from test_expectations.txt.
Attachments
Add attachment proposed patch, testcase, etc.
Dirk Pranke
Comment 1 2011-05-06 10:41:54 PDT
Hm. Well, a lame workaround would be to mark the test as TEXT MISSING. I will mull this over further.
Ojan Vafai
Comment 2 2011-05-06 13:52:38 PDT
I've actually wanted us to get rid of MISSING and instead having MISSING-TEXT, MISSING-IMAGE, MISSING-IMAGE+TEXT, MISSING-AUDIO. This would be a step in the direction of having the JSON be a 1:1 map to the TestFailure types. I'd still be OK with allowing people to put MISSING in the test_expectations file and it would just be syntactic sugar for any of the above. We'd be able to remove is_missing_audio, is_missing_text and is_missing_image from the json we output at http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py#L164
Ojan Vafai
Comment 3 2011-05-06 13:54:32 PDT
I guess that doesn't actually solve this problem. Ugh. Sorry for the noise.
Dirk Pranke
Comment 4 2011-05-06 14:02:45 PDT
Having thought about this further, I think the lame workaround should be good enough. MISSING is an unusual expectation that is trivially fixed, and I don't think it makes sense to alter the code to do anything about it. What do others think?
Ojan Vafai
Comment 5 2011-05-06 14:21:43 PDT
I agree. We need to work to simplify the test_expectations syntax as much as possible, not complicate it. I'm OK with Tony's first solution of still checking for a png file when running --no-pixel if it's not a dumpAsText test.
Tony Chang
Comment 6 2011-05-06 14:25:48 PDT
Updating test_expectations.txt is fine with me. I think platform/gtk/fonts/fontconfig-synthetic-bold.html is the only one left now. I poked the other test author and I think a png got checked in for fast/forms/date-input-visible-strings.html.
Dirk Pranke
Comment 7 2011-05-06 14:34:46 PDT
(In reply to comment #5) > I agree. We need to work to simplify the test_expectations syntax as much as possible, not complicate it. > > I'm OK with Tony's first solution of still checking for a png file when running --no-pixel if it's not a dumpAsText test. Unfortunately NRWT doesn't know if a test is a dumpAsText()/dumpAsMarkup() test. It can't distinguish a test that will never generate PNGs from a test with missing PNGs except insofar as it'll see that there are audio or reftest expectations instead.
Eric Seidel (no email)
Comment 8 2011-07-12 15:23:07 PDT
This seems related to bug 64367?
Eric Seidel (no email)
Comment 9 2011-10-12 14:35:02 PDT
I assume this is still an issue?
Dirk Pranke
Comment 10 2011-10-12 14:49:39 PDT
nothing changed in the code to address this bug. However, I'm not clear if this is still considered a bug worth doing anything about.
Dirk Pranke
Comment 11 2012-06-08 16:13:49 PDT
I'm going to close this as WONTFIX; please re-open or file a new bug if anyone would like to see something different happen here.
Note You need to log in before you can comment on or make changes to this bug.