Bug 85446 - new-run-webkit-tests reports unexpected pass of pixel tests when pixel testing is disabled
Summary: new-run-webkit-tests reports unexpected pass of pixel tests when pixel testin...
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Dirk Pranke
Depends on:
Blocks: 88680
  Show dependency treegraph
Reported: 2012-05-02 18:45 PDT by Filip Pizlo
Modified: 2012-06-19 11:03 PDT (History)
6 users (show)

See Also:

Screenshot (58.61 KB, image/png)
2012-06-18 17:35 PDT, Simon Fraser (smfr)
no flags Details
Patch (3.71 KB, patch)
2012-06-18 18:45 PDT, Dirk Pranke
simon.fraser: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Filip Pizlo 2012-05-02 18:45:45 PDT
If new-run-webkit-tests does not run a test, it should not report anything about that test; in particular, reporting that it unexpectedly passed when it wasn't run at all is very confusing.
Comment 1 Dirk Pranke 2012-06-18 17:26:01 PDT
We used to have a bug where we would log 'unexpected pass'  for expected image failures (bug 36771), but that was fixed years ago.

When I run NRWT now, I don't see any "unexpected pass" messages for tests expected to fail with IMAGE failures.

However, I'm not sure I understand what you mean by "wasn't run at all"; you mean if a test foo.html is skipped, you're seeing unexpected pass for foo.html ?

If you can still reproduce this, can you tell me what command line you're running with?
Comment 2 Simon Fraser (smfr) 2012-06-18 17:35:47 PDT
Created attachment 148211 [details]

Here's the issue. I did run-webkit-tests --debug (i.e. NO pixel testing) but the results page said that sometests were expected to fail, but passed.

That's nonsensical, because only IMAGE failures were expected, but I'm not testing pixel results.
Comment 3 Dirk Pranke 2012-06-18 18:25:45 PDT
I see, I didn't realize we were talking about results.html ...

this is happening because the html/js page doesn't know if pixel testing was disabled or not. This should be easy to fix.
Comment 4 Dirk Pranke 2012-06-18 18:45:46 PDT
Created attachment 148221 [details]
Comment 5 Dirk Pranke 2012-06-19 11:03:36 PDT
Committed r120736: <http://trac.webkit.org/changeset/120736>