Bug 76243 - [chromium] Reftests expected to be image mismatch still generate image mismatch when ran by NRWT --no-pixel-tests.
Summary: [chromium] Reftests expected to be image mismatch still generate image mismat...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Hao Zheng
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-01-12 20:33 PST by Hao Zheng
Modified: 2012-01-17 10:56 PST (History)
4 users (show)

See Also:


Attachments
Proposed patch. (3.46 KB, patch)
2012-01-12 23:45 PST, Hao Zheng
tony: review-
tony: commit-queue-
Details | Formatted Diff | Diff
Add test. (5.82 KB, patch)
2012-01-15 23:28 PST, Hao Zheng
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Hao Zheng 2012-01-12 20:33:52 PST
For example, fast/harness/sample-fail-mismatch-reftest.html
When ran by 'NRWT fast/harness/sample-fail-mismatch-reftest.html', all is fine: All 1 tests ran as expected.
When ran by 'NRWT --no-pixel-tests fast/harness/sample-fail-mismatch-reftest.html', output:
Regressions: Unexpected image mismatch : (1)
  fast/harness/sample-fail-mismatch-reftest.html = IMAGE

In TestExpectations.matches_an_expected_result, it removes pixel expectation when pixel tests is disabled.
        if not pixel_tests_are_enabled:
            expected_results = remove_pixel_failures(expected_results)
However, in test_failures.determine_result_type, reftest failure is treated as image failure.
        elif is_image_failure or is_reftest_failure:
            return test_expectations.IMAGE
Comment 1 Hao Zheng 2012-01-12 23:45:00 PST
Created attachment 122380 [details]
Proposed patch.
Comment 2 Tony Chang 2012-01-13 10:14:00 PST
Comment on attachment 122380 [details]
Proposed patch.

This change looks correct, but can you add a unit test for the change to manager.py?
Comment 3 Hao Zheng 2012-01-15 23:28:47 PST
Created attachment 122596 [details]
Add test.
Comment 4 WebKit Review Bot 2012-01-17 10:56:08 PST
Comment on attachment 122596 [details]
Add test.

Clearing flags on attachment: 122596

Committed r105169: <http://trac.webkit.org/changeset/105169>
Comment 5 WebKit Review Bot 2012-01-17 10:56:12 PST
All reviewed patches have been landed.  Closing bug.