| Summary: | [chromium] Reftests expected to be image mismatch still generate image mismatch when ran by NRWT --no-pixel-tests. | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Hao Zheng <zhenghao> | ||||||
| Component: | Tools / Tests | Assignee: | Hao Zheng <zhenghao> | ||||||
| Status: | RESOLVED FIXED | ||||||||
| Severity: | Normal | CC: | abarth, ojan, tony, webkit.review.bot | ||||||
| Priority: | P2 | ||||||||
| Version: | 528+ (Nightly build) | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Attachments: |
|
||||||||
Created attachment 122380 [details]
Proposed patch.
Comment on attachment 122380 [details]
Proposed patch.
This change looks correct, but can you add a unit test for the change to manager.py?
Created attachment 122596 [details]
Add test.
Comment on attachment 122596 [details] Add test. Clearing flags on attachment: 122596 Committed r105169: <http://trac.webkit.org/changeset/105169> All reviewed patches have been landed. Closing bug. |
For example, fast/harness/sample-fail-mismatch-reftest.html When ran by 'NRWT fast/harness/sample-fail-mismatch-reftest.html', all is fine: All 1 tests ran as expected. When ran by 'NRWT --no-pixel-tests fast/harness/sample-fail-mismatch-reftest.html', output: Regressions: Unexpected image mismatch : (1) fast/harness/sample-fail-mismatch-reftest.html = IMAGE In TestExpectations.matches_an_expected_result, it removes pixel expectation when pixel tests is disabled. if not pixel_tests_are_enabled: expected_results = remove_pixel_failures(expected_results) However, in test_failures.determine_result_type, reftest failure is treated as image failure. elif is_image_failure or is_reftest_failure: return test_expectations.IMAGE