I'm probably just doing it wrong, but I couldn't fine any sample expected mismatch reftests to compare to. I made foo.html and foo-expected-mismatch.html and made their contents identical (just "asdf"). The test passed, but should have failed, right?
reftests only apply to the image part of the test, not the text part.
there are (artificial) samples in webkitpy.layout_tests.port/test.py that you can play with via new-run-webkit-tests --platform test, if that's helpful. Of course, they don't generate real output, so it probably isn't for what you're trying to do.
(In reply to comment #1) > reftests only apply to the image part of the test, not the text part. But still, the test should pass, right? I was running with -p. (In reply to comment #2) > there are (artificial) samples in webkitpy.layout_tests.port/test.py that you can play with via new-run-webkit-tests --platform test, if that's helpful. Of course, they don't generate real output, so it probably isn't for what you're trying to do. I'll take a look.
(In reply to comment #3) > (In reply to comment #1) > > reftests only apply to the image part of the test, not the text part. > > But still, the test should pass, right? I was running with -p. > Oh, I see, I read that as you had made the text output the same. Sorry. Yes, I would expect such a test to fail.
I'll take a look. There is no sample 'real' mismatch reftests in LayoutTest directory. So I might miss something. I'll try to submit a sample mismatch reftest, expecting to find a issue. (In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #1) > > > reftests only apply to the image part of the test, not the text part. > > > > But still, the test should pass, right? I was running with -p. > > > > Oh, I see, I read that as you had made the text output the same. Sorry. Yes, I would expect such a test to fail.
Created attachment 90142 [details] add-sample-reftests
I posted a patch which adds two sample reftests to the fast/harness directory. It seems a mismatch-reftest works correctly on my environment. I cannot reproduce the wrong behavior. Hmm. It would be great if you could help finding an issue about mismatch reftests. (In reply to comment #6) > Created an attachment (id=90142) [details] > add-sample-reftests
Comment on attachment 90142 [details] add-sample-reftests When I aply this patch and run: build-webkit new-run-webkit-tests LayoutTests/fast/harness/ I get: Regressions: Unexpected image mismatch : (2) fast/harness/sample-fail-mismatch-reftest.html = IMAGE fast/harness/sample-mismatch-reftest.html = IMAGE The images are similar for first test and different for the second, but they both fail due to image mismatch. Note this is not a Chromium DRT, it's the regular mac build of WebKit. I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file.
Hi Dmitry, Thank you for testing that. I can reproduce that on mac build of WebKit. I'll take a look further. Sorry for the inconvenience. (In reply to comment #8) > (From update of attachment 90142 [details]) > When I aply this patch and run: > > build-webkit > new-run-webkit-tests LayoutTests/fast/harness/ > > I get: > > Regressions: Unexpected image mismatch : (2) > fast/harness/sample-fail-mismatch-reftest.html = IMAGE > fast/harness/sample-mismatch-reftest.html = IMAGE > > The images are similar for first test and different for the second, but they both fail due to image mismatch. > Note this is not a Chromium DRT, it's the regular mac build of WebKit. > > I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file. (In reply to comment #8) > (From update of attachment 90142 [details]) > When I aply this patch and run: > > build-webkit > new-run-webkit-tests LayoutTests/fast/harness/ > > I get: > > Regressions: Unexpected image mismatch : (2) > fast/harness/sample-fail-mismatch-reftest.html = IMAGE > fast/harness/sample-mismatch-reftest.html = IMAGE > > The images are similar for first test and different for the second, but they both fail due to image mismatch. > Note this is not a Chromium DRT, it's the regular mac build of WebKit. > > I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file.
Let me note what I figured out: - WebKitPort, which is used in new-run-webkit-tests, is trying to disable 'pixel_tests' in default. - We should skip reftests if pixel_tests is disabled. - Even if '--pixel-tests' in turned on explicitly on WebKitPort, there is a difference between ChromiumLinuxDriver and WebKitDriver about how to handle 'image_hash' value. Such a different behavior makes a correct reftests fail on Mac build of WebKit. I'll file bugs for each.
I filed the two bugs: - https://bugs.webkit.org/show_bug.cgi?id=59078 - https://bugs.webkit.org/show_bug.cgi?id=59188
I posted a patch to fix a WebKitDriver so that reftests work in bug 59188. After the patch in bug 59188 is landed, I am going to add these sample reftests too.
Bug 59188 was landed. I'd like to commit these sample mismatch reftests to detect a regression. Should I rename the subject of this bugzilla entry or create a new bug for these sample tests? (In reply to comment #12) > I posted a patch to fix a WebKitDriver so that reftests work in bug 59188. > After the patch in bug 59188 is landed, I am going to add these sample reftests too.
It might be better to close this bug. A mismatch refrests should work on all platfoms after fixing bug 59188. I'll file a separate bug for adding sample mismatch reftests.