Bug 58858 – reftest mismatch doesn't work

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[Ojan Vafai]

I'm probably just doing it wrong, but I couldn't fine any sample expected mismatch reftests to compare to. I made foo.html and foo-expected-mismatch.html and made their contents identical (just "asdf"). The test passed, but should have failed, right?

[Dirk Pranke]

reftests only apply to the image part of the test, not the text part.

[Dirk Pranke]

there are (artificial) samples in webkitpy.layout_tests.port/test.py that you can play with via new-run-webkit-tests --platform test, if that's helpful. Of course, they don't generate real output, so it probably isn't for what you're trying to do.

[Ojan Vafai]

(In reply to comment #1)
> reftests only apply to the image part of the test, not the text part.

But still, the test should pass, right? I was running with -p.

(In reply to comment #2)
> there are (artificial) samples in webkitpy.layout_tests.port/test.py that you can play with via new-run-webkit-tests --platform test, if that's helpful. Of course, they don't generate real output, so it probably isn't for what you're trying to do.

I'll take a look.

[Dirk Pranke]

(In reply to comment #3)
> (In reply to comment #1)
> > reftests only apply to the image part of the test, not the text part.
> 
> But still, the test should pass, right? I was running with -p.
> 

Oh, I see, I read that as you had made the text output the same. Sorry. Yes, I would expect such a test to fail.

[Hayato Ito]

I'll take a look.

There is no sample 'real' mismatch reftests in LayoutTest directory. So I might miss something.
I'll try to submit a sample mismatch reftest, expecting to find a issue.

(In reply to comment #4)
> (In reply to comment #3)
> > (In reply to comment #1)
> > > reftests only apply to the image part of the test, not the text part.
> > 
> > But still, the test should pass, right? I was running with -p.
> > 
> 
> Oh, I see, I read that as you had made the text output the same. Sorry. Yes, I would expect such a test to fail.

[Hayato Ito]

Created attachment 90142 [details]
add-sample-reftests

[Hayato Ito]

I posted a patch which adds two sample reftests to the fast/harness directory.
It seems a mismatch-reftest works correctly on my environment. I cannot reproduce the wrong behavior. Hmm.

It would be great if you could help finding an issue about mismatch reftests.

(In reply to comment #6)
> Created an attachment (id=90142) [details]
> add-sample-reftests

[Dmitry Titov]

Comment on attachment 90142 [details]
add-sample-reftests

When I aply this patch and run:

build-webkit
new-run-webkit-tests LayoutTests/fast/harness/

I get:

Regressions: Unexpected image mismatch : (2)
  fast/harness/sample-fail-mismatch-reftest.html = IMAGE
  fast/harness/sample-mismatch-reftest.html = IMAGE

The images are similar for first test and different for the second, but they both fail due to image mismatch.
Note this is not a Chromium DRT, it's the regular mac build of WebKit.

I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file.

[Hayato Ito]

Hi Dmitry,

Thank you for testing that. I can reproduce that on mac build of WebKit. I'll take a look further. Sorry for the inconvenience.

(In reply to comment #8)
> (From update of attachment 90142 [details])
> When I aply this patch and run:
> 
> build-webkit
> new-run-webkit-tests LayoutTests/fast/harness/
> 
> I get:
> 
> Regressions: Unexpected image mismatch : (2)
>   fast/harness/sample-fail-mismatch-reftest.html = IMAGE
>   fast/harness/sample-mismatch-reftest.html = IMAGE
> 
> The images are similar for first test and different for the second, but they both fail due to image mismatch.
> Note this is not a Chromium DRT, it's the regular mac build of WebKit.
> 
> I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file.

(In reply to comment #8)
> (From update of attachment 90142 [details])
> When I aply this patch and run:
> 
> build-webkit
> new-run-webkit-tests LayoutTests/fast/harness/
> 
> I get:
> 
> Regressions: Unexpected image mismatch : (2)
>   fast/harness/sample-fail-mismatch-reftest.html = IMAGE
>   fast/harness/sample-mismatch-reftest.html = IMAGE
> 
> The images are similar for first test and different for the second, but they both fail due to image mismatch.
> Note this is not a Chromium DRT, it's the regular mac build of WebKit.
> 
> I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file.

[Hayato Ito]

Let me note what I figured out:
- WebKitPort, which is used in new-run-webkit-tests, is trying to disable 'pixel_tests' in default.
- We should skip reftests if pixel_tests is disabled.
- Even if '--pixel-tests' in turned on explicitly on WebKitPort,  there is a difference between ChromiumLinuxDriver and WebKitDriver about how to handle 'image_hash' value. Such a different behavior makes a correct reftests fail on Mac build of WebKit.

I'll file bugs for each.

[Hayato Ito]

I filed the two bugs:
 - https://bugs.webkit.org/show_bug.cgi?id=59078
 - https://bugs.webkit.org/show_bug.cgi?id=59188

[Hayato Ito]

I posted a patch to fix a WebKitDriver so that reftests work in bug 59188.
After the patch in bug 59188 is landed, I am going to add these sample reftests too.

[Hayato Ito]

Bug 59188 was landed. I'd like to commit these sample mismatch reftests to detect a regression.
Should I rename the subject of this bugzilla entry or create a new bug for these sample tests?

(In reply to comment #12)
> I posted a patch to fix a WebKitDriver so that reftests work in bug 59188.
> After the patch in bug 59188 is landed, I am going to add these sample reftests too.

[Hayato Ito]

It might be better to close this bug. A mismatch refrests should work on all platfoms after fixing bug 59188.

I'll file a separate bug for adding sample mismatch reftests.