Bug 66218 – Add two sample mismatch reftests to make sure that mismatch reftests work.

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Hayato Ito]

This is a separated patch from https://bugs.webkit.org/show_bug.cgi?id=58858

Adding sample mismatch reftests to LayoutTests/fast/harness.

[Hayato Ito]

Created attachment 103902 [details]
add sample mismatch reftests.

[Tony Chang]

Comment on attachment 103902 [details]
add sample mismatch reftests.

Do we need to add fast/harness/sample-fail-mismatch-reftest.html to the other test_expectations.txt or Skipped files?  What about ports that still use ORWT (Win, Lion, wk2, I think).

It's probably OK to skip fast/harness/sample-fail-mismatch-reftest.html on other ports since they use tests to check for regressions, not correctness.

[WebKit Review Bot]

Attachment 103902 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'LayoutTests/ChangeLog', u'LayoutTests/fast..." exit_code: 1

ERROR: FAILURES FOR <lucid, x86_64, release, cpu>
ERROR: Line:261 Path does not exist. platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html
LayoutTests/platform/chromium/test_expectations.txt:261:  Path does not exist. platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html  [test/expectations] [2]
Total errors found: 1 in 6 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Hayato Ito]

Created attachment 103991 [details]
Update test_expectations.txt on all platforms.

[Hayato Ito]

Hi Tony, thank you for the review.

(In reply to comment #2)
> (From update of attachment 103902 [details])
> Do we need to add fast/harness/sample-fail-mismatch-reftest.html to the other test_expectations.txt or Skipped files?  What about ports that still use ORWT (Win, Lion, wk2, I think).

Yes. We need to do. I've updated test_expectations.txt on all platforms.
old-run-webkit-tests skips reftests automatically. So I've not touched SKipped files.


> It's probably OK to skip fast/harness/sample-fail-mismatch-reftest.html on other ports since they use tests to check for regressions, not correctness.

I'm not sure what 'they use tests to check for regressions, not correctness' mean.
I'd like to mark the test as WONTFIX on all platforms if that's okay. If that causes problems, please let me know. I am not 100% sure how other ports treat the test results.

[Tony Chang]

Comment on attachment 103991 [details]
Update test_expectations.txt on all platforms.

View in context: https://bugs.webkit.org/attachment.cgi?id=103991&action=review

> LayoutTests/platform/chromium/test_expectations.txt:467
> +WONTFIX : fast/harness/sample-fail-mismatch-reftest.html = IMAGE

Oh, can you just add these to Skipped files instead of test_expectations.txt?  On the other ports, NRWT uses both Skipped and test_expectations.txt and I think so far the other ports are primarily using Skipped.

Sorry for the back and forth.

[Hayato Ito]

Created attachment 104125 [details]
skipped in other ports

[Hayato Ito]

Thank you for the review.

(In reply to comment #6)
> (From update of attachment 103991 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=103991&action=review
> 
> > LayoutTests/platform/chromium/test_expectations.txt:467
> > +WONTFIX : fast/harness/sample-fail-mismatch-reftest.html = IMAGE
> 
> Oh, can you just add these to Skipped files instead of test_expectations.txt?  On the other ports, NRWT uses both Skipped and test_expectations.txt and I think so far the other ports are primarily using Skipped.

Okay. I've updated Skipped files in other ports instead of test_expectations.txt. Thank you.

[WebKit Review Bot]

Comment on attachment 104125 [details]
skipped in other ports

Rejecting attachment 104125 [details] from commit-queue.

Failed to run "['./Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--bot-id=ec2-cq-02', '--port..." exit_code: 2

Last 500 characters of output:
ded at 462 (offset -1 lines).
patching file LayoutTests/platform/gtk/Skipped
patching file LayoutTests/platform/mac/Skipped
patching file LayoutTests/platform/qt/Skipped
Hunk #1 FAILED at 2362.
1 out of 1 hunk FAILED -- saving rejects to file LayoutTests/platform/qt/Skipped.rej
patching file LayoutTests/platform/win/Skipped
Hunk #1 succeeded at 1386 (offset 8 lines).

Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', u'--reviewer', u'Tony Chang', u'--force']" exit_code: 1

Full output: http://queues.webkit.org/results/9401574

[Hayato Ito]

Committed r93187: <http://trac.webkit.org/changeset/93187>