This is a separated patch from https://bugs.webkit.org/show_bug.cgi?id=58858 Adding sample mismatch reftests to LayoutTests/fast/harness.
Created attachment 103902 [details] add sample mismatch reftests.
Comment on attachment 103902 [details] add sample mismatch reftests. Do we need to add fast/harness/sample-fail-mismatch-reftest.html to the other test_expectations.txt or Skipped files? What about ports that still use ORWT (Win, Lion, wk2, I think). It's probably OK to skip fast/harness/sample-fail-mismatch-reftest.html on other ports since they use tests to check for regressions, not correctness.
Attachment 103902 [details] did not pass style-queue: Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'LayoutTests/ChangeLog', u'LayoutTests/fast..." exit_code: 1 ERROR: FAILURES FOR <lucid, x86_64, release, cpu> ERROR: Line:261 Path does not exist. platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html LayoutTests/platform/chromium/test_expectations.txt:261: Path does not exist. platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html [test/expectations] [2] Total errors found: 1 in 6 files If any of these errors are false positives, please file a bug against check-webkit-style.
Created attachment 103991 [details] Update test_expectations.txt on all platforms.
Hi Tony, thank you for the review. (In reply to comment #2) > (From update of attachment 103902 [details]) > Do we need to add fast/harness/sample-fail-mismatch-reftest.html to the other test_expectations.txt or Skipped files? What about ports that still use ORWT (Win, Lion, wk2, I think). Yes. We need to do. I've updated test_expectations.txt on all platforms. old-run-webkit-tests skips reftests automatically. So I've not touched SKipped files. > It's probably OK to skip fast/harness/sample-fail-mismatch-reftest.html on other ports since they use tests to check for regressions, not correctness. I'm not sure what 'they use tests to check for regressions, not correctness' mean. I'd like to mark the test as WONTFIX on all platforms if that's okay. If that causes problems, please let me know. I am not 100% sure how other ports treat the test results.
Comment on attachment 103991 [details] Update test_expectations.txt on all platforms. View in context: https://bugs.webkit.org/attachment.cgi?id=103991&action=review > LayoutTests/platform/chromium/test_expectations.txt:467 > +WONTFIX : fast/harness/sample-fail-mismatch-reftest.html = IMAGE Oh, can you just add these to Skipped files instead of test_expectations.txt? On the other ports, NRWT uses both Skipped and test_expectations.txt and I think so far the other ports are primarily using Skipped. Sorry for the back and forth.
Created attachment 104125 [details] skipped in other ports
Thank you for the review. (In reply to comment #6) > (From update of attachment 103991 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=103991&action=review > > > LayoutTests/platform/chromium/test_expectations.txt:467 > > +WONTFIX : fast/harness/sample-fail-mismatch-reftest.html = IMAGE > > Oh, can you just add these to Skipped files instead of test_expectations.txt? On the other ports, NRWT uses both Skipped and test_expectations.txt and I think so far the other ports are primarily using Skipped. Okay. I've updated Skipped files in other ports instead of test_expectations.txt. Thank you.
Comment on attachment 104125 [details] skipped in other ports Rejecting attachment 104125 [details] from commit-queue. Failed to run "['./Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--bot-id=ec2-cq-02', '--port..." exit_code: 2 Last 500 characters of output: ded at 462 (offset -1 lines). patching file LayoutTests/platform/gtk/Skipped patching file LayoutTests/platform/mac/Skipped patching file LayoutTests/platform/qt/Skipped Hunk #1 FAILED at 2362. 1 out of 1 hunk FAILED -- saving rejects to file LayoutTests/platform/qt/Skipped.rej patching file LayoutTests/platform/win/Skipped Hunk #1 succeeded at 1386 (offset 8 lines). Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', u'--reviewer', u'Tony Chang', u'--force']" exit_code: 1 Full output: http://queues.webkit.org/results/9401574
Committed r93187: <http://trac.webkit.org/changeset/93187>