Bug 66218 - Add two sample mismatch reftests to make sure that mismatch reftests work.
Summary: Add two sample mismatch reftests to make sure that mismatch reftests work.
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Hayato Ito
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-08-15 04:40 PDT by Hayato Ito
Modified: 2011-08-16 18:29 PDT (History)
4 users (show)

See Also:


Attachments
add sample mismatch reftests. (3.54 KB, patch)
2011-08-15 04:42 PDT, Hayato Ito
no flags Details | Formatted Diff | Diff
Update test_expectations.txt on all platforms. (6.56 KB, patch)
2011-08-15 18:45 PDT, Hayato Ito
no flags Details | Formatted Diff | Diff
skipped in other ports (5.94 KB, patch)
2011-08-16 17:16 PDT, Hayato Ito
tony: review+
webkit.review.bot: commit-queue-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Hayato Ito 2011-08-15 04:40:41 PDT
This is a separated patch from https://bugs.webkit.org/show_bug.cgi?id=58858

Adding sample mismatch reftests to LayoutTests/fast/harness.
Comment 1 Hayato Ito 2011-08-15 04:42:52 PDT
Created attachment 103902 [details]
add sample mismatch reftests.
Comment 2 Tony Chang 2011-08-15 11:10:06 PDT
Comment on attachment 103902 [details]
add sample mismatch reftests.

Do we need to add fast/harness/sample-fail-mismatch-reftest.html to the other test_expectations.txt or Skipped files?  What about ports that still use ORWT (Win, Lion, wk2, I think).

It's probably OK to skip fast/harness/sample-fail-mismatch-reftest.html on other ports since they use tests to check for regressions, not correctness.
Comment 3 WebKit Review Bot 2011-08-15 11:23:03 PDT
Attachment 103902 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'LayoutTests/ChangeLog', u'LayoutTests/fast..." exit_code: 1

ERROR: FAILURES FOR <lucid, x86_64, release, cpu>
ERROR: Line:261 Path does not exist. platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html
LayoutTests/platform/chromium/test_expectations.txt:261:  Path does not exist. platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html  [test/expectations] [2]
Total errors found: 1 in 6 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 4 Hayato Ito 2011-08-15 18:45:07 PDT
Created attachment 103991 [details]
Update test_expectations.txt on all platforms.
Comment 5 Hayato Ito 2011-08-15 18:54:28 PDT
Hi Tony, thank you for the review.

(In reply to comment #2)
> (From update of attachment 103902 [details])
> Do we need to add fast/harness/sample-fail-mismatch-reftest.html to the other test_expectations.txt or Skipped files?  What about ports that still use ORWT (Win, Lion, wk2, I think).

Yes. We need to do. I've updated test_expectations.txt on all platforms.
old-run-webkit-tests skips reftests automatically. So I've not touched SKipped files.


> It's probably OK to skip fast/harness/sample-fail-mismatch-reftest.html on other ports since they use tests to check for regressions, not correctness.

I'm not sure what 'they use tests to check for regressions, not correctness' mean.
I'd like to mark the test as WONTFIX on all platforms if that's okay. If that causes problems, please let me know. I am not 100% sure how other ports treat the test results.
Comment 6 Tony Chang 2011-08-16 09:25:48 PDT
Comment on attachment 103991 [details]
Update test_expectations.txt on all platforms.

View in context: https://bugs.webkit.org/attachment.cgi?id=103991&action=review

> LayoutTests/platform/chromium/test_expectations.txt:467
> +WONTFIX : fast/harness/sample-fail-mismatch-reftest.html = IMAGE

Oh, can you just add these to Skipped files instead of test_expectations.txt?  On the other ports, NRWT uses both Skipped and test_expectations.txt and I think so far the other ports are primarily using Skipped.

Sorry for the back and forth.
Comment 7 Hayato Ito 2011-08-16 17:16:50 PDT
Created attachment 104125 [details]
skipped in other ports
Comment 8 Hayato Ito 2011-08-16 17:18:05 PDT
Thank you for the review.

(In reply to comment #6)
> (From update of attachment 103991 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=103991&action=review
> 
> > LayoutTests/platform/chromium/test_expectations.txt:467
> > +WONTFIX : fast/harness/sample-fail-mismatch-reftest.html = IMAGE
> 
> Oh, can you just add these to Skipped files instead of test_expectations.txt?  On the other ports, NRWT uses both Skipped and test_expectations.txt and I think so far the other ports are primarily using Skipped.

Okay. I've updated Skipped files in other ports instead of test_expectations.txt. Thank you.
Comment 9 WebKit Review Bot 2011-08-16 18:17:54 PDT
Comment on attachment 104125 [details]
skipped in other ports

Rejecting attachment 104125 [details] from commit-queue.

Failed to run "['./Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--bot-id=ec2-cq-02', '--port..." exit_code: 2

Last 500 characters of output:
ded at 462 (offset -1 lines).
patching file LayoutTests/platform/gtk/Skipped
patching file LayoutTests/platform/mac/Skipped
patching file LayoutTests/platform/qt/Skipped
Hunk #1 FAILED at 2362.
1 out of 1 hunk FAILED -- saving rejects to file LayoutTests/platform/qt/Skipped.rej
patching file LayoutTests/platform/win/Skipped
Hunk #1 succeeded at 1386 (offset 8 lines).

Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', u'--reviewer', u'Tony Chang', u'--force']" exit_code: 1

Full output: http://queues.webkit.org/results/9401574
Comment 10 Hayato Ito 2011-08-16 18:29:39 PDT
Committed r93187: <http://trac.webkit.org/changeset/93187>