Bug 63675

Summary: new-run-webkit-tests should tell you when it creates new expectations
Product: WebKit Reporter: Adam Barth <abarth>
Component: New BugsAssignee: Adam Barth <abarth>
Status: RESOLVED FIXED    
Severity: Normal CC: dpranke, eric, webkit.review.bot
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch eric: review+, dpranke: commit-queue-

Description Adam Barth 2011-06-29 17:53:58 PDT
new-run-webkit-tests should tell you when it creates new expectations
Comment 1 Adam Barth 2011-06-29 17:55:07 PDT
Created attachment 99196 [details]
Patch
Comment 2 Eric Seidel (no email) 2011-06-29 17:57:21 PDT
Comment on attachment 99196 [details]
Patch

OK.  No way to test this with our current logging infrastructure I assume?
Comment 3 Adam Barth 2011-06-29 18:17:29 PDT
Comment on attachment 99196 [details]
Patch

Yeah, if I could figure out how to test it, I could figure how to stop it from spamming the console during testing.
Comment 4 Dirk Pranke 2011-06-29 18:30:34 PDT
Comment on attachment 99196 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=99196&action=review

> Tools/Scripts/webkitpy/layout_tests/layout_package/single_test_runner.py:194
> +        _log.warning('Writing new expected result "%s"' % result_name)

This should be log.info(); these aren't warnings.
Comment 5 Adam Barth 2011-06-29 19:02:10 PDT
Committed r90079: <http://trac.webkit.org/changeset/90079>
Comment 6 Dirk Pranke 2011-06-30 16:26:30 PDT
Oh, by the way, it occurred to me that if DRT happens to fail on a text-only test in such a way that dumpAsText() doesn't get called and we don't realize the test is test-only, then the code will incorrectly think that the image is missing and generate a new (wrong) baseline for it.

This probably doesn't happen all that often, but is an argument for --no-new-test-results being on by default.

Maybe this should be filed as a separate bug somewhere, but I'm not sure how we can actually fix the DRT flaw so that we would know from NRWT's side definitively whether a test should be text-only or not.