Bug 63675 - new-run-webkit-tests should tell you when it creates new expectations
Summary: new-run-webkit-tests should tell you when it creates new expectations
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Adam Barth
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-06-29 17:53 PDT by Adam Barth
Modified: 2011-06-30 16:26 PDT (History)
3 users (show)

See Also:


Attachments
Patch (1.67 KB, patch)
2011-06-29 17:55 PDT, Adam Barth
eric: review+
dpranke: commit-queue-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Adam Barth 2011-06-29 17:53:58 PDT
new-run-webkit-tests should tell you when it creates new expectations
Comment 1 Adam Barth 2011-06-29 17:55:07 PDT
Created attachment 99196 [details]
Patch
Comment 2 Eric Seidel (no email) 2011-06-29 17:57:21 PDT
Comment on attachment 99196 [details]
Patch

OK.  No way to test this with our current logging infrastructure I assume?
Comment 3 Adam Barth 2011-06-29 18:17:29 PDT
Comment on attachment 99196 [details]
Patch

Yeah, if I could figure out how to test it, I could figure how to stop it from spamming the console during testing.
Comment 4 Dirk Pranke 2011-06-29 18:30:34 PDT
Comment on attachment 99196 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=99196&action=review

> Tools/Scripts/webkitpy/layout_tests/layout_package/single_test_runner.py:194
> +        _log.warning('Writing new expected result "%s"' % result_name)

This should be log.info(); these aren't warnings.
Comment 5 Adam Barth 2011-06-29 19:02:10 PDT
Committed r90079: <http://trac.webkit.org/changeset/90079>
Comment 6 Dirk Pranke 2011-06-30 16:26:30 PDT
Oh, by the way, it occurred to me that if DRT happens to fail on a text-only test in such a way that dumpAsText() doesn't get called and we don't realize the test is test-only, then the code will incorrectly think that the image is missing and generate a new (wrong) baseline for it.

This probably doesn't happen all that often, but is an argument for --no-new-test-results being on by default.

Maybe this should be filed as a separate bug somewhere, but I'm not sure how we can actually fix the DRT flaw so that we would know from NRWT's side definitively whether a test should be text-only or not.