Bug 63675 – new-run-webkit-tests should tell you when it creates new expectations

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Adam Barth]

new-run-webkit-tests should tell you when it creates new expectations

[Adam Barth]

Created attachment 99196 [details]
Patch

[Eric Seidel (no email)]

Comment on attachment 99196 [details]
Patch

OK.  No way to test this with our current logging infrastructure I assume?

[Adam Barth]

Comment on attachment 99196 [details]
Patch

Yeah, if I could figure out how to test it, I could figure how to stop it from spamming the console during testing.

[Dirk Pranke]

Comment on attachment 99196 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=99196&action=review

> Tools/Scripts/webkitpy/layout_tests/layout_package/single_test_runner.py:194
> +        _log.warning('Writing new expected result "%s"' % result_name)

This should be log.info(); these aren't warnings.

[Adam Barth]

Committed r90079: <http://trac.webkit.org/changeset/90079>

[Dirk Pranke]

Oh, by the way, it occurred to me that if DRT happens to fail on a text-only test in such a way that dumpAsText() doesn't get called and we don't realize the test is test-only, then the code will incorrectly think that the image is missing and generate a new (wrong) baseline for it.

This probably doesn't happen all that often, but is an argument for --no-new-test-results being on by default.

Maybe this should be filed as a separate bug somewhere, but I'm not sure how we can actually fix the DRT flaw so that we would know from NRWT's side definitively whether a test should be text-only or not.