Bug 172322 – webkitpy: Layout tests which have no output will succeed

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Jonathan Bedard]

If a layout test has no output, it will succeed.  Such tests should fail.

[Jonathan Bedard]

Tracked internally at <rdar://problem/32258350>

[Jonathan Bedard]

Created attachment 310568 [details]
Patch

[David Kilzer (:ddkilzer)]

Comment on attachment 310568 [details]
Patch

r=me

[Jonathan Bedard]

Comment on attachment 310568 [details]
Patch

Waiting till EWS comes back before landing.

[WebKit Commit Bot]

Comment on attachment 310568 [details]
Patch

Clearing flags on attachment: 310568

Committed r217130: <http://trac.webkit.org/changeset/217130>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Daniel Bates]

Should we take a similar approach in SingleTestRunner._compare_audio() when there is no actual result?

[Daniel Bates]

How are these tests classified in the layout test results results.html file?

[Daniel Bates]

(In reply to Daniel Bates from comment #8)
> How are these tests classified in the layout test results results.html file?

They are classified as test_expectations.MISSING in determine_result_type() (defined in Tools/Scripts/webkitpy/layout_tests/models/test_failures.py). Tests that have no actual result more than likely represent a bug in the test tool/infrastructure and we should segregate them from tests that are missing their expected results.

[Daniel Bates]

Comment on attachment 310568 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=310568&action=review

> Tools/Scripts/webkitpy/layout_tests/models/test_failures.py:170
> +        return 'test was not run'

Is this correct? I mean, did we really not run the test or did we run the test and it had no output?

[Jonathan Bedard]

(In reply to Daniel Bates from comment #10)
> Comment on attachment 310568 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=310568&action=review
> 
> > Tools/Scripts/webkitpy/layout_tests/models/test_failures.py:170
> > +        return 'test was not run'
> 
> Is this correct? I mean, did we really not run the test or did we run the
> test and it had no output?

We attempted to run the test and had no output.  This could mean that we actually ran the test and had to output, or it could mean that the test was not run at all.  In my experience, this failure is usually caused by a test failing to run, although, the tests associated with https://bugs.webkit.org/show_bug.cgi?id=172378 likely have different causes.