Bug 220863 – [GTK] run-gtk-tests: Crashing and non-existent tests should not count as a pass

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Alicia Boya García]

The current implementation doesn't account for test binaries that have
not emitted any subtests. This is the case when a test binary doesn't
exist or it crashes.

Also, in the latter case, the stderr of the crashed processes was not
being outputted to the user, masking a crashing test binary as a
passing test.

This patch fixes both issues: It adds several warning prints when
binaries can't be run or fail to add any subtest, emitting the stderr
generated by the failing or crashing binary.

[Alicia Boya García]

Created attachment 418152 [details]
Patch

[Alicia Boya García]

Created attachment 418421 [details]
Patch

[Alicia Boya García]

Uploaded an updated version that marks these tests as CRASH. I expect the EWS to fail on the test suites that are currently crashing.

[Alicia Boya García]

(In reply to Alicia Boya García from comment #3)
> Uploaded an updated version that marks these tests as CRASH. I expect the
> EWS to fail on the test suites that are currently crashing.

Well, EWS didn't fail because the crashes have been fixed in the meantime. Now the previously crashing test suites are reporting subtests again.

TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestCookieManager...
  /webkit/WebKitCookieManager/accept-policy:                          PASS
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestBackForwardList...
  /webkit/BackForwardList/navigation:                                 PASS
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestLoaderClient...
  /webkit/WebKitWebView/loading-status:                               PASS

... and so on.

[Alicia Boya García]

Created attachment 418538 [details]
Patch

[Alicia Boya García]

Reuploaded removing the ChangeLog line about changing expectations since it's no longer necessary.

[Michael Catanzaro]

Remember to CC [e-mail] if you're not using the GTK or WPE component, so it doesn't get missed!

[Michael Catanzaro]

Comment on attachment 418538 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=418538&action=review

> Tools/glib/api_test_runner.py:308
> +                    # No subtests where emitted, either the test binary didn't exist, or we don't know how to run it, or it crashed.

were

[Alicia Boya García]

Created attachment 418550 [details]
Patch

[EWS]

Committed r271955: <https://trac.webkit.org/changeset/271955>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 418550 [details].

[Radar WebKit Bug Importer]

<rdar://problem/73664773>