Bug 220863 - [GTK] run-gtk-tests: Crashing and non-existent tests should not count as a pass
Summary: [GTK] run-gtk-tests: Crashing and non-existent tests should not count as a pass
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alicia Boya García
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-01-22 08:59 PST by Alicia Boya García
Modified: 2021-01-27 09:17 PST (History)
3 users (show)

See Also:


Attachments
Patch (3.35 KB, patch)
2021-01-22 09:00 PST, Alicia Boya García
no flags Details | Formatted Diff | Diff
Patch (3.60 KB, patch)
2021-01-26 08:01 PST, Alicia Boya García
no flags Details | Formatted Diff | Diff
Patch (3.50 KB, patch)
2021-01-27 06:27 PST, Alicia Boya García
no flags Details | Formatted Diff | Diff
Patch (3.50 KB, patch)
2021-01-27 08:46 PST, Alicia Boya García
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alicia Boya García 2021-01-22 08:59:28 PST
The current implementation doesn't account for test binaries that have
not emitted any subtests. This is the case when a test binary doesn't
exist or it crashes.

Also, in the latter case, the stderr of the crashed processes was not
being outputted to the user, masking a crashing test binary as a
passing test.

This patch fixes both issues: It adds several warning prints when
binaries can't be run or fail to add any subtest, emitting the stderr
generated by the failing or crashing binary.
Comment 1 Alicia Boya García 2021-01-22 09:00:22 PST
Created attachment 418152 [details]
Patch
Comment 2 Alicia Boya García 2021-01-26 08:01:32 PST
Created attachment 418421 [details]
Patch
Comment 3 Alicia Boya García 2021-01-26 08:06:02 PST
Uploaded an updated version that marks these tests as CRASH. I expect the EWS to fail on the test suites that are currently crashing.
Comment 4 Alicia Boya García 2021-01-27 06:03:14 PST
(In reply to Alicia Boya García from comment #3)
> Uploaded an updated version that marks these tests as CRASH. I expect the
> EWS to fail on the test suites that are currently crashing.

Well, EWS didn't fail because the crashes have been fixed in the meantime. Now the previously crashing test suites are reporting subtests again.

TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestCookieManager...
  /webkit/WebKitCookieManager/accept-policy:                          PASS
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestBackForwardList...
  /webkit/BackForwardList/navigation:                                 PASS
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestLoaderClient...
  /webkit/WebKitWebView/loading-status:                               PASS

... and so on.
Comment 5 Alicia Boya García 2021-01-27 06:27:04 PST
Created attachment 418538 [details]
Patch
Comment 6 Alicia Boya García 2021-01-27 06:27:38 PST
Reuploaded removing the ChangeLog line about changing expectations since it's no longer necessary.
Comment 7 Michael Catanzaro 2021-01-27 06:49:42 PST
Remember to CC bugs-noreply@webkitgtk.org if you're not using the GTK or WPE component, so it doesn't get missed!
Comment 8 Michael Catanzaro 2021-01-27 06:50:29 PST
Comment on attachment 418538 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=418538&action=review

> Tools/glib/api_test_runner.py:308
> +                    # No subtests where emitted, either the test binary didn't exist, or we don't know how to run it, or it crashed.

were
Comment 9 Alicia Boya García 2021-01-27 08:46:57 PST
Created attachment 418550 [details]
Patch
Comment 10 EWS 2021-01-27 09:14:20 PST
Committed r271955: <https://trac.webkit.org/changeset/271955>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 418550 [details].
Comment 11 Radar WebKit Bug Importer 2021-01-27 09:17:59 PST
<rdar://problem/73664773>