Bug 217466 - [webkitpy] Allow passing subtests to GLIB API testrunner
Summary: [webkitpy] Allow passing subtests to GLIB API testrunner
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Diego Pino
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-10-08 01:30 PDT by Diego Pino
Modified: 2020-10-14 02:34 PDT (History)
2 users (show)

See Also:


Attachments
Patch (3.67 KB, patch)
2020-10-08 01:44 PDT, Diego Pino
no flags Details | Formatted Diff | Diff
Patch (3.74 KB, patch)
2020-10-08 02:50 PDT, Diego Pino
no flags Details | Formatted Diff | Diff
Patch (4.01 KB, patch)
2020-10-12 01:46 PDT, Diego Pino
no flags Details | Formatted Diff | Diff
Patch (4.01 KB, patch)
2020-10-12 01:51 PDT, Diego Pino
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Diego Pino 2020-10-08 01:30:57 PDT
[webkitpy] Allow passing subtests argument to glib testrunner API
Comment 1 Diego Pino 2020-10-08 01:44:11 PDT
Created attachment 410818 [details]
Patch
Comment 2 Diego Pino 2020-10-08 02:50:49 PDT
Created attachment 410824 [details]
Patch
Comment 3 Diego Pino 2020-10-08 05:34:11 PDT
For instance, now it's possible to run a GLIB API subtest like this:

```
$ python Tools/Scripts/run-gtk-tests --debug WebKitBuild/GTK/Debug/bin/TestWebKitAPI/WebKit2Gtk/TestUIClient -p /webkit/WebKitWebView/mouse-target
Could not start accessibility bus, so disabling TestWebKitAccessibility
TEST: WebKitBuild/GTK/Debug/bin/TestWebKitAPI/WebKit2Gtk/TestUIClient...
  /webkit/WebKitWebView/mouse-target:                                 FAIL
GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ‘gio-vfs’
ERROR:../../Tools/TestWebKitAPI/Tests/WebKitGLib/TestUIClient.cpp:817:void testWebViewMouseTarget(UIClientTest*, gconstpointer): 'webkit_hit_test_result_context_is_media(hitTestResult)' should be TRUE

Ran 1 tests of 1 with 1 successful
```
Comment 4 Carlos Garcia Campos 2020-10-08 09:21:49 PDT
Comment on attachment 410824 [details]
Patch

But subtests are specific to a test program, and they are passed to all test programs to be run, no?
Comment 5 Diego Pino 2020-10-09 06:10:14 PDT
(In reply to Carlos Garcia Campos from comment #4)
> Comment on attachment 410824 [details]
> Patch
> 
> But subtests are specific to a test program, and they are passed to all test
> programs to be run, no?

Yes, so in the case of launching the runner with a command like:

```
$ python Tools/Scripts/run-gtk-tests --debug -p /webkit/WebKitWebView/mouse-target
```

The runner iterates over all the tests running only the subtests passed. So if a test doesn't feature a subtest, nothing is run. For instance:

```
...
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestDOMNode...
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestWebKitWebContext...
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestResources...
GLib-GObject-FATAL-CRITICAL: g_object_unref: assertion 'G_IS_OBJECT (object)' failed
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestAuthentication...
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestWebKitSettings...
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestAutocleanups...
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestAutomationSession...
GLib-DEBUG: setenv()/putenv() are not thread-safe and should not be used after threads are created
GLib-DEBUG: unsetenv() is not thread-safe and should not be used after threads are created
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestUIClient...
  /webkit/WebKitWebView/mouse-target:                                 FAIL
GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ‘gio-vfs’
ERROR:../../Tools/TestWebKitAPI/Tests/WebKitGLib/TestUIClient.cpp:817:void testWebViewMouseTarget(UIClientTest*, gconstpointer): 'webkit_hit_test_result_context_is_media(hitTestResult)' should be TRUE
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestLoaderClient...
TEST: /app/webkit/Tools/glib/../../WebKitBuild/Release/bin/TestWebKitAPI/WebKit2Gtk/TestDOMElement...
...
```

For me, this is still better than not being able to run subtests at all from the runner. Is there's any convenient way to figure out whether a subtest belongs to a test? (so I could skip running the test in case not).
Comment 6 Diego Pino 2020-10-09 07:10:06 PDT
Perhaps an easier alternative is to require a test when one or more subtests are passed.
Comment 7 Carlos Garcia Campos 2020-10-09 08:04:26 PDT
I think we already collect the subtests for the skipped tests.
Comment 8 Diego Pino 2020-10-12 01:46:23 PDT
Created attachment 411102 [details]
Patch
Comment 9 Diego Pino 2020-10-12 01:51:32 PDT
Created attachment 411103 [details]
Patch
Comment 10 EWS 2020-10-14 02:33:29 PDT
Committed r268447: <https://trac.webkit.org/changeset/268447>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 411103 [details].
Comment 11 Radar WebKit Bug Importer 2020-10-14 02:34:16 PDT
<rdar://problem/70285212>