Bug 136339 – [GTK] test case filter support in run-gtk-tests

• Status: NEW
• Component: Tools / Tests

[Philippe Normand]

If a test program bundles multiple test-cases it'd be nice to be able to run only a subset of those, especially when adding new test cases.

[Philippe Normand]

Created attachment 237304 [details]
patch

[Philippe Normand]

Created attachment 237306 [details]
patch

Restored executable permissions on file.

[Carlos Garcia Campos]

Comment on attachment 237304 [details]
patch

View in context: https://bugs.webkit.org/attachment.cgi?id=237304&action=review

Thanks for working on this, I have few comments/questions.

> Tools/Scripts/run-gtk-tests:325
> +        if self._options.filter and self._options.filter.startswith('/'):
> +            tester_command.extend(['-p', self._options.filter])

So, for glib we only support one test case? What happens if the same test is added and skipped? I guess we should ifgnore skipped tests in this case.

> Tools/Scripts/run-gtk-tests:364
> +        if self._options.filter:
> +            filters = self._options.filter.split(',')
> +            basedir = os.path.basename(os.path.dirname(test_program))
> +            subtests = [ f for f in filters if f.startswith(basedir) ]
> +        if not subtests:
> +            subtests = self._get_tests_from_google_test_suite(test_program)

I wouldn't add the tests if we fail to parse the given test in the filter, if there user gave a bad test or whatever, we should not run any test. So here I would do if self._options.filter: else: subtests = self._get_tests_from_google_test_suite(test_program)

> Tools/Scripts/run-gtk-tests:468
> +    option_parser.add_option('-f', '--filter',
> +                             action='store', type='string', dest='filter', default='',
> +                             help='Execute only the specified test(s) of the suite')

I'm not sure filter is the best name, because skipped tests are also filters, and filter also sounds to me like you might be able to filter out. Maybe -t, --tests? Also the help description says test(s) but it's not clear how to specify more than one test

[Build Bot]

Comment on attachment 237306 [details]
patch

Attachment 237306 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.appspot.com/results/5397164192694272

New failing tests:
compositing/checkerboard.html
compositing/bounds-in-flipped-writing-mode.html
accessibility/adjacent-continuations-cause-assertion-failure.html
canvas/philip/tests/2d.clearRect+fillRect.alpha0.html
http/tests/appcache/abort-cache-onchecking-resource-404.html
animations/animation-border-overflow.html
animations/3d/transform-origin-vs-functions.html
http/tests/appcache/abort-cache-onchecking.html
accessibility/accessibility-object-detached.html
http/tests/appcache/abort-cache-onchecking-manifest-404.html
canvas/philip/tests/2d.clearRect+fillRect.basic.html
animations/animation-add-events-in-handler.html
animations/animation-controller-drt-api.html
animations/3d/replace-filling-transform.html
animations/3d/state-at-end-event-transform.html
canvas/philip/tests/2d.clearRect+fillRect.alpha0.5.html
accessibility/alt-tag-on-image-with-nonimage-role.html
compositing/absolute-position-changed-with-composited-parent-layer.html

[Build Bot]

Created attachment 237312 [details]
Archive of layout-test-results from webkit-ews-14 for mac-mountainlion-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: webkit-ews-14  Port: mac-mountainlion-wk2  Platform: Mac OS X 10.8.5

[Michael Catanzaro]

Comment on attachment 237306 [details]
patch

Removing this from request-queue; please respond to Carlos's suggestions! :)