Bug 224638 - Migrate Python tests to pytest
Summary: Migrate Python tests to pytest
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
Keywords: InRadar
Depends on: 224627 224687 224877 226017
  Show dependency treegraph
Reported: 2021-04-15 18:30 PDT by Sam Sneddon [:gsnedders]
Modified: 2024-04-06 15:58 PDT (History)
4 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Sam Sneddon [:gsnedders] 2021-04-15 18:30:45 PDT
There has been occasional talk about moving our Python unit testing from the stdlib unittest library to pytest, which we already have for the sake of WPT's WebDriver tests.

pytest adds a variety of useful features (various ways to do test filtering, better debug integration, much better skip/xfail support, a de-facto standard parallel test-execution method (pytest-xdist), less boilerplate in tests), and avoids us having to reimplement much in our own wrapper around unittest.

One open question is whether we can break the specific arguments used by ./Tools/Scripts/test-webkitpy{,-python2} today?

Aside from that, there's a few more technical issues:

 * pytest's unittest support uses the stdlib unittest.TestLoader class, which we subclass in webkitpy.test.main, to optionally load methods beginning with serial_test_*, integration_test_*, and serial_integration_test_* as well as test_*; logically these would become marks in pytest

 * we have support for uploading test results (via --report) and our own JSON format (--json-output); we could either implement post-processing results from pytest or implement our own output format

 * pytest doesn't handle running tests in very different directories particularly well due to its rootdir and confdir handling (and for Source/WebKit/Scripts/webkit and Tools/Scripts/webkitpy the common root in the root of the WebKit repo, and we likely don't want to put any files there for pytest!); we could just have multiple invocations of pytest?

 * pytest-xdist doesn't support running a subset of tests sequentially (https://github.com/pytest-dev/pytest-xdist/issues/385), which we currently do for serial_* tests; this is another argument for multiple invocations of pytest
Comment 1 Radar WebKit Bug Importer 2021-04-15 18:31:05 PDT
Comment 2 Jonathan Bedard 2021-04-16 07:31:04 PDT
I like this idea, but we should probably grow this as a totally separate script from test-webkitpy and then replace at some future date.

I also feel like, if we can get away with it, we should try to library-ize our purest integration. By that, I mean have a function in webkitcorepy that is basically "run all of the Python tests in these directories", and then pytest-webkitpy is basically a script that just calls that function with the directories it cares about. This approach helps us improve buildbot and other services tests as well, because they can use the exact same "run all of the Python tests in these directories" and just define different directories.
Comment 3 Sam Sneddon [:gsnedders] 2024-04-06 15:58:17 PDT
I've been toying with thoughts around this again recently; I think the right approach here in the short-term is to make webkitpy.test invoke pytest (as a library), rather than aiming for some better solution here.

This maintains most of the same downsides that we have today (we end up with a very strange sys.path!), but is a smaller change.