There has been occasional talk about moving our Python unit testing from the stdlib unittest library to pytest, which we already have for the sake of WPT's WebDriver tests. pytest adds a variety of useful features (various ways to do test filtering, better debug integration, much better skip/xfail support, a de-facto standard parallel test-execution method (pytest-xdist), less boilerplate in tests), and avoids us having to reimplement much in our own wrapper around unittest. One open question is whether we can break the specific arguments used by ./Tools/Scripts/test-webkitpy{,-python2} today? Aside from that, there's a few more technical issues: * pytest's unittest support uses the stdlib unittest.TestLoader class, which we subclass in webkitpy.test.main, to optionally load methods beginning with serial_test_*, integration_test_*, and serial_integration_test_* as well as test_*; logically these would become marks in pytest * we have support for uploading test results (via --report) and our own JSON format (--json-output); we could either implement post-processing results from pytest or implement our own output format * pytest doesn't handle running tests in very different directories particularly well due to its rootdir and confdir handling (and for Source/WebKit/Scripts/webkit and Tools/Scripts/webkitpy the common root in the root of the WebKit repo, and we likely don't want to put any files there for pytest!); we could just have multiple invocations of pytest? * pytest-xdist doesn't support running a subset of tests sequentially (https://github.com/pytest-dev/pytest-xdist/issues/385), which we currently do for serial_* tests; this is another argument for multiple invocations of pytest
<rdar://problem/76735312>
I like this idea, but we should probably grow this as a totally separate script from test-webkitpy and then replace at some future date. I also feel like, if we can get away with it, we should try to library-ize our purest integration. By that, I mean have a function in webkitcorepy that is basically "run all of the Python tests in these directories", and then pytest-webkitpy is basically a script that just calls that function with the directories it cares about. This approach helps us improve buildbot and other services tests as well, because they can use the exact same "run all of the Python tests in these directories" and just define different directories.
I've been toying with thoughts around this again recently; I think the right approach here in the short-term is to make webkitpy.test invoke pytest (as a library), rather than aiming for some better solution here. This maintains most of the same downsides that we have today (we end up with a very strange sys.path!), but is a smaller change.