Bug 64142 – test-webkitpy needs to support integration tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Dirk Pranke]

There is a class of tests in webkitpy that are "integration tests": they talk to real filesystems, require code to have been built, and start and stop servers. We have no way to automatically run and test them all.

I could have sworn I had either already implemented support for this in test-wekbitpy, or at least filed a bug, but I'm not seeing either now.

At any rate, there are several issues we should probably agree on:

1) What exactly is the difference between a unit test and an integration test? Is it that a unit test is very fast, or that the unit test does not require any system-dependent real resources, or that the test is tightly scoped to as few methods/modules as necessary?

For example, most of the tests in run_webkit_tests_integrationtest.py do not require real resources, and while they individually run relatively fast, there are a bunch of them. Perhaps more importantly, I'll call these "functional" tests rather than unit or integration tests, because they are testing what happens if you run NRWT with a particular set of command line options (but don't use real resources).

2) Can a single python file contain both integration and unit tests? I think the answer might want to be yes, because it's easy to imagine wanting to write tests that can run in both modes to get better coverage.

3) What should the test files be called? Currently we have _unittest.py for files that are (mostly?) unit tests, and _integrationtest.py for files that are (mostly?) integration tests.

4) Where do we want to run these tests? On the bots, and optionally interactively but not by default interactively?

I propose the following:

unit tests are platform-independent. integration tests are platform-dependent and may not function correctly if all of the dependencies aren't available.

Separately, we should separate our tests into "fast" and "slow", where we can run a "fast" set of tests that gets some basic, shallow coverage but may not be comprehensive. I would argue that any test that takes, say, more than 10ms to complete is "slow", and we can annotate those with a descriptor like @slow or something.

There should be a way to run either only unit tests, or unit + integration tests.

There should be an orthogonal way to run only the fast subset vs. the slow, comprehensive subset. The bots should run the slow set of tests, and all integration tests (although this may preclude running multiple test-webkitpys concurrently, I think it's important that at least some bots should cover this)

It is easy for me to write patches for whatever the desired combination of answers to the above are, but I haven't done so yet.

[Dirk Pranke]

Patch posted in bug 92925.

> At any rate, there are several issues we should probably agree on:
> 
> 1) What exactly is the difference between a unit test and an integration test? Is it that a unit test is very fast, or that the unit test does not require any system-dependent real resources, or that the test is tightly scoped to as few methods/modules as necessary?
>

For purposes of that patch, I will define "integration test" as a test that either affects the machine it is running on or is highly dependent on it (needs particular files checked out, writes to the filesystem, etc.) or may have issues with tests running concurrently in other processes (since we don't run tests concurrently in a single process).

This is independent of the test execution time (which may be slow but plenty of unit tests may be slower than plenty of integration tests), and, perhaps more controversially, independent of how much code is exercised by the test.

> 2) Can a single python file contain both integration and unit tests? 

Yes.

> 
> 3) What should the test files be called? Currently we have _unittest.py for files that are (mostly?) unit tests, and _integrationtest.py for files that are (mostly?) integration tests.
> 

Both will be supported.

> 4) Where do we want to run these tests? On the bots, and optionally interactively but not by default interactively?

We should aim for running them all by default both interactively and on the bots. Currently they are too slow and not stable enough to be on anywhere, but I will track that in separate bugs.

Conceivably we might want a way to mark tests as "slow" but we can cross that bridge when we come to it.