Bug 263260 – Move testharness.js tests to using a timeout_multiplier

• Status: NEW
• Component: Tools / Tests

[Sam Sneddon [:gsnedders]]

Currently we set the internal testharness.js timeout in our testharnessreport.js, totally overriding testharness.js's default behaviours:

  setTimeout(timeout, testRunner.timeout * 0.9);

For Apple ports (though see also bug 263252), this gives for non-slow tests in most configurations a 27s timeout, and for slow tests in most configurations a 135s timeout.

Upstream wptrunner uses a 10s timeout for most tests, or a 60s timeout for "long" (i.e., slow) tests. That said, this is configurable in a bunch of ways; debug Firefox for example uses a 3x multiplier and therefore ends up with 30s/180s timeouts. (Within WebKit, the glib port uses double the timeout for debug, which would mean 20s/120s timeouts, still both shorter than what we currently use.)

In general, we should probably also be setting timeout_multiplier rather than overriding the timeout directly: this allows tests to use test.step_timeout() which scales its timeout based on the multiplier, which hopefully saves us some flakiness.

There's two main approaches we could go with here:

One is to set it based on the current TestRunner timeout and the default WPT timeout (something like testRunner.timeout * 0.9 / wptTimeout would maintain our current behaviour, but would always result in >1.0 multipliers, which is probably not necessary so we might want a smaller constant than 0.9).

The other is to actually change how we deal with timeouts throughout run-webkit-tests (which probably isn't a bad idea in general!) to have a base timeout and then just repeatedly increase the multiplier, and pass both into the test runner.

[Radar WebKit Bug Importer]

<rdar://problem/117083423>