Bug 223020 – Slower tests than [ Slow ] cannot be tested in consistent way

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[Kimmo Kinnunen]

Slower tests than [ Slow ] cannot be tested in consistent way

I would like to say the equivalent of: 
  
  webgl/2.0.0/deqp [ Pass Slow ]
  [ Debug ] webgl/2.0.0/deqp [ Timeout Pass Slow ]

Here I say:
 - I expect deqp tests pass, but they are slow
 - I expect deqp tests pass on debug, but they are sometimes slower than our slow timeout limit
 - I expect deqp tests very rarely to regress on a real timeout, so given what is implemented now in the test runner, I trade off detecting real timeout regressions with less flaky results
 - I don't want to show "this test is flaky" due to slow timeouts, and I don't want the tests to be re-run during a run, since reruns are pointlessly slowing down the testing and the flaky status is anyway just ignored

To which the run-webkit-test says:
  LayoutTests/webgl/TestExpectations:2 A test can not be both SLOW and TIMEOUT. If it times out indefinitely, then it should be just TIMEOUT. webgl/2.0.0/deqp


How can I say roughly what I intend?

Ideally I'd perhaps say:
  webgl/2.0.0/deqp [ Pass Slow ]
  [ Debug ] webgl/2.0.0/deqp [ Pass VerySlow ]

or

  webgl/2.0.0/deqp [ Pass VerySlow ]


Where VerySlow would perhaps be "Slow*2" or something such.

[Jonathan Bedard]

Is there any way to speed up these tests or maybe split them up? My concern here is that "Slow" already gives a test a timeout of minutes, which can have a meaningful impact on the time it takes CI to run things. If we're dealing with tests that are longer than that, it starts to cause issues with our sharing logic and efficiently using machine time.

[Alexey Proskuryakov]

I agree, even Slow tests are problematic, and overall test suite time has become excessive already.

[Sam Sneddon [:gsnedders]]

We already have different timeouts for debug and release, right?

[Kimmo Kinnunen]

(In reply to Jonathan Bedard from comment #1)
> Is there any way to speed up these tests or maybe split them up? My concern
> here is that "Slow" already gives a test a timeout of minutes, which can
> have a meaningful impact on the time it takes CI to run things.
>(In reply to Alexey Proskuryakov from comment #2)
> I agree, even Slow tests are problematic, and overall test suite time has
> become excessive already.

I don't think these comments are productive.

I'm looking for a technical solution for an actual technical problem that exist today and is unaddressed.

I'm not looking to expand the test suite time. I'm looking to shrink the time.

What is in the test suite is a policy decision. I'm not looking to discuss this.

So let me rephrase:
Currently random amount of tests timeout because they're slower than the slow timeout.
This cause the tests to be re-run at the end, sequentially.
This causes longer test runs and flaky results.
This is an unilateral problem for development as well as bot infrastructure.

I'm looking for a solution for this.

One solution to this is that these could be tagged as VerySlow, increasing their timeout. Increasing these tests timeout would not cause any further increase of test suite time, because the timeouts are due to slowness, not hangs. Since the increased time would be executed in parallel instead of the extra sequential runs in the end, we can increase the timeout up to Slow*2 and probably not see any increase in runtime. If there's a problem wrt sequencing, the deqp bot and the developers can be instructed to use --order=random.

Other, only partial, solution is to allow [ Timeout Slow ] as the result. 

There may be other solutions, what do you think?

Note, for these test being discussed, deqp tests, the tests are not even run on main bots or anybody in particular. They're run by (probably mostly only) *me* and the deqp bot.

The tests are already split quite a lot. Spliting them more is quite an effort, as they're part of the standardisation effort. Splitting tests do not make the test run faster.

The tests are slow, because they (for the most part) simulate GPU pixel rasterisation in JavaScript. So they execute a lot of JavaScript in Debug process.

There is a related slowness problem for the main WebGL tests. However, in this bug I'd like to address the slowness and timeouts of deqp tests.

[Jonathan Bedard]

(In reply to Kimmo Kinnunen from comment #4)
> ...

This context makes more sense. I guess the follow-up question is should such tests ever actually timeout? If you run them with --no-timeout, how long do they take? Originally, "Slow" was supposed to be a catch-all for this sort of thing...I guess we could start adding on "Very" to crank up the timeout.

[Alexey Proskuryakov]

What is important is that we do not run tests that are slower than Slow on the bots. Any tests that are flakily slower than Slow must be skipped or deleted, there is indeed nothing to discuss here.

When you run additional tests locally, just pass a --timeout that makes them work for you.