Bug 60094 – fast/encoding/parser-tests often timeout on debug bots

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Dirk Pranke]

Despite the changes in bug 51721, the tests are still slow and frequently timeout.

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=fast%2Fencoding%2Fparser-tests.html%20fast%2Fencoding%2Fparser-tests-10.html%20fast%2Fencoding%2Fparser-tests-20.html%20fast%2Fencoding%2Fparser-tests-30.html%20fast%2Fencoding%2Fparser-tests-40.html%20fast%2Fencoding%2Fparser-tests-50.html%20fast%2Fencoding%2Fparser-tests-60.html%20fast%2Fencoding%2Fparser-tests-70.html%20fast%2Fencoding%2Fparser-tests-80.html%20fast%2Fencoding%2Fparser-tests-90.html%20fast%2Fencoding%2Fparser-tests-100.html%20fast%2Fencoding%2Fparser-tests-110.html%20fast%2Fencoding%2Fparser-tests-120.html%20fast%2Fencoding%2Fparser-tests-130.html%20&group=%40ToT%20-%20chromium.org

[Jenn Braithwaite]

I will halve the size of each batch of tests again.

[Jenn Braithwaite]

Created attachment 92351 [details]
patch

[Jenn Braithwaite]

Adam, could you review this as you reviewed the similar change in bug 51721 - this time I further halved each batch of tests.

[Adam Barth]

Comment on attachment 92351 [details]
patch

I don't really understand the details of what this patch is doing, but it looks fine to me.

[Jenn Braithwaite]

(In reply to comment #4)
> (From update of attachment 92351 [details])
> I don't really understand the details of what this patch is doing, but it looks fine to me.


Each batch used to run 10 tests, it now runs 5.  The script takes the first and last test, e.g. runtests(10, 19) is changed to runtests(10, 14).  A new test is added to invoke runtests(15, 19).  The expected results files were also updated so that the expected failures are in the expected results file for the correct batch.

[Dirk Pranke]

I'm sorry, the bug report wasn't clear (editing to fix).

It's not the batch size that's the problem. If you look at the numbers on the dashboard, you'll see that most of the time the batches run in a couple of seconds, and once in a while they'll take a really long time and time out.

We need to figure out why they're timing out, not speed up the average case.

[Jenn Braithwaite]

(In reply to comment #6)
> I'm sorry, the bug report wasn't clear (editing to fix).
> 
> It's not the batch size that's the problem. If you look at the numbers on the dashboard, you'll see that most of the time the batches run in a couple of seconds, and once in a while they'll take a really long time and time out.
> 
> We need to figure out why they're timing out, not speed up the average case.

Is there a way to tell from the logs whether the machine was under load when the timeouts occurred?  Reducing the batch sizes will reduce timeouts that occur due to the machine being under load, so it is still worth doing.

[Dirk Pranke]

(In reply to comment #7)
> (In reply to comment #6)
> > I'm sorry, the bug report wasn't clear (editing to fix).
> > 
> > It's not the batch size that's the problem. If you look at the numbers on the dashboard, you'll see that most of the time the batches run in a couple of seconds, and once in a while they'll take a really long time and time out.
> > 
> > We need to figure out why they're timing out, not speed up the average case.
> 
> Is there a way to tell from the logs whether the machine was under load when the timeouts occurred?  Reducing the batch sizes will reduce timeouts that occur due to the machine being under load, so it is still worth doing.

Not really, but because of the way new-run-webkit-tests works, you should assume the machine is under load all the time (because that's the design goal, to get through the tests as quickly as possible).

It would be interesting to see if the tests never timed out if the machine wasn't under load; there is a patch floating around to try that, but it hasn't landed yet.

[Dirk Pranke]

Sure. bug 59570 is the one to watch.

[Eric Seidel (no email)]

Do we know why these tests are slow?

[Jenn Braithwaite]

(In reply to comment #10)
> Do we know why these tests are slow?

I do not know why theses tests are occasionally slow.

[Dimitri Glazkov (Google)]

It's interesting, but these tests never failed on upstream Chromium bots: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20webkit.org&tests=fast%2Fencoding%2Fparser-tests

[Jenn Braithwaite]

*** Bug 64498 has been marked as a duplicate of this bug. ***

[Adam Barth]

Comment on attachment 92351 [details]
patch

Please re-nominate for review if this is still a problem.

[Dirk Pranke]

This has been resolved in Blink; it's not clear if this was ever an issue for non-Chromium ports given that they had a much longer timeout.