Bug 160057 – Benchmark test content-animation timeouts always

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Carlos Alberto Lopez Perez]

The benchmark test content-animation added in https://trac.webkit.org/changeset/199805 timeouts.

The GTK+ performance bots is running all the benchmark tests plans, and this is one is constantly timing out. Check: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Perf%29 It uses the "--allplans" command line switch that runs all *.plan one after other.

I tried running it on the GTK+ platform:

$ Tools/Scripts/run-benchmark --platform gtk --browser minibrowser --plan content-animation

and it timeouts

I tried running it also on Mac platform:

$ Tools/Scripts/run-benchmark --platform osx --browser safari --plan content-animation

and it also timeouts.

... looking at the changelog of r199805 it says:

"""
When run as a benchmark, test completion involves a couple of bounces between the test and MobileSafari.
Test completion is initiated by a setTimeout() in the test, at which point location.hash is set to "#done".
MobileSafari detects that, and asynchronously dumps performance data. Once this is complete, MobileSafari
changes the URL has to "#submit", which triggers the XHR to the test relay that indicates the test is complete.
"""

So, if I understand things correctly, this tests depends on some black-magic that only happens on MobileSafari to work?

This is very unfortunate.

I think we should either:

Option 1) Fix this test so it can run with any browser.
Option 2) Add a way to skip test plans by platform/browser or make some tests to only run on some very specific platform/browser.

Opinions?

[Simon Fraser (smfr)]

I thought I skipped them but can't find that commit now.

[Simon Fraser (smfr)]

Ah, I did in https://trac.webkit.org/changeset/199906

[Carlos Alberto Lopez Perez]

(In reply to comment #2)
> Ah, I did in https://trac.webkit.org/changeset/199906

That is for performance tests (that are run with the Script Tools/Scripts/run-perf-tests

Here I talk about the benchmark test that is ran with the script Tools/Scripts/run-benchmark

The GTK perf test bot runs both kind of performance tests. It has 2 steps. First runs perf-tests and after that it runs benchmark-tests

[Carlos Alberto Lopez Perez]

Created attachment 285248 [details]
Patch

[Carlos Alberto Lopez Perez]

Comment on attachment 285248 [details]
Patch

Clearing flags on attachment: 285248

Committed r204087: <http://trac.webkit.org/changeset/204087>

[Carlos Alberto Lopez Perez]

All reviewed patches have been landed.  Closing bug.