Bug 78289 – META: speed up tests in webkitpy and/or annotate them as 'slow' so we can skip them

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Dirk Pranke]

Per discussion in bug 76022, we have a number of "slow" tests that are run during test-webkitpy. This bug analyzes where they are and what to do about them.

Roughly speaking, we should fix the really slow ones as much as possible, and perhaps annotate the others with 'slow' (for some definition of slow) so that we can do a 'fast' run that skips them.

Building on the newly-enhanced test-webkitpy (rollup patch for it in bug 77687; not ready for review) and the attached script to bucket the output of -t  (test-webkitpy -t 2>&1 | egrep 'passed|failed' | egrep -v 'all passed' | awk '{ printf("%s %f", $1, $4); }' | buckets) , we get the following ouput:

For what it's worth, here's the output of said script:

bucket             count ccount %ccount  time  ctime %time %ctime
0.00001 -  0.00003    80     80  0.06    0.00   0.00  0.00   0.00
0.00003 -  0.00010   131    211  0.16    0.01   0.01  0.00   0.00
0.00010 -  0.00032   227    438  0.34    0.04   0.05  0.00   0.00
0.00032 -  0.00100   175    613  0.47    0.10   0.15  0.00   0.00
0.00100 -  0.00316   205    818  0.63    0.36   0.51  0.01   0.01
0.00316 -  0.01000    83    901  0.70    0.45   0.96  0.01   0.02
0.01000 -  0.03162   219   1120  0.87    4.16   5.11  0.07   0.09
0.03162 -  0.10000    88   1208  0.93    4.69   9.81  0.08   0.17
0.10000 -  0.31623    48   1256  0.97    8.71  18.51  0.15   0.32
0.31623 -  1.00000    18   1274  0.99   10.33  28.85  0.18   0.50
1.00000 -  3.16228    17   1291  1.00   24.87  53.72  0.43   0.94
3.16228 - 10.00000     1   1292  1.00    3.63  57.35  0.06   1.00

legend:
bucket: time it took to run the test in seconds
count: # of tests in the bucket
ccount: cumulative # of tests
%ccount: cumulative % of tests in this plus the preceding buckets
time: total time of all tests in this bucket
ctime: cumulative time in this plus the preceding buckets
%time: time in this bucket as a %age of total execution time
%ctime: cumulative time of tests

So, we see, for example, that 87% of our tests complete in under 32 milliseconds each (five seconds total), and 18 tests (~1%) account for half of the total test run time, with another 18 accounting for another sixth.

It seems like a reasonable definition of a 'slow' test would therefore be one that takes more than somewhere between 32-320ms, depending on how aggressive we wanted to be. I'd probably fall closer to the 32ms end.

[Dirk Pranke]

marking as blocking bug 76022 for the 'slow' decorator part of this.

[Dirk Pranke]

Note that this includes all of the tests run by test-webkitpy by default, so it includes the tests in run_webkit_tests_integrationtest, but does not include scm_unittest or the integration_test_* routines in webkitpy.layout_tests.*

[Eric Seidel (no email)]

Can we have a list of the slow tests?

[Dirk Pranke]

Created attachment 126395 [details]
test times for webkitpy sorted by execution time as of r107291

[Dirk Pranke]

(In reply to comment #3)
> Can we have a list of the slow tests?

Surely. attached.

[Dirk Pranke]

I think most tests are fast enough now. Closing. Maybe at some point I'll get around to adding the @slow decorator once my other test-webkitpy changes have landed.