Bug 67177 – JavaScriptCore does not have a good mechanism for performance regression testing

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Filip Pizlo]

Performance testing in JSC involves running some benchmark suites in browser or on the command-line.  In-browser tests are irreplaceable, since they give us the official results.  But we often use command-line benchmark results as a way of rapidly determining if a change affects performance the way we want (either progression or no change).  Command-line testing is meant to be convenient, but it often isn't, for the followiong reasons:

1) It's necessary to run benchmark suites separately.  There is one script for SunSpider, two sets of scripts for V8, and a totally separate script for Kraken.

2) It's necessary to execute three commands to compare VMs: one command to run benchmarks on one VM, another command to run benchmarks on another VM, and a third command (which involves copy-pasting text from the results of the previous two commands) to compare results.

3) The VMs are run entirely separately with no interleaving.  A slight change in behavior of the system (for example a change in network behavior, or some schedules system job) could systematically affect the results in favor of one VM or another.  This can be avoided with interleaving the VMs at fine granularity.  That way, systematic changes in system behavior are less likely to systematically affect results and conclusions.

It would be ideal if we had a script to automatically compare two (or more) VMs on all benchmark suites at once, interleave VM execution, and print complete summary results.

[Filip Pizlo]

Created attachment 105571 [details]
the patch

[Darin Adler]

Comment on attachment 105571 [details]
the patch

View in context: https://bugs.webkit.org/attachment.cgi?id=105571&action=review

Seems OK to land this.

> Tools/Scripts/bencher:3
> +# Copyright (C) 2010 Apple Inc. All rights reserved.

It’s 2011

[Filip Pizlo]

Created attachment 105670 [details]
the patch (fix review)

[WebKit Review Bot]

Comment on attachment 105670 [details]
the patch (fix review)

Clearing flags on attachment: 105670

Committed r94103: <http://trac.webkit.org/changeset/94103>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.