Bug 137245 – It should be fun and easy to run every possible JavaScript benchmark from the command line

• Status: RESOLVED
• Resolution: FIXED
• Component: JavaScriptCore

[Filip Pizlo]

We previously had Tools/Scripts/bencher.  Then we stopped adding things to it because we weren't sure about the licensing of things like Kraken and Octane.  Various people ended up having their own private scripts for doing benchmark runs, and didn't share them in the open source community, because of fears about the shady licensing of the benchmarks suites that they were running.

We should fix this by having a script that just takes a configuration file with paths to the various benchmark scripts that cannot be safely checked into the WebKit repository.  It's up to the person running the benchmarks to download the benchmark suites from wherever they can find them.

[Filip Pizlo]

Created attachment 238908 [details]
the patch

[Saam Barati]

Comment on attachment 238908 [details]
the patch

View in context: https://bugs.webkit.org/attachment.cgi?id=238908&action=review

In initial testing, it seems to work exactly as expected. I'll be using it more extensively as I test the performance of my upcoming DFG patch and I'll let you know if I run into any problems.

Thanks :)

> Tools/Scripts/run-jsc-benchmarks:275
> +  puts "of the form Conf#<n> will be ascribed to the configuration automatically."

Maybe it's worth also adding in that each vm can be given a set of specific CLI options just for that vm, because I didn't know this until you mentioned it to me.

Maybe something like:
...
or of the string of the form <name>:<options>:<path> or <name>:<path>
where <path> is the path to the executable, <options> are the command line options you want passed into this specific vm (i.e, JSC_useJIT=0), and <name> is the name that you would like to give the configuration for the purpose of reporting.

[Filip Pizlo]

(In reply to comment #2)
> (From update of attachment 238908 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=238908&action=review
> 
> In initial testing, it seems to work exactly as expected. I'll be using it more extensively as I test the performance of my upcoming DFG patch and I'll let you know if I run into any problems.
> 
> Thanks :)
> 
> > Tools/Scripts/run-jsc-benchmarks:275
> > +  puts "of the form Conf#<n> will be ascribed to the configuration automatically."
> 
> Maybe it's worth also adding in that each vm can be given a set of specific CLI options just for that vm, because I didn't know this until you mentioned it to me.
> 
> Maybe something like:
> ...
> or of the string of the form <name>:<options>:<path> or <name>:<path>
> where <path> is the path to the executable, <options> are the command line options you want passed into this specific vm (i.e, JSC_useJIT=0), and <name> is the name that you would like to give the configuration for the purpose of reporting.

Yup, I added some text about this!

[Filip Pizlo]

Landed in http://trac.webkit.org/changeset/174123