Bug 137245 - It should be fun and easy to run every possible JavaScript benchmark from the command line
Summary: It should be fun and easy to run every possible JavaScript benchmark from the...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: JavaScriptCore (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Filip Pizlo
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-09-29 21:11 PDT by Filip Pizlo
Modified: 2014-09-30 14:12 PDT (History)
16 users (show)

See Also:


Attachments
the patch (188.30 KB, patch)
2014-09-29 21:32 PDT, Filip Pizlo
oliver: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Filip Pizlo 2014-09-29 21:11:08 PDT
We previously had Tools/Scripts/bencher.  Then we stopped adding things to it because we weren't sure about the licensing of things like Kraken and Octane.  Various people ended up having their own private scripts for doing benchmark runs, and didn't share them in the open source community, because of fears about the shady licensing of the benchmarks suites that they were running.

We should fix this by having a script that just takes a configuration file with paths to the various benchmark scripts that cannot be safely checked into the WebKit repository.  It's up to the person running the benchmarks to download the benchmark suites from wherever they can find them.
Comment 1 Filip Pizlo 2014-09-29 21:32:16 PDT
Created attachment 238908 [details]
the patch
Comment 2 Saam Barati 2014-09-30 00:05:10 PDT
Comment on attachment 238908 [details]
the patch

View in context: https://bugs.webkit.org/attachment.cgi?id=238908&action=review

In initial testing, it seems to work exactly as expected. I'll be using it more extensively as I test the performance of my upcoming DFG patch and I'll let you know if I run into any problems.

Thanks :)

> Tools/Scripts/run-jsc-benchmarks:275
> +  puts "of the form Conf#<n> will be ascribed to the configuration automatically."

Maybe it's worth also adding in that each vm can be given a set of specific CLI options just for that vm, because I didn't know this until you mentioned it to me.

Maybe something like:
...
or of the string of the form <name>:<options>:<path> or <name>:<path>
where <path> is the path to the executable, <options> are the command line options you want passed into this specific vm (i.e, JSC_useJIT=0), and <name> is the name that you would like to give the configuration for the purpose of reporting.
Comment 3 Filip Pizlo 2014-09-30 14:10:53 PDT
(In reply to comment #2)
> (From update of attachment 238908 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=238908&action=review
> 
> In initial testing, it seems to work exactly as expected. I'll be using it more extensively as I test the performance of my upcoming DFG patch and I'll let you know if I run into any problems.
> 
> Thanks :)
> 
> > Tools/Scripts/run-jsc-benchmarks:275
> > +  puts "of the form Conf#<n> will be ascribed to the configuration automatically."
> 
> Maybe it's worth also adding in that each vm can be given a set of specific CLI options just for that vm, because I didn't know this until you mentioned it to me.
> 
> Maybe something like:
> ...
> or of the string of the form <name>:<options>:<path> or <name>:<path>
> where <path> is the path to the executable, <options> are the command line options you want passed into this specific vm (i.e, JSC_useJIT=0), and <name> is the name that you would like to give the configuration for the purpose of reporting.

Yup, I added some text about this!
Comment 4 Filip Pizlo 2014-09-30 14:12:59 PDT
Landed in http://trac.webkit.org/changeset/174123