Bug 131118 – Let's benchmark malloc

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Geoffrey Garen]

Let's benchmark malloc

[Geoffrey Garen]

Created attachment 228412 [details]
patch

[Geoffrey Garen]

Note: There's a binary file of opcodes that goes with the Facebook benchmark. I haven't uploaded it here because Bugzilla.

[WebKit Commit Bot]

Attachment 228412 [details] did not pass style-queue:


ERROR: PerformanceTests/MallocBench/MallocBench/big.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/big.cpp:27:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/big.cpp:33:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/big.cpp:42:  benchmark_big is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/tree.cpp:26:  Found header this file implements before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/tree.cpp:31:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/tree.cpp:178:  benchmark_tree_allocate is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/tree.cpp:193:  benchmark_tree_traverse is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/tree.cpp:208:  benchmark_tree_churn is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Interpreter.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Interpreter.cpp:27:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Interpreter.cpp:41:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/tree.h:29:  benchmark_tree_allocate is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/tree.h:30:  benchmark_tree_traverse is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/tree.h:31:  benchmark_tree_churn is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:27:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:32:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:113:  benchmark_message_one is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:126:  Missing space before {  [whitespace/braces] [5]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:133:  Missing space before {  [whitespace/braces] [5]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:136:  Missing space before {  [whitespace/braces] [5]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:141:  benchmark_message_many is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:158:  Missing space before {  [whitespace/braces] [5]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:168:  Missing space before {  [whitespace/braces] [5]
ERROR: PerformanceTests/MallocBench/MallocBench/message.cpp:172:  Missing space before {  [whitespace/braces] [5]
ERROR: PerformanceTests/MallocBench/MallocBench/medium.h:29:  benchmark_medium is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/list.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/list.cpp:27:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/list.cpp:31:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/list.cpp:99:  benchmark_list_allocate is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/list.cpp:127:  benchmark_list_traverse is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/medium.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/medium.cpp:27:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/medium.cpp:33:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/medium.cpp:42:  benchmark_medium is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/list.h:29:  benchmark_list_allocate is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/list.h:30:  benchmark_list_traverse is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Benchmark.cpp:26:  Found header this file implements before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Benchmark.cpp:38:  Streams are highly discouraged.  [readability/streams] [3]
ERROR: PerformanceTests/MallocBench/MallocBench/Benchmark.cpp:47:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Benchmark.cpp:145:  Missing space before {  [whitespace/braces] [5]
ERROR: PerformanceTests/MallocBench/MallocBench/Benchmark.cpp:212:  vm_info is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Benchmark.cpp:213:  vm_size is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/fragment.h:29:  benchmark_fragment is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/fragment.h:30:  benchmark_fragment_iterate is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/fragment.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/fragment.cpp:27:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/fragment.cpp:31:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/fragment.cpp:82:  benchmark_fragment is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/fragment.cpp:110:  benchmark_fragment_iterate is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/big.h:29:  benchmark_big is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/balloon.h:29:  benchmark_balloon is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/balloon.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/balloon.cpp:28:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/balloon.cpp:34:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/balloon.cpp:36:  benchmark_balloon is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/facebook.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/facebook.cpp:28:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/facebook.cpp:31:  "cstddef" already included at PerformanceTests/MallocBench/MallocBench/facebook.cpp:30  [build/include] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/facebook.cpp:43:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/facebook.cpp:45:  benchmark_facebook is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/CPUCount.cpp:26:  Found header this file implements before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/CPUCount.cpp:41:  When wrapping a line, only indent 4 spaces.  [whitespace/indent] [3]
ERROR: PerformanceTests/MallocBench/MallocBench/mbmalloc.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/realloc.h:29:  benchmark_realloc is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/realloc.cpp:26:  Found header this file implements before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/realloc.cpp:30:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/realloc.cpp:32:  benchmark_realloc is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/CommandLine.cpp:26:  Found header this file implements before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/CommandLine.cpp:28:  Streams are highly discouraged.  [readability/streams] [3]
ERROR: PerformanceTests/MallocBench/MallocBench/CommandLine.cpp:30:  CommandLine::long_options is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/CommandLine.cpp:46:  option_index is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/CommandLine.cpp:50:  This { should be at the end of the previous line  [whitespace/braces] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/main.cpp:28:  Streams are highly discouraged.  [readability/streams] [3]
ERROR: PerformanceTests/MallocBench/MallocBench/facebook.h:29:  benchmark_facebook is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Interpreter.h:39:  enum members should use InterCaps with an initial capital letter.  [readability/enum_casing] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Interpreter.h:40:  More than one command on the same line  [whitespace/newline] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/Interpreter.h:41:  More than one command on the same line  [whitespace/newline] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/message.h:29:  benchmark_message_one is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/message.h:30:  benchmark_message_many is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/churn.h:29:  benchmark_churn is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/churn.cpp:26:  Found other header before WebCore config.h. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/churn.cpp:27:  Found header this file implements after other header. Should be: config.h, primary header, blank line, and then alphabetically sorted.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/churn.cpp:31:  Bad include order. Mixing system and custom headers.  [build/include_order] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/churn.cpp:37:  Should be indented on a separate line, with the colon or comma first on that line.  [whitespace/indent] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/churn.cpp:42:  benchmark_churn is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: PerformanceTests/MallocBench/MallocBench/CommandLine.h:41:  long_options is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 88 in 34 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Mark Hahnenberg]

Comment on attachment 228412 [details]
patch

View in context: https://bugs.webkit.org/attachment.cgi?id=228412&action=review

r=me with questions

> PerformanceTests/MallocBench/MallocBench/CPUCount.cpp:34
> +size_t cpuCount()

Don't we have something in WTF already that computes how many cores you have?

> PerformanceTests/MallocBench/MallocBench/facebook.cpp:49
> +    Interpreter interpreter("facebook.ops");

Is there a way to record these binary files for arbitrary workloads?

> PerformanceTests/MallocBench/MallocBench/realloc.cpp:34
> +void benchmark_realloc(bool isParallel)
> +{
> +}

Is this supposed to do anything yet?

[Mark Hahnenberg]

(In reply to comment #4)
> (From update of attachment 228412 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=228412&action=review
> 
> r=me with questions
> 
> > PerformanceTests/MallocBench/MallocBench/CPUCount.cpp:34
> > +size_t cpuCount()
> 
> Don't we have something in WTF already that computes how many cores you have?
> 
> > PerformanceTests/MallocBench/MallocBench/facebook.cpp:49
> > +    Interpreter interpreter("facebook.ops");
> 
> Is there a way to record these binary files for arbitrary workloads?
> 
> > PerformanceTests/MallocBench/MallocBench/realloc.cpp:34
> > +void benchmark_realloc(bool isParallel)
> > +{
> > +}
> 
> Is this supposed to do anything yet?

Along with style fixes where appropriate.

[Geoffrey Garen]

Created attachment 228417 [details]
Example Output Comparing SystemMalloc vs NanoMalloc

[Mark Hahnenberg]

(In reply to comment #6)
> Created an attachment (id=228417) [details]
> Example Output Comparing SystemMalloc vs NanoMalloc

One stylistic thing about the output. It might look nicer if we right-justified the columns with numbers in them.

[Mark Hahnenberg]

(In reply to comment #6)
> Created an attachment (id=228417) [details]
> Example Output Comparing SystemMalloc vs NanoMalloc

Also, it would be nice if we could somehow factor out our confidence interval calculation stuff from bencher/run-jsc-benchmarks so that it could be reused in other scripts. Might be a good followup bug. I know I've wished for that before when writing new performance analysis tools.

[Geoffrey Garen]

> Don't we have something in WTF already that computes how many cores you have?

Yes: wtf/NumberOfCores.cpp.

I wanted to avoid linking against other WebKit stuff because I wanted something stand-alone that I could give to non-WebKit developers to measure outside the WebKit context.

> > PerformanceTests/MallocBench/MallocBench/facebook.cpp:49
> > +    Interpreter interpreter("facebook.ops");
> 
> Is there a way to record these binary files for arbitrary workloads?

Sadly, in a fit of silliness, I seem to have lost the FastMalloc patch that does recordings. For any malloc, though, it's pretty easy to produce a recording: Duplicate the "Op" struct, put each allocation+size+slot in a hash table, and dump each malloc/free Op to a binary file as you go.

> > PerformanceTests/MallocBench/MallocBench/realloc.cpp:34
> > +void benchmark_realloc(bool isParallel)
> > +{
> > +}
> 
> Is this supposed to do anything yet?

No -- I haven't filled it out yet, and the script doesn't run it :(.

[Geoffrey Garen]

> Along with style fixes where appropriate.

Looks like the main thing that needs fixing is order of system header includes. I'll fix that. 

The other cases seem to be the bot being a bot.

[Geoffrey Garen]

> One stylistic thing about the output. It might look nicer if we right-justified the columns with numbers in them.

Will do.

[Geoffrey Garen]

> Also, it would be nice if we could somehow factor out our confidence interval calculation stuff from bencher/run-jsc-benchmarks so that it could be reused in other scripts. Might be a good followup bug. I know I've wished for that before when writing new performance analysis tools.

Yeah. I spent about 10 minutes trying to do this when I started the project, and then I reached the limits of my Ruby. We should try again: <https://bugs.webkit.org/show_bug.cgi?id=131122>.

[Geoffrey Garen]

> Looks like the main thing that needs fixing is order of system header includes. I'll fix that. 

Actually, I think it's complaining about mbmalloc.h following other things. mbmalloc.h needs to be last because it undefs the malloc APIs to catch errors in benchmarks.

Another complaint is "benchmark_" -- I think that's a special case because it's a global convention for mapping strings to function names.

Another complaint is "^{" -- but that's correct for blocks.

Another complaint is long_options. Fixed.

[Geoffrey Garen]

Committed revision 166667.

[Ryosuke Niwa]

We need to add MallocBench to PerformanceTests/MallocBench since run-perf-tests can't parse results from this test.

[Ryosuke Niwa]

Oh, I guess we don't need to since we don't have any .html, .xhtml, and .svg files.