Bug 155551 – Pathological performance on first execution of a function called with mismatched argument count

• Status: NEW
• Component: JavaScriptCore

[Oliver Hunt]

So, i have a case of strange performance if i don’t pass an argument
function f(j) { for (var i = 0 ; i < 10000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(1); var end = new Date; print("1st: " + (end - start) + "ms"); var start = new Date; f(1); var end = new Date; print("2nd: " + (end - start) + "ms")
1st: 245ms
2nd: 223ms

If we do it again, without passing a parameter we get:
function f(j) { for (var i = 0 ; i < 10000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; 
var start = new Date; f(); var end = new Date; print("1st: " + (end - start) + "ms"); 
var start = new Date; f(); var end = new Date; print("2nd: " + (end - start) + "ms")

1st: 11919ms
2nd: 29ms

I tried a couple of permutations, and in general it appears the first run through without passing a parameter never tiers up.

[Oliver Hunt]

Let's try again with the same function in both
function f(j) { for (var i = 0 ; i < 10000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(1); var end = new Date; print("1st: " + (end - start) + "ms"); var start = new Date; f(1); var end = new Date; print("2nd: " + (end - start) + "ms")
1st: 30
2nd: 23ms

If we do it again, without passing a parameter we get:
function f(j) { for (var i = 0 ; i < 10000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; 
var start = new Date; f(); var end = new Date; print("1st: " + (end - start) + "ms"); 
var start = new Date; f(); var end = new Date; print("2nd: " + (end - start) + "ms")

1st: 11919ms
2nd: 29ms

[Oliver Hunt]

Ok, so passing extra is still penalized, but nowhere near the same extent.

Also, if we call with extra arguments initially the no arguments case remains much more expensive.

Derp, timings in initial report were in a debug build, but that just makes the failure more apparent. Below timings are in a trunk release build and they're still showing a 4x hit.

A more curious failure is 

function f(j) { for (var i = 0 ; i < 10000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; 
var start = new Date; f(1,2); var end = new Date; print("1st: " + (end - start) + "ms"); 
var start = new Date; f(); var end = new Date; print("2nd: " + (end - start) + "ms"); 
var start = new Date; f(); var end = new Date; print("3rd: " + (end - start) + "ms"); 
var start = new Date; f(); var end = new Date; print("4th: " + (end - start) + "ms"); 
var start = new Date; f(1,2); var end = new Date; print("5th: " + (end - start) + "ms"); 
var start = new Date; f(1); var end = new Date; print("6th: " + (end - start) + "ms")


Produces 

1st: 79ms
2nd: 410ms
3rd: 414ms
4th: 411ms
5th: 121ms
6th: 111ms

Implying hard failure after the initial excess argument compilation.

[Filip Pizlo]

(In reply to comment #2)
> Ok, so passing extra is still penalized, but nowhere near the same extent.
> 
> Also, if we call with extra arguments initially the no arguments case
> remains much more expensive.
> 
> Derp, timings in initial report were in a debug build, but that just makes
> the failure more apparent. Below timings are in a trunk release build and
> they're still showing a 4x hit.
> 
> A more curious failure is 
> 
> function f(j) { for (var i = 0 ; i < 10000000; i++) j = (j ?  j * i : 1) ^
> (i & j); }; 
> var start = new Date; f(1,2); var end = new Date; print("1st: " + (end -
> start) + "ms"); 
> var start = new Date; f(); var end = new Date; print("2nd: " + (end - start)
> + "ms"); 
> var start = new Date; f(); var end = new Date; print("3rd: " + (end - start)
> + "ms"); 
> var start = new Date; f(); var end = new Date; print("4th: " + (end - start)
> + "ms"); 
> var start = new Date; f(1,2); var end = new Date; print("5th: " + (end -
> start) + "ms"); 
> var start = new Date; f(1); var end = new Date; print("6th: " + (end -
> start) + "ms")
> 
> 
> Produces 
> 
> 1st: 79ms
> 2nd: 410ms
> 3rd: 414ms
> 4th: 411ms
> 5th: 121ms
> 6th: 111ms
> 
> Implying hard failure after the initial excess argument compilation.

I'm not sure I see a bug here.

This probably has nothing to do with the number of passed arguments.

In the first call to f() you pass an integer for j.

In the second, third, and fourth calls you pass undefined for j.  Let's make sure we're clear on this.  This most likely has nothing to do with the number of arguments passed.  Passing undefined directly in those calls will likely have the same results.

The fifth and sixth calls to f() pass integers again.  This will now run slower than the first time, since the second, third, and fourth calls probably caused us to recompile f() with j being inferred to be something other than integer.  So, the fifth and sixth calls run at a decent speed (because integers) but not as fast as they could (because type inference stopped assuming that j will be an integer).

[Oliver Hunt]

Here's the case that originally struct me as wrong:

function f(j) { for (var i = 0 ; i < 10000000; i++) j = (j ?  j * i : 1) ^ (i & j); };

var start = new Date; f(); var end = new Date; print("2nd: " + (end - start) + "ms");
var start = new Date; f(); var end = new Date; print("3rd: " + (end - start) + "ms");

First run is 433ms, second run is 137ms.

[Oliver Hunt]

(In reply to comment #4)
> Here's the case that originally struct me as wrong:
> 
> function f(j) { for (var i = 0 ; i < 10000000; i++) j = (j ?  j * i : 1) ^
> (i & j); };
> 
> var start = new Date; f(); var end = new Date; print("2nd: " + (end - start)
> + "ms");
> var start = new Date; f(); var end = new Date; print("3rd: " + (end - start)
> + "ms");
> 
> First run is 433ms, second run is 137ms.

Essentially we're spending 300ms longer in the first run than the second. To me that implies that the first run doesn't get to jump into the optimized path. This is further reinforced if i increase the iteration count:

function f(j) { for (var i = 0 ; i < 500000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(); var end = new Date; print("1st: " + (end - start) + "ms"); var start = new Date; f(); var end = new Date; print("2nd: " + (end - start) + "ms");

Producing
1st: 18392ms
2nd: 4166ms

The scaling of runtime implies that the first run is not tiering up in any meaningful way, but we are producing code that is optimised: we hit it the second time we run the function.

[Oliver Hunt]

Oooh, had a thought and tried passing undefined explicitly:
function f(j) { for (var i = 0 ; i < 500000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(undefined); var end = new Date; print("1st: " + (end - start) + "ms"); var start = new Date; f(); var end = new Date; print("2nd: " + (end - start) + "ms");
1st: 4313ms
2nd: 4305ms

Note that there isn't a performance difference here.

The behaviour i'm seeing is specific to the 0 parameter call.

[Oliver Hunt]

If i make the function take 0 parameters and use arguments object there's no perf delta either:
function f() { var j = arguments[0]; for (var i = 0 ; i < 500000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(1); var end = new Date; print("1st: " + (end - start) + "ms"); var start = new Date; f(); var end = new Date; print("2nd: " + (end - start) + "ms");
1st: 3915ms
2nd: 4081ms

This behaviour is only occurring if i pass zero parameters in the first call to the function.

[Filip Pizlo]

Sorry, I still don't see anything actionable here.

[Oliver Hunt]

Actually, i managing to trigger a bunch of performance cliffs. if we change f(j) to f(j,k) performance falls off a cliff:
function f(j) { for (var i = 0 ; i < 50000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(1); var end = new Date; print("1st: " + (end - start) + "ms");var start = new Date; f(1); var end = new Date; print("2nd: " + (end - start) + "ms")

1st: 375ms

vs.

function f(j,k) { for (var i = 0 ; i < 50000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(1); var end = new Date; print("1st: " + (end - start) + "ms"); 
1st: 1885ms


There is clearly some kind of perf cliff going on. At least for the first call specifically, and varying arty mismatches in general.

[Oliver Hunt]

(In reply to comment #8)
> Sorry, I still don't see anything actionable here.

Really? If i have a function f(i), and then do f();f(); it's reasonable for the first call to f to take vastly longer than the second? but if i do f(undefined);f(undefined); that won't happen.

[Oliver Hunt]

(In reply to comment #10)
> (In reply to comment #8)
> > Sorry, I still don't see anything actionable here.
> 
> Really? If i have a function f(i), and then do f();f(); it's reasonable for
> the first call to f to take vastly longer than the second? but if i do
> f(undefined);f(undefined); that won't happen.


I mean seriously, it seems unreasonable to get this behaviour:

function f(j) { for (var i = 0 ; i < 50000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(); var end = new Date; print("1st: " + (end - start) + "ms");var start = new Date; f(); var end = new Date; print("2nd: " + (end - start) + "ms")
1st: 1793ms
2nd: 487ms


vs. this behaviour:
function f(j) { for (var i = 0 ; i < 50000000; i++) j = (j ?  j * i : 1) ^ (i & j); }; var start = new Date; f(undefined); var end = new Date; print("1st: " + (end - start) + "ms");var start = new Date; f(undefined); var end = new Date; print("2nd: " + (end - start) + "ms")
1st: 439ms
2nd: 417ms

[Filip Pizlo]

We don't usually spend a lot of time tuning for microbenchmarks, because such work doesn't end up benefiting real-world code.  We only tune for microbenchmarks if the microbenchmark is carefully designed to emulate a VM pathology that occurred in real code.

I don't think that trying to optimize this tiny integer loop for cases where you didn't pass integers is going to have a high likelihood of helping users.  I'd be more inclined to look at this if it was a bigger program.

[David Kilzer (:ddkilzer)]

<rdar://problem/32005732>