Bug 114852 – Whenever it is cheap and non-invasive, SunSpider tests should validate their results to ensure that the browser runs them correctly

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Filip Pizlo]

Patch forthcoming.

[Filip Pizlo]

Created attachment 198802 [details]
work in progress

Not yet ready for review.

[Filip Pizlo]

Example output when a browser fails tests (in this case FROM is Chrome, which apparently does math differently than everyone else):

TEST                   COMPARISON               FROM                 TO             DETAILS

===============================================================================

** TOTAL **:           ??                      ERROR        134.8ms +/- 1.2%      invalid runs detected

===============================================================================

  3d:                  ??                      ERROR         19.7ms +/- 4.5%      invalid runs detected
    cube:              1.56x as fast     10.9ms +/- 36.8%     7.0ms +/- 11.8%     significant
    morph:             ??                      ERROR          5.6ms +/- 8.9%      invalid runs detected
    raytrace:          1.32x as fast      9.4ms +/- 8.2%      7.1ms +/- 3.2%      significant

  access:              1.59x as fast     19.1ms +/- 23.4%    12.0ms +/- 4.9%      significant
    binary-trees:      -                  2.3ms +/- 106.7%     1.1ms +/- 20.5% 
    fannkuch:          1.70x as fast      8.5ms +/- 18.7%     5.0ms +/- 0.0%      significant
    nbody:             -                  5.3ms +/- 70.2%     3.0ms +/- 11.2% 
    nsieve:            -                  3.0ms +/- 50.2%     2.9ms +/- 14.0% 

  bitops:              1.96x as fast     14.5ms +/- 24.6%     7.4ms +/- 6.8%      significant
    3bit-bits-in-byte: ??                 0.9ms +/- 25.1%     1.1ms +/- 20.5%     not conclusive: might be *1.22x as slow*
    bits-in-byte:      2.00x as fast      3.6ms +/- 13.9%     1.8ms +/- 16.7%     significant
    bitwise-and:       1.50x as fast      2.4ms +/- 15.4%     1.6ms +/- 23.1%     significant
    nsieve-bits:       2.62x as fast      7.6ms +/- 49.2%     2.9ms +/- 14.0%     significant

  controlflow:         -                  1.7ms +/- 20.3%     1.7ms +/- 20.3% 
    recursive:         -                  1.7ms +/- 20.3%     1.7ms +/- 20.3% 

  crypto:              1.46x as fast     15.0ms +/- 7.8%     10.3ms +/- 4.7%      significant
    aes:               -                  5.7ms +/- 13.3%     5.2ms +/- 5.8%  
    md5:               1.48x as fast      4.3ms +/- 17.6%     2.9ms +/- 7.8%      significant
    sha1:              2.27x as fast      5.0ms +/- 21.3%     2.2ms +/- 13.7%     significant

  date:                -                 22.0ms +/- 24.9%    19.0ms +/- 5.0%  
    format-tofte:      ??                 9.6ms +/- 35.0%    10.8ms +/- 2.8%      not conclusive: might be *1.125x as slow*
    format-xparb:      1.51x as fast     12.4ms +/- 33.5%     8.2ms +/- 10.7%     significant

  math:                ??                      ERROR         10.8ms +/- 5.2%      invalid runs detected
    cordic:            -                  3.2ms +/- 17.6%     2.7ms +/- 17.9% 
    partial-sums:      ??                      ERROR          6.1ms +/- 3.7%      invalid runs detected
    spectral-norm:     -                  3.3ms +/- 63.0%     2.0ms +/- 0.0%  

  regexp:              ??                 6.2ms +/- 4.9%      6.4ms +/- 5.8%      not conclusive: might be *1.032x as slow*
    dna:               ??                 6.2ms +/- 4.9%      6.4ms +/- 5.8%      not conclusive: might be *1.032x as slow*

  string:              1.175x as fast    55.8ms +/- 9.9%     47.5ms +/- 1.1%      significant
    base64:            ??                 3.4ms +/- 14.7%     3.6ms +/- 10.3%     not conclusive: might be *1.059x as slow*
    fasta:             *1.91x as slow*    5.3ms +/- 12.8%    10.1ms +/- 2.2%      significant
    tagcloud:          1.74x as fast     16.9ms +/- 6.1%      9.7ms +/- 3.6%      significant
    unpack-code:       -                 21.8ms +/- 13.1%    19.2ms +/- 3.8%  
    validate-input:    -                  8.4ms +/- 46.8%     4.9ms +/- 4.6%

[Filip Pizlo]

Example output in analyze mode (again this is Chrome):


============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                   ERROR: Some tests failed.
--------------------------------------------

  3d:                    ERROR: Some tests failed.
    cube:               10.9ms +/- 36.8%
    morph:               ERROR: Invalid test run.
    raytrace:            9.4ms +/- 8.2%

  access:               19.1ms +/- 23.4%
    binary-trees:        2.3ms +/- 106.7%
    fannkuch:            8.5ms +/- 18.7%
    nbody:               5.3ms +/- 70.2%
    nsieve:              3.0ms +/- 50.2%

  bitops:               14.5ms +/- 24.6%
    3bit-bits-in-byte:   0.9ms +/- 25.1%
    bits-in-byte:        3.6ms +/- 13.9%
    bitwise-and:         2.4ms +/- 15.4%
    nsieve-bits:         7.6ms +/- 49.2%

  controlflow:           1.7ms +/- 20.3%
    recursive:           1.7ms +/- 20.3%

  crypto:               15.0ms +/- 7.8%
    aes:                 5.7ms +/- 13.3%
    md5:                 4.3ms +/- 17.6%
    sha1:                5.0ms +/- 21.3%

  date:                 22.0ms +/- 24.9%
    format-tofte:        9.6ms +/- 35.0%
    format-xparb:       12.4ms +/- 33.5%

  math:                  ERROR: Some tests failed.
    cordic:              3.2ms +/- 17.6%
    partial-sums:        ERROR: Invalid test run.
    spectral-norm:       3.3ms +/- 63.0%

  regexp:                6.2ms +/- 4.9%
    dna:                 6.2ms +/- 4.9%

  string:               55.8ms +/- 9.9%
    base64:              3.4ms +/- 14.7%
    fasta:               5.3ms +/- 12.8%
    tagcloud:           16.9ms +/- 6.1%
    unpack-code:        21.8ms +/- 13.1%
    validate-input:      8.4ms +/- 46.8%

[Filip Pizlo]

Created attachment 198803 [details]
the patch

[Filip Pizlo]

Comment on attachment 198803 [details]
the patch

Clearing r? while I investigate more, whether the expected results are correct.

[Filip Pizlo]

Actually, I think both Safari/Firefox and Chrome were both "right" in their different results - ECMAScript does not specify what pow(), sin(), or cos() return. :-/

[Filip Pizlo]

Created attachment 198812 [details]
the patch

[Oliver Hunt]

(In reply to comment #6)
> Actually, I think both Safari/Firefox and Chrome were both "right" in their different results - ECMAScript does not specify what pow(), sin(), or cos() return. :-/

O_o

Are firefox and chrome consistent with each other?

[Filip Pizlo]

(In reply to comment #8)
> (In reply to comment #6)
> > Actually, I think both Safari/Firefox and Chrome were both "right" in their different results - ECMAScript does not specify what pow(), sin(), or cos() return. :-/
> 
> O_o
> 
> Are firefox and chrome consistent with each other?

No.

Firefox is consistent with Safari.  Chrome differs.

[Geoffrey Garen]

Comment on attachment 198812 [details]
the patch

View in context: https://bugs.webkit.org/attachment.cgi?id=198812&action=review

r=me

> PerformanceTests/SunSpider/tests/sunspider-1.0/3d-morph.js:56
> +// This has to be an approximate test since ECMAscript doesn't 

Missed the end of the sentence here.

[Filip Pizlo]

(In reply to comment #10)
> (From update of attachment 198812 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=198812&action=review
> 
> r=me
> 
> > PerformanceTests/SunSpider/tests/sunspider-1.0/3d-morph.js:56
> > +// This has to be an approximate test since ECMAscript doesn't 
> 
> Missed the end of the sentence here.

Fixed.

[Filip Pizlo]

Landed in http://trac.webkit.org/changeset/148784