Bug 170389 - JetStream bug: some +- deviations are greater than the average
Summary: JetStream bug: some +- deviations are greater than the average
Status: RESOLVED INVALID
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: All All
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-04-02 08:32 PDT by Daniel Gutson
Modified: 2017-04-02 11:12 PDT (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Gutson 2017-04-02 08:32:28 PDT
After running JetStream on Chrome 56.0.2924.87 / Ubuntu 16.10 (Linux xxx 4.8.0-45-generic #48-Ubuntu SMP Fri Mar 24 11:46:39 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux) some deviations are greater than the average:
mandreel-latency	84.26 ± 105.6
mandreel	115.3 ± 142.2
bigfib.cpp	241.7 ± 253.8
box2d	107.2 ± 186.0
splay	121.1 ± 136.3


Just in case, here are all the results:

Score
122.58
± 41.822


Benchmark	Average Score	Benchmark	Average Score	Benchmark	Average Score
Latency	75.410 ± 17.469	regex-dna	128.7 ± 4.499	gcc-loops.cpp	351.2 ± 71.42
3d-cube	59.11 ± 6.473	splay-latency	255.9 ± 129.7	hash-map	119.6 ± 66.36
3d-raytrace	70.21 ± 1.745	tagcloud	65.74 ± 12.12	mandreel	115.3 ± 142.2
base64	69.42 ± 1.424	typescript	51.16 ± 51.08	n-body.c	199.0 ± 119.1
cdjs	63.98 ± 3.945	Throughput	178.59 ± 76.212	navier-stokes	130.5 ± 89.12
code-first-load	62.32 ± 7.358	bigfib.cpp	241.7 ± 253.8	pdfjs	108.6 ± 80.48
code-multi-load	52.85 ± 35.35	box2d	107.2 ± 186.0	proto-raytracer	167.0 ± 47.36
crypto-aes	114.5 ± 7.219	container.cpp	200.5 ± 158.1	quicksort.c	236.1 ± 58.82
crypto-md5	72.58 ± 1.264	crypto	116.1 ± 44.74	regexp-2010	267.4 ± 72.76
crypto-sha1	50.00 ± 1.514	delta-blue	296.4 ± 43.91	richards	167.7 ± 28.27
date-format-tofte	53.28 ± 5.979	dry.c	326.7 ± 49.59	splay	121.1 ± 136.3
date-format-xparb	58.33 ± 10.15	earley-boyer	113.1 ± 4.630	towers.c	159.4 ± 23.60
mandreel-latency	84.26 ± 105.6	float-mm.c	337.9 ± 14.01	zlib	242.5 ± 22.73
n-body	131.5 ± 11.04	gbemu	224.5 ± 18.94	Geometric Mean	122.58 ± 41.822


I think that some calculation is wrong, potentially turning the final score doubtful.
Comment 1 Filip Pizlo 2017-04-02 11:12:05 PDT
Since these are confidence intervals computed using 3 samples, you can expect some individual scores to have intervals that are larger than the average. To be clear, your complaint would be valid if we were reporting stddev. We're not. We're reporting 95% confints. 

Without specific evidence suggesting otherwise, I'm gonna say your machine is just noisy. More likely, Chrome just got super noisy with the ignition/turbofan kerfuffle.