Bug 114895 - SunSpider/string-tagcloud should be more lenient in validating its results, since they depend on Math.log, which isn't formally specified
Summary: SunSpider/string-tagcloud should be more lenient in validating its results, s...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Filip Pizlo
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2013-04-19 19:04 PDT by Filip Pizlo
Modified: 2013-04-20 13:10 PDT (History)
10 users (show)

See Also:


Attachments
the patch (1.52 KB, patch)
2013-04-19 19:09 PDT, Filip Pizlo
msaboff: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Filip Pizlo 2013-04-19 19:04:46 PDT
Patch forthcoming.

<rdar://13699853>
Comment 1 Filip Pizlo 2013-04-19 19:09:24 PDT
Created attachment 198923 [details]
the patch
Comment 2 Michael Saboff 2013-04-20 06:50:18 PDT
Comment on attachment 198923 [details]
the patch

View in context: https://bugs.webkit.org/attachment.cgi?id=198923&action=review

r=me

> PerformanceTests/SunSpider/tests/sunspider-1.0/string-tagcloud.js:266
> +var expectedMinLength = 315000;

How did you come up with this being enough in all cases?
Comment 3 Filip Pizlo 2013-04-20 13:09:12 PDT
(In reply to comment #2)
> (From update of attachment 198923 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=198923&action=review
> 
> r=me
> 
> > PerformanceTests/SunSpider/tests/sunspider-1.0/string-tagcloud.js:266
> > +var expectedMinLength = 315000;
> 
> How did you come up with this being enough in all cases?

The differences I observed on the various platforms and browsers I tested were never greater than ~20.  So I think this is much a much bigger tolerance than we need, based on empirical data.

But also, this is a comparison of string length.  The string length will only if Math.log returns a result that is in a different order of magnitude.  So I think that it would be hard for someone to have a Math.log function that is so wrong that you get 241 fewer digits in the string generated by tagcloud, than you would get in the systems I tested.

Finally, I'd be happy to make this even more tolerant, if you or others think it's a good idea.  The point of this string compare at this point is just to force the JS engine to actually run the benchmark and generate the string - since you can only prove that the string wasn't shorter by actually running the thing.  (Well I guess the browser could still spoof the benchmark, in which case we can't really do anything; or the browser could solve the much harder combinatoric problem of proving that the string must be at least that big without actually running the code, but if the browser can solve that problem faster than running the code then I suppose such a browser deserves to be given an epic victory.)
Comment 4 Filip Pizlo 2013-04-20 13:10:28 PDT
Landed in http://trac.webkit.org/changeset/148796