| Summary: | SunSpider/string-tagcloud should be more lenient in validating its results, since they depend on Math.log, which isn't formally specified | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Filip Pizlo <fpizlo> | ||||
| Component: | Tools / Tests | Assignee: | Filip Pizlo <fpizlo> | ||||
| Status: | RESOLVED FIXED | ||||||
| Severity: | Normal | CC: | barraclough, commit-queue, ggaren, mark.lam, mhahnenberg, mjs, msaboff, oliver, rniwa, sam | ||||
| Priority: | P2 | Keywords: | InRadar | ||||
| Version: | 528+ (Nightly build) | ||||||
| Hardware: | All | ||||||
| OS: | All | ||||||
| Attachments: |
|
||||||
|
Description
Filip Pizlo
2013-04-19 19:04:46 PDT
Created attachment 198923 [details]
the patch
Comment on attachment 198923 [details] the patch View in context: https://bugs.webkit.org/attachment.cgi?id=198923&action=review r=me > PerformanceTests/SunSpider/tests/sunspider-1.0/string-tagcloud.js:266 > +var expectedMinLength = 315000; How did you come up with this being enough in all cases? (In reply to comment #2) > (From update of attachment 198923 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=198923&action=review > > r=me > > > PerformanceTests/SunSpider/tests/sunspider-1.0/string-tagcloud.js:266 > > +var expectedMinLength = 315000; > > How did you come up with this being enough in all cases? The differences I observed on the various platforms and browsers I tested were never greater than ~20. So I think this is much a much bigger tolerance than we need, based on empirical data. But also, this is a comparison of string length. The string length will only if Math.log returns a result that is in a different order of magnitude. So I think that it would be hard for someone to have a Math.log function that is so wrong that you get 241 fewer digits in the string generated by tagcloud, than you would get in the systems I tested. Finally, I'd be happy to make this even more tolerant, if you or others think it's a good idea. The point of this string compare at this point is just to force the JS engine to actually run the benchmark and generate the string - since you can only prove that the string wasn't shorter by actually running the thing. (Well I guess the browser could still spoof the benchmark, in which case we can't really do anything; or the browser could solve the much harder combinatoric problem of proving that the string must be at least that big without actually running the code, but if the browser can solve that problem faster than running the code then I suppose such a browser deserves to be given an epic victory.) Landed in http://trac.webkit.org/changeset/148796 |