Bug 174201 - Speedometer 2.0: Use production builds where possible
Summary: Speedometer 2.0: Use production builds where possible
Status: RESOLVED WONTFIX
Alias: None
Product: WebKit
Classification: Unclassified
Component: JavaScriptCore (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on: 175278
Blocks: 172339
  Show dependency treegraph
 
Reported: 2017-07-06 06:15 PDT by Mathias Bynens
Modified: 2017-08-16 11:47 PDT (History)
8 users (show)

See Also:


Attachments
Patch (5.66 MB, patch)
2017-07-06 06:44 PDT, Mathias Bynens
no flags Details | Formatted Diff | Diff
Patch (5.66 MB, patch)
2017-07-06 06:49 PDT, Mathias Bynens
no flags Details | Formatted Diff | Diff
Patch (5.66 MB, patch)
2017-07-06 09:11 PDT, Mathias Bynens
rniwa: review-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Mathias Bynens 2017-07-06 06:15:14 PDT
Speedometer 2.0 should use production builds of frameworks and libraries whenever possible.
Comment 1 Mathias Bynens 2017-07-06 06:44:01 PDT
Created attachment 314720 [details]
Patch
Comment 2 Mathias Bynens 2017-07-06 06:49:05 PDT
Created attachment 314721 [details]
Patch
Comment 3 Mathias Bynens 2017-07-06 09:11:46 PDT
Created attachment 314725 [details]
Patch
Comment 4 Mathias Bynens 2017-07-06 09:18:48 PDT
Should we skip style checks on `PerformanceTests/Speedometer/resources/todomvc/`?
Comment 5 Addy Osmani 2017-07-06 12:23:20 PDT
> Speedometer 2.0 should use production builds of frameworks and libraries whenever possible.

We might want to file separate bugs for each app-specific change we're making here and treat 174201 as a meta-bug that is blocked on them landing. A similar approach was suggested by rniwa@ when we landed Speedometer 2 initially.

I'll defer to him for the call here however.

> Should we skip style checks on `PerformanceTests/Speedometer/resources/todomvc/`?

I believe rniwa@'s preference was the contents of Speedometer/resources matching the current style guide but will defer to him again for whether this is something that is possible. I applied a tab->space transform when working on the apps previously.
Comment 6 Ryosuke Niwa 2017-07-06 13:12:31 PDT
This patch is way too big to review. Please split per framework. I also don't agree with the premise that we should be always using production libraries.

For each library and framework, we need to carefully evaluate whether it makes sense to use the production version or not.

Remember, what we're trying to do is to measure the performance of browser, not production websites.
Comment 7 Ryosuke Niwa 2017-07-06 13:17:15 PDT
Comment on attachment 314725 [details]
Patch

r- because the patch is way too big to review, and we ned to understand the implication of changes to each libraries & framework separately.
Comment 8 Harald Kirschner 2017-07-12 16:31:20 PDT
(In reply to Ryosuke Niwa from comment #6)
> Remember, what we're trying to do is to measure the performance of browser,
> not production websites.

Addy asked me for feedback on this issue as I gave input the initial v2 changes and as the Spidermonkey team started looking at Speedometer v2.

I understand it is hard to clearly separate Speedometer from the frameworks it tests from the real world. Speedometer was announced [1] as "a benchmark that measures the responsiveness of real-world web applications". While best practices could differ between frameworks (I could not find any examples), the default premise should be that real-world web apps use production builds.

[1]: https://webkit.org/blog/3395/speedometer-benchmark-for-web-app-responsiveness/
Comment 9 Ryosuke Niwa 2017-07-16 18:56:41 PDT
(In reply to Harald Kirschner from comment #8)
> (In reply to Ryosuke Niwa from comment #6)
> > Remember, what we're trying to do is to measure the performance of browser,
> > not production websites.
> 
> Addy asked me for feedback on this issue as I gave input the initial v2
> changes and as the Spidermonkey team started looking at Speedometer v2.
> 
> I understand it is hard to clearly separate Speedometer from the frameworks
> it tests from the real world. Speedometer was announced [1] as "a benchmark
> that measures the responsiveness of real-world web applications". While best
> practices could differ between frameworks (I could not find any examples),
> the default premise should be that real-world web apps use production builds.

I wouldn't make such an assumption. When I first wrote Speedometer, many real world Ember apps were using debug builds. We really need to do a careful assessment of each framework & library separately.