Bug 199054 - [perf.webkit.org] When calculating a ratio for the summary page, use a rolling window for last available data instead of only the last available data point
Summary: [perf.webkit.org] When calculating a ratio for the summary page, use a rollin...
Status: RESOLVED LATER
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P3 Normal
Assignee: Dean Johnson
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-06-19 19:00 PDT by Dean Johnson
Modified: 2019-06-20 14:45 PDT (History)
3 users (show)

See Also:


Attachments
Patch (3.69 KB, patch)
2019-06-19 22:56 PDT, Dean Johnson
no flags Details | Formatted Diff | Diff
Patch (24.10 KB, patch)
2019-06-19 22:56 PDT, Dean Johnson
no flags Details | Formatted Diff | Diff
Patch (2.72 KB, patch)
2019-06-19 23:11 PDT, Dean Johnson
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dean Johnson 2019-06-19 19:00:26 PDT
Right now when calculating a ratio for the summary page, we do the following:
current = "last 24 hours of non-baseline data" if any data exists in the last 24 hours, otherwise "last available data point"
baseline = "last 24 hours of baseline data" if any data exists in the last 24 hours, otherwise "last available data point"
ratio = (median(current) / median(baseline))

Since we set the "last 24 hours" window to the current time, and it can take hours between a commit landing, build products becoming available, and testing to occur, this 24 hour window will always have a margin that contains no data to evaluate. Additionally, using only a single data point when no data is available in the past 24 hours can lead to misrepresented results due to a single outlier run before a configuration stopped producing results.

I propose that we change the calculation to:
current = "last 24 hours of non-baseline data since the last available data point"
baseline = "last 24 hours of baseline data since the last available data point"
ratio = (median(current) / median(baseline))

Overall, I think this should improve the analysis of data when representing results on Summary pages, since it will take more data points into account.
Comment 1 Dean Johnson 2019-06-19 19:00:47 PDT
<rdar://problem/51763999>
Comment 2 Dean Johnson 2019-06-19 22:56:21 PDT
Created attachment 372534 [details]
Patch
Comment 3 Dean Johnson 2019-06-19 22:56:54 PDT
Created attachment 372535 [details]
Patch
Comment 4 Dean Johnson 2019-06-19 23:06:36 PDT
Uploaded patches are incorrect due to https://bugs.webkit.org/show_bug.cgi?id=199058.

Uploading new patch soon.
Comment 5 Dean Johnson 2019-06-19 23:11:32 PDT
Created attachment 372541 [details]
Patch
Comment 6 Dean Johnson 2019-06-19 23:15:18 PDT
Spoke with Ryosuke in-person about this. There's another downside I did not consider initially - this change would make it even more difficult to explain how perf-safari is doing its calculations for Summary pages / understand exactly how a number was calculated.

Because we're already making changes to the summary page calculations in https://bugs.webkit.org/show_bug.cgi?id=199050, we will hold off on this patch until we have stronger evidence something like it is needed. As it stands, I do not feel strongly about landing it.
Comment 7 Ryosuke Niwa 2019-06-20 14:45:47 PDT
Resolving this as Later for now. We can reopen it once we decide this is useful.