Bug 199054 – [perf.webkit.org] When calculating a ratio for the summary page, use a rolling window for last available data instead of only the last available data point

• Component: Tools / Tests
• Status: RESOLVED LATER

[Dean Johnson]

Right now when calculating a ratio for the summary page, we do the following:
current = "last 24 hours of non-baseline data" if any data exists in the last 24 hours, otherwise "last available data point"
baseline = "last 24 hours of baseline data" if any data exists in the last 24 hours, otherwise "last available data point"
ratio = (median(current) / median(baseline))

Since we set the "last 24 hours" window to the current time, and it can take hours between a commit landing, build products becoming available, and testing to occur, this 24 hour window will always have a margin that contains no data to evaluate. Additionally, using only a single data point when no data is available in the past 24 hours can lead to misrepresented results due to a single outlier run before a configuration stopped producing results.

I propose that we change the calculation to:
current = "last 24 hours of non-baseline data since the last available data point"
baseline = "last 24 hours of baseline data since the last available data point"
ratio = (median(current) / median(baseline))

Overall, I think this should improve the analysis of data when representing results on Summary pages, since it will take more data points into account.

[Dean Johnson]

<rdar://problem/51763999>

[Dean Johnson]

Created attachment 372534 [details]
Patch

[Dean Johnson]

Created attachment 372535 [details]
Patch

[Dean Johnson]

Uploaded patches are incorrect due to https://bugs.webkit.org/show_bug.cgi?id=199058.

Uploading new patch soon.

[Dean Johnson]

Created attachment 372541 [details]
Patch

[Dean Johnson]

Spoke with Ryosuke in-person about this. There's another downside I did not consider initially - this change would make it even more difficult to explain how perf-safari is doing its calculations for Summary pages / understand exactly how a number was calculated.

Because we're already making changes to the summary page calculations in https://bugs.webkit.org/show_bug.cgi?id=199050, we will hold off on this patch until we have stronger evidence something like it is needed. As it stands, I do not feel strongly about landing it.

[Ryosuke Niwa]

Resolving this as Later for now. We can reopen it once we decide this is useful.