Bug 257393
| Summary: | Speedometer's flight test has timeouts, which lead to non-determinism | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Scott Violet <sky> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | RESOLVED CONFIGURATION CHANGED | ||
| Severity: | Normal | CC: | ap, mark.lam, rniwa, ysuzuki |
| Priority: | P2 | ||
| Version: | Other | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
Scott Violet
See https://github.com/WebKit/WebKit/blob/main/PerformanceTests/Speedometer2.1/resources/todomvc/dependency-examples/flight/flight/app/js/main.js#L27 . The test does 6 timeouts, which can lead to non-determinism. If the test needs to wait until something is loaded, that should not be included in test time.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Ryosuke Niwa
Speedometer 3 benchmark, which is being developed at https://github.com/WebKit/Speedometer/pull/144 does not have this workload.
Scott Violet
Isn't that ~6 months out? Seems like it would be good to fix this before the next version of Speedometer.
Ryosuke Niwa
(In reply to Scott Violet from comment #2)
> Isn't that ~6 months out? Seems like it would be good to fix this before the
> next version of Speedometer.
This workload is disabled by default anyway so it won't affect the official benchmark result.
Ryosuke Niwa
(In reply to Ryosuke Niwa from comment #3)
> (In reply to Scott Violet from comment #2)
> > Isn't that ~6 months out? Seems like it would be good to fix this before the
> > next version of Speedometer.
>
> This workload is disabled by default anyway so it won't affect the official
> benchmark result.
Actually, that's not true. This is about FlightJS's TodoMVC, which is included in the official result. Regardless, we're not going to fix this for 2.1.