Bug 76504 - run-perf-tests should generate a json file that summaries the result
Summary: run-perf-tests should generate a json file that summaries the result
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Ryosuke Niwa
URL:
Keywords:
Depends on:
Blocks: 10266
  Show dependency treegraph
 
Reported: 2012-01-17 17:50 PST by Ryosuke Niwa
Modified: 2012-01-18 11:44 PST (History)
5 users (show)

See Also:


Attachments
fixes the bug (16.09 KB, patch)
2012-01-18 00:52 PST, Ryosuke Niwa
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ryosuke Niwa 2012-01-17 17:50:53 PST
Now that I've made a progress on webkit-perf.appspot.com, we're ready to start submitting data there. In order to do this, we need to have run-perf-tests generate a json file that summaries the results.
Comment 1 Ryosuke Niwa 2012-01-17 17:57:37 PST
Format:
{
  "password": "test",
  "timestamp": 1326851283,
  "revision": 104856,
  "summary": {
    "webkit_style_test": {
      "average": 100,
      "median": 102,
      "stdev": 5,
      "min": 90,
      "max": 110
    },
    "some_test": 54
  }
}

password: given per bot to avoid someone adversely reporting wrong values.
timestamp: the time test started running
revision: the webkit revision for which the tests were ran
summary: this is the main part of the json. It's a dictionary where the key is the name of a test and the value is either an integer (simple/chromium style tests) or a dictionary that contains average, median, standard deviation, min, and max values.
Comment 2 Ryosuke Niwa 2012-01-17 17:58:45 PST
If we end up having lots of tests and wanted to reduce the size of json, we can use an array (as supposed to a dictionary) for the webkit style test.
Comment 3 Ryosuke Niwa 2012-01-18 00:52:19 PST
Created attachment 122884 [details]
fixes the bug
Comment 4 Ryosuke Niwa 2012-01-18 00:56:39 PST
(In reply to comment #1)
> Format:
> {
>   "password": "test",
>   "timestamp": 1326851283,
>   "revision": 104856,
>   "summary": {

I'm changing my mind and renaming it to "results".
Comment 5 Adam Barth 2012-01-18 01:08:16 PST
Comment on attachment 122884 [details]
fixes the bug

View in context: https://bugs.webkit.org/attachment.cgi?id=122884&action=review

> Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py:32
> +import json

This isn't Python 2.5 compat, but I don't think you should worry about that.
Comment 6 Ryosuke Niwa 2012-01-18 01:47:11 PST
Thanks for the review!

(In reply to comment #5)
> (From update of attachment 122884 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=122884&action=review
> 
> > Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py:32
> > +import json
> 
> This isn't Python 2.5 compat, but I don't think you should worry about that.

Yeah, I'll worry about only if someone complains.
Comment 7 WebKit Review Bot 2012-01-18 02:53:01 PST
Comment on attachment 122884 [details]
fixes the bug

Clearing flags on attachment: 122884

Committed r105256: <http://trac.webkit.org/changeset/105256>
Comment 8 WebKit Review Bot 2012-01-18 02:53:06 PST
All reviewed patches have been landed.  Closing bug.
Comment 9 Ilya Tikhonovsky 2012-01-18 05:03:44 PST
Comment on attachment 122884 [details]
fixes the bug

View in context: https://bugs.webkit.org/attachment.cgi?id=122884&action=review

> Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py:55
> +        self._port = port or self._host.port_factory.get(self._options.platform, self._options)
> +        self._host = self._port.host

there is dependency cycle between port and host.
I've fixed it at r105263
Comment 10 Ryosuke Niwa 2012-01-18 11:44:52 PST
(In reply to comment #9)
> (From update of attachment 122884 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=122884&action=review
> 
> > Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py:55
> > +        self._port = port or self._host.port_factory.get(self._options.platform, self._options)
> > +        self._host = self._port.host
> 
> there is dependency cycle between port and host.
> I've fixed it at r105263

Thanks for the fix.