Bug 76504 – run-perf-tests should generate a json file that summaries the result

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Ryosuke Niwa]

Now that I've made a progress on webkit-perf.appspot.com, we're ready to start submitting data there. In order to do this, we need to have run-perf-tests generate a json file that summaries the results.

[Ryosuke Niwa]

Format:
{
  "password": "test",
  "timestamp": 1326851283,
  "revision": 104856,
  "summary": {
    "webkit_style_test": {
      "average": 100,
      "median": 102,
      "stdev": 5,
      "min": 90,
      "max": 110
    },
    "some_test": 54
  }
}

password: given per bot to avoid someone adversely reporting wrong values.
timestamp: the time test started running
revision: the webkit revision for which the tests were ran
summary: this is the main part of the json. It's a dictionary where the key is the name of a test and the value is either an integer (simple/chromium style tests) or a dictionary that contains average, median, standard deviation, min, and max values.

[Ryosuke Niwa]

If we end up having lots of tests and wanted to reduce the size of json, we can use an array (as supposed to a dictionary) for the webkit style test.

[Ryosuke Niwa]

Created attachment 122884 [details]
fixes the bug

[Ryosuke Niwa]

(In reply to comment #1)
> Format:
> {
>   "password": "test",
>   "timestamp": 1326851283,
>   "revision": 104856,
>   "summary": {

I'm changing my mind and renaming it to "results".

[Adam Barth]

Comment on attachment 122884 [details]
fixes the bug

View in context: https://bugs.webkit.org/attachment.cgi?id=122884&action=review

> Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py:32
> +import json

This isn't Python 2.5 compat, but I don't think you should worry about that.

[Ryosuke Niwa]

Thanks for the review!

(In reply to comment #5)
> (From update of attachment 122884 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=122884&action=review
> 
> > Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py:32
> > +import json
> 
> This isn't Python 2.5 compat, but I don't think you should worry about that.

Yeah, I'll worry about only if someone complains.

[WebKit Review Bot]

Comment on attachment 122884 [details]
fixes the bug

Clearing flags on attachment: 122884

Committed r105256: <http://trac.webkit.org/changeset/105256>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.

[Ilya Tikhonovsky]

Comment on attachment 122884 [details]
fixes the bug

View in context: https://bugs.webkit.org/attachment.cgi?id=122884&action=review

> Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py:55
> +        self._port = port or self._host.port_factory.get(self._options.platform, self._options)
> +        self._host = self._port.host

there is dependency cycle between port and host.
I've fixed it at r105263

[Ryosuke Niwa]

(In reply to comment #9)
> (From update of attachment 122884 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=122884&action=review
> 
> > Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py:55
> > +        self._port = port or self._host.port_factory.get(self._options.platform, self._options)
> > +        self._host = self._port.host
> 
> there is dependency cycle between port and host.
> I've fixed it at r105263

Thanks for the fix.