Bug 10266 – WebKit needs public performance tests running on public bots

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Eric Seidel (no email)]

Move Safari performance scripts into the WebKit public repository

Safari should move all (or at least most) of its performance testing scripts into the public repository.  At least enough to support the PLT build slaves and allow external developers to run external PLT suites (such as the SVG PLT suites as part of bug 10265).

[Jeremy Orlow]

jorlow: can a page cycler be easily hooked up to DRT?
[5:48pm] dglazkov: jorlow: it's not easy, because we can't keep snapshots of sites in public repo.
[5:48pm] • dglazkov is not a lawyer.
[5:49pm] jorlow: dglazkov: um..why do we need to put it in a public repo?
[5:49pm] dglazkov: jorlow: well.. that's a good question! 
[5:49pm] jorlow: chromium doesn't but runs page cycler bots publicly
[5:49pm] jorlow: the most important thing is knowing that a regression happened with perf stuff
[5:50pm] jorlow: it does suck that we can't release that (and apple can't release theirs, etc) but....something is better than nothing
[5:50pm] dglazkov: jorlow: let's file a bug and start talking about it there -- maybe we'll figure something out
[5:50pm] smfr: can't we write a script that bowdlerizes the test pages?
[5:50pm] smfr: and replace all the images with dummy images or something?
[5:51pm] jorlow: smfr: what test pages?
[5:51pm] jorlow: the layout tests?
[5:51pm] smfr: pages used by a page cycler
[5:51pm] dglazkov: smfr: hmm, that may work. We just need to make sure we still have meaningful tests 
[5:51pm] jorlow: well, then you're doing network requests
[5:51pm] smfr: e.g. the infamous PLT tests
[5:51pm] dglazkov: what are the PLT tests, btw?
[5:52pm] smfr: page load tests
[5:52pm] smfr: a bunch of pages scraped back in 2000 and cleaned up
[5:53pm] zecke: dglazkov: did you guys look into Mozilla's talos? at least they use GFDL licensed content... (even if it is not really representative for the web)
[5:53pm] dglazkov: smfr: well, duh  I mean, are they public?
[5:53pm] smfr: dglazkov: no
[5:53pm] jorlow: exactly  
[5:53pm] jorlow: zecke: I haven't heard of that
[5:53pm] jorlow: does sound interesting tho
[5:53pm] dglazkov: smfr: I quite like the bowdlerizer idea
[5:54pm] zecke: jorlow: IIRC: https://wiki.mozilla.org/StandaloneTalos
[5:54pm] jorlow: dglazkov:  ^^^
[5:54pm] jorlow: seems like a good start
[5:55pm] jorlow: er....dglazkov, smfr
[5:55pm] zecke: and back when working for Nokia, I did this: http://trac.webkit.org/wiki/QtWebKitPerformanceUtilities
[5:56pm] zecke: this way the scaper is public, the content is not.

[Eric Seidel (no email)]

run-page-load-test is already in teh public repository.  But it's not as useful w/o Apple's "autovicki" harness which knows how to restart your machine in a new user, clear the caches, and turn off the network before running the tests.  None of this is useful at all w/o data.  run-page-load-test uses the SVG PLT (which is public) in svn.webkit.org for now, but we need some HTML PLTs too.

[Jeremy Orlow]

Thanks for the info, Eric!  It made me realize I forgot to mention that we should be running this stuff on the bots.

Note that in the IRC log I posted there were some suggestions for how we could do HTML PLTestss.

[Adam Barth]

http://trac.webkit.org/browser/trunk/WebCore/benchmarks/parser/html-parser.html

might be a good place to start since it's a fully public benchmark.  It tests a relatively narrow slice of the code though.

[Adam Roben (:aroben)]

<rdar://problem/9509295>

[Ryosuke Niwa]

I intend to work on this bug this quarter.

[Ryosuke Niwa]

Sadly more issues are popping up as I integrate things :(

[Ryosuke Niwa]

Created attachment 123686 [details]
Adds a Chromium Mac Perf bot

[Ryosuke Niwa]

+darin, +fishd just to let them know we now have an ability to add more perf tests & perf bots.

[Adam Barth]

Comment on attachment 123686 [details]
Adds a Chromium Mac Perf bot

View in context: https://bugs.webkit.org/attachment.cgi?id=123686&action=review

> Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg:654
> +    command = ["curl", "--request", "POST", "--data", "@perf-test-results.json",
> +        ]

You can drop the trailing "," if you want to put the ] on the same line and pass PEP8.

[Tony Chang]

Comment on attachment 123686 [details]
Adds a Chromium Mac Perf bot

View in context: https://bugs.webkit.org/attachment.cgi?id=123686&action=review

>> Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg:654
>> +        ]
> 
> You can drop the trailing "," if you want to put the ] on the same line and pass PEP8.

Hmm, I wonder if curl is installed on all the bots.  It looks like DownloadBuiltProduct uses curl, but maybe that's not used by all the bots.

[Ryosuke Niwa]

Comment on attachment 123686 [details]
Adds a Chromium Mac Perf bot

View in context: https://bugs.webkit.org/attachment.cgi?id=123686&action=review

>>> Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg:654
>>> +        ]
>> 
>> You can drop the trailing "," if you want to put the ] on the same line and pass PEP8.
> 
> Hmm, I wonder if curl is installed on all the bots.  It looks like DownloadBuiltProduct uses curl, but maybe that's not used by all the bots.

Oops, this command is never used :( It's a left over before I added --test-results-server to run-perf-tests. Will remove this command before landing it.

[Ryosuke Niwa]

Committed r105848: <http://trac.webkit.org/changeset/105848>