NRWT 40% slower than ORWT on Qt Requested by abarth|zZz on #webkit.
Comment #16 From Andras Becsi 2011-07-27 07:20:46 PST (-) [reply] (In reply to comment #15) > NRWT was slow when we first turned it on, and we changed a few things to make it faster. If it's still slow, please let me know and we'll make it faster. In single-child mode, NRWT should be about 5% slower than ORWT. Anything more than is something we want to fix irrespective of whether we support running multiple instances on one machine. You can see a good comparison between ORWT: http://build.webkit.sed.hu/waterfall?show=x86-32%20Linux%20Qt-4.8.x%20Release NRWT: http://build.webkit.org/waterfall?show=Qt%20Linux%20Release The first bot is still using ORWT (NRWT does not understand qt-4.8) and runs the tests with few failing tests (and 3 additionaly skipped) in approximately 700 seconds whereas the release bot runs NRWT in approximately 1100s which is almost 40% slower on average.
(Discussion from Bug 64898.)
(In reply to comment #2) > (Discussion from Bug 64898.) Thanks Adam for opening this bug, I assumed this is the expected slowdown resulting from the way NRWT works.
This is not an expected slowdown. NRWT should be up-to 10% slower when run with --child-processes=1 mode (which is the current mode during this transition), but it shouldn't be 40% slower! NRWT is of course much much much faster than ORWT when run with parallel processes. :)
I got one reason of this bug. After switching to NRWT there are 6 fast/workers failing tests with 30 secs notify done timeout. It is a DRT sideeffect bug: https://bugs.webkit.org/show_bug.cgi?id=64002 and it increased the runtime with 6*30secs = 180 secs http://build.webkit.org/results/Qt%20Linux%20Release/r92952%20%2836428%29/results.html If we can fix bug64002, the NRWT runtime will be ~900 secs, but ORWT is ~700 secs. So NRWT will be ~28% slower than ORWT.
Thank you for the update. We also have some trouble with timeouts being reported as failures in NRWT due to bug 63981.
Looking at: python -mcProfile Tools/Scripts/new-run-webkit-tests --child-processes=1 html5lib It's obvious that not much work gets done in the master process these days: 787 function calls (784 primitive calls) in 12.208 seconds ncalls tottime percall cumtime percall filename:lineno(function) 1 12.200 12.200 12.200 12.200 {posix.waitpid} We'll have to profile one of the child processes.
We might want to wire up a flag which automatically has the worker children dump a cProfile into a per-worker log file.
Dirk pointed out in another bug that it's easy to profile NRWT using --worker-model=inline (which will run the worker code in the main process).
Moving this to the polish category since it hasn't actually blocked us from switching these bots.
After switch qt-mac platform to NRWT I checked its performance: NRWT is 43% slower on Qt-Mac than ORWT. :-/
:( The big win is to move to parallel execution. But regardless, we should make this faster. Will investigate.
(In reply to comment #11) > After switch qt-mac platform to NRWT I checked its performance: NRWT is 43% slower on Qt-Mac than ORWT. :-/ I think qt-mac is not as the main platform for us like qt-linux, additionally we are moving to a wk2 api based qtwebkit api so I'm not sure that we will run layouttests for qt-mac with qt5+wk2 api awhile. Does the 28% slowdown still stand for qt-linux? (In reply to comment #12) > :( The big win is to move to parallel execution. But regardless, we should make this faster. Will investigate. Eric please let us know where we can help you!
(In reply to comment #13) > (In reply to comment #12) > > :( The big win is to move to parallel execution. But regardless, we should make this faster. Will investigate. > > Eric please let us know where we can help you! Once https://bugs.webkit.org/show_bug.cgi?id=34984 lands it will be possible to opt-in to parallel testing on the bots by default. I would strongly encourage Qt to consider doing this. It will require some minor updates to the Skipped and test_expectation files, but it will be a huge improvement in overall runtime on the bots.
(In reply to comment #14) > > Eric please let us know where we can help you! > > Once https://bugs.webkit.org/show_bug.cgi?id=34984 lands it will be possible to opt-in to parallel testing on the bots by default. I would strongly encourage Qt to consider doing this. It will require some minor updates to the Skipped and test_expectation files, but it will be a huge improvement in overall runtime on the bots. Now it's super-easy to opt-in to parallel testing on your bots: http://trac.webkit.org/browser/trunk/Tools/Scripts/run-webkit-tests#L68 I would encourage you to try it. You'll have to wade through a few rounds of skipping tests and updating test_expectations.txt due to some bad tests we have in our tree (which bleed results into later tests). But once that's done it should work fine. Chromium has used parallel testing for years.
Is this still an issue? It looks like the only Qt bot still running ORWT is the arm bot, which has been offline for more than a month? http://build.webkit.sed.hu/builders/ARMv5%20Linux%20Qt%20Release%20%28Test%29 I don't necessarily have any reason to think things have gotten better, and I'm happy to look into this further. Should I be able to reproduce this locally?
I'm going to close this as fixed, since we have sped things up over the year since this was filed. If anyone can tell me how to reproduce this or thinks this still matters, please let me know and/or reopen. Thanks!