Bug 65268

Summary: NRWT 28% slower than ORWT on Qt
Product: WebKit Reporter: WebKit Review Bot <webkit.review.bot>
Component: New BugsAssignee: WebKit Review Bot <webkit.review.bot>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth, abecsi, dpranke, eric, loki, ossy, rgabor, zoltan
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 65797    
Bug Blocks: 88680    

Description WebKit Review Bot 2011-07-27 10:49:55 PDT
NRWT 40% slower than ORWT on Qt
Requested by abarth|zZz on #webkit.
Comment 1 Adam Barth 2011-07-27 10:50:36 PDT
 Comment #16 From Andras Becsi 2011-07-27 07:20:46 PST (-) [reply] 
(In reply to comment #15)
> NRWT was slow when we first turned it on, and we changed a few things to make it faster.  If it's still slow, please let me know and we'll make it faster.  In single-child mode, NRWT should be about 5% slower than ORWT.  Anything more than is something we want to fix irrespective of whether we support running multiple instances on one machine.

You can see a good comparison between

ORWT: http://build.webkit.sed.hu/waterfall?show=x86-32%20Linux%20Qt-4.8.x%20Release

NRWT: http://build.webkit.org/waterfall?show=Qt%20Linux%20Release

The first bot is still using ORWT (NRWT does not understand qt-4.8) and runs the tests with few failing tests (and 3 additionaly skipped) in approximately 700 seconds whereas the release bot runs NRWT in approximately 1100s which is almost 40% slower on average.
Comment 2 Adam Barth 2011-07-27 10:51:00 PDT
(Discussion from Bug 64898.)
Comment 3 Andras Becsi 2011-07-27 11:51:24 PDT
(In reply to comment #2)
> (Discussion from Bug 64898.)

Thanks Adam for opening this bug, I assumed this is the expected slowdown resulting from the way NRWT works.
Comment 4 Eric Seidel (no email) 2011-08-08 10:06:46 PDT
This is not an expected slowdown.  NRWT should be up-to 10% slower when run with --child-processes=1 mode (which is the current mode during this transition), but it shouldn't be 40% slower!

NRWT is of course much much much faster than ORWT when run with parallel processes. :)
Comment 5 Csaba Osztrogon√°c 2011-08-12 04:38:48 PDT
I got one reason of this bug. After switching to NRWT there are 6 fast/workers failing tests with 30 secs notify done timeout. It is a DRT sideeffect bug: https://bugs.webkit.org/show_bug.cgi?id=64002 and it increased the runtime with 6*30secs = 180 secs

http://build.webkit.org/results/Qt%20Linux%20Release/r92952%20%2836428%29/results.html

If we can fix bug64002, the NRWT runtime will be ~900 secs, but ORWT is ~700 secs. So NRWT will be ~28% slower than ORWT.
Comment 6 Eric Seidel (no email) 2011-08-12 09:26:58 PDT
Thank you for the update.  We also have some trouble with timeouts being reported as failures in NRWT due to bug 63981.
Comment 7 Eric Seidel (no email) 2011-10-11 13:45:16 PDT
Looking at:
python -mcProfile Tools/Scripts/new-run-webkit-tests --child-processes=1 html5lib 

It's obvious that not much work gets done in the master process these days:
         787 function calls (784 primitive calls) in 12.208 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1   12.200   12.200   12.200   12.200 {posix.waitpid}

We'll have to profile one of the child processes.
Comment 8 Eric Seidel (no email) 2011-10-11 13:49:46 PDT
We might want to wire up a flag which automatically has the worker children dump a cProfile into a per-worker log file.
Comment 9 Eric Seidel (no email) 2011-10-24 10:39:20 PDT
Dirk pointed out in another bug that it's easy to profile NRWT using --worker-model=inline (which will run the worker code in the main process).
Comment 10 Eric Seidel (no email) 2011-10-24 16:02:55 PDT
Moving this to the polish category since it hasn't actually blocked us from switching these bots.
Comment 11 Csaba Osztrogon√°c 2011-10-27 10:15:09 PDT
After switch qt-mac platform to NRWT I checked its performance: NRWT is 43% slower on Qt-Mac than ORWT. :-/
Comment 12 Eric Seidel (no email) 2011-10-27 14:29:32 PDT
:(  The big win is to move to parallel execution.  But regardless, we should make this faster.  Will investigate.
Comment 13 Zoltan Horvath 2011-10-28 00:52:43 PDT
(In reply to comment #11)
> After switch qt-mac platform to NRWT I checked its performance: NRWT is 43% slower on Qt-Mac than ORWT. :-/

I think qt-mac is not as the main platform for us like qt-linux, additionally we are moving to a wk2 api based qtwebkit api so I'm not sure that we will run layouttests for qt-mac with qt5+wk2 api awhile.

Does the 28% slowdown still stand for qt-linux?

(In reply to comment #12)
> :(  The big win is to move to parallel execution.  But regardless, we should make this faster.  Will investigate.

Eric please let us know where we can help you!
Comment 14 Eric Seidel (no email) 2011-10-28 15:19:12 PDT
(In reply to comment #13)
> (In reply to comment #12)
> > :(  The big win is to move to parallel execution.  But regardless, we should make this faster.  Will investigate.
> 
> Eric please let us know where we can help you!

Once https://bugs.webkit.org/show_bug.cgi?id=34984 lands it will be possible to opt-in to parallel testing on the bots by default.  I would strongly encourage Qt to consider doing this.  It will require some minor updates to the Skipped and test_expectation files, but it will be a huge improvement in overall runtime on the bots.
Comment 15 Eric Seidel (no email) 2011-10-29 00:28:23 PDT
(In reply to comment #14)
> > Eric please let us know where we can help you!
> 
> Once https://bugs.webkit.org/show_bug.cgi?id=34984 lands it will be possible to opt-in to parallel testing on the bots by default.  I would strongly encourage Qt to consider doing this.  It will require some minor updates to the Skipped and test_expectation files, but it will be a huge improvement in overall runtime on the bots.

Now it's super-easy to opt-in to parallel testing on your bots:
http://trac.webkit.org/browser/trunk/Tools/Scripts/run-webkit-tests#L68

I would encourage you to try it.  You'll have to wade through a few rounds of skipping tests and updating test_expectations.txt due to some bad tests we have in our tree (which bleed results into later tests).   But once that's done it should work fine.  Chromium has used parallel testing for years.
Comment 16 Dirk Pranke 2012-06-08 17:14:40 PDT
Is this still an issue? It looks like the only Qt bot still running ORWT is the arm bot, which has been offline for more than a month?

http://build.webkit.sed.hu/builders/ARMv5%20Linux%20Qt%20Release%20%28Test%29

I don't necessarily have any reason to think things have gotten better, and I'm happy to look into this further. Should I be able to reproduce this locally?
Comment 17 Dirk Pranke 2012-06-18 14:38:51 PDT
I'm going to close this as fixed, since we have sped things up over the year since this was filed. If anyone can tell me how to reproduce this or thinks this still matters, please let me know and/or reopen. Thanks!