Bug 65268 – NRWT 28% slower than ORWT on Qt

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[WebKit Review Bot]

NRWT 40% slower than ORWT on Qt
Requested by abarth|zZz on #webkit.

[Adam Barth]

 Comment #16 From Andras Becsi 2011-07-27 07:20:46 PST (-) [reply] 
(In reply to comment #15)
> NRWT was slow when we first turned it on, and we changed a few things to make it faster.  If it's still slow, please let me know and we'll make it faster.  In single-child mode, NRWT should be about 5% slower than ORWT.  Anything more than is something we want to fix irrespective of whether we support running multiple instances on one machine.

You can see a good comparison between

ORWT: http://build.webkit.sed.hu/waterfall?show=x86-32%20Linux%20Qt-4.8.x%20Release

NRWT: http://build.webkit.org/waterfall?show=Qt%20Linux%20Release

The first bot is still using ORWT (NRWT does not understand qt-4.8) and runs the tests with few failing tests (and 3 additionaly skipped) in approximately 700 seconds whereas the release bot runs NRWT in approximately 1100s which is almost 40% slower on average.

[Adam Barth]

(Discussion from Bug 64898.)

[Andras Becsi]

(In reply to comment #2)
> (Discussion from Bug 64898.)

Thanks Adam for opening this bug, I assumed this is the expected slowdown resulting from the way NRWT works.

[Eric Seidel (no email)]

This is not an expected slowdown.  NRWT should be up-to 10% slower when run with --child-processes=1 mode (which is the current mode during this transition), but it shouldn't be 40% slower!

NRWT is of course much much much faster than ORWT when run with parallel processes. :)

[Csaba Osztrogonác]

I got one reason of this bug. After switching to NRWT there are 6 fast/workers failing tests with 30 secs notify done timeout. It is a DRT sideeffect bug: https://bugs.webkit.org/show_bug.cgi?id=64002 and it increased the runtime with 6*30secs = 180 secs

http://build.webkit.org/results/Qt%20Linux%20Release/r92952%20%2836428%29/results.html

If we can fix bug64002, the NRWT runtime will be ~900 secs, but ORWT is ~700 secs. So NRWT will be ~28% slower than ORWT.

[Eric Seidel (no email)]

Thank you for the update.  We also have some trouble with timeouts being reported as failures in NRWT due to bug 63981.

[Eric Seidel (no email)]

Looking at:
python -mcProfile Tools/Scripts/new-run-webkit-tests --child-processes=1 html5lib 

It's obvious that not much work gets done in the master process these days:
         787 function calls (784 primitive calls) in 12.208 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1   12.200   12.200   12.200   12.200 {posix.waitpid}

We'll have to profile one of the child processes.

[Eric Seidel (no email)]

We might want to wire up a flag which automatically has the worker children dump a cProfile into a per-worker log file.

[Eric Seidel (no email)]

Dirk pointed out in another bug that it's easy to profile NRWT using --worker-model=inline (which will run the worker code in the main process).

[Eric Seidel (no email)]

Moving this to the polish category since it hasn't actually blocked us from switching these bots.

[Csaba Osztrogonác]

After switch qt-mac platform to NRWT I checked its performance: NRWT is 43% slower on Qt-Mac than ORWT. :-/

[Eric Seidel (no email)]

:(  The big win is to move to parallel execution.  But regardless, we should make this faster.  Will investigate.

[Zoltan Horvath]

(In reply to comment #11)
> After switch qt-mac platform to NRWT I checked its performance: NRWT is 43% slower on Qt-Mac than ORWT. :-/

I think qt-mac is not as the main platform for us like qt-linux, additionally we are moving to a wk2 api based qtwebkit api so I'm not sure that we will run layouttests for qt-mac with qt5+wk2 api awhile.

Does the 28% slowdown still stand for qt-linux?

(In reply to comment #12)
> :(  The big win is to move to parallel execution.  But regardless, we should make this faster.  Will investigate.

Eric please let us know where we can help you!

[Eric Seidel (no email)]

(In reply to comment #13)
> (In reply to comment #12)
> > :(  The big win is to move to parallel execution.  But regardless, we should make this faster.  Will investigate.
> 
> Eric please let us know where we can help you!

Once https://bugs.webkit.org/show_bug.cgi?id=34984 lands it will be possible to opt-in to parallel testing on the bots by default.  I would strongly encourage Qt to consider doing this.  It will require some minor updates to the Skipped and test_expectation files, but it will be a huge improvement in overall runtime on the bots.

[Eric Seidel (no email)]

(In reply to comment #14)
> > Eric please let us know where we can help you!
> 
> Once https://bugs.webkit.org/show_bug.cgi?id=34984 lands it will be possible to opt-in to parallel testing on the bots by default.  I would strongly encourage Qt to consider doing this.  It will require some minor updates to the Skipped and test_expectation files, but it will be a huge improvement in overall runtime on the bots.

Now it's super-easy to opt-in to parallel testing on your bots:
http://trac.webkit.org/browser/trunk/Tools/Scripts/run-webkit-tests#L68

I would encourage you to try it.  You'll have to wade through a few rounds of skipping tests and updating test_expectations.txt due to some bad tests we have in our tree (which bleed results into later tests).   But once that's done it should work fine.  Chromium has used parallel testing for years.

[Dirk Pranke]

Is this still an issue? It looks like the only Qt bot still running ORWT is the arm bot, which has been offline for more than a month?

http://build.webkit.sed.hu/builders/ARMv5%20Linux%20Qt%20Release%20%28Test%29

I don't necessarily have any reason to think things have gotten better, and I'm happy to look into this further. Should I be able to reproduce this locally?

[Dirk Pranke]

I'm going to close this as fixed, since we have sped things up over the year since this was filed. If anyone can tell me how to reproduce this or thinks this still matters, please let me know and/or reopen. Thanks!