Bug 34984 – new-run-webkit-tests: MASTER BUG: Switch all webkit.org bots over

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Eric Seidel (no email)]

MASTER BUG: Switch webkit.org to using new python run-webkit-tests

There are a lot of blocking issues.  This will be a multi-week or possibly multi-month project.  But I'm creating this tracking bug to help keep tabs on all the blocking issues.

[Mark Rowe (bdash)]

Should there be bugs about making this work on each of the supported platforms?  Mac OS X 10.4 in particular comes to mind as it has the oldest version of Python of the supported platforms.

[Eric Seidel (no email)]

Perhaps.  I figure we'll cross that bridge when we come to it.  I have no insight into Apple's deprecation plans, but its also possible that Tiger support will be dropped from WebKit before this work is completed. :)

[Dirk Pranke]

Taking ownership ... I plan to start a push to actually get this done, hopefully starting as early as next week (I just need to wrap up another project first, which should happen today or tomorrow).

[Adam Barth]

(In reply to comment #3)
> Taking ownership ... I plan to start a push to actually get this done, hopefully starting as early as next week (I just need to wrap up another project first, which should happen today or tomorrow).

That's exciting!

[Adam Roben (:aroben)]

<rdar://problem/9509321>

[Dirk Pranke]

Eric, I'm punting this to you now :)

[Eric Seidel (no email)]

(In reply to comment #6)
> Eric, I'm punting this to you now :)

Thank you again Dirk for all your hard work on this.  I'll pick this up next week.

[Eric Seidel (no email)]

NRWT is enabled for all bots on build.webkit.org except Windows and WebKit2 as of:
http://trac.webkit.org/changeset/90440

Hopefully it will stick!

[Eric Seidel (no email)]

The change stuck.  All bots except for Apple Win, WebKit2 and --leaks are running NRWT.  Fixes for the remaining bots are underway.

I'm taking vacation the next two weeks, so the remaining bots are unlikely to switch until August.

[Eric Seidel (no email)]

I'm back.  My goal is to switch WebKit2 and --leaks this week.  I'll coordinate with Adam Roben and other windows folks to make sure we're moving forward on Win32 support, but it's not on my immediate goal list for this week.

A stretch goal is to move some/all the existing bots over to parallel execution (which is the real benefit of this work).

[Balazs Kelemen]

I have a good news on the topic: The Qt WebKit2 bot (hosted at http://build.webkit.sed.hu/) has already successfully switched.

[Eric Seidel (no email)]

Update:
Fixed a bunch of random bugs last week and switched the --leaks bot.
WK2 works, but I've not switched the bots yet.
Win and Lion theoretically work, but I've not tested.

I will be working on bidi-isolate (bug 65617) this week.  I will resume work on NRWT Aug. 22nd, with the gaol of switching the WK2 bots as soon as possible.


WK2 (and a bunch of these other bugs) boil down to one issue that NRWT uses a ServerProcess abstraction which reads line-by-line only from stdout.  stderr is collected in one long string and processed after the process is done.

This means that we can't detect timeouts (since they're sent via #timeout over stderr) or WebProcess crashes (for wk2).

I just need to redesign the ServerProcess abstraction to allow reading from both stdout and stderr in a line-by-line fashion.  This will be the first thing I do when I resume NRWT work in a week.

[Eric Seidel (no email)]

OK.  My work on bug 65617 is basically done (just waiting for review).  Back to working on NRWT full time.

[Eric Seidel (no email)]

Created attachment 112919 [details]
Use a blacklist of unsupported platforms instead of a whitelist

[Eric Seidel (no email)]

Once this last patch lands, all ports will use NRWT by default, with two specific exceptions (win and qt-arm), and I think we should close this bug.   I'm working my way through the polish bugs now.

[WebKit Review Bot]

Comment on attachment 112919 [details]
Use a blacklist of unsupported platforms instead of a whitelist

Clearing flags on attachment: 112919

Committed r98780: <http://trac.webkit.org/changeset/98780>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.

[Ryosuke Niwa]

It appears that this patch broke Qt bots:
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/39090/steps/layout-test/logs/stdio

[Ryosuke Niwa]

I'm going to rollout the patch since Qt port's losing 100% of the test coverage due to this bug.

[Ryosuke Niwa]

Reopen the bug since the patch was rolled out in r98801.

[Csaba Osztrogonác]

Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .

[Ryosuke Niwa]

(In reply to comment #21)
> Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .

Oh, could you reland the patch with that fix in that case?

[Balazs Kelemen]

(In reply to comment #22)
> (In reply to comment #21)
> > Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .
> 
> Oh, could you reland the patch with that fix in that case?

Relanded in r98804.

[Eric Seidel (no email)]

Thank you arm.

[Raphael Kubo da Costa (:rakuco)]

Can we close this one now?

[Ryosuke Niwa]

(In reply to comment #25)
> Can we close this one now?

Has Windows port moved to new-run-webkit-tests yet?

[Dirk Pranke]

Nope, the Apple win port is still running ORWT. I would like to fix it, but the Apple win port is generally pretty unhappy these days so it's hard to isolate which issues are NRWT-related and which are just broken. Hopefully that will be fixed soon.

[Csaba Osztrogonác]

All bots run new-run-webkit-tests now, which is simple run-webkit-tests :)