Bug 34984

Summary: new-run-webkit-tests: MASTER BUG: Switch all webkit.org bots over
Product: WebKit Reporter: Eric Seidel <eric>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth, abecsi, aroben, darin, dglazkov, dpranke, galpeter, hayato, joepeck, kbalazs, laszlo.gombos, leandro, mjs, mrowe, ossy, rakuco, rgabor, rniwa, tonyg, vestbo, webkit.review.bot, zoltan
Priority: P2 Keywords: InRadar, NRWT, ToolsHitList
Version: 528+ (Nightly build)   
Hardware: All   
OS: All   
Bug Depends on: 34826, 34983, 34985, 34986, 35001, 35006, 35049, 35053, 35055, 35265, 35266, 35358, 35359, 37007, 37736, 37738, 37739, 38063, 38756, 43565, 55907, 56047, 56729, 56730, 56731, 57640, 58293, 58625, 59894, 60162, 62178, 62194, 62425, 62497, 63199, 63494, 63523, 63549, 63786, 63791, 63832, 63838, 63839, 63845, 63956, 63973, 63983, 64071, 64086, 64092, 64346, 64347, 64445, 64514, 64564, 64812, 65719, 66227, 66228, 67216, 67268, 70416, 71087, 71173, 71364    
Bug Blocks: 57310    
Attachments:
Description Flags
Use a blacklist of unsupported platforms instead of a whitelist none

Description Eric Seidel 2010-02-16 13:26:56 PST
MASTER BUG: Switch webkit.org to using new python run-webkit-tests

There are a lot of blocking issues.  This will be a multi-week or possibly multi-month project.  But I'm creating this tracking bug to help keep tabs on all the blocking issues.
Comment 1 Mark Rowe (bdash) 2010-02-19 17:12:46 PST
Should there be bugs about making this work on each of the supported platforms?  Mac OS X 10.4 in particular comes to mind as it has the oldest version of Python of the supported platforms.
Comment 2 Eric Seidel 2010-02-24 12:27:14 PST
Perhaps.  I figure we'll cross that bridge when we come to it.  I have no insight into Apple's deprecation plans, but its also possible that Tiger support will be dropped from WebKit before this work is completed. :)
Comment 3 Dirk Pranke 2011-03-10 16:29:00 PST
Taking ownership ... I plan to start a push to actually get this done, hopefully starting as early as next week (I just need to wrap up another project first, which should happen today or tomorrow).
Comment 4 Adam Barth 2011-03-10 16:30:04 PST
(In reply to comment #3)
> Taking ownership ... I plan to start a push to actually get this done, hopefully starting as early as next week (I just need to wrap up another project first, which should happen today or tomorrow).

That's exciting!
Comment 5 Adam Roben (:aroben) 2011-05-26 11:45:23 PDT
<rdar://problem/9509321>
Comment 6 Dirk Pranke 2011-06-21 21:15:14 PDT
Eric, I'm punting this to you now :)
Comment 7 Eric Seidel 2011-06-22 11:15:40 PDT
(In reply to comment #6)
> Eric, I'm punting this to you now :)

Thank you again Dirk for all your hard work on this.  I'll pick this up next week.
Comment 8 Eric Seidel 2011-07-06 00:00:58 PDT
NRWT is enabled for all bots on build.webkit.org except Windows and WebKit2 as of:
http://trac.webkit.org/changeset/90440

Hopefully it will stick!
Comment 9 Eric Seidel 2011-07-19 01:32:18 PDT
The change stuck.  All bots except for Apple Win, WebKit2 and --leaks are running NRWT.  Fixes for the remaining bots are underway.

I'm taking vacation the next two weeks, so the remaining bots are unlikely to switch until August.
Comment 10 Eric Seidel 2011-08-08 10:04:23 PDT
I'm back.  My goal is to switch WebKit2 and --leaks this week.  I'll coordinate with Adam Roben and other windows folks to make sure we're moving forward on Win32 support, but it's not on my immediate goal list for this week.

A stretch goal is to move some/all the existing bots over to parallel execution (which is the real benefit of this work).
Comment 11 Balazs Kelemen 2011-08-08 15:43:12 PDT
I have a good news on the topic: The Qt WebKit2 bot (hosted at http://build.webkit.sed.hu/) has already successfully switched.
Comment 12 Eric Seidel 2011-08-15 09:51:57 PDT
Update:
Fixed a bunch of random bugs last week and switched the --leaks bot.
WK2 works, but I've not switched the bots yet.
Win and Lion theoretically work, but I've not tested.

I will be working on bidi-isolate (bug 65617) this week.  I will resume work on NRWT Aug. 22nd, with the gaol of switching the WK2 bots as soon as possible.


WK2 (and a bunch of these other bugs) boil down to one issue that NRWT uses a ServerProcess abstraction which reads line-by-line only from stdout.  stderr is collected in one long string and processed after the process is done.

This means that we can't detect timeouts (since they're sent via #timeout over stderr) or WebProcess crashes (for wk2).

I just need to redesign the ServerProcess abstraction to allow reading from both stdout and stderr in a line-by-line fashion.  This will be the first thing I do when I resume NRWT work in a week.
Comment 13 Eric Seidel 2011-08-29 12:11:46 PDT
OK.  My work on bug 65617 is basically done (just waiting for review).  Back to working on NRWT full time.
Comment 14 Eric Seidel 2011-10-28 15:17:36 PDT
Created attachment 112919 [details]
Use a blacklist of unsupported platforms instead of a whitelist
Comment 15 Eric Seidel 2011-10-28 15:24:44 PDT
Once this last patch lands, all ports will use NRWT by default, with two specific exceptions (win and qt-arm), and I think we should close this bug.   I'm working my way through the polish bugs now.
Comment 16 WebKit Review Bot 2011-10-28 16:32:00 PDT
Comment on attachment 112919 [details]
Use a blacklist of unsupported platforms instead of a whitelist

Clearing flags on attachment: 112919

Committed r98780: <http://trac.webkit.org/changeset/98780>
Comment 17 WebKit Review Bot 2011-10-28 16:32:10 PDT
All reviewed patches have been landed.  Closing bug.
Comment 18 Ryosuke Niwa 2011-10-29 00:55:25 PDT
It appears that this patch broke Qt bots:
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/39090/steps/layout-test/logs/stdio
Comment 19 Ryosuke Niwa 2011-10-29 01:18:56 PDT
I'm going to rollout the patch since Qt port's losing 100% of the test coverage due to this bug.
Comment 20 Ryosuke Niwa 2011-10-29 01:29:16 PDT
Reopen the bug since the patch was rolled out in r98801.
Comment 21 Csaba Osztrogon√°c 2011-10-29 02:07:08 PDT
Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .
Comment 22 Ryosuke Niwa 2011-10-29 02:09:20 PDT
(In reply to comment #21)
> Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .

Oh, could you reland the patch with that fix in that case?
Comment 23 Balazs Kelemen 2011-10-29 07:46:05 PDT
(In reply to comment #22)
> (In reply to comment #21)
> > Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .
> 
> Oh, could you reland the patch with that fix in that case?

Relanded in r98804.
Comment 24 Eric Seidel 2011-10-29 11:25:09 PDT
Thank you arm.
Comment 25 Raphael Kubo da Costa (:rakuco) 2012-05-13 15:09:38 PDT
Can we close this one now?
Comment 26 Ryosuke Niwa 2012-05-13 15:10:47 PDT
(In reply to comment #25)
> Can we close this one now?

Has Windows port moved to new-run-webkit-tests yet?
Comment 27 Dirk Pranke 2012-05-14 11:54:59 PDT
Nope, the Apple win port is still running ORWT. I would like to fix it, but the Apple win port is generally pretty unhappy these days so it's hard to isolate which issues are NRWT-related and which are just broken. Hopefully that will be fixed soon.
Comment 28 Csaba Osztrogon√°c 2014-02-06 07:52:49 PST
All bots run new-run-webkit-tests now, which is simple run-webkit-tests :)