Bug 34984

Summary: new-run-webkit-tests: MASTER BUG: Switch all webkit.org bots over
Product: WebKit Reporter: Eric Seidel <eric@webkit.org>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned@lists.webkit.org>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth@webkit.org, abecsi@webkit.org, aroben@webkit.org, darin@apple.com, dglazkov@chromium.org, dpranke@chromium.org, galpeter@inf.u-szeged.hu, hayato@chromium.org, joepeck@webkit.org, kbalazs@webkit.org, laszlo.gombos@webkit.org, leandro@profusion.mobi, mjs@apple.com, mrowe@apple.com, ossy@webkit.org, rakuco@webkit.org, rgabor@webkit.org, rniwa@webkit.org, tonyg@chromium.org, vestbo@webkit.org, webkit.review.bot@gmail.com, zoltan@webkit.org
Priority: P2 Keywords: InRadar, NRWT, ToolsHitList
Version: 528+ (Nightly build)   
Hardware: All   
OS: All   
Bug Depends on: 34826, 34983, 34985, 34986, 35001, 35006, 35049, 35053, 35055, 35265, 35266, 35358, 35359, 37007, 37736, 37738, 37739, 38063, 38756, 43565, 55907, 56047, 56729, 56730, 56731, 57640, 58293, 58625, 59894, 60162, 62178, 62194, 62425, 62497, 63199, 63494, 63523, 63549, 63786, 63791, 63832, 63838, 63839, 63845, 63956, 63973, 63983, 64071, 64086, 64092, 64346, 64347, 64445, 64514, 64564, 64812, 65719, 66227, 66228, 67216, 67268, 70416, 71087, 71173, 71364    
Bug Blocks: 57310    
Attachments:
Description Flags
Use a blacklist of unsupported platforms instead of a whitelist none

Description From 2010-02-16 13:26:56 PST
MASTER BUG: Switch webkit.org to using new python run-webkit-tests

There are a lot of blocking issues.  This will be a multi-week or possibly multi-month project.  But I'm creating this tracking bug to help keep tabs on all the blocking issues.
------- Comment #1 From 2010-02-19 17:12:46 PST -------
Should there be bugs about making this work on each of the supported platforms?  Mac OS X 10.4 in particular comes to mind as it has the oldest version of Python of the supported platforms.
------- Comment #2 From 2010-02-24 12:27:14 PST -------
Perhaps.  I figure we'll cross that bridge when we come to it.  I have no insight into Apple's deprecation plans, but its also possible that Tiger support will be dropped from WebKit before this work is completed. :)
------- Comment #3 From 2011-03-10 16:29:00 PST -------
Taking ownership ... I plan to start a push to actually get this done, hopefully starting as early as next week (I just need to wrap up another project first, which should happen today or tomorrow).
------- Comment #4 From 2011-03-10 16:30:04 PST -------
(In reply to comment #3)
> Taking ownership ... I plan to start a push to actually get this done, hopefully starting as early as next week (I just need to wrap up another project first, which should happen today or tomorrow).

That's exciting!
------- Comment #5 From 2011-05-26 11:45:23 PST -------
<rdar://problem/9509321>
------- Comment #6 From 2011-06-21 21:15:14 PST -------
Eric, I'm punting this to you now :)
------- Comment #7 From 2011-06-22 11:15:40 PST -------
(In reply to comment #6)
> Eric, I'm punting this to you now :)

Thank you again Dirk for all your hard work on this.  I'll pick this up next week.
------- Comment #8 From 2011-07-06 00:00:58 PST -------
NRWT is enabled for all bots on build.webkit.org except Windows and WebKit2 as of:
http://trac.webkit.org/changeset/90440

Hopefully it will stick!
------- Comment #9 From 2011-07-19 01:32:18 PST -------
The change stuck.  All bots except for Apple Win, WebKit2 and --leaks are running NRWT.  Fixes for the remaining bots are underway.

I'm taking vacation the next two weeks, so the remaining bots are unlikely to switch until August.
------- Comment #10 From 2011-08-08 10:04:23 PST -------
I'm back.  My goal is to switch WebKit2 and --leaks this week.  I'll coordinate with Adam Roben and other windows folks to make sure we're moving forward on Win32 support, but it's not on my immediate goal list for this week.

A stretch goal is to move some/all the existing bots over to parallel execution (which is the real benefit of this work).
------- Comment #11 From 2011-08-08 15:43:12 PST -------
I have a good news on the topic: The Qt WebKit2 bot (hosted at http://build.webkit.sed.hu/) has already successfully switched.
------- Comment #12 From 2011-08-15 09:51:57 PST -------
Update:
Fixed a bunch of random bugs last week and switched the --leaks bot.
WK2 works, but I've not switched the bots yet.
Win and Lion theoretically work, but I've not tested.

I will be working on bidi-isolate (bug 65617) this week.  I will resume work on NRWT Aug. 22nd, with the gaol of switching the WK2 bots as soon as possible.


WK2 (and a bunch of these other bugs) boil down to one issue that NRWT uses a ServerProcess abstraction which reads line-by-line only from stdout.  stderr is collected in one long string and processed after the process is done.

This means that we can't detect timeouts (since they're sent via #timeout over stderr) or WebProcess crashes (for wk2).

I just need to redesign the ServerProcess abstraction to allow reading from both stdout and stderr in a line-by-line fashion.  This will be the first thing I do when I resume NRWT work in a week.
------- Comment #13 From 2011-08-29 12:11:46 PST -------
OK.  My work on bug 65617 is basically done (just waiting for review).  Back to working on NRWT full time.
------- Comment #14 From 2011-10-28 15:17:36 PST -------
Created an attachment (id=112919) [details]
Use a blacklist of unsupported platforms instead of a whitelist
------- Comment #15 From 2011-10-28 15:24:44 PST -------
Once this last patch lands, all ports will use NRWT by default, with two specific exceptions (win and qt-arm), and I think we should close this bug.   I'm working my way through the polish bugs now.
------- Comment #16 From 2011-10-28 16:32:00 PST -------
(From update of attachment 112919 [details])
Clearing flags on attachment: 112919

Committed r98780: <http://trac.webkit.org/changeset/98780>
------- Comment #17 From 2011-10-28 16:32:10 PST -------
All reviewed patches have been landed.  Closing bug.
------- Comment #18 From 2011-10-29 00:55:25 PST -------
It appears that this patch broke Qt bots:
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/39090/steps/layout-test/logs/stdio
------- Comment #19 From 2011-10-29 01:18:56 PST -------
I'm going to rollout the patch since Qt port's losing 100% of the test coverage due to this bug.
------- Comment #20 From 2011-10-29 01:29:16 PST -------
Reopen the bug since the patch was rolled out in r98801.
------- Comment #21 From 2011-10-29 02:07:08 PST -------
Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .
------- Comment #22 From 2011-10-29 02:09:20 PST -------
(In reply to comment #21)
> Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .

Oh, could you reland the patch with that fix in that case?
------- Comment #23 From 2011-10-29 07:46:05 PST -------
(In reply to comment #22)
> (In reply to comment #21)
> > Thanks for rolling out and unbreak Qt bots. :) I checked this fail, it was only a typo: isArm() vs isARM() .
> 
> Oh, could you reland the patch with that fix in that case?

Relanded in r98804.
------- Comment #24 From 2011-10-29 11:25:09 PST -------
Thank you arm.
------- Comment #25 From 2012-05-13 15:09:38 PST -------
Can we close this one now?
------- Comment #26 From 2012-05-13 15:10:47 PST -------
(In reply to comment #25)
> Can we close this one now?

Has Windows port moved to new-run-webkit-tests yet?
------- Comment #27 From 2012-05-14 11:54:59 PST -------
Nope, the Apple win port is still running ORWT. I would like to fix it, but the Apple win port is generally pretty unhappy these days so it's hard to isolate which issues are NRWT-related and which are just broken. Hopefully that will be fixed soon.
------- Comment #28 From 2014-02-06 07:52:49 PST -------
All bots run new-run-webkit-tests now, which is simple run-webkit-tests :)