Bug 142166

Summary: [buildbot] Add regression test to check which builder runs which tests
Product: WebKit Reporter: Csaba Osztrogonác <ossy>
Component: Tools / TestsAssignee: Csaba Osztrogonác <ossy>
Status: RESOLVED FIXED    
Severity: Normal CC: ap, commit-queue, dburkart, lforschler, matthew_hanson, mmirman, ossy, rniwa
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch none

Description Csaba Osztrogonác 2015-03-02 07:46:44 PST
It would be good to have a buildstep list for each builder to make 
master.cfg hacking easier and it can ensure that a change won't
break the list of the steps accidentally.
Comment 1 Csaba Osztrogonác 2015-03-02 07:56:39 PST
Created attachment 247668 [details]
Patch
Comment 2 WebKit Commit Bot 2015-03-02 07:58:24 PST
Attachment 247668 [details] did not pass style-queue:


ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:368:  whitespace before ':'  [pep8/E203] [5]
ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:415:  [BuildStepsTest.generateTests] Undefined variable 'c'  [pylint/E0602] [5]
ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:432:  [BuildStepsTest.test_unnecessary_expected_results] Undefined variable 'c'  [pylint/E0602] [5]
Total errors found: 3 in 2 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 3 Csaba Osztrogonác 2015-03-02 08:02:16 PST
note: inspired by bug142094
Comment 4 Alexey Proskuryakov 2015-03-02 09:05:59 PST
Comment on attachment 247668 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=247668&action=review

Seems like a good idea. What runs these tests, does buildbot's checkconfig do that?

> Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:406
> +    "GTK Linux ARM Release" : ["configure build", "svn", "kill old processes", "delete WebKitBuild directory", "delete stale build files", "jhbuild", "compile-webkit", "jscore-test", "webkitpy-test", "webkitperl-test", "bindings-generation-tests", "API tests", "WebKit GObject DOM bindings API break tests"],

"WebKit GObject DOM bindings API break tests"? The dashboard doesn't know about this one.
Comment 5 Csaba Osztrogonác 2015-03-02 09:53:52 PST
Comment on attachment 247668 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=247668&action=review

>> Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:406
>> +    "GTK Linux ARM Release" : ["configure build", "svn", "kill old processes", "delete WebKitBuild directory", "delete stale build files", "jhbuild", "compile-webkit", "jscore-test", "webkitpy-test", "webkitperl-test", "bindings-generation-tests", "API tests", "WebKit GObject DOM bindings API break tests"],
> 
> "WebKit GObject DOM bindings API break tests"? The dashboard doesn't know about this one.

I don't know what is this test for, but GTK bots run it with RunGtkWebKitGObjectDOMBindingsAPIBreakTests():
http://trac.webkit.org/browser/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg?rev=180288#L600
Comment 6 Csaba Osztrogonác 2015-03-02 10:02:34 PST
(In reply to comment #4)
> Comment on attachment 247668 [details]
> Patch
> 
> Seems like a good idea. What runs these tests, does buildbot's checkconfig
> do that?

Nor checkconfig runs it, nor Tools/Scripts/test-webkitpy, 
but it can be run by hand after a little patching.

The actually used buildmaster is too old for actual twisted on 
Mavericks and Ubuntu 14.04. Unfortunately you have to fix the 
autoinstalled buildbot locally to be able to run the unittests.

Tools/Scripts/webkitpy/thirdparty/autoinstalled/buildbot/
* status/web/changes.py:
* status/web/logs.py:
* status/web/slaves.py:

-from twisted.web.error import NoResource
+from twisted.web.resource import NoResource
Comment 7 Alexey Proskuryakov 2015-03-02 10:14:30 PST
I think that it goes without saying that we need to run these tests automatically; there is no way anyone would remember to do it manually with hacks.
Comment 8 Csaba Osztrogonác 2015-03-02 11:06:02 PST
(In reply to comment #7)
> I think that it goes without saying that we need to run these tests
> automatically; there is no way anyone would remember to do it manually with
> hacks.

I agree, will investigate how can we do it easily. I think we should try
to upgrade the buildmaster. Now we still use 0.8.6p1 which was released
was released on March 25, 2012 - nearly 3 years ago. 

I'll check which newer buildmaster release can we use on Mavericks 
and Ubuntu 14.04 too without too much adjust needed in master.cfg.
Comment 9 Csaba Osztrogonác 2015-03-02 11:13:33 PST
Comment on attachment 247668 [details]
Patch

cq- now, I'm going to tweak a little bit to be able update resulults easily.
Comment 10 Csaba Osztrogonác 2015-03-03 09:22:36 PST
Created attachment 247764 [details]
Patch

Additionally dumping the actual result if expected result is missing (new bot added). And updated results after bug142094 (not landed yet).
Comment 11 WebKit Commit Bot 2015-03-03 09:24:51 PST
Attachment 247764 [details] did not pass style-queue:


ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:368:  whitespace before ':'  [pep8/E203] [5]
ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:417:  [BuildStepsTest.generateTests] Undefined variable 'c'  [pylint/E0602] [5]
ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:434:  [BuildStepsTest.test_unnecessary_expected_results] Undefined variable 'c'  [pylint/E0602] [5]
Total errors found: 3 in 2 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 12 Csaba Osztrogonác 2015-03-03 09:31:40 PST
(In reply to comment #7)
> I think that it goes without saying that we need to run these tests
> automatically; there is no way anyone would remember to do it manually with
> hacks.

I filed a new bug report to make it runnable out-of-the-box: bug142219
Comment 13 WebKit Commit Bot 2015-03-05 00:03:47 PST
Comment on attachment 247764 [details]
Patch

Clearing flags on attachment: 247764

Committed r181070: <http://trac.webkit.org/changeset/181070>
Comment 14 WebKit Commit Bot 2015-03-05 00:03:53 PST
All reviewed patches have been landed.  Closing bug.