Bug 142166 – [buildbot] Add regression test to check which builder runs which tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Csaba Osztrogonác]

It would be good to have a buildstep list for each builder to make 
master.cfg hacking easier and it can ensure that a change won't
break the list of the steps accidentally.

[Csaba Osztrogonác]

Created attachment 247668 [details]
Patch

[WebKit Commit Bot]

Attachment 247668 [details] did not pass style-queue:


ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:368:  whitespace before ':'  [pep8/E203] [5]
ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:415:  [BuildStepsTest.generateTests] Undefined variable 'c'  [pylint/E0602] [5]
ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:432:  [BuildStepsTest.test_unnecessary_expected_results] Undefined variable 'c'  [pylint/E0602] [5]
Total errors found: 3 in 2 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Csaba Osztrogonác]

note: inspired by bug142094

[Alexey Proskuryakov]

Comment on attachment 247668 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=247668&action=review

Seems like a good idea. What runs these tests, does buildbot's checkconfig do that?

> Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:406
> +    "GTK Linux ARM Release" : ["configure build", "svn", "kill old processes", "delete WebKitBuild directory", "delete stale build files", "jhbuild", "compile-webkit", "jscore-test", "webkitpy-test", "webkitperl-test", "bindings-generation-tests", "API tests", "WebKit GObject DOM bindings API break tests"],

"WebKit GObject DOM bindings API break tests"? The dashboard doesn't know about this one.

[Csaba Osztrogonác]

Comment on attachment 247668 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=247668&action=review

>> Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:406
>> +    "GTK Linux ARM Release" : ["configure build", "svn", "kill old processes", "delete WebKitBuild directory", "delete stale build files", "jhbuild", "compile-webkit", "jscore-test", "webkitpy-test", "webkitperl-test", "bindings-generation-tests", "API tests", "WebKit GObject DOM bindings API break tests"],
> 
> "WebKit GObject DOM bindings API break tests"? The dashboard doesn't know about this one.

I don't know what is this test for, but GTK bots run it with RunGtkWebKitGObjectDOMBindingsAPIBreakTests():
http://trac.webkit.org/browser/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg?rev=180288#L600

[Csaba Osztrogonác]

(In reply to comment #4)
> Comment on attachment 247668 [details]
> Patch
> 
> Seems like a good idea. What runs these tests, does buildbot's checkconfig
> do that?

Nor checkconfig runs it, nor Tools/Scripts/test-webkitpy, 
but it can be run by hand after a little patching.

The actually used buildmaster is too old for actual twisted on 
Mavericks and Ubuntu 14.04. Unfortunately you have to fix the 
autoinstalled buildbot locally to be able to run the unittests.

Tools/Scripts/webkitpy/thirdparty/autoinstalled/buildbot/
* status/web/changes.py:
* status/web/logs.py:
* status/web/slaves.py:

-from twisted.web.error import NoResource
+from twisted.web.resource import NoResource

[Alexey Proskuryakov]

I think that it goes without saying that we need to run these tests automatically; there is no way anyone would remember to do it manually with hacks.

[Csaba Osztrogonác]

(In reply to comment #7)
> I think that it goes without saying that we need to run these tests
> automatically; there is no way anyone would remember to do it manually with
> hacks.

I agree, will investigate how can we do it easily. I think we should try
to upgrade the buildmaster. Now we still use 0.8.6p1 which was released
was released on March 25, 2012 - nearly 3 years ago. 

I'll check which newer buildmaster release can we use on Mavericks 
and Ubuntu 14.04 too without too much adjust needed in master.cfg.

[Csaba Osztrogonác]

Comment on attachment 247668 [details]
Patch

cq- now, I'm going to tweak a little bit to be able update resulults easily.

[Csaba Osztrogonác]

Created attachment 247764 [details]
Patch

Additionally dumping the actual result if expected result is missing (new bot added). And updated results after bug142094 (not landed yet).

[WebKit Commit Bot]

Attachment 247764 [details] did not pass style-queue:


ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:368:  whitespace before ':'  [pep8/E203] [5]
ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:417:  [BuildStepsTest.generateTests] Undefined variable 'c'  [pylint/E0602] [5]
ERROR: Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:434:  [BuildStepsTest.test_unnecessary_expected_results] Undefined variable 'c'  [pylint/E0602] [5]
Total errors found: 3 in 2 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Csaba Osztrogonác]

(In reply to comment #7)
> I think that it goes without saying that we need to run these tests
> automatically; there is no way anyone would remember to do it manually with
> hacks.

I filed a new bug report to make it runnable out-of-the-box: bug142219

[WebKit Commit Bot]

Comment on attachment 247764 [details]
Patch

Clearing flags on attachment: 247764

Committed r181070: <http://trac.webkit.org/changeset/181070>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.