Bug 155053 – Add iOS simulator debug bots

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Alexey Proskuryakov]

.

[Alexey Proskuryakov]

Created attachment 273057 [details]
proposed patch

[Lucas Forschler]

Comment on attachment 273057 [details]
proposed patch

I think you need to add a scheduler for the new configurations.

[Lucas Forschler]

Looks correct, my mistake.
(In reply to comment #2)
> Comment on attachment 273057 [details]
> proposed patch
> 
> I think you need to add a scheduler for the new configurations.

[WebKit Commit Bot]

Comment on attachment 273057 [details]
proposed patch

Rejecting attachment 273057 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-02', 'validate-changelog', '--check-oops', '--non-interactive', 273057, '--port=mac']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

/Volumes/Data/EWS/WebKit/Tools/ChangeLog neither lists a valid reviewer nor contains the string "Unreviewed" or "Rubber stamp" (case insensitive).

Full output: http://webkit-queues.webkit.org/results/924166

[Alexey Proskuryakov]

Committed http://trac.webkit.org/r197601

[Csaba Osztrogonác]

(In reply to comment #5)
> Committed http://trac.webkit.org/r197601

It broke the unittest of master.cfg. Could you possibly
add the new bots to mastercfg_unittest.py too?

======================================================================
FAIL: test_builder Apple iOS 9 Simulator Debug (Build) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./mastercfg_unittest.py", line 438, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple iOS 9 Simulator Debug (Build)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'compile-webkit', 'archive-built-product', 'upload', 'trigger']

======================================================================
FAIL: test_builder Apple iOS 9 Simulator Debug WK1 (Tests) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./mastercfg_unittest.py", line 438, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple iOS 9 Simulator Debug WK1 (Tests)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'download-built-product', 'extract-built-product', 'layout-test', 'run-api-tests', 'webkitpy-test', 'webkitperl-test', 'bindings-generation-tests', 'archive-test-results', 'upload', 'MasterShellCommand']

======================================================================
FAIL: test_builder Apple iOS 9 Simulator Debug WK2 (Tests) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./mastercfg_unittest.py", line 438, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple iOS 9 Simulator Debug WK2 (Tests)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'download-built-product', 'extract-built-product', 'layout-test', 'run-api-tests', 'webkitpy-test', 'webkitperl-test', 'bindings-generation-tests', 'archive-test-results', 'upload', 'MasterShellCommand']

----------------------------------------------------------------------
Ran 643 tests in 0.045s

FAILED (failures=3)

[Alexey Proskuryakov]

Added in http://trac.webkit.org/r198293

I have some comments about this test:

1. It should run on bots at least, if not EWS. Hidden secret tests are not good.

2. It's not a very good test, because it needed to be updated for an obviously irrelevant change like this one. What it tests couldn't have possibly be broken here, so it shouldn't have required maintenance.

3. It prints a warning when I run it locally:

$ python ./mastercfg_unittest.py 
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py:186: ImportWarning: Not importing directory '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/mpl_toolkits': missing __init__.py
  file, filename, etc = imp.find_module(subname, path)
/Users/ap/Safari/OpenSource/Tools/Scripts/webkitpy/thirdparty/autoinstalled/twisted/twisted/spread/jelly.py:102: DeprecationWarning: the sets module is deprecated
  import sets as _sets
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 643 tests in 0.022s

OK

[Csaba Osztrogonác]

(In reply to comment #7)
> Added in http://trac.webkit.org/r198293
> 
> I have some comments about this test:
> 
> 1. It should run on bots at least, if not EWS. Hidden secret tests are not
> good.

It's not a hidden test, it's next to master.cfg and there 
are only a couple of developers who regularly touch master.cfg.

But I agree, it would be great to run it automatically. I won't have
time for it, but filed a bug report, maybe somebody else will pick it up:
bug155647
 
> 2. It's not a very good test, because it needed to be updated for an
> obviously irrelevant change like this one. What it tests couldn't have
> possibly be broken here, so it shouldn't have required maintenance.
I started to simplify this test in bug155648

> 3. It prints a warning when I run it locally:
> 
> $ python ./mastercfg_unittest.py 
> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/
> pkgutil.py:186: ImportWarning: Not importing directory
> '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/
> mpl_toolkits': missing __init__.py
>   file, filename, etc = imp.find_module(subname, path)
> /Users/ap/Safari/OpenSource/Tools/Scripts/webkitpy/thirdparty/autoinstalled/
> twisted/twisted/spread/jelly.py:102: DeprecationWarning: the sets module is
> deprecated
>   import sets as _sets
> .......
> ----------------------------------------------------------------------
> Ran 643 tests in 0.022s
> 
> OK

We can't do anything with this warning. It's come from the 
ancient twisted package which is used by build.webkit.org

[Alexey Proskuryakov]

> We can't do anything with this warning.

Certainly it is possible filter it out of output? May not be worth the effort though.