Bug 155053 - Add iOS simulator debug bots
Summary: Add iOS simulator debug bots
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alexey Proskuryakov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-04 16:45 PST by Alexey Proskuryakov
Modified: 2016-03-18 11:25 PDT (History)
3 users (show)

See Also:


Attachments
proposed patch (5.96 KB, patch)
2016-03-04 16:46 PST, Alexey Proskuryakov
lforschler: review+
commit-queue: commit-queue-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2016-03-04 16:45:12 PST
.
Comment 1 Alexey Proskuryakov 2016-03-04 16:46:17 PST
Created attachment 273057 [details]
proposed patch
Comment 2 Lucas Forschler 2016-03-04 16:48:11 PST
Comment on attachment 273057 [details]
proposed patch

I think you need to add a scheduler for the new configurations.
Comment 3 Lucas Forschler 2016-03-04 16:49:59 PST
Looks correct, my mistake.
(In reply to comment #2)
> Comment on attachment 273057 [details]
> proposed patch
> 
> I think you need to add a scheduler for the new configurations.
Comment 4 WebKit Commit Bot 2016-03-04 16:51:29 PST
Comment on attachment 273057 [details]
proposed patch

Rejecting attachment 273057 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-02', 'validate-changelog', '--check-oops', '--non-interactive', 273057, '--port=mac']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

/Volumes/Data/EWS/WebKit/Tools/ChangeLog neither lists a valid reviewer nor contains the string "Unreviewed" or "Rubber stamp" (case insensitive).

Full output: http://webkit-queues.webkit.org/results/924166
Comment 5 Alexey Proskuryakov 2016-03-04 17:06:27 PST
Committed http://trac.webkit.org/r197601
Comment 6 Csaba Osztrogonác 2016-03-16 11:35:35 PDT
(In reply to comment #5)
> Committed http://trac.webkit.org/r197601

It broke the unittest of master.cfg. Could you possibly
add the new bots to mastercfg_unittest.py too?

======================================================================
FAIL: test_builder Apple iOS 9 Simulator Debug (Build) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./mastercfg_unittest.py", line 438, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple iOS 9 Simulator Debug (Build)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'compile-webkit', 'archive-built-product', 'upload', 'trigger']

======================================================================
FAIL: test_builder Apple iOS 9 Simulator Debug WK1 (Tests) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./mastercfg_unittest.py", line 438, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple iOS 9 Simulator Debug WK1 (Tests)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'download-built-product', 'extract-built-product', 'layout-test', 'run-api-tests', 'webkitpy-test', 'webkitperl-test', 'bindings-generation-tests', 'archive-test-results', 'upload', 'MasterShellCommand']

======================================================================
FAIL: test_builder Apple iOS 9 Simulator Debug WK2 (Tests) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./mastercfg_unittest.py", line 438, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple iOS 9 Simulator Debug WK2 (Tests)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'download-built-product', 'extract-built-product', 'layout-test', 'run-api-tests', 'webkitpy-test', 'webkitperl-test', 'bindings-generation-tests', 'archive-test-results', 'upload', 'MasterShellCommand']

----------------------------------------------------------------------
Ran 643 tests in 0.045s

FAILED (failures=3)
Comment 7 Alexey Proskuryakov 2016-03-16 12:50:08 PDT
Added in http://trac.webkit.org/r198293

I have some comments about this test:

1. It should run on bots at least, if not EWS. Hidden secret tests are not good.

2. It's not a very good test, because it needed to be updated for an obviously irrelevant change like this one. What it tests couldn't have possibly be broken here, so it shouldn't have required maintenance.

3. It prints a warning when I run it locally:

$ python ./mastercfg_unittest.py 
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py:186: ImportWarning: Not importing directory '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/mpl_toolkits': missing __init__.py
  file, filename, etc = imp.find_module(subname, path)
/Users/ap/Safari/OpenSource/Tools/Scripts/webkitpy/thirdparty/autoinstalled/twisted/twisted/spread/jelly.py:102: DeprecationWarning: the sets module is deprecated
  import sets as _sets
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 643 tests in 0.022s

OK
Comment 8 Csaba Osztrogonác 2016-03-18 10:26:25 PDT
(In reply to comment #7)
> Added in http://trac.webkit.org/r198293
> 
> I have some comments about this test:
> 
> 1. It should run on bots at least, if not EWS. Hidden secret tests are not
> good.

It's not a hidden test, it's next to master.cfg and there 
are only a couple of developers who regularly touch master.cfg.

But I agree, it would be great to run it automatically. I won't have
time for it, but filed a bug report, maybe somebody else will pick it up:
bug155647
 
> 2. It's not a very good test, because it needed to be updated for an
> obviously irrelevant change like this one. What it tests couldn't have
> possibly be broken here, so it shouldn't have required maintenance.
I started to simplify this test in bug155648

> 3. It prints a warning when I run it locally:
> 
> $ python ./mastercfg_unittest.py 
> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/
> pkgutil.py:186: ImportWarning: Not importing directory
> '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/
> mpl_toolkits': missing __init__.py
>   file, filename, etc = imp.find_module(subname, path)
> /Users/ap/Safari/OpenSource/Tools/Scripts/webkitpy/thirdparty/autoinstalled/
> twisted/twisted/spread/jelly.py:102: DeprecationWarning: the sets module is
> deprecated
>   import sets as _sets
> .......
> ----------------------------------------------------------------------
> Ran 643 tests in 0.022s
> 
> OK

We can't do anything with this warning. It's come from the 
ancient twisted package which is used by build.webkit.org
Comment 9 Alexey Proskuryakov 2016-03-18 11:25:06 PDT
> We can't do anything with this warning.

Certainly it is possible filter it out of output? May not be worth the effort though.