191864 2018-11-20 10:47:52 -0800 EWS iOS-sim bots sometimes fails to shutdown simulator 2018-11-29 12:44:37 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 aakash_jain jbedard ap bburg commit-queue ews-watchlist glenn jbedard lforschler rniwa ryanhaddad webkit-bug-importer oldest_to_newest 1480534 0 aakash_jain 2018-11-20 10:47:52 -0800 EWS iOS bots sometimes fails to shutdown simulator. Following error is seen in the logs: RuntimeError raised: Timed out while waiting for F856DA6C-99A9-4F53-A577-FF6214181E63 to shut down Traceback (most recent call last): File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 85, in main run_details = run(port, options, args, stderr) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 448, in run run_details = manager.run(args) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 249, in run initial_results, retry_results, enabled_pixel_tests_in_retry = self._run_test_subset(default_device_tests, tests_to_skip) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 293, in _run_test_subset self._clean_up_run() File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 342, in _clean_up_run self._port.clean_up_test_run() File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/port/ios_simulator.py", line 103, in clean_up_test_run SimulatedDeviceManager.tear_down(self.host) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 444, in tear_down device.platform_device._tear_down(deadline - time.time()) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 538, in _tear_down self._shut_down(deadline - time.time()) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 524, in _shut_down raise RuntimeError('Timed out while waiting for {} to shut down'.format(self.udid)) RuntimeError: Timed out while waiting for F856DA6C-99A9-4F53-A577-FF6214181E63 to shut down Stopping HTTP server ... Stopping WebSocket server ... e.g.: https://webkit-queues.webkit.org/results/10084446 1480539 1 aakash_jain 2018-11-20 10:55:27 -0800 This seems to be happening on all the iOS simulator bots (checked on ews121, ews122, ews123, ews124, ews125, ews126). e.g.: ews121:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 326 ews122:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 215 ews123:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 36 ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 481 ews125:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 285 ews126:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 36 Also it seems to be happening for a while, atleast for 2 months. e.g.: ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log ios-sim-ews_2018-09-24_21-09.log:1212: raise RuntimeError('Timed out while waiting for {} to shut down'.format(self.udid)) 1481165 2 jbedard 2018-11-23 11:18:42 -0800 (In reply to Aakash Jain from comment #1) > This seems to be happening on all the iOS simulator bots (checked on ews121, > ews122, ews123, ews124, ews125, ews126). > > > e.g.: > ews121:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 326 > ews122:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 215 > ews123:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 36 > ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 481 > ews125:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 285 > ews126:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 36 > > > Also it seems to be happening for a while, atleast for 2 months. > > e.g.: > ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log > ios-sim-ews_2018-09-24_21-09.log:1212: raise RuntimeError('Timed out > while waiting for {} to shut down'.format(self.udid)) These logs alone aren't really actionable. How often is this happening and do we have spindumps or sysdiagnoses from any of these runs? Basically what's happening here is that we're getting stuck when shutting down one of our managed simulators. This is especially strange since this simulator was (apparently) healthy enough to actually run tests. 1481167 3 jbedard 2018-11-23 11:21:07 -0800 Side note for future diagnosis: according to <https://webkit-queues.webkit.org/results/10084446>, this is definitely one of the managed simulators. We used 10 simulators, so the simulator which failed to shutdown was booted by the instance of run-webkit-tests which failed to shut it down. In other words, the simulator in question WAS NOT a stale simulator from a previous run. 1483055 4 jbedard 2018-11-29 11:38:47 -0800 I think I stumbled upon a solution to this while working on some related simulator/device stuff. 1483057 5 356028 jbedard 2018-11-29 11:42:17 -0800 Created attachment 356028 Patch 1483095 6 356028 commit-queue 2018-11-29 12:43:07 -0800 Comment on attachment 356028 Patch Clearing flags on attachment: 356028 Committed r238684: <https://trac.webkit.org/changeset/238684> 1483096 7 commit-queue 2018-11-29 12:43:09 -0800 All reviewed patches have been landed. Closing bug. 1483100 8 webkit-bug-importer 2018-11-29 12:44:37 -0800 <rdar://problem/46350950> 356028 2018-11-29 11:42:17 -0800 2018-11-29 12:43:07 -0800 Patch bug-191864-20181129114216.patch text/plain 1987 jbedard SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDIzODY3OSkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE0IEBACisyMDE4LTExLTI5ICBKb25hdGhhbiBCZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNv bT4KKworICAgICAgICBFV1MgaU9TLXNpbSBib3RzIHNvbWV0aW1lcyBmYWlscyB0byBzaHV0ZG93 biBzaW11bGF0b3IKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dp P2lkPTE5MTg2NAorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAg ICAgICogU2NyaXB0cy93ZWJraXRweS94Y29kZS9zaW11bGF0ZWRfZGV2aWNlLnB5OgorICAgICAg ICAoU2ltdWxhdGVkRGV2aWNlLl9zaHV0X2Rvd24pOiBJbmNyZWFzZSB0aW1lIHRvIHdhaXQgZm9y IHNpbXVsYXRlZCBkZXZpY2UgdG8KKyAgICAgICAgc2h1dCBkb3duLCBpZ25vcmUgZXhpdCBjb2Rl IGluZGljYXRpbmcgYSBkZXZpY2UgaXMgYWxyZWFkeSBzaHV0IGRvd24uCisKIDIwMTgtMTEtMjkg IEpvbmF0aGFuIEJlZGFyZCAgPGpiZWRhcmRAYXBwbGUuY29tPgogCiAgICAgICAgIHdlYmtpdHB5 OiBVbmlmeSBpb3NfdmVyc2lvbi93YXRjaG9zX3ZlcnNpb24gY29kZQpJbmRleDogVG9vbHMvU2Ny aXB0cy93ZWJraXRweS94Y29kZS9zaW11bGF0ZWRfZGV2aWNlLnB5Cj09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRv b2xzL1NjcmlwdHMvd2Via2l0cHkveGNvZGUvc2ltdWxhdGVkX2RldmljZS5weQkocmV2aXNpb24g MjM4Njc5KQorKysgVG9vbHMvU2NyaXB0cy93ZWJraXRweS94Y29kZS9zaW11bGF0ZWRfZGV2aWNl LnB5CSh3b3JraW5nIGNvcHkpCkBAIC01MTIsMTEgKzUxMiwxMyBAQCBjbGFzcyBTaW11bGF0ZWRE ZXZpY2Uob2JqZWN0KToKICAgICAgICAgICAgICAgICByZXR1cm4gVHJ1ZQogICAgICAgICByZXR1 cm4gRmFsc2UKIAotICAgIGRlZiBfc2h1dF9kb3duKHNlbGYsIHRpbWVvdXQ9MTAuMCk6CisgICAg ZGVmIF9zaHV0X2Rvd24oc2VsZiwgdGltZW91dD0zMC4wKToKICAgICAgICAgZGVhZGxpbmUgPSB0 aW1lLnRpbWUoKSArIHRpbWVvdXQKIAotICAgICAgICBpZiBzZWxmLnN0YXRlKGZvcmNlX3VwZGF0 ZT1UcnVlKSAhPSBTaW11bGF0ZWREZXZpY2UuRGV2aWNlU3RhdGUuU0hVVF9ET1dOIGFuZCBzZWxm LnN0YXRlICE9IFNpbXVsYXRlZERldmljZS5EZXZpY2VTdGF0ZS5TSFVUVElOR19ET1dOOgotICAg ICAgICAgICAgc2VsZi5leGVjdXRpdmUucnVuX2NvbW1hbmQoW1NpbXVsYXRlZERldmljZU1hbmFn ZXIueGNydW4sICdzaW1jdGwnLCAnc2h1dGRvd24nLCBzZWxmLnVkaWRdKQorICAgICAgICAjIEVp dGhlciBzaHV0ZG93biBpcyBzdWNjZXNzZnVsLCBvciB0aGUgZGV2aWNlIHdhcyBhbHJlYWR5IHNo dXRkb3duIHdoZW4gd2UgYXR0ZW1wdGVkIHRvIHNodXQgaXQgZG93bi4KKyAgICAgICAgZXhpdF9j b2RlID0gc2VsZi5leGVjdXRpdmUucnVuX2NvbW1hbmQoW1NpbXVsYXRlZERldmljZU1hbmFnZXIu eGNydW4sICdzaW1jdGwnLCAnc2h1dGRvd24nLCBzZWxmLnVkaWRdLCByZXR1cm5fZXhpdF9jb2Rl PVRydWUpCisgICAgICAgIGlmIGV4aXRfY29kZSAhPSAwIGFuZCBleGl0X2NvZGUgIT0gMTY0Ogor ICAgICAgICAgICAgcmFpc2UgUnVudGltZUVycm9yKCdGYWlsZWQgdG8gc2h1dGRvd24ge30gd2l0 aCBleGl0IGNvZGUge30nLmZvcm1hdChzZWxmLnVkaWQsIGV4aXRfY29kZSkpCiAKICAgICAgICAg d2hpbGUgc2VsZi5zdGF0ZShmb3JjZV91cGRhdGU9VHJ1ZSkgIT0gU2ltdWxhdGVkRGV2aWNlLkRl dmljZVN0YXRlLlNIVVRfRE9XTjoKICAgICAgICAgICAgIHRpbWUuc2xlZXAoLjUpCg==