EWS iOS bots sometimes fails to shutdown simulator. Following error is seen in the logs: RuntimeError raised: Timed out while waiting for F856DA6C-99A9-4F53-A577-FF6214181E63 to shut down Traceback (most recent call last): File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 85, in main run_details = run(port, options, args, stderr) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 448, in run run_details = manager.run(args) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 249, in run initial_results, retry_results, enabled_pixel_tests_in_retry = self._run_test_subset(default_device_tests, tests_to_skip) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 293, in _run_test_subset self._clean_up_run() File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 342, in _clean_up_run self._port.clean_up_test_run() File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/port/ios_simulator.py", line 103, in clean_up_test_run SimulatedDeviceManager.tear_down(self.host) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 444, in tear_down device.platform_device._tear_down(deadline - time.time()) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 538, in _tear_down self._shut_down(deadline - time.time()) File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 524, in _shut_down raise RuntimeError('Timed out while waiting for {} to shut down'.format(self.udid)) RuntimeError: Timed out while waiting for F856DA6C-99A9-4F53-A577-FF6214181E63 to shut down Stopping HTTP server ... Stopping WebSocket server ... e.g.: https://webkit-queues.webkit.org/results/10084446
This seems to be happening on all the iOS simulator bots (checked on ews121, ews122, ews123, ews124, ews125, ews126). e.g.: ews121:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 326 ews122:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 215 ews123:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 36 ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 481 ews125:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 285 ews126:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l 36 Also it seems to be happening for a while, atleast for 2 months. e.g.: ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log ios-sim-ews_2018-09-24_21-09.log:1212: raise RuntimeError('Timed out while waiting for {} to shut down'.format(self.udid))
(In reply to Aakash Jain from comment #1) > This seems to be happening on all the iOS simulator bots (checked on ews121, > ews122, ews123, ews124, ews125, ews126). > > > e.g.: > ews121:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 326 > ews122:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 215 > ews123:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 36 > ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 481 > ews125:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 285 > ews126:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log | wc -l > 36 > > > Also it seems to be happening for a while, atleast for 2 months. > > e.g.: > ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" > *.log > ios-sim-ews_2018-09-24_21-09.log:1212: raise RuntimeError('Timed out > while waiting for {} to shut down'.format(self.udid)) These logs alone aren't really actionable. How often is this happening and do we have spindumps or sysdiagnoses from any of these runs? Basically what's happening here is that we're getting stuck when shutting down one of our managed simulators. This is especially strange since this simulator was (apparently) healthy enough to actually run tests.
Side note for future diagnosis: according to <https://webkit-queues.webkit.org/results/10084446>, this is definitely one of the managed simulators. We used 10 simulators, so the simulator which failed to shutdown was booted by the instance of run-webkit-tests which failed to shut it down. In other words, the simulator in question WAS NOT a stale simulator from a previous run.
I think I stumbled upon a solution to this while working on some related simulator/device stuff.
Created attachment 356028 [details] Patch
Comment on attachment 356028 [details] Patch Clearing flags on attachment: 356028 Committed r238684: <https://trac.webkit.org/changeset/238684>
All reviewed patches have been landed. Closing bug.
<rdar://problem/46350950>