Bug 191864 – EWS iOS-sim bots sometimes fails to shutdown simulator

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

EWS iOS bots sometimes fails to shutdown simulator. Following error is seen in the logs:

RuntimeError raised: Timed out while waiting for F856DA6C-99A9-4F53-A577-FF6214181E63 to shut down
Traceback (most recent call last):
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 85, in main
    run_details = run(port, options, args, stderr)
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 448, in run
    run_details = manager.run(args)
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 249, in run
    initial_results, retry_results, enabled_pixel_tests_in_retry = self._run_test_subset(default_device_tests, tests_to_skip)
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 293, in _run_test_subset
    self._clean_up_run()
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 342, in _clean_up_run
    self._port.clean_up_test_run()
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/port/ios_simulator.py", line 103, in clean_up_test_run
    SimulatedDeviceManager.tear_down(self.host)
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 444, in tear_down
    device.platform_device._tear_down(deadline - time.time())
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 538, in _tear_down
    self._shut_down(deadline - time.time())
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 524, in _shut_down
    raise RuntimeError('Timed out while waiting for {} to shut down'.format(self.udid))
RuntimeError: Timed out while waiting for F856DA6C-99A9-4F53-A577-FF6214181E63 to shut down
Stopping HTTP server ...
Stopping WebSocket server ...



e.g.: https://webkit-queues.webkit.org/results/10084446

[Aakash Jain]

This seems to be happening on all the iOS simulator bots (checked on ews121, ews122, ews123, ews124, ews125, ews126).


e.g.:
ews121:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l
     326
ews122:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l
     215
ews123:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l
      36
ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l
     481
ews125:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l
     285
ews126:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log | wc -l
      36


Also it seems to be happening for a while, atleast for 2 months.

e.g.:
ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for" *.log
ios-sim-ews_2018-09-24_21-09.log:1212:    raise RuntimeError('Timed out while waiting for {} to shut down'.format(self.udid))

[Jonathan Bedard]

(In reply to Aakash Jain from comment #1)
> This seems to be happening on all the iOS simulator bots (checked on ews121,
> ews122, ews123, ews124, ews125, ews126).
> 
> 
> e.g.:
> ews121:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for"
> *.log | wc -l
>      326
> ews122:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for"
> *.log | wc -l
>      215
> ews123:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for"
> *.log | wc -l
>       36
> ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for"
> *.log | wc -l
>      481
> ews125:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for"
> *.log | wc -l
>      285
> ews126:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for"
> *.log | wc -l
>       36
> 
> 
> Also it seems to be happening for a while, atleast for 2 months.
> 
> e.g.:
> ews124:ios-sim-ews-logs buildbot$ grep -rn "Timed out while waiting for"
> *.log
> ios-sim-ews_2018-09-24_21-09.log:1212:    raise RuntimeError('Timed out
> while waiting for {} to shut down'.format(self.udid))

These logs alone aren't really actionable. How often is this happening and do we have spindumps or sysdiagnoses from any of these runs?

Basically what's happening here is that we're getting stuck when shutting down one of our managed simulators. This is especially strange since this simulator was (apparently) healthy enough to actually run tests.

[Jonathan Bedard]

Side note for future diagnosis: according to <https://webkit-queues.webkit.org/results/10084446>, this is definitely one of the managed simulators. We used 10 simulators, so the simulator which failed to shutdown was booted by the instance of run-webkit-tests which failed to shut it down. In other words, the simulator in question WAS NOT a stale simulator from a previous run.

[Jonathan Bedard]

I think I stumbled upon a solution to this while working on some related simulator/device stuff.

[Jonathan Bedard]

Created attachment 356028 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 356028 [details]
Patch

Clearing flags on attachment: 356028

Committed r238684: <https://trac.webkit.org/changeset/238684>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/46350950>