Exception in run-webkit-tests: Bad file descriptor e.g.: https://build.webkit.org/#/builders/45/builds/2047/steps/11/logs/stdio OSError raised: [Errno 9] Bad file descriptor (from worker/2) Traceback (most recent call last): File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main run_details = run(port, options, args, stderr) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 494, in run run_details = manager.run(args) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 179, in run_tests pool.wait() File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 423, in wait self.queue.receive()(self) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 133, in __call__ reraise(*self.exc_info) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/autoinstalled/python-3/six.py", line 702, in reraise raise value.with_traceback(tb) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 296, in main queue.send(_Result(value=task(None), id=task.id)) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 56, in __call__ return self.function(*self.args, **self.kwargs) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard return Worker.instance.run_tests(shard) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests Worker.instance.run_test(input, shard.name) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 358, in run_test self._clean_up_after_test(test_input, result) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 411, in _clean_up_after_test self._kill_driver() File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 398, in _kill_driver driver.stop() File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/driver.py", line 843, in stop self._driver.stop() File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/driver.py", line 515, in stop self._server_process.stop(self._port.driver_stop_timeout()) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/simulator_process.py", line 140, in stop return self._wait_for_stop(timeout_secs) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/server_process.py", line 428, in _wait_for_stop self._wait_for_data_and_update_buffers_using_select(now, stopping=True) File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/server_process.py", line 286, in _wait_for_data_and_update_buffers_using_select read_fds, _, _ = select.select(select_fds, [], select_fds, max(deadline - time.time(), 0)) OSError: [Errno 9] Bad file descriptor (from worker/2)
Seems to be happening frequently on https://build.webkit.org/#/builders/45 queue https://build.webkit.org/#/builders/45/builds/1916 https://build.webkit.org/#/builders/45/builds/2048 https://build.webkit.org/#/builders/45/builds/2040 https://build.webkit.org/#/builders/45/builds/2050
Similar issue on: https://build.webkit.org/#/builders/Apple-iOS-14-Simulator-Debug-WK2-Tests https://build.webkit.org/#/builders/Apple-iPadOS-14-Simulator-Debug-WK2-Tests
Similar issue on EWS in https://ews-build.webkit.org/#/builders/51/builds/20712/steps/15/logs/stdio
Created attachment 437506 [details] Patch
Created attachment 437507 [details] Patch
Comment on attachment 437506 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=437506&action=review > Tools/Scripts/webkitpy/port/server_process.py:287 > + except OSError: we should definitely log something here (maybe with some more data which might help us debug the root-cause). Ignoring the issue completely doesn't sounds good. > Tools/Scripts/webkitpy/port/server_process.py:317 > + except (IOError, OSError): is this needed or just proactive?
(In reply to Aakash Jain from comment #6) > Comment on attachment 437506 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=437506&action=review > > > Tools/Scripts/webkitpy/port/server_process.py:287 > > + except OSError: > > we should definitely log something here (maybe with some more data which > might help us debug the root-cause). Ignoring the issue completely doesn't > sounds good. I actually think ignoring is the right move because of the comment right bellow: # We can ignore EINVAL since it's likely the process just crashed and we'll # figure that out the next time through the loop in _read(). I'm pretty sure that's what's happening here. > > > Tools/Scripts/webkitpy/port/server_process.py:317 > > + except (IOError, OSError): > > is this needed or just proactive? Just proactive.
Created attachment 437511 [details] Patch
Comment on attachment 437511 [details] Patch rs=me
Committed r282091 (241391@main): <https://commits.webkit.org/241391@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 437511 [details].
<rdar://problem/82826083>
Reopening to attach new patch.
Created attachment 437664 [details] Patch
(In reply to Jonathan Bedard from comment #13) > Created attachment 437664 [details] > Patch An effort to address https://build.webkit.org/#/builders/45/builds/2070
Comment on attachment 437664 [details] Patch I don't feel comfortable with hiding issue while launching the simulator itself. Let's try it out, but let's not make it a permanent fix.
Created attachment 437683 [details] Patch for landing
(In reply to Aakash Jain from comment #15) > Comment on attachment 437664 [details] > Patch > > I don't feel comfortable with hiding issue while launching the simulator > itself. Let's try it out, but let's not make it a permanent fix. We're not exactly hiding the issue, or rather, we're noting hiding the issue any more than we already hide similar issues. Basically, if we fail to launch an app for too much time, we will raise an exception. I'm assuming that our file descriptor errors are essentially the same as a failed-to-launch-app error.
Committed r282185 (241474@main): <https://commits.webkit.org/241474@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 437683 [details].
Created attachment 437757 [details] Patch
(In reply to Jonathan Bedard from comment #20) > Created attachment 437757 [details] > Patch An effort to address https://build.webkit.org/#/builders/45/builds/2083
Comment on attachment 437757 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=437757&action=review > Tools/Scripts/webkitpy/common/system/filesystem.py:296 > + pass Looking beyond the issue seen with simulators, can this unexpectedly bite us in other ways?
(In reply to Ryan Haddad from comment #22) > Comment on attachment 437757 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=437757&action=review > > > Tools/Scripts/webkitpy/common/system/filesystem.py:296 > > + pass > > Looking beyond the issue seen with simulators, can this unexpectedly bite us > in other ways? I don't think we ever expect rmtree with ignore_errors=True to ever raise an exception, the fact that it does is probably a Python bug, but I'm not sure we can give a good enough reproduction for that.
Created attachment 437868 [details] Patch for landing
Committed r282268 (241546@main): <https://commits.webkit.org/241546@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 437868 [details].
Created attachment 438367 [details] Patch
(In reply to Jonathan Bedard from comment #27) > Created attachment 438367 [details] > Patch Targeting issues like this: https://build.webkit.org/#/builders/45/builds/2142
Committed r282625 (241782@main): <https://commits.webkit.org/241782@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 438367 [details].
Created attachment 439232 [details] Patch
Comment on attachment 439232 [details] Patch Reverting everything I landed to attempt to work around this problem, this is a regression due to Python 3. While we try and figure out the root cause, I'm going to revert run-webkit-tests back to Python 2.
Committed r283126 (242184@main): <https://commits.webkit.org/242184@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 439232 [details].