Bug 229994 – Exception in run-webkit-tests: Bad file descriptor

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

Exception in run-webkit-tests: Bad file descriptor
e.g.: https://build.webkit.org/#/builders/45/builds/2047/steps/11/logs/stdio

OSError raised: [Errno 9] Bad file descriptor (from worker/2)
Traceback (most recent call last):
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main
    run_details = run(port, options, args, stderr)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 494, in run
    run_details = manager.run(args)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run
    temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset
    initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests
    return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 179, in run_tests
    pool.wait()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 423, in wait
    self.queue.receive()(self)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 133, in __call__
    reraise(*self.exc_info)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/autoinstalled/python-3/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 296, in main
    queue.send(_Result(value=task(None), id=task.id))
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 56, in __call__
    return self.function(*self.args, **self.kwargs)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard
    return Worker.instance.run_tests(shard)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests
    Worker.instance.run_test(input, shard.name)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 358, in run_test
    self._clean_up_after_test(test_input, result)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 411, in _clean_up_after_test
    self._kill_driver()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 398, in _kill_driver
    driver.stop()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/driver.py", line 843, in stop
    self._driver.stop()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/driver.py", line 515, in stop
    self._server_process.stop(self._port.driver_stop_timeout())
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/simulator_process.py", line 140, in stop
    return self._wait_for_stop(timeout_secs)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/server_process.py", line 428, in _wait_for_stop
    self._wait_for_data_and_update_buffers_using_select(now, stopping=True)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/server_process.py", line 286, in _wait_for_data_and_update_buffers_using_select
    read_fds, _, _ = select.select(select_fds, [], select_fds, max(deadline - time.time(), 0))
OSError: [Errno 9] Bad file descriptor (from worker/2)

[Aakash Jain]

Seems to be happening frequently on https://build.webkit.org/#/builders/45 queue

https://build.webkit.org/#/builders/45/builds/1916
https://build.webkit.org/#/builders/45/builds/2048
https://build.webkit.org/#/builders/45/builds/2040
https://build.webkit.org/#/builders/45/builds/2050

[Aakash Jain]

Similar issue on: 
https://build.webkit.org/#/builders/Apple-iOS-14-Simulator-Debug-WK2-Tests
https://build.webkit.org/#/builders/Apple-iPadOS-14-Simulator-Debug-WK2-Tests

[Aakash Jain]

Similar issue on EWS in https://ews-build.webkit.org/#/builders/51/builds/20712/steps/15/logs/stdio

[Jonathan Bedard]

Created attachment 437506 [details]
Patch

[Jonathan Bedard]

Created attachment 437507 [details]
Patch

[Aakash Jain]

Comment on attachment 437506 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=437506&action=review

> Tools/Scripts/webkitpy/port/server_process.py:287
> +        except OSError:

we should definitely log something here (maybe with some more data which might help us debug the root-cause). Ignoring the issue completely doesn't sounds good.

> Tools/Scripts/webkitpy/port/server_process.py:317
> +        except (IOError, OSError):

is this needed or just proactive?

[Jonathan Bedard]

(In reply to Aakash Jain from comment #6)
> Comment on attachment 437506 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=437506&action=review
> 
> > Tools/Scripts/webkitpy/port/server_process.py:287
> > +        except OSError:
> 
> we should definitely log something here (maybe with some more data which
> might help us debug the root-cause). Ignoring the issue completely doesn't
> sounds good.

I actually think ignoring is the right move because of the comment right bellow:

# We can ignore EINVAL since it's likely the process just crashed and we'll
# figure that out the next time through the loop in _read().

I'm pretty sure that's what's happening here.

> 
> > Tools/Scripts/webkitpy/port/server_process.py:317
> > +        except (IOError, OSError):
> 
> is this needed or just proactive?

Just proactive.

[Jonathan Bedard]

Created attachment 437511 [details]
Patch

[Aakash Jain]

Comment on attachment 437511 [details]
Patch

rs=me

[EWS]

Committed r282091 (241391@main): <https://commits.webkit.org/241391@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437511 [details].

[Radar WebKit Bug Importer]

<rdar://problem/82826083>

[Jonathan Bedard]

Reopening to attach new patch.

[Jonathan Bedard]

Created attachment 437664 [details]
Patch

[Jonathan Bedard]

(In reply to Jonathan Bedard from comment #13)
> Created attachment 437664 [details]
> Patch

An effort to address https://build.webkit.org/#/builders/45/builds/2070

[Aakash Jain]

Comment on attachment 437664 [details]
Patch

I don't feel comfortable with hiding issue while launching the simulator itself. Let's try it out, but let's not make it a permanent fix.

[Jonathan Bedard]

Created attachment 437683 [details]
Patch for landing

[Jonathan Bedard]

(In reply to Aakash Jain from comment #15)
> Comment on attachment 437664 [details]
> Patch
> 
> I don't feel comfortable with hiding issue while launching the simulator
> itself. Let's try it out, but let's not make it a permanent fix.

We're not exactly hiding the issue, or rather, we're noting hiding the issue any more than we already hide similar issues. Basically, if we fail to launch an app for too much time, we will raise an exception. I'm assuming that our file descriptor errors are essentially the same as a failed-to-launch-app error.

[EWS]

Committed r282185 (241474@main): <https://commits.webkit.org/241474@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437683 [details].

[Jonathan Bedard]

Reopening to attach new patch.

[Jonathan Bedard]

Created attachment 437757 [details]
Patch

[Jonathan Bedard]

(In reply to Jonathan Bedard from comment #20)
> Created attachment 437757 [details]
> Patch

An effort to address https://build.webkit.org/#/builders/45/builds/2083

[Ryan Haddad]

Comment on attachment 437757 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=437757&action=review

> Tools/Scripts/webkitpy/common/system/filesystem.py:296
> +            pass

Looking beyond the issue seen with simulators, can this unexpectedly bite us in other ways?

[Jonathan Bedard]

(In reply to Ryan Haddad from comment #22)
> Comment on attachment 437757 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=437757&action=review
> 
> > Tools/Scripts/webkitpy/common/system/filesystem.py:296
> > +            pass
> 
> Looking beyond the issue seen with simulators, can this unexpectedly bite us
> in other ways?

I don't think we ever expect rmtree with ignore_errors=True to ever raise an exception, the fact that it does is probably a Python bug, but I'm not sure we can give a good enough reproduction for that.

[Jonathan Bedard]

Created attachment 437868 [details]
Patch for landing

[EWS]

Committed r282268 (241546@main): <https://commits.webkit.org/241546@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437868 [details].

[Jonathan Bedard]

Reopening to attach new patch.

[Jonathan Bedard]

Created attachment 438367 [details]
Patch

[Jonathan Bedard]

(In reply to Jonathan Bedard from comment #27)
> Created attachment 438367 [details]
> Patch

Targeting issues like this: https://build.webkit.org/#/builders/45/builds/2142

[EWS]

Committed r282625 (241782@main): <https://commits.webkit.org/241782@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 438367 [details].

[Jonathan Bedard]

Reopening to attach new patch.

[Jonathan Bedard]

Created attachment 439232 [details]
Patch

[Jonathan Bedard]

Comment on attachment 439232 [details]
Patch

Reverting everything I landed to attempt to work around this problem, this is a regression due to Python 3.

While we try and figure out the root cause, I'm going to revert run-webkit-tests back to Python 2.

[EWS]

Committed r283126 (242184@main): <https://commits.webkit.org/242184@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 439232 [details].