Bug 229994 - Exception in run-webkit-tests: Bad file descriptor
Summary: Exception in run-webkit-tests: Bad file descriptor
Status: REOPENED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Jonathan Bedard
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-09-07 04:16 PDT by Aakash Jain
Modified: 2021-09-24 20:33 PDT (History)
10 users (show)

See Also:


Attachments
Patch (2.42 KB, patch)
2021-09-07 08:16 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (2.43 KB, patch)
2021-09-07 08:17 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (2.71 KB, patch)
2021-09-07 08:33 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (2.25 KB, patch)
2021-09-08 14:19 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch for landing (2.32 KB, patch)
2021-09-08 16:52 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (1.41 KB, patch)
2021-09-09 10:22 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch for landing (1.43 KB, patch)
2021-09-10 07:32 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (4.57 KB, patch)
2021-09-16 10:37 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (9.04 KB, patch)
2021-09-24 20:33 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aakash Jain 2021-09-07 04:16:41 PDT
Exception in run-webkit-tests: Bad file descriptor
e.g.: https://build.webkit.org/#/builders/45/builds/2047/steps/11/logs/stdio

OSError raised: [Errno 9] Bad file descriptor (from worker/2)
Traceback (most recent call last):
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main
    run_details = run(port, options, args, stderr)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 494, in run
    run_details = manager.run(args)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run
    temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset
    initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests
    return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 179, in run_tests
    pool.wait()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 423, in wait
    self.queue.receive()(self)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 133, in __call__
    reraise(*self.exc_info)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/autoinstalled/python-3/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 296, in main
    queue.send(_Result(value=task(None), id=task.id))
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 56, in __call__
    return self.function(*self.args, **self.kwargs)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard
    return Worker.instance.run_tests(shard)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests
    Worker.instance.run_test(input, shard.name)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 358, in run_test
    self._clean_up_after_test(test_input, result)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 411, in _clean_up_after_test
    self._kill_driver()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 398, in _kill_driver
    driver.stop()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/driver.py", line 843, in stop
    self._driver.stop()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/driver.py", line 515, in stop
    self._server_process.stop(self._port.driver_stop_timeout())
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/simulator_process.py", line 140, in stop
    return self._wait_for_stop(timeout_secs)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/server_process.py", line 428, in _wait_for_stop
    self._wait_for_data_and_update_buffers_using_select(now, stopping=True)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/server_process.py", line 286, in _wait_for_data_and_update_buffers_using_select
    read_fds, _, _ = select.select(select_fds, [], select_fds, max(deadline - time.time(), 0))
OSError: [Errno 9] Bad file descriptor (from worker/2)
Comment 3 Aakash Jain 2021-09-07 05:29:55 PDT
Similar issue on EWS in https://ews-build.webkit.org/#/builders/51/builds/20712/steps/15/logs/stdio
Comment 4 Jonathan Bedard 2021-09-07 08:16:03 PDT
Created attachment 437506 [details]
Patch
Comment 5 Jonathan Bedard 2021-09-07 08:17:20 PDT
Created attachment 437507 [details]
Patch
Comment 6 Aakash Jain 2021-09-07 08:20:18 PDT
Comment on attachment 437506 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=437506&action=review

> Tools/Scripts/webkitpy/port/server_process.py:287
> +        except OSError:

we should definitely log something here (maybe with some more data which might help us debug the root-cause). Ignoring the issue completely doesn't sounds good.

> Tools/Scripts/webkitpy/port/server_process.py:317
> +        except (IOError, OSError):

is this needed or just proactive?
Comment 7 Jonathan Bedard 2021-09-07 08:22:43 PDT
(In reply to Aakash Jain from comment #6)
> Comment on attachment 437506 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=437506&action=review
> 
> > Tools/Scripts/webkitpy/port/server_process.py:287
> > +        except OSError:
> 
> we should definitely log something here (maybe with some more data which
> might help us debug the root-cause). Ignoring the issue completely doesn't
> sounds good.

I actually think ignoring is the right move because of the comment right bellow:

# We can ignore EINVAL since it's likely the process just crashed and we'll
# figure that out the next time through the loop in _read().

I'm pretty sure that's what's happening here.

> 
> > Tools/Scripts/webkitpy/port/server_process.py:317
> > +        except (IOError, OSError):
> 
> is this needed or just proactive?

Just proactive.
Comment 8 Jonathan Bedard 2021-09-07 08:33:51 PDT
Created attachment 437511 [details]
Patch
Comment 9 Aakash Jain 2021-09-07 08:36:50 PDT
Comment on attachment 437511 [details]
Patch

rs=me
Comment 10 EWS 2021-09-07 09:37:40 PDT
Committed r282091 (241391@main): <https://commits.webkit.org/241391@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437511 [details].
Comment 11 Radar WebKit Bug Importer 2021-09-07 09:38:32 PDT
<rdar://problem/82826083>
Comment 12 Jonathan Bedard 2021-09-08 14:19:15 PDT
Reopening to attach new patch.
Comment 13 Jonathan Bedard 2021-09-08 14:19:17 PDT
Created attachment 437664 [details]
Patch
Comment 14 Jonathan Bedard 2021-09-08 16:27:20 PDT
(In reply to Jonathan Bedard from comment #13)
> Created attachment 437664 [details]
> Patch

An effort to address https://build.webkit.org/#/builders/45/builds/2070
Comment 15 Aakash Jain 2021-09-08 16:42:12 PDT
Comment on attachment 437664 [details]
Patch

I don't feel comfortable with hiding issue while launching the simulator itself. Let's try it out, but let's not make it a permanent fix.
Comment 16 Jonathan Bedard 2021-09-08 16:52:03 PDT
Created attachment 437683 [details]
Patch for landing
Comment 17 Jonathan Bedard 2021-09-08 16:54:23 PDT
(In reply to Aakash Jain from comment #15)
> Comment on attachment 437664 [details]
> Patch
> 
> I don't feel comfortable with hiding issue while launching the simulator
> itself. Let's try it out, but let's not make it a permanent fix.

We're not exactly hiding the issue, or rather, we're noting hiding the issue any more than we already hide similar issues. Basically, if we fail to launch an app for too much time, we will raise an exception. I'm assuming that our file descriptor errors are essentially the same as a failed-to-launch-app error.
Comment 18 EWS 2021-09-08 17:20:26 PDT
Committed r282185 (241474@main): <https://commits.webkit.org/241474@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437683 [details].
Comment 19 Jonathan Bedard 2021-09-09 10:22:11 PDT
Reopening to attach new patch.
Comment 20 Jonathan Bedard 2021-09-09 10:22:12 PDT
Created attachment 437757 [details]
Patch
Comment 21 Jonathan Bedard 2021-09-09 10:35:01 PDT
(In reply to Jonathan Bedard from comment #20)
> Created attachment 437757 [details]
> Patch

An effort to address https://build.webkit.org/#/builders/45/builds/2083
Comment 22 Ryan Haddad 2021-09-09 14:24:38 PDT
Comment on attachment 437757 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=437757&action=review

> Tools/Scripts/webkitpy/common/system/filesystem.py:296
> +            pass

Looking beyond the issue seen with simulators, can this unexpectedly bite us in other ways?
Comment 23 Jonathan Bedard 2021-09-09 16:13:45 PDT
(In reply to Ryan Haddad from comment #22)
> Comment on attachment 437757 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=437757&action=review
> 
> > Tools/Scripts/webkitpy/common/system/filesystem.py:296
> > +            pass
> 
> Looking beyond the issue seen with simulators, can this unexpectedly bite us
> in other ways?

I don't think we ever expect rmtree with ignore_errors=True to ever raise an exception, the fact that it does is probably a Python bug, but I'm not sure we can give a good enough reproduction for that.
Comment 24 Jonathan Bedard 2021-09-10 07:32:16 PDT
Created attachment 437868 [details]
Patch for landing
Comment 25 EWS 2021-09-10 08:07:58 PDT
Committed r282268 (241546@main): <https://commits.webkit.org/241546@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437868 [details].
Comment 26 Jonathan Bedard 2021-09-16 10:37:45 PDT
Reopening to attach new patch.
Comment 27 Jonathan Bedard 2021-09-16 10:37:47 PDT
Created attachment 438367 [details]
Patch
Comment 28 Jonathan Bedard 2021-09-16 10:38:37 PDT
(In reply to Jonathan Bedard from comment #27)
> Created attachment 438367 [details]
> Patch

Targeting issues like this: https://build.webkit.org/#/builders/45/builds/2142
Comment 29 EWS 2021-09-16 18:16:12 PDT
Committed r282625 (241782@main): <https://commits.webkit.org/241782@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 438367 [details].
Comment 30 Jonathan Bedard 2021-09-24 20:33:56 PDT
Reopening to attach new patch.
Comment 31 Jonathan Bedard 2021-09-24 20:33:59 PDT
Created attachment 439232 [details]
Patch