Bug 229994

Summary:

Exception in run-webkit-tests: Bad file descriptor

Product:

WebKit

Reporter:

Aakash Jain <aakash_jain>

Component:

Tools / Tests

Assignee:

Jonathan Bedard <jbedard>

Status:

RESOLVED FIXED

Severity:

Normal

CC:

aakash_jain, ap, dewei_zhu, ews-watchlist, glenn, jbedard, ryanhaddad, slewis, tsavell, webkit-bug-importer

Priority:

Keywords:

InRadar

Version:

WebKit Nightly Build

Hardware:

Unspecified

OS:

Unspecified

See Also:

https://bugs.webkit.org/show_bug.cgi?id=230513

Attachments:

Description	Flags
Patch	none
Patch	none
Patch	none
Patch	none
Patch for landing	none
Patch	none
Patch for landing	none
Patch	none
Patch	none

Description Aakash Jain 2021-09-07 04:16:41 PDT

Exception in run-webkit-tests: Bad file descriptor
e.g.: https://build.webkit.org/#/builders/45/builds/2047/steps/11/logs/stdio

OSError raised: [Errno 9] Bad file descriptor (from worker/2)
Traceback (most recent call last):
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main
    run_details = run(port, options, args, stderr)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 494, in run
    run_details = manager.run(args)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run
    temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset
    initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests
    return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 179, in run_tests
    pool.wait()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 423, in wait
    self.queue.receive()(self)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 133, in __call__
    reraise(*self.exc_info)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/autoinstalled/python-3/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 296, in main
    queue.send(_Result(value=task(None), id=task.id))
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 56, in __call__
    return self.function(*self.args, **self.kwargs)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard
    return Worker.instance.run_tests(shard)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests
    Worker.instance.run_test(input, shard.name)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 358, in run_test
    self._clean_up_after_test(test_input, result)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 411, in _clean_up_after_test
    self._kill_driver()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 398, in _kill_driver
    driver.stop()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/driver.py", line 843, in stop
    self._driver.stop()
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/driver.py", line 515, in stop
    self._server_process.stop(self._port.driver_stop_timeout())
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/simulator_process.py", line 140, in stop
    return self._wait_for_stop(timeout_secs)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/server_process.py", line 428, in _wait_for_stop
    self._wait_for_data_and_update_buffers_using_select(now, stopping=True)
  File "/Volumes/Data/worker/ipados-simulator-14-release-tests-wk2/build/Tools/Scripts/webkitpy/port/server_process.py", line 286, in _wait_for_data_and_update_buffers_using_select
    read_fds, _, _ = select.select(select_fds, [], select_fds, max(deadline - time.time(), 0))
OSError: [Errno 9] Bad file descriptor (from worker/2)

Comment 1 Aakash Jain 2021-09-07 04:17:02 PDT

Seems to be happening frequently on https://build.webkit.org/#/builders/45 queue

https://build.webkit.org/#/builders/45/builds/1916
https://build.webkit.org/#/builders/45/builds/2048
https://build.webkit.org/#/builders/45/builds/2040
https://build.webkit.org/#/builders/45/builds/2050

Comment 2 Aakash Jain 2021-09-07 04:20:51 PDT

Similar issue on: 
https://build.webkit.org/#/builders/Apple-iOS-14-Simulator-Debug-WK2-Tests
https://build.webkit.org/#/builders/Apple-iPadOS-14-Simulator-Debug-WK2-Tests

Comment 3 Aakash Jain 2021-09-07 05:29:55 PDT

Similar issue on EWS in https://ews-build.webkit.org/#/builders/51/builds/20712/steps/15/logs/stdio

Comment 4 Jonathan Bedard 2021-09-07 08:16:03 PDT

Created attachment 437506 [details]
Patch

Comment 5 Jonathan Bedard 2021-09-07 08:17:20 PDT

Created attachment 437507 [details]
Patch

Comment 6 Aakash Jain 2021-09-07 08:20:18 PDT

Comment on attachment 437506 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=437506&action=review

> Tools/Scripts/webkitpy/port/server_process.py:287
> +        except OSError:

we should definitely log something here (maybe with some more data which might help us debug the root-cause). Ignoring the issue completely doesn't sounds good.

> Tools/Scripts/webkitpy/port/server_process.py:317
> +        except (IOError, OSError):

is this needed or just proactive?

Comment 7 Jonathan Bedard 2021-09-07 08:22:43 PDT

(In reply to Aakash Jain from comment #6)
> Comment on attachment 437506 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=437506&action=review
> 
> > Tools/Scripts/webkitpy/port/server_process.py:287
> > +        except OSError:
> 
> we should definitely log something here (maybe with some more data which
> might help us debug the root-cause). Ignoring the issue completely doesn't
> sounds good.

I actually think ignoring is the right move because of the comment right bellow:

# We can ignore EINVAL since it's likely the process just crashed and we'll
# figure that out the next time through the loop in _read().

I'm pretty sure that's what's happening here.

> 
> > Tools/Scripts/webkitpy/port/server_process.py:317
> > +        except (IOError, OSError):
> 
> is this needed or just proactive?

Just proactive.

Comment 8 Jonathan Bedard 2021-09-07 08:33:51 PDT

Created attachment 437511 [details]
Patch

Comment 9 Aakash Jain 2021-09-07 08:36:50 PDT

Comment on attachment 437511 [details]
Patch

rs=me

Comment 10 EWS 2021-09-07 09:37:40 PDT

Committed r282091 (241391@main): <https://commits.webkit.org/241391@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437511 [details].

Comment 11 Radar WebKit Bug Importer 2021-09-07 09:38:32 PDT

<rdar://problem/82826083>

Comment 12 Jonathan Bedard 2021-09-08 14:19:15 PDT

Reopening to attach new patch.

Comment 13 Jonathan Bedard 2021-09-08 14:19:17 PDT

Created attachment 437664 [details]
Patch

Comment 14 Jonathan Bedard 2021-09-08 16:27:20 PDT

(In reply to Jonathan Bedard from comment #13)
> Created attachment 437664 [details]
> Patch

An effort to address https://build.webkit.org/#/builders/45/builds/2070

Comment 15 Aakash Jain 2021-09-08 16:42:12 PDT

Comment on attachment 437664 [details]
Patch

I don't feel comfortable with hiding issue while launching the simulator itself. Let's try it out, but let's not make it a permanent fix.

Comment 16 Jonathan Bedard 2021-09-08 16:52:03 PDT

Created attachment 437683 [details]
Patch for landing

Comment 17 Jonathan Bedard 2021-09-08 16:54:23 PDT

(In reply to Aakash Jain from comment #15)
> Comment on attachment 437664 [details]
> Patch
> 
> I don't feel comfortable with hiding issue while launching the simulator
> itself. Let's try it out, but let's not make it a permanent fix.

We're not exactly hiding the issue, or rather, we're noting hiding the issue any more than we already hide similar issues. Basically, if we fail to launch an app for too much time, we will raise an exception. I'm assuming that our file descriptor errors are essentially the same as a failed-to-launch-app error.

Comment 18 EWS 2021-09-08 17:20:26 PDT

Committed r282185 (241474@main): <https://commits.webkit.org/241474@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437683 [details].

Comment 19 Jonathan Bedard 2021-09-09 10:22:11 PDT

Reopening to attach new patch.

Comment 20 Jonathan Bedard 2021-09-09 10:22:12 PDT

Created attachment 437757 [details]
Patch

Comment 21 Jonathan Bedard 2021-09-09 10:35:01 PDT

(In reply to Jonathan Bedard from comment #20)
> Created attachment 437757 [details]
> Patch

An effort to address https://build.webkit.org/#/builders/45/builds/2083

Comment 22 Ryan Haddad 2021-09-09 14:24:38 PDT

Comment on attachment 437757 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=437757&action=review

> Tools/Scripts/webkitpy/common/system/filesystem.py:296
> +            pass

Looking beyond the issue seen with simulators, can this unexpectedly bite us in other ways?

Comment 23 Jonathan Bedard 2021-09-09 16:13:45 PDT

(In reply to Ryan Haddad from comment #22)
> Comment on attachment 437757 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=437757&action=review
> 
> > Tools/Scripts/webkitpy/common/system/filesystem.py:296
> > +            pass
> 
> Looking beyond the issue seen with simulators, can this unexpectedly bite us
> in other ways?

I don't think we ever expect rmtree with ignore_errors=True to ever raise an exception, the fact that it does is probably a Python bug, but I'm not sure we can give a good enough reproduction for that.

Comment 24 Jonathan Bedard 2021-09-10 07:32:16 PDT

Created attachment 437868 [details]
Patch for landing

Comment 25 EWS 2021-09-10 08:07:58 PDT

Committed r282268 (241546@main): <https://commits.webkit.org/241546@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437868 [details].

Comment 26 Jonathan Bedard 2021-09-16 10:37:45 PDT

Reopening to attach new patch.

Comment 27 Jonathan Bedard 2021-09-16 10:37:47 PDT

Created attachment 438367 [details]
Patch

Comment 28 Jonathan Bedard 2021-09-16 10:38:37 PDT

(In reply to Jonathan Bedard from comment #27)
> Created attachment 438367 [details]
> Patch

Targeting issues like this: https://build.webkit.org/#/builders/45/builds/2142

Comment 29 EWS 2021-09-16 18:16:12 PDT

Committed r282625 (241782@main): <https://commits.webkit.org/241782@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 438367 [details].

Comment 30 Jonathan Bedard 2021-09-24 20:33:56 PDT

Reopening to attach new patch.

Comment 31 Jonathan Bedard 2021-09-24 20:33:59 PDT

Created attachment 439232 [details]
Patch

Comment 32 Jonathan Bedard 2021-09-27 10:00:15 PDT

Comment on attachment 439232 [details]
Patch

Reverting everything I landed to attempt to work around this problem, this is a regression due to Python 3.

While we try and figure out the root cause, I'm going to revert run-webkit-tests back to Python 2.

Comment 33 EWS 2021-09-27 10:18:20 PDT

Committed r283126 (242184@main): <https://commits.webkit.org/242184@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 439232 [details].