248533 2022-11-30 06:47:05 -0800 [GTK][WPE][run-webkit-tests] layout tests are deadlocking when a test crashes due to unhandled BrokenPipeError 2022-12-07 20:39:37 -0800 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=247843 https://bugs.webkit.org/show_bug.cgi?id=248904 InRadar P2 Normal --- 1 clopez clopez bugs-noreply dpino lmoura webkit-bug-importer oldest_to_newest 1915599 0 clopez 2022-11-30 06:47:05 -0800 We are having lots of deadlocks in the test runs recently. 1) Check the EWS queue: https://ews-build.webkit.org/#/builders/35 Many of them are ending with "Unexpected infrastructure issue: The layout-test run with change generated no list of results and exited with error, retrying with the hope it was a random infrastructure error." that is caused because the layout test steps deadlocks and the timeout of the step triggers 2) Same for the post-commit bots. Example WPE release tests: https://build.webkit.org/#/builders/40 The layout test is not finishing, deadlocks I have been debugging this and if you run the tests with only one worker (NUMBER_OF_PROCESSORS=1) instead of the deadlock you get this backtrace: BrokenPipeError raised: [Errno 32] Broken pipe Traceback (most recent call last): File "/app/webkit/Tools/Scripts/webkitpy/port/server_process.py", line 186, in write self._proc.stdin.flush() BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main run_details = run(port, options, args, stderr) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 508, in run run_details = manager.run(args) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 570, in _run_tests return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 176, in run_tests pool.do( File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 405, in do result = function(*args, **kwargs) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 77, in run_shard return Worker.instance.run_tests(shard) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 319, in run_tests Worker.instance.run_test(input, shard.name) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 347, in run_test result = self._run_test_with_or_without_timeout(test_input, test_timeout_sec, stop_when_done) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 404, in _run_test_with_or_without_timeout return self._run_test_in_this_thread(test_input, stop_when_done) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 490, in _run_test_in_this_thread return self._run_single_test(self._driver, test_input, stop_when_done) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 493, in _run_single_test return single_test_runner.run_single_test( File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 48, in run_single_test return runner.run() File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 128, in run return self._run_compare_test() File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 131, in _run_compare_test driver_output = self._driver.run_test(self._driver_input(), self._stop_when_done) File "/app/webkit/Tools/Scripts/webkitpy/port/driver.py", line 863, in run_test return self._driver.run_test(driver_input, stop_when_done) File "/app/webkit/Tools/Scripts/webkitpy/port/driver.py", line 233, in run_test self._server_process.write(command) File "/app/webkit/Tools/Scripts/webkitpy/port/server_process.py", line 188, in write self.stop(0.0) File "/app/webkit/Tools/Scripts/webkitpy/port/server_process.py", line 403, in stop self._proc.stdin.close() BrokenPipeError: [Errno 32] Broken pipe The issue seems to be we are not handling correctly the BrokenPipeError() when a test crashes. Not sure what change lead to this but maybe it was a Python upgrade in the Flatpak SDK We are running Python 3.10.6 right now 1915601 1 clopez 2022-11-30 07:01:23 -0800 Pull request: https://github.com/WebKit/WebKit/pull/6970 1915644 2 ews-feeder 2022-11-30 10:59:26 -0800 Committed 257186@main (025ed0ce9ed6): <https://commits.webkit.org/257186@main> Reviewed commits have been landed. Closing PR #6970 and removing active labels. 1915646 3 webkit-bug-importer 2022-11-30 11:00:20 -0800 <rdar://problem/102820603>