RESOLVED CONFIGURATION CHANGED 243989
Tools/Scripts/test-webkitpy-python2 is flaky (Cannot send message to closed queue) 
https://bugs.webkit.org/show_bug.cgi?id=243989
Summary Tools/Scripts/test-webkitpy-python2 is flaky (Cannot send message to closed q...
Kimmo Kinnunen
Reported 2022-08-16 05:33:35 PDT
Tools/Scripts/test-webkitpy-python2 is flaky (Cannot send message to closed queue) Blocked my commit: https://ews-build.webkit.org/#/builders/5/builds/89742 Local runs produce flaky results: 1. success 2. repro of the Cannot send message to closed queue 3. exception but success kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2 Suppressing most webkitpy logging while running unit tests. Deleting orphan *.pyc file: /Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/xcode/__pycache__/__init__.cpython-39.pyc .... Deleting orphan *.pyc file: /Users/kkinnunen/WebKit/OpenSource/Source/WebKit/Scripts/webkit/__pycache__/parser.cpython-39.pyc Skipping tests in the following modules or packages because they are really, really, slow: webkitpy.common.checkout.scm.scm_unittest (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include) [752/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_reftest_driver_should_run_expected_mismatch_html passedProcess Process-8:4: Traceback (most recent call last): File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 313, in main cls.queue.close() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 161, in close self.outgoing.join_thread() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 171, in join_thread self._jointhread() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 207, in __call__ res = self._callback(*self._args, **self._kwargs) File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 218, in _finalize_join thread.join() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 940, in join self.__block.wait() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 340, in wait waiter.acquire() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler with StackTraceFileContext(output_file=output_file) as file: File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 54, in __enter__ self.file = open(self.file_name, 'w') IOError: [Errno 2] No such file or directory: '/tmp/layout-test-results/41087-python_stack_trace.txt' [789/2279] webkitpy.layout_tests.views.metered_stream_unittest.RegularTest.test_log_after_update passedworker/0 starting worker/1 starting worker/0 stopping worker/1 stopping [810/2279] webkitpy.common.system.executive_unittest.ExecutiveTest.test_run_command_with_unicode (+8)Unknown browser mosaic. Defaulting to Cog and MiniBrowser selection Ran 2279 tests in 8.825s OK kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2 Suppressing most webkitpy logging while running unit tests. Skipping tests in the following modules or packages because they are really, really, slow: webkitpy.common.checkout.scm.scm_unittest (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include) [748/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_reftest_expected_html_should_be_ignored passedUnable to flush stdout: Broken pipe Unable to flush stdout: Broken pipe Unable to flush stdout: Broken pipe Unable to flush stdout: Broken pipe Unknown browser mosaic. Defaulting to Cog and MiniBrowser selection Unable to flush stdout: Broken pipe Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Cannot send message to closed queue Traceback (most recent call last): File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/main.py", line 366, in <module> sys.exit(main()) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/main.py", line 115, in main return not tester.run() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/main.py", line 188, in run return self._run_tests(names) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/main.py", line 228, in _run_tests test_runner.run(parallel_tests, getattr(self._options, 'child_processes', 1)) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/runner.py", line 53, in run pool.run(('test', test_name) for test_name in test_names) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/message_pool.py", line 104, in run self.wait() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/message_pool.py", line 139, in wait self._loop(block=True) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/message_pool.py", line 199, in _loop self._caller.handle(message.name, message.src, *message.args) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/runner.py", line 71, in handle self.printer.print_finished_test(source, test_name, delay, failures, errors) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/printer.py", line 149, in print_finished_test self.running_tests.remove(test_name) ValueError: list.remove(x): x not in list kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2 Suppressing most webkitpy logging while running unit tests. Skipping tests in the following modules or packages because they are really, really, slow: webkitpy.common.checkout.scm.scm_unittest (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include) [783/2279] webkitpy.performance_tests.perftest_unittest.TestPerfTest.test_parse_output_with_description passedworker/1 starting worker/0 starting worker/1 stopping worker/0 stopping [817/2279] webkitpy.performance_tests.perftest_unittest.TestSingleProcessPerfTest.test_use_only_one_process passedUnknown browser mosaic. Defaulting to Cog and MiniBrowser selection Ran 2279 tests in 8.181s OK kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2 Suppressing most webkitpy logging while running unit tests. Skipping tests in the following modules or packages because they are really, really, slow: webkitpy.common.checkout.scm.scm_unittest (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include) [732/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_ews_corner_case_skipped_test passedProcess Process-7:7: Traceback (most recent call last): File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 313, in main cls.queue.close() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 161, in close self.outgoing.join_thread() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 171, in join_thread self._jointhread() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 207, in __call__ res = self._callback(*self._args, **self._kwargs) File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 218, in _finalize_join thread.join() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 940, in join self.__block.wait() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 340, in wait waiter.acquire() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler with StackTraceFileContext(output_file=output_file) as file: File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 54, in __enter__ self.file = open(self.file_name, 'w') IOError: [Errno 2] No such file or directory: '/tmp/layout-test-results/43178-python_stack_trace.txt' [756/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RebaselineTest.test_missing_results passedworker/1 starting worker/0 starting worker/1 stopping worker/0 stopping [802/2279] webkitpy.common.system.executive_unittest.ExecutiveTest.test_run_command_with_unicode (+7)Unknown browser mosaic. Defaulting to Cog and MiniBrowser selection Ran 2279 tests in 8.286s OK kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2 Suppressing most webkitpy logging while running unit tests. Skipping tests in the following modules or packages because they are really, really, slow: webkitpy.common.checkout.scm.scm_unittest (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include) [742/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_keyboard_interrupt passedworker/1 starting worker/0 starting worker/0 stopping worker/1 stopping [827/2279] webkitpy.performance_tests.perftestsrunner_unittest.MainTest.test_parse_args passedUnknown browser mosaic. Defaulting to Cog and MiniBrowser selection Ran 2279 tests in 8.270s OK kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2 Suppressing most webkitpy logging while running unit tests. Skipping tests in the following modules or packages because they are really, really, slow: webkitpy.common.checkout.scm.scm_unittest (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include) [727/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_ews_corner_case_failing_test passedProcess Process-5:6: Traceback (most recent call last): File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 313, in main cls.queue.close() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 161, in close self.outgoing.join_thread() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 171, in join_thread self._jointhread() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 207, in __call__ res = self._callback(*self._args, **self._kwargs) File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 218, in _finalize_join thread.join() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 940, in join self.__block.wait() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 340, in wait waiter.acquire() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler with StackTraceFileContext(output_file=output_file) as file: File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 54, in __enter__ self.file = open(self.file_name, 'w') IOError: [Errno 2] No such file or directory: '/tmp/layout-test-results/44593-python_stack_trace.txt' [777/2279] webkitpy.common.system.executive_unittest.ExecutiveTest.test_run_command_with_unicode (+6)worker/0 starting worker/1 starting worker/0 stopping worker/1 stopping [836/2279] webkitpy.performance_tests.perftestsrunner_integrationtest.MainTest.test_run_crash_test passedUnknown browser mosaic. Defaulting to Cog and MiniBrowser selection Ran 2279 tests in 8.277s OK kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2 Suppressing most webkitpy logging while running unit tests. Skipping tests in the following modules or packages because they are really, really, slow: webkitpy.common.checkout.scm.scm_unittest (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include) [722/2279] webkitpy.layout_tests.models.test_run_results_unittest.SummarizedResultsTest.test_summarized_results_world_leaks_disabled passedProcess Process-5:7: Traceback (most recent call last): File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 313, in main cls.queue.close() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 161, in close self.outgoing.join_thread() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 171, in join_thread self._jointhread() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 207, in __call__ res = self._callback(*self._args, **self._kwargs) File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 218, in _finalize_join thread.join() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 940, in join self.__block.wait() File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 340, in wait waiter.acquire() File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler with StackTraceFileContext(output_file=output_file) as file: File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 54, in __enter__ self.file = open(self.file_name, 'w') IOError: [Errno 2] No such file or directory: '/tmp/layout-test-results/45397-python_stack_trace.txt' [783/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_random_order_test_specified_multiple_times passedworker/1 starting worker/0 starting worker/1 stopping worker/0 stopping [842/2279] webkitpy.performance_tests.perftestsrunner_integrationtest.MainTest.test_run_test_set_for_parser_tests passedUnknown browser mosaic. Defaulting to Cog and MiniBrowser selection Ran 2279 tests in 8.423s OK kkinnunen@mpb4-local OpenSource %
Attachments
Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2022-08-23 05:34:15 PDT
<rdar://problem/99028920>
Aakash Jain
Comment 2 2022-08-23 08:05:37 PDT
> Blocked my commit Note that webkitpy queue doesn't block PRs. Some other queue (e.g.: builder or layout-test queue) would have blocked the PR.
Sam Sneddon [:gsnedders]
Comment 3 2024-03-22 07:47:09 PDT
My guess is that one of the subprocesses is crashing (and older multiprocessing doesn't handle workers going away unexpectedly), causing this. I do wonder if we should just move to running tests with -j1 on Python 2, like we already do on Windows. Locally this is a decent perf regression, but maybe worth it for the stability: ``` % hyperfine -r1 -L children 1,8 './Tools/Scripts/test-webkitpy-python2 -j{children}' Benchmark 1: ./Tools/Scripts/test-webkitpy-python2 -j1 Time (abs ≡): 89.726 s [User: 41.462 s, System: 15.612 s] Benchmark 2: ./Tools/Scripts/test-webkitpy-python2 -j8 Time (abs ≡): 25.100 s [User: 43.881 s, System: 20.178 s] Summary ./Tools/Scripts/test-webkitpy-python2 -j8 ran 3.57 times faster than ./Tools/Scripts/test-webkitpy-python2 -j1 ``` This is especially true when the majority of PRs don't run the webkitpy tests at all, so the bot is largely free. (Note however my local machine is much faster than what we're running in EWS: the -j8 time is nearly twice as fast, but a 4x multiplier would take us from 1 minute on EWS to run the Python 2 tests to 4 minutes, which is probably still acceptable.)
Sam Sneddon [:gsnedders]
Comment 4 2024-11-16 19:17:01 PST
The script and Python 2 support is long gone.
Note You need to log in before you can comment on or make changes to this bug.