Bug 234487 – Running layout tests sometimes throws an AssertionError in python2.7/multiprocessing

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Simon Fraser (smfr)]

On 21D29a, iMac Pro, webkit r287227:

OpenSource % wkquicktests LayoutTests/fast/scrolling 
Using port 'mac-monterey-wk2'
Test configuration: <monterey, x86_64, debug>
Placing test results in /Volumes/Data/Development/system/webkit/OpenSource/WebKitBuild/Debug/layout-test-results
Using Debug build
Pixel tests disabled
Regular timeout: 30000, slow test timeout: 150000
Command line: /Volumes/Data/Development/system/webkit/OpenSource/WebKitBuild/Debug/WebKitTestRunner -

Found 317 tests; running 203, skipping 114.
                  
Verbose baseline search path: platform/mac-monterey-wk2 -> platform/mac-monterey -> platform/mac-wk2 -> platform/mac -> platform/wk2 -> generic

Baseline search path: platform/mac-wk2 -> platform/mac -> platform/wk2 -> generic

Running 203 tests

Running 4 WebKitTestRunners in parallel.

worker/0 starting     
worker/1 starting
worker/2 starting
worker/3 starting
worker/3 stopping                                                                             
worker/1 stopping                                                                                               
worker/2 stopping                                                                                      
worker/0 stopping                                                                                              

AssertionError raised: 
Traceback (most recent call last):
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main
    run_details = run(port, options, args, stderr)
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 495, in run
    run_details = manager.run(args)
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run
    temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type)
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset
    initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type)
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests
    return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type)
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 172, in run_tests
    ), teardown=teardown_shard,
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 388, in __enter__
    worker.start()
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler
    with StackTraceFileContext(output_file=output_file) as file:
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 55, in __enter__
    _log.critical('Stack trace saved to {}'.format(self.file_name))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1219, in critical
    self._log(CRITICAL, msg, args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1293, in _log
    self.handle(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1303, in handle
    self.callHandlers(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1343, in callHandlers
    hdlr.handle(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 766, in handle
    self.emit(record)
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 186, in emit
    self._queue.send(_Log(record))
  File "/Volumes/Data/Development/system/webkit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 142, in send
    return self.outgoing.put(object)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 100, in put
    assert not self._closed
AssertionError
All 203 tests ran as expected.

[Radar WebKit Bug Importer]

<rdar://problem/86919726>

[Jonathan Bedard]

Created attachment 448294 [details]
Patch

[Jonathan Bedard]

So, this is because we have a function triggered on process exit, which looks like it's happening after we tear down the message queues. I think the answer is to log an error when we encounter this problem instead of throwing an exception.

We might be able to trigger the callbacks early, but I'm a bit concerned about our ability to do that reliably.

[EWS]

Committed r287614 (245741@main): <https://commits.webkit.org/245741@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 448294 [details].