NEW239780
A python error causes results.html to be missing from layout test results 
https://bugs.webkit.org/show_bug.cgi?id=239780
Summary A python error causes results.html to be missing from layout test results
Simon Fraser (smfr)
Reported 2022-04-26 11:40:12 PDT
This EWS run <https://ews-build.webkit.org/#/builders/68/builds/13819/steps/17/logs/stdio> shows an error shutting down the simulator: 10:08:27.729 681 Removing device 'Managed 7' 10:08:27.897 681 "/usr/bin/xcrun simctl delete 3882575F-7118-4051-9D0F-105AEEC707AE" took 0.17s TypeError raised: 'NoneType' object is not subscriptable Traceback (most recent call last): File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main run_details = run(port, options, args, stderr) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 495, in run run_details = manager.run(args) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 506, in _run_test_subset retry_results = self._run_tests(retry_test_inputs, File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 175, in run_tests pool.do( File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 400, in do result = function(*args, **kwargs) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard return Worker.instance.run_tests(shard) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests Worker.instance.run_test(input, shard.name) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 346, in run_test result = self._run_test_with_or_without_timeout(test_input, test_timeout_sec, stop_when_done) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 403, in _run_test_with_or_without_timeout return self._run_test_in_this_thread(test_input, stop_when_done) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 489, in _run_test_in_this_thread return self._run_single_test(self._driver, test_input, stop_when_done) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 492, in _run_single_test return single_test_runner.run_single_test( File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 48, in run_single_test return runner.run() File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 123, in run return self._run_reftest() File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 340, in _run_reftest test_result = self._compare_output_with_reference(reference_output, test_output, reference_filename, expectation == '!=') File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 446, in _compare_output_with_reference diff_result.passed = self._test_passes_fuzzy_matching(fuzzy_tolerance, diff_result.fuzzy_data) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 401, in _test_passes_fuzzy_matching actualMaxDifference = fuzzy_result['max_difference'] TypeError: 'NoneType' object is not subscriptable 10:08:28.673 681 Stopping HTTP server ... 10:08:28.929 681 Stopping WebSocket server ... 10:08:29.139 681 Stopping Web Platform Test server ... program finished with exit code 254 and this means that the script fails to generate results.html, so the entire run is wasted.
Attachments
Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2022-05-03 11:41:15 PDT
<rdar://problem/92683094>
Jonathan Bedard
Comment 2 2022-05-03 13:28:49 PDT
That doesn't look like a failure shutting down simulators, that's a failure in one of our child processes in Python code when retrying, this bit is probably the key: EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 401, in _test_passes_fuzzy_matching actualMaxDifference = fuzzy_result['max_difference'] TypeError: 'NoneType' object is not subscriptable
Simon Fraser (smfr)
Comment 3 2022-05-03 13:39:59 PDT
Ah I missed that. I wonder why this just happens sometimes.
Note You need to log in before you can comment on or make changes to this bug.