239780 2022-04-26 11:40:12 -0700 A python error causes results.html to be missing from layout test results 2023-04-25 18:50:57 -0700 1 1 1 Unclassified WebKit Tools / Tests Safari Technology Preview Unspecified Unspecified NEW https://bugs.webkit.org/show_bug.cgi?id=255873 InRadar P2 Normal --- 1 simon.fraser jbedard aakash_jain ap jbedard simon.fraser webkit-bug-importer oldest_to_newest 1864543 0 simon.fraser 2022-04-26 11:40:12 -0700 This EWS run <https://ews-build.webkit.org/#/builders/68/builds/13819/steps/17/logs/stdio> shows an error shutting down the simulator: 10:08:27.729 681 Removing device 'Managed 7' 10:08:27.897 681 "/usr/bin/xcrun simctl delete 3882575F-7118-4051-9D0F-105AEEC707AE" took 0.17s TypeError raised: 'NoneType' object is not subscriptable Traceback (most recent call last): File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main run_details = run(port, options, args, stderr) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 495, in run run_details = manager.run(args) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 506, in _run_test_subset retry_results = self._run_tests(retry_test_inputs, File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 175, in run_tests pool.do( File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 400, in do result = function(*args, **kwargs) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard return Worker.instance.run_tests(shard) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests Worker.instance.run_test(input, shard.name) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 346, in run_test result = self._run_test_with_or_without_timeout(test_input, test_timeout_sec, stop_when_done) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 403, in _run_test_with_or_without_timeout return self._run_test_in_this_thread(test_input, stop_when_done) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 489, in _run_test_in_this_thread return self._run_single_test(self._driver, test_input, stop_when_done) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 492, in _run_single_test return single_test_runner.run_single_test( File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 48, in run_single_test return runner.run() File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 123, in run return self._run_reftest() File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 340, in _run_reftest test_result = self._compare_output_with_reference(reference_output, test_output, reference_filename, expectation == '!=') File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 446, in _compare_output_with_reference diff_result.passed = self._test_passes_fuzzy_matching(fuzzy_tolerance, diff_result.fuzzy_data) File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 401, in _test_passes_fuzzy_matching actualMaxDifference = fuzzy_result['max_difference'] TypeError: 'NoneType' object is not subscriptable 10:08:28.673 681 Stopping HTTP server ... 10:08:28.929 681 Stopping WebSocket server ... 10:08:29.139 681 Stopping Web Platform Test server ... program finished with exit code 254 and this means that the script fails to generate results.html, so the entire run is wasted. 1866202 1 webkit-bug-importer 2022-05-03 11:41:15 -0700 <rdar://problem/92683094> 1866218 2 jbedard 2022-05-03 13:28:49 -0700 That doesn't look like a failure shutting down simulators, that's a failure in one of our child processes in Python code when retrying, this bit is probably the key: EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 401, in _test_passes_fuzzy_matching actualMaxDifference = fuzzy_result['max_difference'] TypeError: 'NoneType' object is not subscriptable 1866224 3 simon.fraser 2022-05-03 13:39:59 -0700 Ah I missed that. I wonder why this just happens sometimes.