255873 2023-04-24 08:54:06 -0700 REGRESSION: iOS layout tests intermittently failing with TypeError: 'NoneType' object is not subscriptable in _test_passes_fuzzy_matching 2023-05-10 11:50:35 -0700 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=239780 InRadar P2 Normal --- 1 ryanhaddad ryanhaddad aakash_jain ap jbedard simon.fraser webkit-bot-watchers-bugzilla webkit-bug-importer oldest_to_newest 1950776 0 ryanhaddad 2023-04-24 08:54:06 -0700 iOS layout tests on EWS appear to be intermittently failing with the following exception: Traceback (most recent call last): File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main run_details = run(port, options, args, stderr) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 522, in run run_details = manager.run(args) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 570, in _run_tests return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 180, in run_tests pool.wait() File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 432, in wait self.queue.receive()(self) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 134, in __call__ reraise(*self.exc_info) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/libraries/autoinstalled/python-3-arm64/six.py", line 702, in reraise raise value.with_traceback(tb) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 303, in main queue.send(_Result(value=task(None), id=task.id)) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 56, in __call__ return self.function(*self.args, **self.kwargs) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 77, in run_shard return Worker.instance.run_tests(shard) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 319, in run_tests Worker.instance.run_test(input, shard.name) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 347, in run_test result = self._run_test_with_or_without_timeout(test_input, test_timeout_sec, stop_when_done) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 404, in _run_test_with_or_without_timeout return self._run_test_in_this_thread(test_input, stop_when_done) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 490, in _run_test_in_this_thread return self._run_single_test(self._driver, test_input, stop_when_done) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 493, in _run_single_test return single_test_runner.run_single_test( File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 48, in run_single_test return runner.run() File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 123, in run return self._run_reftest() File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 370, in _run_reftest test_result = self._compare_output_with_reference(reference_output, test_output, reference_filename, expectation == '!=') File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 476, in _compare_output_with_reference diff_result.passed = self._test_passes_fuzzy_matching(fuzzy_tolerance, diff_result.fuzzy_data) File "/Volumes/Data/worker/iOS-16-Simulator-WPT-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 431, in _test_passes_fuzzy_matching actualMaxDifference = fuzzy_result['max_difference'] TypeError: 'NoneType' object is not subscriptable (from worker/0) https://ews-build.webkit.org/#/builders/39/builds/1913/steps/17/logs/stdio 1950777 1 ryanhaddad 2023-04-24 08:54:38 -0700 I think this is a new issue that may have started when we updated to iOS 16.4 and/or moved to new macOS hardware. 1950778 2 webkit-bug-importer 2023-04-24 08:55:00 -0700 <rdar://problem/108455505> 1951356 3 ryanhaddad 2023-04-26 10:07:57 -0700 https://ews-build.webkit.org/#/builders/15/builds/4729 happened here on all three layout test attempts. 1952712 4 aakash_jain 2023-05-03 13:10:41 -0700 Pull request: https://github.com/WebKit/WebKit/pull/13412 1952723 5 ews-feeder 2023-05-03 13:49:45 -0700 Committed 263649@main (2e9ec190c7f6): <https://commits.webkit.org/263649@main> Reviewed commits have been landed. Closing PR #13412 and removing active labels. 1954419 6 ryanhaddad 2023-05-10 11:44:59 -0700 Re-opening for pull request https://github.com/apple/WebKit/pull/593 1954427 7 ews-feeder 2023-05-10 11:50:30 -0700 Committed 259548.750@safari-7615-branch (47c1f159af10): <https://commits.webkit.org/259548.750@safari-7615-branch> Reviewed commits have been landed. Closing PR #593 and removing active labels.