229788 2021-09-01 18:35:26 -0700 [webkitpy] Occasional exception thrown in change_result_to_failure when repeating flaky layout tests 2021-10-14 19:20:58 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=231790 InRadar P2 Normal --- 1 lmoura lmoura ews-watchlist glenn jbedard webkit-bug-importer oldest_to_newest 1789581 0 lmoura 2021-09-01 18:35:26 -0700 It's hard to reproduce with run-webkit-tests, but sometimes, when running multiple iterations of the same layout test, the run is aborted with the following backtrace (from running a single test with --iterations=100): KeyError raised: 'fast/mediastream/RTCPeerConnection-iceconnectionstatechange-event.html' Traceback (most recent call last): File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main run_details = run(port, options, args, stderr) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 494, in run run_details = manager.run(args) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 175, in run_tests pool.do( File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 397, in do result = function(*args, **kwargs) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard return Worker.instance.run_tests(shard) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests Worker.instance.run_test(input, shard.name) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 353, in run_test TaskPool.Process.queue.send(TaskPool.Task( File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 165, in send object(None) File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 56, in __call__ return self.function(*self.args, **self.kwargs) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 81, in handle_finished_test LayoutTestRunner.instance.update_summary_with_result(result) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 241, in update_summary_with_result self._current_run_results.change_result_to_failure(existing, result, existing_expectation, expected) File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py", line 102, in change_result_to_failure self.tests_by_expectation[existing_result.type].remove(existing_result.test_name) KeyError: 'fast/mediastream/RTCPeerConnection-iceconnectionstatechange-event.html' When this happens, a common sequence of events is: * Unexpected results appears (e.g. Failure in an expected Timeout) * Another kind of unexpected results appears (e.g. Pass in an expected Timeout) * The same kind of unexpected results appears (e.g. Pass in an expected Timeout) * EXCEPTION This seems to happen due to the `TestResults` instance stored in `TestRunResults.result_by_name` not being updated from Failure to Pass, as `TestResults.convert_to_failure` (from r235467) just extends the list of failures (IIUC, to keep existing failures information across test runs). This causes the Failure test result to be removed from TestRunResults.test_by_expectation[Failure] inside change_result_to_failure, but still stored as the current result. In the next Pass result, the code tries to remove the Failure it again, raising KeyError. Tentative patch in the next comment. 1789582 1 437102 lmoura 2021-09-01 18:44:14 -0700 Created attachment 437102 Tentative patch 1789701 2 437102 jbedard 2021-09-02 07:48:19 -0700 Comment on attachment 437102 Tentative patch View in context: https://bugs.webkit.org/attachment.cgi?id=437102&action=review > Tools/ChangeLog:10 > + Failure (likely due to r235467). This may cause change_result_to_failure I would be inclined to blame 240653@main / r281216, but I don't think that effects the validity of this change. 1791723 3 webkit-bug-importer 2021-09-08 18:36:24 -0700 <rdar://problem/82902569> 1804639 4 437102 lmoura 2021-10-14 18:41:44 -0700 Comment on attachment 437102 Tentative patch Patch still valid after testing locally. 1804653 5 ews-feeder 2021-10-14 19:20:57 -0700 Committed r284227 (243036@main): <https://commits.webkit.org/243036@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 437102. 437102 2021-09-01 18:44:14 -0700 2021-10-14 19:20:58 -0700 Tentative patch bug-229788-20210901224412.patch text/plain 4547 lmoura U3VidmVyc2lvbiBSZXZpc2lvbjogMjgxODUwCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggYWU4ZGUyOTJkZGRjZGQzYTg1YmYyMDkzYjdiMTJkOWE0 MzYzNjJkNC4uNzdiNDBhMThhY2E0ZDRmMWVlYjMxZGM4MDY4MzM5MDZhMTM1YjYwMSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDI1 IEBACisyMDIxLTA5LTAxICBMYXVybyBNb3VyYSAgPGxtb3VyYUBpZ2FsaWEuY29tPgorCisgICAg ICAgIFt3ZWJraXRweV0gT2NjYXNpb25hbCBleGNlcHRpb24gdGhyb3duIGluIGNoYW5nZV9yZXN1 bHRfdG9fZmFpbHVyZSB3aGVuIHJlcGVhdGluZyBmbGFreSBsYXlvdXQgdGVzdHMKKyAgICAgICAg aHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTIyOTc4OAorCisgICAgICAg IFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFdoZW4gYSBpdGVyYXRlZC9y ZXBlYXRlZCBmbGFreSB0ZXN0IG1vdmVzIGZyb20gYW4gdW5leHBlY3RlZCBGYWlsdXJlIHRvCisg ICAgICAgIHN1Y2Nlc3NpdmUgdW5leHBlY3RlZCBQYXNzLCB0aGUgY3VycmVudGx5IHN0b3JlZCBy ZXN1bHQgaXMgc3RpbGwgdGhlCisgICAgICAgIEZhaWx1cmUgKGxpa2VseSBkdWUgdG8gcjIzNTQ2 NykuIFRoaXMgbWF5IGNhdXNlIGNoYW5nZV9yZXN1bHRfdG9fZmFpbHVyZQorICAgICAgICB0byBy YWlzZSBLZXlFcnJvciB0cnlpbmcgdG8gZm9yY2libHkgcmVtb3ZlKCkgdGhlIEZhaWx1cmUgcmVz dWx0IGZyb20KKyAgICAgICAgdGhlIHRlc3RfYnlfZXhwZWN0YXRpb25zIHNldCB0d2ljZS4KKwor ICAgICAgICBUaGlzIGNvbW1pdCByZWxheGVzIHRoZSBiZWhhdmlvciB0byBqdXN0IGFzayB0byBk aXNjYXJkIHRoZSB2YWx1ZSBmcm9tCisgICAgICAgIHRoZSBzZXQuCisKKyAgICAgICAgKiBTY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9sYXlvdXRfdGVzdF9ydW5uZXJf dW5pdHRlc3QucHk6CisgICAgICAgIChMYXlvdXRUZXN0UnVubmVyVGVzdHMudGVzdF91cGRhdGVf c3VtbWFyeV93aXRoX3Jlc3VsdCk6IEFkZGVkIHRlc3QgY2FzZS4KKyAgICAgICAgKiBTY3JpcHRz L3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weToKKyAgICAg ICAgKFRlc3RSdW5SZXN1bHRzLmNoYW5nZV9yZXN1bHRfdG9fZmFpbHVyZSk6IFVzZSBzZXQuZGlz Y2FyZCB0byBhdm9pZAorICAgICAgICBLZXlFcnJvcnMuCisKIDIwMjEtMDktMDEgIEpvbmF0aGFu IEJlZGFyZCAgPGpiZWRhcmRAYXBwbGUuY29tPgogCiAgICAgICAgIFtjb250cmlidXRvcnMuanNv bl0gUmVsb2NhdGlvbiAoUGFydCAxKQpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRw eS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbGF5b3V0X3Rlc3RfcnVubmVyX3VuaXR0ZXN0LnB5 IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbGF5b3V0 X3Rlc3RfcnVubmVyX3VuaXR0ZXN0LnB5CmluZGV4IDUxODNiMjExNzRjMGNhNWYxMGJiODQ1NmEx Y2RhMmJmY2EyMmM5OTIuLjM1NmE0MDdjZjUxNDg0ODhjZWQxZDUzMDg5M2FiNjU0YjM1MDY4MmUg MTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xs ZXJzL2xheW91dF90ZXN0X3J1bm5lcl91bml0dGVzdC5weQorKysgYi9Ub29scy9TY3JpcHRzL3dl YmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9sYXlvdXRfdGVzdF9ydW5uZXJfdW5pdHRl c3QucHkKQEAgLTEyNiw3ICsxMjYsOCBAQCBjbGFzcyBMYXlvdXRUZXN0UnVubmVyVGVzdHModW5p dHRlc3QuVGVzdENhc2UpOgogICAgICAgICBydW5uZXIuX29wdGlvbnMud29ybGRfbGVha3MgPSBG YWxzZQogICAgICAgICB0ZXN0ID0gJ2ZhaWx1cmVzL2V4cGVjdGVkL3JlZnRlc3QuaHRtbCcKICAg ICAgICAgbGVha190ZXN0ID0gJ2ZhaWx1cmVzL2V4cGVjdGVkL2xlYWsuaHRtbCcKLSAgICAgICAg ZXhwZWN0YXRpb25zID0gVGVzdEV4cGVjdGF0aW9ucyhydW5uZXIuX3BvcnQsIHRlc3RzPVt0ZXN0 LCBsZWFrX3Rlc3RdKQorICAgICAgICB0aW1lb3V0X3Rlc3QgPSAnZmFpbHVyZXMvZXhwZWN0ZWQv dGltZW91dC5odG1sJworICAgICAgICBleHBlY3RhdGlvbnMgPSBUZXN0RXhwZWN0YXRpb25zKHJ1 bm5lci5fcG9ydCwgdGVzdHM9W3Rlc3QsIGxlYWtfdGVzdCwgdGltZW91dF90ZXN0XSkKICAgICAg ICAgZXhwZWN0YXRpb25zLnBhcnNlX2FsbF9leHBlY3RhdGlvbnMoKQogICAgICAgICBydW5uZXIu X2V4cGVjdGF0aW9ucyA9IGV4cGVjdGF0aW9ucwogCkBAIC0xNDgsNiArMTQ5LDI0IEBAIGNsYXNz IExheW91dFRlc3RSdW5uZXJUZXN0cyh1bml0dGVzdC5UZXN0Q2FzZSk6CiAgICAgICAgIHNlbGYu YXNzZXJ0RXF1YWwoMSwgcnVubmVyLl9jdXJyZW50X3J1bl9yZXN1bHRzLmV4cGVjdGVkKQogICAg ICAgICBzZWxmLmFzc2VydEVxdWFsKDAsIHJ1bm5lci5fY3VycmVudF9ydW5fcmVzdWx0cy51bmV4 cGVjdGVkKQogCisgICAgICAgIHJ1bm5lci5fY3VycmVudF9ydW5fcmVzdWx0cyA9IFRlc3RSdW5S ZXN1bHRzKGV4cGVjdGF0aW9ucywgMykKKyAgICAgICAgcmVzdWx0ID0gVGVzdFJlc3VsdCh0aW1l b3V0X3Rlc3QsIGZhaWx1cmVzPVtdKQorICAgICAgICBydW5uZXIudXBkYXRlX3N1bW1hcnlfd2l0 aF9yZXN1bHQocmVzdWx0KQorICAgICAgICBzZWxmLmFzc2VydEVxdWFsKDAsIHJ1bm5lci5fY3Vy cmVudF9ydW5fcmVzdWx0cy5leHBlY3RlZCkKKyAgICAgICAgc2VsZi5hc3NlcnRFcXVhbCgxLCBy dW5uZXIuX2N1cnJlbnRfcnVuX3Jlc3VsdHMudW5leHBlY3RlZCkKKyAgICAgICAgcmVzdWx0ID0g VGVzdFJlc3VsdCh0aW1lb3V0X3Rlc3QsIGZhaWx1cmVzPVt0ZXN0X2ZhaWx1cmVzLkZhaWx1cmVU ZXh0TWlzbWF0Y2goKV0pCisgICAgICAgIHJ1bm5lci51cGRhdGVfc3VtbWFyeV93aXRoX3Jlc3Vs dChyZXN1bHQpCisgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWwoMCwgcnVubmVyLl9jdXJyZW50X3J1 bl9yZXN1bHRzLmV4cGVjdGVkKQorICAgICAgICBzZWxmLmFzc2VydEVxdWFsKDIsIHJ1bm5lci5f Y3VycmVudF9ydW5fcmVzdWx0cy51bmV4cGVjdGVkKQorICAgICAgICByZXN1bHQgPSBUZXN0UmVz dWx0KHRpbWVvdXRfdGVzdCwgZmFpbHVyZXM9W10pCisgICAgICAgIHJ1bm5lci51cGRhdGVfc3Vt bWFyeV93aXRoX3Jlc3VsdChyZXN1bHQpCisgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWwoMCwgcnVu bmVyLl9jdXJyZW50X3J1bl9yZXN1bHRzLmV4cGVjdGVkKQorICAgICAgICBzZWxmLmFzc2VydEVx dWFsKDMsIHJ1bm5lci5fY3VycmVudF9ydW5fcmVzdWx0cy51bmV4cGVjdGVkKQorICAgICAgICBy ZXN1bHQgPSBUZXN0UmVzdWx0KHRpbWVvdXRfdGVzdCwgZmFpbHVyZXM9W10pCisgICAgICAgIHJ1 bm5lci51cGRhdGVfc3VtbWFyeV93aXRoX3Jlc3VsdChyZXN1bHQpCisgICAgICAgIHNlbGYuYXNz ZXJ0RXF1YWwoMCwgcnVubmVyLl9jdXJyZW50X3J1bl9yZXN1bHRzLmV4cGVjdGVkKQorICAgICAg ICBzZWxmLmFzc2VydEVxdWFsKDQsIHJ1bm5lci5fY3VycmVudF9ydW5fcmVzdWx0cy51bmV4cGVj dGVkKQorCiAgICAgZGVmIHRlc3Rfc2VydmVyc19zdGFydGVkKHNlbGYpOgogCiAgICAgICAgIGRl ZiBzdGFydF9odHRwX3NlcnZlcigpOgpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRw eS9sYXlvdXRfdGVzdHMvbW9kZWxzL3Rlc3RfcnVuX3Jlc3VsdHMucHkgYi9Ub29scy9TY3JpcHRz L3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weQppbmRleCBi ODUwMzVjMDFmNDAwYWNjMGRiZmY5NmY5YmNkOWZkYzVhOTI1OTUyLi5lNWE0ZWNlMzE1N2E5MTFj NzRiYjkyMGU3ZjAxOGZhNDdhOWIyZmEwIDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weQorKysgYi9Ub29scy9T Y3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weQpA QCAtOTksNyArOTksNyBAQCBjbGFzcyBUZXN0UnVuUmVzdWx0cyhvYmplY3QpOgogICAgICAgICBp ZiBleGlzdGluZ19yZXN1bHQudHlwZSBpcyBuZXdfcmVzdWx0LnR5cGU6CiAgICAgICAgICAgICBy ZXR1cm4KIAotICAgICAgICBzZWxmLnRlc3RzX2J5X2V4cGVjdGF0aW9uW2V4aXN0aW5nX3Jlc3Vs dC50eXBlXS5yZW1vdmUoZXhpc3RpbmdfcmVzdWx0LnRlc3RfbmFtZSkKKyAgICAgICAgc2VsZi50 ZXN0c19ieV9leHBlY3RhdGlvbltleGlzdGluZ19yZXN1bHQudHlwZV0uZGlzY2FyZChleGlzdGlu Z19yZXN1bHQudGVzdF9uYW1lKQogICAgICAgICBzZWxmLnRlc3RzX2J5X2V4cGVjdGF0aW9uW25l d19yZXN1bHQudHlwZV0uYWRkKG5ld19yZXN1bHQudGVzdF9uYW1lKQogCiAgICAgICAgIGhhZF9m YWlsdXJlcyA9IGxlbihleGlzdGluZ19yZXN1bHQuZmFpbHVyZXMpID4gMAo=