Bug 229788 – [webkitpy] Occasional exception thrown in change_result_to_failure when repeating flaky layout tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Lauro Moura]

It's hard to reproduce with run-webkit-tests, but sometimes, when running multiple iterations of the same layout test, the run is aborted with the following backtrace (from running a single test with --iterations=100):

KeyError raised: 'fast/mediastream/RTCPeerConnection-iceconnectionstatechange-event.html'                                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                                            
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main                                                                                                                                                
    run_details = run(port, options, args, stderr)                                                                                                                                                                                            
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 494, in run                                                                                                                                                
    run_details = manager.run(args)                                                                                                                                                                                                           
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run                                                                                                                                             
    temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type)                                                                                                 
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset                                                                                                                                
    initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type)                                                          
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests               
    return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type)                                                                                                                       
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 175, in run_tests     
    pool.do(                                                                                                                                                                                                                                  
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 397, in do                                                                                                                                          
    result = function(*args, **kwargs)                                                                                                                                                                                                        
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard                                                                                                                             
    return Worker.instance.run_tests(shard)                                                                                                                                                                                                   
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests                                                                                                                            
    Worker.instance.run_test(input, shard.name)                                                                                                                                                                                               
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 353, in run_test                                                                                                                             
    TaskPool.Process.queue.send(TaskPool.Task(                                                                                                                                                                                                
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 165, in send                 
    object(None)                                                                                                       
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 56, in __call__              
    return self.function(*self.args, **self.kwargs)                                                                    
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 81, in handle_finished_test                                                                                                                  
    LayoutTestRunner.instance.update_summary_with_result(result)                                                       
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 241, in update_summary_with_result                                                                                                           
    self._current_run_results.change_result_to_failure(existing, result, existing_expectation, expected)               
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py", line 102, in change_result_to_failure                                                                                                                    
    self.tests_by_expectation[existing_result.type].remove(existing_result.test_name)                                                                                                                                                         
KeyError: 'fast/mediastream/RTCPeerConnection-iceconnectionstatechange-event.html'

When this happens, a common sequence of events is:

* Unexpected results appears (e.g. Failure in an expected Timeout)
* Another kind of unexpected results appears (e.g. Pass in an expected Timeout)
* The same kind of unexpected results appears (e.g. Pass in an expected Timeout)
* EXCEPTION 

This seems to happen due to the `TestResults` instance stored in `TestRunResults.result_by_name` not being updated from Failure to Pass, as `TestResults.convert_to_failure` (from r235467) just extends the list of failures (IIUC, to keep existing failures information across test runs).

This causes the Failure test result to be removed from TestRunResults.test_by_expectation[Failure] inside change_result_to_failure, but still stored as the current result. In the next Pass result, the code tries to remove the Failure it again, raising KeyError.

Tentative patch in the next comment.

[Lauro Moura]

Created attachment 437102 [details]
Tentative patch

[Jonathan Bedard]

Comment on attachment 437102 [details]
Tentative patch

View in context: https://bugs.webkit.org/attachment.cgi?id=437102&action=review

> Tools/ChangeLog:10
> +        Failure (likely due to r235467). This may cause change_result_to_failure

I would be inclined to blame 240653@main / r281216, but I don't think that effects the validity of this change.

[Radar WebKit Bug Importer]

<rdar://problem/82902569>

[Lauro Moura]

Comment on attachment 437102 [details]
Tentative patch

Patch still valid after testing locally.

[EWS]

Committed r284227 (243036@main): <https://commits.webkit.org/243036@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437102 [details].