Bug 63984

Summary: new-run-webkit-tests is not correctly catching/printing TestRunInterruptedException
Product: WebKit Reporter: Eric Seidel (no email) <eric>
Component: Tools / TestsAssignee: Dirk Pranke <dpranke>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth, aroben, dpranke, rniwa
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 64218    
Bug Blocks: 64491    

Description Eric Seidel (no email) 2011-07-06 03:36:23 PDT 
new-run-webkit-tests is not correctly catching/printing TestRunInterruptedException

http://build.webkit.org/builders/GTK%20Linux%2032-bit%20Debug/builds/16226/steps/layout-test/logs/stdio

2011-07-06 03:24:10,796 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/worker.py", line 91, in run
2011-07-06 03:24:10,796 11347 stack_utils.py:67 DEBUG       self._worker_connection.run_message_loop()
2011-07-06 03:24:10,797 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/message_broker.py", line 192, in run_message_loop
2011-07-06 03:24:10,797 11347 stack_utils.py:67 DEBUG       self._broker.run_message_loop(self._run_topic, self._client, delay_secs)
2011-07-06 03:24:10,797 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/message_broker.py", line 128, in run_message_loop
2011-07-06 03:24:10,797 11347 stack_utils.py:67 DEBUG       self._run_loop(topic_name, client, block=True, delay_secs=delay_secs)
2011-07-06 03:24:10,797 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/message_broker.py", line 142, in _run_loop
2011-07-06 03:24:10,797 11347 stack_utils.py:67 DEBUG       self._dispatch_message(msg, client)
2011-07-06 03:24:10,797 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/message_broker.py", line 151, in _dispatch_message
2011-07-06 03:24:10,798 11347 stack_utils.py:67 DEBUG       message_handler(message.src, *optargs)
2011-07-06 03:24:10,798 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/worker.py", line 120, in handle_test_list
2011-07-06 03:24:10,798 11347 stack_utils.py:67 DEBUG       self._worker_connection.yield_to_broker()
2011-07-06 03:24:10,798 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/manager_worker_broker.py", line 238, in yield_to_broker
2011-07-06 03:24:10,798 11347 stack_utils.py:67 DEBUG       self._broker.run_all_pending(MANAGER_TOPIC, self._manager_client)
2011-07-06 03:24:10,798 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/message_broker.py", line 132, in run_all_pending
2011-07-06 03:24:10,798 11347 stack_utils.py:67 DEBUG       self._run_loop(topic_name, client, block=False, delay_secs=None)
2011-07-06 03:24:10,799 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/message_broker.py", line 142, in _run_loop
2011-07-06 03:24:10,799 11347 stack_utils.py:67 DEBUG       self._dispatch_message(msg, client)
2011-07-06 03:24:10,799 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/message_broker.py", line 151, in _dispatch_message
2011-07-06 03:24:10,799 11347 stack_utils.py:67 DEBUG       message_handler(message.src, *optargs)
2011-07-06 03:24:10,799 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py", line 1462, in handle_finished_test
2011-07-06 03:24:10,799 11347 stack_utils.py:67 DEBUG       self._update_summary_with_result(self._current_result_summary, result)
2011-07-06 03:24:10,799 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py", line 1017, in _update_summary_with_result
2011-07-06 03:24:10,800 11347 stack_utils.py:67 DEBUG       self._interrupt_if_at_failure_limits(result_summary)
2011-07-06 03:24:10,800 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py", line 1005, in _interrupt_if_at_failure_limits
2011-07-06 03:24:10,800 11347 stack_utils.py:67 DEBUG       "Exiting early after %d crashes and %d timeouts." % (result_summary.unexpected_crashes, result_summary.unexpected_timeouts))
2011-07-06 03:24:10,800 11347 stack_utils.py:67 DEBUG     File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py", line 993, in interrupt_if_at_failure_limit
2011-07-06 03:24:10,800 11347 stack_utils.py:67 DEBUG       raise TestRunInterruptedException(message)

It should be silently catching the exception and printing the log message.
Comment 1 Dirk Pranke 2011-07-06 13:46:56 PDT 
Hm, 

It looks like the code mostly worked. The debug code prints the stack trace of any exception it gets on the worker. Perhaps your comment about "silently" means you wouldn't expect to see this?

But, you're right that we don't seem to be catching the exception and printing the log message, so it's unclear which code path fired on the way out.

It looks like the gtk port is running w/o multiple workers, using --worker-model=inline, so this should be trivial to test/reproduce. We have a test for this in run_webkit_tests_integrationtest.py: test_exit_after_n_crashes that you can probably debug or try on the command line to see what happens.
Comment 2 Dirk Pranke 2011-07-08 17:35:44 PDT 
Okay, this bug has to do with the interactions of exceptions (the way they are sent across the message queue) and the worker models.

The patch I've uploaded to bug 64218 should fix this as well.
Comment 3 Dirk Pranke 2011-07-08 17:36:26 PDT 
Also, this only happens with --worker-model=inline, which we probably shouldn't be using on the bots. 

There's no reason for this to block the NRWT cutover, so I'm clearing that.
Comment 4 Eric Seidel (no email) 2011-08-08 10:15:36 PDT 
This affects how crashes are displayed on the bots.
Comment 5 Dirk Pranke 2011-08-16 16:39:55 PDT 
I'm closing this, since I think this was fixed in r90679. Please re-open (or file a new bug) if there's still an issue.
Comment 6 Eric Seidel (no email) 2011-08-17 11:10:48 PDT 
OK.  We'll know if the bots get confused by many crashes.