Bug 50263 - Exception in flaky test reporting code in commit-queue
Summary: Exception in flaky test reporting code in commit-queue
Status: RESOLVED DUPLICATE of bug 51272
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: PC OS X 10.5
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on: 51272
Blocks:
  Show dependency treegraph
 
Reported: 2010-11-30 13:00 PST by Eric Seidel (no email)
Modified: 2010-12-20 20:35 PST (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Eric Seidel (no email) 2010-11-30 13:00:54 PST
Exception in flaky test reporting code in commit-queue

From the log:

=== BUILD AGGREGATE TARGET All OF PROJECT DumpRenderTree WITH CONFIGURATION Release ===
Check dependencies
** BUILD SUCCEEDED **

Compiling Java tests
make: Nothing to be done for `default'.
Running tests from /Projects/CommitQueue/LayoutTests
Testing 22075 test cases.
accessibility .....................................................................................
animations ...........................................................
animations/stop-animation-on-suspend.html -> failed

Exiting early after 1 failures. 144 tests run.
44.81s total testing time

143 test cases (99%) succeeded
1 test case (<1%) had incorrect layout
1 test case (<1%) had stderr output
Patch does not pass tests
Traceback (most recent call last):
  File "/Projects/CommitQueue/WebKitTools/Scripts/webkitpy/tool/bot/queueengine.py", line 108, in run
    if not self._delegate.process_work_item(work_item):
  File "/Projects/CommitQueue/WebKitTools/Scripts/webkitpy/tool/commands/queues.py", line 265, in process_work_item
    if task.run():
  File "/Projects/CommitQueue/WebKitTools/Scripts/webkitpy/tool/bot/commitqueuetask.py", line 188, in run
    if not self._test_patch():
  File "/Projects/CommitQueue/WebKitTools/Scripts/webkitpy/tool/bot/commitqueuetask.py", line 172, in _test_patch
    self._report_flaky_tests(first_failing_tests + second_failing_tests)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'list'
Exception while preparing queue Sleeping until 2010-11-30 12:57:20 (2 mins).
Comment 1 Eric Seidel (no email) 2010-11-30 15:04:28 PST
This would happen any time the tests fail on the first run, but we fail to get results from that failure.

The code in question:

    def _test_patch(self):
        if self._patch.is_rollout():
            return True
        if self._test():
            return True

        first_failing_tests = self._failing_tests_from_last_run()
        if self._test():
            self._report_flaky_tests(first_failing_tests)
            return True

        second_failing_tests = self._failing_tests_from_last_run()
        if first_failing_tests != second_failing_tests:
            self._report_flaky_tests(first_failing_tests + second_failing_tests)
            return False

The fix is straightforward.  We just make "first_failing_tests != second_failing_tests" into "first_failing_tests and (first_failing_tests != second_failing_tests)".  However we might also want to add some special logging to track this case, as the layout tests shoudl not fail w/o us being able to get the results from the run!
Comment 2 Eric Seidel (no email) 2010-12-20 20:35:00 PST
I expect this to be fixed by bug 51272.

*** This bug has been marked as a duplicate of bug 51272 ***