Bug 51371 – commit-queue can't upload failure diffs from the first flake on a double-flake

• Status: NEW
• Component: Tools / Tests

[Eric Seidel (no email)]

commit-queue can't upload failure diffs from the first flake on a double-flake

def _test_patch(self):
    if self._patch.is_rollout():
        return True
    if self._test():
        return True

    first_failing_tests = self._failing_tests_from_last_run()
    if self._test():
        self._report_flaky_tests(first_failing_tests)
        return True

    second_failing_tests = self._failing_tests_from_last_run()
    if first_failing_tests != second_failing_tests:
        self._report_flaky_tests(first_failing_tests + second_failing_tests)
        return False

    if self._build_and_test_without_patch():
        raise self._script_error  # The error from the previous ._test() run is real, report it.
    return False  # Tree must be red, just retry later.


Currently _failing_tests_from_last_run() only returns the test names (paths) for the failures.  It does not bottle up the results information.  So when we run the tests a second time, old-run-webkit-tests will delete the results from the first run.  Then when we go to try and report the test as flaky, we don't have any results diff to upload.

[Eric Seidel (no email)]

We already have a TestResult command in layout_tests/layout_package, so I'd rather not re-invent the wheel. But it doesn't look like it's meant for dealing with a single result from a single test.

[Eric Seidel (no email)]

We still need better handling here.  However, double-flake reporting was removed as part of bug 51272.

[Eric Seidel (no email)]

Actually in fixing bug 52048 I realized that double-flakes were the *only* times we were able to upload failure diffs. :)  However, once bug 52048 lands, we could (carefully) re-enable double-flake reporting and uploads would work.