commit-queue can't upload failure diffs from the first flake on a double-flake def _test_patch(self): if self._patch.is_rollout(): return True if self._test(): return True first_failing_tests = self._failing_tests_from_last_run() if self._test(): self._report_flaky_tests(first_failing_tests) return True second_failing_tests = self._failing_tests_from_last_run() if first_failing_tests != second_failing_tests: self._report_flaky_tests(first_failing_tests + second_failing_tests) return False if self._build_and_test_without_patch(): raise self._script_error # The error from the previous ._test() run is real, report it. return False # Tree must be red, just retry later. Currently _failing_tests_from_last_run() only returns the test names (paths) for the failures. It does not bottle up the results information. So when we run the tests a second time, old-run-webkit-tests will delete the results from the first run. Then when we go to try and report the test as flaky, we don't have any results diff to upload.
We already have a TestResult command in layout_tests/layout_package, so I'd rather not re-invent the wheel. But it doesn't look like it's meant for dealing with a single result from a single test.
We still need better handling here. However, double-flake reporting was removed as part of bug 51272.
Actually in fixing bug 52048 I realized that double-flakes were the *only* times we were able to upload failure diffs. :) However, once bug 52048 lands, we could (carefully) re-enable double-flake reporting and uploads would work.