Bug 204707 – Add a unit-test to EWS for scenario when there is flakiness with patch and also on clean tree

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

Add a unit-test to old EWS for scenario when there is flakiness with patch and also on clean tree.

[Aakash Jain]

Created attachment 384518 [details]
Unit-test for old EWS

[Aakash Jain]

Committed r252943: <https://trac.webkit.org/changeset/252943>

[Radar WebKit Bug Importer]

<rdar://problem/57533998>

[Aakash Jain]

Created attachment 384519 [details]
Similar unit-test for new EWS

[Aakash Jain]

.

[Aakash Jain]

Committed r252946: <https://trac.webkit.org/changeset/252946>

[Alexey Proskuryakov]

I think that I know why I didn't believe that this was the old EWS behavior.

Looking at the "if self._results_failed_different_tests" branch in PatchAnalysisTask._retry_layout_tests, I see that it calls self._should_defer_patch_or_throw, which says that the result should be a PASS. But then we ignore the result, and make it a DEFER anyway!

            if tests_that_consistently_failed:
                if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive,
                                                     first_script_error, first_failure_status_id):
                    return False  # Defer patch

            # At this point we know that at least one test flaked, but no consistent failures
            # were introduced. This is a bit of a grey-zone.
            return False  # Defer patch


What's the point of the first "return False" here if we are going to defer either way?