Bug 310315 – [WKCI][WPE][GTK] Make the steps for GTK/WPE WK2 testing on the EWS less likely to mark as good patches introducing new flakes.

• Status: REOPENED
• Component: WPE WebKit

[Carlos Alberto Lopez Perez]

Historical context:
------------------

The current classes for WK2 testing on the EWS that are used for GTK and WPE ports were introduced in 244752@main (bug 231999).

In that moment the status of the GTK and WPE layout tests was bad: always red and with a number of unexpected failures that could be anywhere from a dozen to a few hundred.

So that class was designed with the idea of trying to avoid reporting false failures.
That is why to be sure that a failure was introduced by the patch it ran each failure 10 times with the patch and 10 times without it and then it would only report as new failures those that failed the 10 times with the patch and not failed even once without the patch on the 10 retries.

The problem with that approach is that is unable to detect new flakes introduced by the patch.
If a patch introduces a bug that makes a given test to fail with a probability rate of 50% then it won't detect it because if it passes only once on the 10 retries with patch is already considered good (not introduced by the patch).


Current situation:
-----------------

On the last year we have put a big effort on being on top of unexpected failures daily, and thanks to that the tree of the WPE and GTK ports is green most of the time. You can check that at https://build.webkit.org/#/builders/WPE-Linux-64-bit-Release-Tests and https://build.webkit.org/#/builders/GTK-Linux-64-bit-Release-Tests

And this effort is going to be maintained over the time.

So to help with that we want to start marking as bad patches on the EWS the ones that cause new flaky tests.
In order to do that, the idea is to modify the class used to remove the step "repeat failures with patch", so a patch will be considered bad if it fails once on the first run with patch and then passes 10 times on the repeated runs without patch.

[Carlos Alberto Lopez Perez]

Pull request: https://github.com/WebKit/WebKit/pull/60973

[EWS]

Committed 309933@main (471453201b4f): <https://commits.webkit.org/309933@main>

Reviewed commits have been landed. Closing PR #60973 and removing active labels.

[WebKit Commit Bot]

Re-opened since this is blocked by bug 311360