Bug 204769 – Do not retry the EWS build due to flaky failures in layout-test

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

In EWS, when layout-tests run fails few tests, we re-run the tests. Then we run these tests on clean-tree, and compare the results of first run, second run, and clean tree run.

In case there is flakiness (different tests fail in first and second runs), and no consistent failure is introduced by the patch being tested, we retry the entire build.

We have this logic for years in old EWS and we copied it in new EWS. See: https://trac.webkit.org/browser/webkit/trunk/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py#L352 (Defer means retrying the build).

'''
350    # At this point we know that at least one test flaked, but no consistent failures
351    # were introduced. This is a bit of a grey-zone.
352    return False  # Defer patch
'''


Retrying the entire build, just because of few flaky tests seems excessive and inefficient. This means that if someone commits a very flaky test (which is not rare), then EWS will keep retrying the patch (almost in a infinite loop), and would be almost stuck until the flakiness is fixed. 

Also, we know that we already have various tests in trunk which are flaky.

In other test-suites like API-tests and JSC-tests, we do not retry the build on flaky test failures, we ignore the flaky tests, and only focus on tests which failed consistently with the patch, and passed without the patch. We should do the similar thing for layout-tests as well.

[Aakash Jain]

For e.g.: In https://ews-build.webkit.org/#/builders/17/builds/7537

first_run_failures: "loader/stateobjects/pushstate-size.html"

second_run_failures: "loader/stateobjects/pushstate-size.html", "webaudio/audioparam-setTargetAtTime.html"

clean_tree_run_failures: "loader/stateobjects/pushstate-size.html"


There was a pre-existing test-failure on trunk, and just because of one flaky test (webaudio/audioparam-setTargetAtTime.html), entire build was retried. This build should have been marked as SUCCESS.

[Aakash Jain]

Created attachment 384663 [details]
Patch

[Jonathan Bedard]

Comment on attachment 384663 [details]
Patch

Seems like this is what we want more often than not, but I still dislike our flakiness story....

[WebKit Commit Bot]

Comment on attachment 384663 [details]
Patch

Clearing flags on attachment: 384663

Committed r253049: <https://trac.webkit.org/changeset/253049>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/57593402>

[Aakash Jain]

Deployed on production server.