Bug 188413 – Add basic flakiness-analyzer

• Status: NEW
• Component: Tools / Tests

[Jonathan Bedard]

This has come in handy enough times over the last 2 months that it's worth committing.

What this script does is read through the stdout of a test run (or runs) and determines which tests were run before some failing test. It then cross-references successful test runs, with the assumption that if a given test preceded a successful run of another test, it is unlikely to be the cause of the failure. We've been able to track down a few flakey test failures using this script, or a predecessor to it.

[Jonathan Bedard]

Created attachment 346782 [details]
Patch

[Jonathan Bedard]

Comment on attachment 346782 [details]
Patch

Need to figure out why these tests failed on the bots but passed locally.

[Jonathan Bedard]

Created attachment 346807 [details]
Patch

[EWS Watchlist]

Comment on attachment 346807 [details]
Patch

Attachment 346807 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/8828955

New failing tests:
http/tests/security/video-poster-cross-origin-crash2.html

[EWS Watchlist]

Created attachment 346963 [details]
Archive of layout-test-results from ews205 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews205  Port: win-future  Platform: CYGWIN_NT-6.1-2.9.0-0.318-5-3-x86_64-64bit

[Ryan Haddad]

Comment on attachment 346807 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=346807&action=review

> Tools/Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer.py:153
> +        index_to_print = int(raw_input('Pick which of the sequences (0-{}) to print: '.format(len(steps_candidates - 1))))

I hit a bug here when trying to test this locally. This should be: .format(len(steps_candidates) - 1))

[Alex Christensen]

Comment on attachment 346807 [details]
Patch

This has been requesting review for more than one year.  If this is still needed, please rebase and re-request review.