Bug 66071 - Teach garden-o-matic about flaky tests
Summary: Teach garden-o-matic about flaky tests
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks: 69227
  Show dependency treegraph
 
Reported: 2011-08-11 10:03 PDT by Dimitri Glazkov (Google)
Modified: 2011-10-17 13:17 PDT (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dimitri Glazkov (Google) 2011-08-11 10:03:06 PDT
We have all the information necessary to recognize flaky tests. We should use it to prompt the gardener to mark the test as flaky.
Comment 1 Dimitri Glazkov (Google) 2011-08-26 10:38:58 PDT
The flakiness should be revealed in the "Failing Tests" view, so that you could have a choice to mark a test as either failing or flaky.

It might also be useful to show an item in the ui.notifications.TestsFailing "causes" list, indicating that a test is flaky -- and provide a "Mark as Flaky" button there, too.

An edge case where you have 400 tests failing, and 300 of them are also flaky, thus making "causes" list be extremely long seems unlikely.
Comment 2 Adam Barth 2011-10-02 10:21:26 PDT
Punt to v2.
Comment 3 Dimitri Glazkov (Google) 2011-10-17 13:17:49 PDT
avoid a trip to the flakiness dashboard

- is the test flaky?
- since when?