Bug 45740

Summary: SheriffBot should announce when the commit-queue hits a flaky test
Product: WebKit Reporter: Eric Seidel (no email) <eric>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED DUPLICATE    
Severity: Normal CC: abarth, dglazkov, dpranke, ojan, victorw
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: PC   
OS: OS X 10.5   

Description Eric Seidel (no email) 2010-09-14 04:01:39 PDT
SheriffBot should announce when the commit-queue hits a flaky test

Flaky tests should be as annoying for the channel as they are for folks trying to use the commit-queue.

We also really just need the flakey test dashboard.
Comment 1 Ojan Vafai 2010-09-14 05:11:54 PDT
How do you distinguish the commit queue hitting a flaky test versus a test that actually fails due to the patch it's trying to commit?

As far as the flakiness dashboard, getting it hooked up is just a matter of finally switching over to new-run-webkit-tests. Alternately, making the JSON generation code work standalone (shouldn't be too hard) and having old-run-webkit-tests call out to it.
Comment 2 Dimitri Glazkov (Google) 2010-09-14 07:35:14 PDT
(In reply to comment #1)
> How do you distinguish the commit queue hitting a flaky test versus a test that actually fails due to the patch it's trying to commit?
> 
> As far as the flakiness dashboard, getting it hooked up is just a matter of finally switching over to new-run-webkit-tests. Alternately, making the JSON generation code work standalone (shouldn't be too hard) and having old-run-webkit-tests call out to it.

+1 to switching to NWRT.
Comment 3 Eric Seidel (no email) 2010-12-10 17:36:31 PST
I think this is better covered by bug 50803.

*** This bug has been marked as a duplicate of bug 50803 ***