Bug 89368

Summary: Mark failing tests as failing
Product: WebKit Reporter: Tony Payne <tpayne>
Component: New BugsAssignee: Tony Payne <tpayne>
Status: NEW ---    
Severity: Normal CC: abarth, tony
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch tony: review-, tony: commit-queue-

Description Tony Payne 2012-06-18 11:48:40 PDT
Mark failing tests as failing
Comment 1 Tony Payne 2012-06-18 11:49:38 PDT
Created attachment 148144 [details]
Patch
Comment 2 Tony Payne 2012-06-18 11:52:05 PDT
Created attachment 148146 [details]
Patch
Comment 3 Adam Barth 2012-06-18 12:05:30 PDT
Tony might have some thoughts about this patch.
Comment 4 Tony Chang 2012-06-18 13:45:32 PDT
Comment on attachment 148146 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=148146&action=review

> LayoutTests/platform/chromium/TestExpectations:1448
> +BUGWK89367 LINUX : platform/chromium-linux/fast/text/international/complex-joining-using-gpos.html = IMAGE
> +BUGWK89367 LINUX : fast/text/international/thai-line-breaks.html = IMAGE

They pass on the waterfall, right?  IIRC, the EWS bots run a different version of Ubuntu than the waterfall bots, so these tests only fail on the EWS bots.  I think these are always going to fail until we switch to Precise in a year or so.
Comment 5 Tony Payne 2012-06-18 13:47:54 PDT
(In reply to comment #4)
> (From update of attachment 148146 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=148146&action=review
> 
> > LayoutTests/platform/chromium/TestExpectations:1448
> > +BUGWK89367 LINUX : platform/chromium-linux/fast/text/international/complex-joining-using-gpos.html = IMAGE
> > +BUGWK89367 LINUX : fast/text/international/thai-line-breaks.html = IMAGE
> 
> They pass on the waterfall, right?  IIRC, the EWS bots run a different version of Ubuntu than the waterfall bots, so these tests only fail on the EWS bots.  I think these are always going to fail until we switch to Precise in a year or so.

These failures really hurt EWS performance since they guarantee the tests will be run 3 times, and they make it impossible to use the CQ when there are red tests, even if the CL is fixing those red tests. I don't think this is a situation that's tenable for a year. It impacts developer productivity significantly.
Comment 6 Adam Barth 2012-06-18 13:49:40 PDT
That's not really accurate.  The EWS remembers the set of failing tests from run to run and tends to ignore these tests anyway.
Comment 7 Tony Payne 2012-06-18 13:50:34 PDT
(In reply to comment #6)
> That's not really accurate.  The EWS remembers the set of failing tests from run to run and tends to ignore these tests anyway.

That doesn't match my experience at all.