Bug 69118

Summary: NRWT should have an option to not exit early because of image failures
Product: WebKit Reporter: Simon Fraser (smfr) <simon.fraser>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: dpranke, eric, simon.fraser
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on:    
Bug Blocks: 88680    

Description Simon Fraser (smfr) 2011-09-29 18:36:01 PDT
NRWT should never exit early because of image failures. When running on a new OS version, you can have lots of image failures without necessarily having layout failures.
Comment 1 Dirk Pranke 2011-09-29 18:56:53 PDT
I'm not sure I agree with this; it depends on whether pixel tests are enabled or not. If they are enabled, it seems like we should exit early (and running on a new/unsupported OS version seems like the exception rather than the rule). You can change the --exit-after-n-failures threshold to some really high number if you want to defeat this, or just turn off pixel tests.

We can certainly change the defaults to do something different if we detect we're on the "future" version of the O/S, though. I'm not yet convinced that that's the right thing to do, either, but it would be easier to convince me of this than to change it for the general case.
Comment 2 Simon Fraser (smfr) 2011-09-29 20:17:14 PDT
OK, I retitled :)
Comment 3 Dirk Pranke 2011-09-30 12:14:46 PDT
I guess I'm not sure what you want ... if you know you're on a new release that doesn't have correct images yet, is it good enough to just turn off pixel tests?
Comment 4 Simon Fraser (smfr) 2011-09-30 12:16:27 PDT
No, because I'd still like to see what the pixel diffs look like.

Maybe --exit-after-n-failure=10000000 is enough for me. I'd prefer a shorthand for 'never exit' though, and didn't see one in the help output.
Comment 5 Dirk Pranke 2011-09-30 12:19:38 PDT
Ah, I see. Would you want to still want to exit early after N text diffs, and just ignore the pixel failures? Otherwise, as you say, --exit-after-n-failures would work.

From a quick glance at the code, I believe that --exit-after-n-failures=0 is equivalent to "never exit". That should probably be documented in the help.
Comment 6 Dirk Pranke 2012-06-08 15:54:53 PDT
smfr ... is --exit-after-n-failures=0 good enough, or would you like image failures to be treated separately?
Comment 7 Dirk Pranke 2012-06-13 19:12:41 PDT
in r120007 we changed the default for --exit-after-n-failures to None (i.e., never exit early) (see bug 73843).

Hopefully this is good enough and we don't need to add another flag.

Please reopen if there's not the case.