Bug 28192 – run-webkit-tests needs a --exit-after-failures=N option

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Eric Seidel (no email)]

I'm not sure --fail-fast is the right name.

But I want a mode where run-webkit-tests exits after the first failure or new result.  This is useful for bugzilla-tool (or possibly some bots) when validating that everything still passes before committing.  In some cases we don't care to know all of the failures, only that something failed (and want to know as fast as possible).

[Mark Rowe (bdash)]

The build bots would be better served by a mode that gave up after a certain number of crashes or failures.  Perhaps a --exit-after-n-failures flag could fill both of these needs?

[Eric Seidel (no email)]

Excellent idea!  (and that's a much better name)

[Eric Seidel (no email)]

Created attachment 34704 [details]
Patch v1

[Darin Adler]

Comment on attachment 34704 [details]
Patch v1

> +    my $testsWhithDidNotMatch = $count - $counts{match};

Which?

[Eric Seidel (no email)]

Created attachment 34705 [details]
Removed extra change

[Eric Seidel (no email)]

Created attachment 34706 [details]
now with misspelling fixed

[Mark Rowe (bdash)]

Comment on attachment 34706 [details]
now with misspelling fixed

> diff --git a/WebKitTools/Scripts/run-webkit-tests b/WebKitTools/Scripts/run-webkit-tests
> index 1f908ec..003ac8d 100755
> --- a/WebKitTools/Scripts/run-webkit-tests
> +++ b/WebKitTools/Scripts/run-webkit-tests
> @@ -109,6 +109,7 @@ sub writeToFile($$);
>  # Argument handling
>  my $addPlatformExceptions = 0;
>  my $complexText = 0;
> +my $exitAfterNFailures = 0;
>  my $guardMalloc = '';
>  my $httpdPort = 8000;
>  my $httpdSSLPort = 8443;
> @@ -193,6 +194,7 @@ Usage: $programName [options] [testdir|testpath ...]
>    --complex-text                  Use the complex text code path for all text (Mac OS X and Windows only)
>    -c|--configuration config       Set DumpRenderTree build configuration
>    -g|--guard-malloc               Enable malloc guard
> +  --exit-after-n-failures         Exit after the first N failures instead of running all the tests

The help for this flag should indicate that it takes an argument.  Is there any reason to put it at this point in the help?  Are they in any specific order?

>      countFinishedTest($test, $base, $result, $isText);
> +
> +    my $testsWhichDidNotMatch = $count - $counts{match};
> +    if ($exitAfterNFailures && $testsWhichDidNotMatch >= $exitAfterNFailures) {
> +        print "\nExiting early after $testsWhichDidNotMatch failures.  $count tests run.";
> +        closeDumpTool();
> +        last;
> +    }

$testsWhichDidNotMatch sounds like an array of tests that didn't match.  Perhaps $failureCount would be a clearer variable name?

r=me

[Eric Seidel (no email)]

(In reply to comment #7)
> (From update of attachment 34706 [details])
> > diff --git a/WebKitTools/Scripts/run-webkit-tests b/WebKitTools/Scripts/run-webkit-tests
> > index 1f908ec..003ac8d 100755
> > --- a/WebKitTools/Scripts/run-webkit-tests
> > +++ b/WebKitTools/Scripts/run-webkit-tests
> > @@ -109,6 +109,7 @@ sub writeToFile($$);
> >  # Argument handling
> >  my $addPlatformExceptions = 0;
> >  my $complexText = 0;
> > +my $exitAfterNFailures = 0;
> >  my $guardMalloc = '';
> >  my $httpdPort = 8000;
> >  my $httpdSSLPort = 8443;
> > @@ -193,6 +194,7 @@ Usage: $programName [options] [testdir|testpath ...]
> >    --complex-text                  Use the complex text code path for all text (Mac OS X and Windows only)
> >    -c|--configuration config       Set DumpRenderTree build configuration
> >    -g|--guard-malloc               Enable malloc guard
> > +  --exit-after-n-failures         Exit after the first N failures instead of running all the tests
> 
> The help for this flag should indicate that it takes an argument.  Is there any
> reason to put it at this point in the help?  Are they in any specific order?

It seemed they were in alphabetical order.  But I'm happy to put it wherever.

> >      countFinishedTest($test, $base, $result, $isText);
> > +
> > +    my $testsWhichDidNotMatch = $count - $counts{match};
> > +    if ($exitAfterNFailures && $testsWhichDidNotMatch >= $exitAfterNFailures) {
> > +        print "\nExiting early after $testsWhichDidNotMatch failures.  $count tests run.";
> > +        closeDumpTool();
> > +        last;
> > +    }
> 
> $testsWhichDidNotMatch sounds like an array of tests that didn't match. 
> Perhaps $failureCount would be a clearer variable name?

Sounds fine to me.  Will change.

[Eric Seidel (no email)]

Committing to http://svn.webkit.org/repository/webkit/trunk ...
	M	WebKitTools/ChangeLog
	M	WebKitTools/Scripts/bugzilla-tool
	M	WebKitTools/Scripts/run-webkit-tests
Committed r47166
	M	WebKitTools/ChangeLog
	M	WebKitTools/Scripts/bugzilla-tool
	M	WebKitTools/Scripts/run-webkit-tests
r47166 = 581871acba07344f98ea6b67c0be8ed60106d917 (trunk)
No changes between current HEAD and refs/remotes/trunk
Resetting to the latest refs/remotes/trunk
http://trac.webkit.org/changeset/47166