Summary: | run-webkit-tests needs a --exit-after-failures=N option | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | Eric Seidel (no email) <eric> | ||||||||
Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> | ||||||||
Status: | RESOLVED FIXED | ||||||||||
Severity: | Normal | CC: | abarth | ||||||||
Priority: | P2 | ||||||||||
Version: | 528+ (Nightly build) | ||||||||||
Hardware: | PC | ||||||||||
OS: | OS X 10.5 | ||||||||||
Attachments: |
|
Description
Eric Seidel (no email)
2009-08-11 15:15:45 PDT
The build bots would be better served by a mode that gave up after a certain number of crashes or failures. Perhaps a --exit-after-n-failures flag could fill both of these needs? Excellent idea! (and that's a much better name) Created attachment 34704 [details]
Patch v1
Comment on attachment 34704 [details] Patch v1 > + my $testsWhithDidNotMatch = $count - $counts{match}; Which? Created attachment 34705 [details]
Removed extra change
Created attachment 34706 [details]
now with misspelling fixed
Comment on attachment 34706 [details] now with misspelling fixed > diff --git a/WebKitTools/Scripts/run-webkit-tests b/WebKitTools/Scripts/run-webkit-tests > index 1f908ec..003ac8d 100755 > --- a/WebKitTools/Scripts/run-webkit-tests > +++ b/WebKitTools/Scripts/run-webkit-tests > @@ -109,6 +109,7 @@ sub writeToFile($$); > # Argument handling > my $addPlatformExceptions = 0; > my $complexText = 0; > +my $exitAfterNFailures = 0; > my $guardMalloc = ''; > my $httpdPort = 8000; > my $httpdSSLPort = 8443; > @@ -193,6 +194,7 @@ Usage: $programName [options] [testdir|testpath ...] > --complex-text Use the complex text code path for all text (Mac OS X and Windows only) > -c|--configuration config Set DumpRenderTree build configuration > -g|--guard-malloc Enable malloc guard > + --exit-after-n-failures Exit after the first N failures instead of running all the tests The help for this flag should indicate that it takes an argument. Is there any reason to put it at this point in the help? Are they in any specific order? > countFinishedTest($test, $base, $result, $isText); > + > + my $testsWhichDidNotMatch = $count - $counts{match}; > + if ($exitAfterNFailures && $testsWhichDidNotMatch >= $exitAfterNFailures) { > + print "\nExiting early after $testsWhichDidNotMatch failures. $count tests run."; > + closeDumpTool(); > + last; > + } $testsWhichDidNotMatch sounds like an array of tests that didn't match. Perhaps $failureCount would be a clearer variable name? r=me (In reply to comment #7) > (From update of attachment 34706 [details]) > > diff --git a/WebKitTools/Scripts/run-webkit-tests b/WebKitTools/Scripts/run-webkit-tests > > index 1f908ec..003ac8d 100755 > > --- a/WebKitTools/Scripts/run-webkit-tests > > +++ b/WebKitTools/Scripts/run-webkit-tests > > @@ -109,6 +109,7 @@ sub writeToFile($$); > > # Argument handling > > my $addPlatformExceptions = 0; > > my $complexText = 0; > > +my $exitAfterNFailures = 0; > > my $guardMalloc = ''; > > my $httpdPort = 8000; > > my $httpdSSLPort = 8443; > > @@ -193,6 +194,7 @@ Usage: $programName [options] [testdir|testpath ...] > > --complex-text Use the complex text code path for all text (Mac OS X and Windows only) > > -c|--configuration config Set DumpRenderTree build configuration > > -g|--guard-malloc Enable malloc guard > > + --exit-after-n-failures Exit after the first N failures instead of running all the tests > > The help for this flag should indicate that it takes an argument. Is there any > reason to put it at this point in the help? Are they in any specific order? It seemed they were in alphabetical order. But I'm happy to put it wherever. > > countFinishedTest($test, $base, $result, $isText); > > + > > + my $testsWhichDidNotMatch = $count - $counts{match}; > > + if ($exitAfterNFailures && $testsWhichDidNotMatch >= $exitAfterNFailures) { > > + print "\nExiting early after $testsWhichDidNotMatch failures. $count tests run."; > > + closeDumpTool(); > > + last; > > + } > > $testsWhichDidNotMatch sounds like an array of tests that didn't match. > Perhaps $failureCount would be a clearer variable name? Sounds fine to me. Will change. Committing to http://svn.webkit.org/repository/webkit/trunk ... M WebKitTools/ChangeLog M WebKitTools/Scripts/bugzilla-tool M WebKitTools/Scripts/run-webkit-tests Committed r47166 M WebKitTools/ChangeLog M WebKitTools/Scripts/bugzilla-tool M WebKitTools/Scripts/run-webkit-tests r47166 = 581871acba07344f98ea6b67c0be8ed60106d917 (trunk) No changes between current HEAD and refs/remotes/trunk Resetting to the latest refs/remotes/trunk http://trac.webkit.org/changeset/47166 |