Bug 37783 – new-run-webkit-tests: add a way to print out or retry the last set of tests that failed

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Dirk Pranke]

I frequently want to re-run new-run-webkit-tests and just re-test the set of tests that failed. It would be nice if there was a way to print the last tests out easily or read them into new-run-webkit-tests. The data is in the unexpected_results.json file, so we just have to process it.

[Dirk Pranke]

Created attachment 53646 [details]
Patch

[Eric Seidel (no email)]

Comment on attachment 53646 [details]
Patch

Looks useful.

[Ojan Vafai]

Comment on attachment 53646 [details]
Patch

> +        optparse.make_option("--print-last-results", action="store_true",
> +            default=False, help="print the tests in the last run that "
> +            "had unexpected results."),

Do we really need an option for this? Lets just always do this. If python had a way of doing "hidden" commands the way the webkit-patch does for commands, I'd be OK with adding all these options. But having them all visible to users of the script by default makes the script harder to use.

> +        optparse.make_option("--retry-last-results", action="store_true",
> +            default=False, help="re-try the tests in the last run that "
> +            "had unexpected results."),

Nit: last-results isn't clear without reading the help text or the code. How about --retry-unexpected-failures?

[Dirk Pranke]

(In reply to comment #3)
> (From update of attachment 53646 [details])
> > +        optparse.make_option("--print-last-results", action="store_true",
> > +            default=False, help="print the tests in the last run that "
> > +            "had unexpected results."),
> 
> Do we really need an option for this? Lets just always do this. If python had a
> way of doing "hidden" commands the way the webkit-patch does for commands, I'd
> be OK with adding all these options. But having them all visible to users of
> the script by default makes the script harder to use.

What do you mean by "always do this"? The patch prints the list and exits if that flag is passed in.

> > +        optparse.make_option("--retry-last-results", action="store_true",
> > +            default=False, help="re-try the tests in the last run that "
> > +            "had unexpected results."),
> 
> Nit: last-results isn't clear without reading the help text or the code. How
> about --retry-unexpected-failures?

Of course, we retry unexpected passes as well. --retry-unexpected-results? I could also live with -failures, as that is the main intent.

-- Dirk

[Dirk Pranke]

Created attachment 53723 [details]
Patch

[Dirk Pranke]

Committed r57843: <http://trac.webkit.org/changeset/57843>