Bug 88702 – [NRWT] Would like an output mode similar to ORWT verbose one

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Alexey Proskuryakov]

When investigating flaky failures, I usually do it with "old-run-webkit-tests -v". The reason is that it runs tests sequentially, and prints each test out (so I can easily see which tests ran prior to failing one, and are the likely culprits).

This is more difficult with NRWT, because (in the order of decreasing importance):

1. its verbose output is way more verbose, to the degree that it's not easy to hunt down where test list is.

2. Retrying failing tests wastes time, and further confuses output.

3. Need to pass extra options (--num-processes) is understandable, but inconvenient nonetheless.

[Dirk Pranke]

(In reply to comment #0)
> When investigating flaky failures, I usually do it with "old-run-webkit-tests -v". The reason is that it runs tests sequentially, and prints each test out (so I can easily see which tests ran prior to failing one, and are the likely culprits).
> 
> This is more difficult with NRWT, because (in the order of decreasing importance):
> 
> 1. its verbose output is way more verbose, to the degree that it's not easy to hunt down where test list is.
> 

I quite agree with this. I have been meaning to implement a simpler format for some time. I have two candidates in mind, and I'd be curious to get your feedback on them.

The first is something along the lines of the difference between test-webkitpy and test-webkitpy -v. The first gives the "progress update" like you get today; the latter prints one line per test with the name of the test and passed/failed (this is also similar to how ninja's output works, and I've been meaning to converge with ninja here).

The second option is more verbose: use new-run-webkit-tests --print trace-everything. That prints out the expected result (in the TestExpectations sense), the location of the baselines that will be used, the actual result, and the time the test took.

I think both forms of output are useful. WDYT?

> 2. Retrying failing tests wastes time, and further confuses output.

You can disable this today with --no-retry-failures, in case you didn't know that.
 
> 3. Need to pass extra options (--num-processes) is understandable, but inconvenient nonetheless.

Yeah. I have no particular objection to adding another command line option that combines some or all o f these things, if that's generally useful. 

I personally recommend people create shell functions or aliases or other wrappers, though, since most people often find they want a particular combination of things (my most common aliases do disable retries, for example) and it's hard to please everyone without making the help even more unmanageably longer.

[Dirk Pranke]

This will actually be implemented by the patch posted to bug 92432.

[Dirk Pranke]

Created attachment 156219 [details]
Patch

[Ryosuke Niwa]

Comment on attachment 156219 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=156219&action=review

rubber stamp.

> Tools/ChangeLog:11
> +        --debug-rwt-logging to get the full debug stream (aka old ORWT
> +        --verbose).

You mean old NRTW --verbose?

[Dirk Pranke]

(In reply to comment #4)
> (From update of attachment 156219 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=156219&action=review
> 
> rubber stamp.
> 
> > Tools/ChangeLog:11
> > +        --debug-rwt-logging to get the full debug stream (aka old ORWT
> > +        --verbose).
> 
> You mean old NRTW --verbose?

er, right.

[Dirk Pranke]

Created attachment 157082 [details]
patch for landing

[Dirk Pranke]

Committed r124967: <http://trac.webkit.org/changeset/124967>

[Csaba Osztrogonác]

(In reply to comment #7)
> Committed r124967: <http://trac.webkit.org/changeset/124967>

It caused a regression. See https://bugs.webkit.org/show_bug.cgi?id=93434 for details.