Bug 153950 – run-webkit-tests header output is wordy, not useful, should not write out by default

• Status: NEW
• Component: Tools / Tests

[Darin Adler]

Here’s what run-webkit-tests writes out at the start:

Using port 'mac-elcapitan-wk2'
Test configuration: <elcapitan, x86_64, debug>
Placing test results in /Users/darin/Build/Debug/layout-test-results
Baseline search path: mac-wk2 -> wk2 -> mac -> generic
Using Debug build
Pixel tests disabled
Regular timeout: 30000, slow test timeout: 150000
Command line: /Users/darin/Build/Debug/WebKitTestRunner -

Found 44215 tests; running 40869, skipping 3346.
Running 6 WebKitTestRunners in parallel.

*None* of this is important to me when I’m running tests. I suggest we not write any of this out unless it’s requested with a command line option.

[Alexey Proskuryakov]

Some of these do answer relevant questions (like "why my expectation that I just added didn't work", or "what happened now, when I forgot to manually add the --debug option").

The number of skipped tests is very relevant when running tests in one directory, I needed to know that many times. Seeing the number of parallel processes frequently helps me realize that I need to add or remove --fully-parallel when trying to reproduce an issue.

And most of these are important when diagnosing an issue that someone has with the tests - it would be wasteful to request re-running them with extra logging enabled.

Only these are the line that I personally don't see much use in, unless verbose logging is enabled:

> Pixel tests disabled
> Regular timeout: 30000, slow test timeout: 150000
> Command line: /Users/darin/Build/Debug/WebKitTestRunner -

[Darin Adler]

(In reply to comment #1)
> Some of these do answer relevant questions (like "why my expectation that I
> just added didn't work", or "what happened now, when I forgot to manually
> add the --debug option").

Questions relevant to who? I am unconvinced. I don’t see how the output answers either of those questions.

> The number of skipped tests is very relevant when running tests in one
> directory, I needed to know that many times.

Sure, but I am not running tests in one directory. We can write this out when doing any special type of test running, but not in the normal case.

> Seeing the number of parallel
> processes frequently helps me realize that I need to add or remove
> --fully-parallel when trying to reproduce an issue.

Yes but it never helps me. Is there some way we can make this clear to you in a less wordy way?

> And most of these are important when diagnosing an issue that someone has
> with the tests - it would be wasteful to request re-running them with extra
> logging enabled.

I’d like some evidence this is true. Could you cite any recent example where someone would have had to re-run? If re-running is the concern, then we should write this data into a file.

You haven’t convinced me that any of this output is helpful in the day-to-day while running tests.

[Alexey Proskuryakov]

One example is bug 153951, where I had to ask a question that could easily be answered from a complete log.

It may be worth talking about some other user scenarios in person. I do get questions about how run-webkit-tests works, and I expect that the level of mystery would unnecessarily increase if we attempted to make the information less discoverable.