Bug 153950
| Summary: | run-webkit-tests header output is wordy, not useful, should not write out by default | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Darin Adler <darin> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | NEW | ||
| Severity: | Normal | CC: | ap, lforschler |
| Priority: | P2 | ||
| Version: | WebKit Local Build | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
Darin Adler
Here’s what run-webkit-tests writes out at the start:
Using port 'mac-elcapitan-wk2'
Test configuration: <elcapitan, x86_64, debug>
Placing test results in /Users/darin/Build/Debug/layout-test-results
Baseline search path: mac-wk2 -> wk2 -> mac -> generic
Using Debug build
Pixel tests disabled
Regular timeout: 30000, slow test timeout: 150000
Command line: /Users/darin/Build/Debug/WebKitTestRunner -
Found 44215 tests; running 40869, skipping 3346.
Running 6 WebKitTestRunners in parallel.
*None* of this is important to me when I’m running tests. I suggest we not write any of this out unless it’s requested with a command line option.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Alexey Proskuryakov
Some of these do answer relevant questions (like "why my expectation that I just added didn't work", or "what happened now, when I forgot to manually add the --debug option").
The number of skipped tests is very relevant when running tests in one directory, I needed to know that many times. Seeing the number of parallel processes frequently helps me realize that I need to add or remove --fully-parallel when trying to reproduce an issue.
And most of these are important when diagnosing an issue that someone has with the tests - it would be wasteful to request re-running them with extra logging enabled.
Only these are the line that I personally don't see much use in, unless verbose logging is enabled:
> Pixel tests disabled
> Regular timeout: 30000, slow test timeout: 150000
> Command line: /Users/darin/Build/Debug/WebKitTestRunner -
Darin Adler
(In reply to comment #1)
> Some of these do answer relevant questions (like "why my expectation that I
> just added didn't work", or "what happened now, when I forgot to manually
> add the --debug option").
Questions relevant to who? I am unconvinced. I don’t see how the output answers either of those questions.
> The number of skipped tests is very relevant when running tests in one
> directory, I needed to know that many times.
Sure, but I am not running tests in one directory. We can write this out when doing any special type of test running, but not in the normal case.
> Seeing the number of parallel
> processes frequently helps me realize that I need to add or remove
> --fully-parallel when trying to reproduce an issue.
Yes but it never helps me. Is there some way we can make this clear to you in a less wordy way?
> And most of these are important when diagnosing an issue that someone has
> with the tests - it would be wasteful to request re-running them with extra
> logging enabled.
I’d like some evidence this is true. Could you cite any recent example where someone would have had to re-run? If re-running is the concern, then we should write this data into a file.
You haven’t convinced me that any of this output is helpful in the day-to-day while running tests.
Alexey Proskuryakov
One example is bug 153951, where I had to ask a question that could easily be answered from a complete log.
It may be worth talking about some other user scenarios in person. I do get questions about how run-webkit-tests works, and I expect that the level of mystery would unnecessarily increase if we attempted to make the information less discoverable.