Bug 105926 - Add a 'verbose' option to show each test result immediately at console.
Summary: Add a 'verbose' option to show each test result immediately at console.
Status: UNCONFIRMED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-01-02 08:37 PST by ChangSeok Oh
Modified: 2014-08-19 09:05 PDT (History)
3 users (show)

See Also:


Attachments
Patch (4.31 KB, patch)
2013-01-02 09:18 PST, ChangSeok Oh
darin: review-
darin: commit-queue-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description ChangSeok Oh 2013-01-02 08:37:14 PST
In this bug, I want to add an option showing each test result at console window for run-javascriptcore-tests.
Comment 1 ChangSeok Oh 2013-01-02 09:18:24 PST
Created attachment 181033 [details]
Patch
Comment 2 Eric Seidel (no email) 2013-01-05 00:29:50 PST
Ccing relevant apple folks
Comment 3 ChangSeok Oh 2013-01-05 03:19:20 PST
@Eric Seidel Thanks eric!
Comment 4 Daniel Bates 2013-01-05 17:16:01 PST
Comment on attachment 181033 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=181033&action=review

> Source/JavaScriptCore/ChangeLog:11
> +        By showing test result immediately at console, we can recognize which test is failed
> +        or not without ransacking the final result file.

Notice that you can run run-javascriptcore-tests with --jsDriver-args=-k or --jsDriver-args=--confail to have test failures logged to the console. I'm unclear of the necessity to "ransack" (see remark (*)) the final results file to determine the tests that failed given these command line options. Disregarding where the output is printed (stderr vs stdout), the difference between -k/--confail and the proposed functionality in this patch is that we print a "passed" message for each test that passed. I take it that such "passed" messages are useful?

(*) I assume that you chose the word "ransack" to emphasize your dislike of the verbose test failure output emitted by jsDriver.pl as opposed to its literal meaning, which implies that the output isn't straightforward to read. If you meant the latter then we should look to clean up the test failure output so that it is straightforward to interpret, including making it straightforward to identify the test(s) that failed.
Comment 5 ChangSeok Oh 2013-01-07 08:50:17 PST
(In reply to comment #4)
> (From update of attachment 181033 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=181033&action=review
> 
> > Source/JavaScriptCore/ChangeLog:11
> > +        By showing test result immediately at console, we can recognize which test is failed
> > +        or not without ransacking the final result file.
> 
> Notice that you can run run-javascriptcore-tests with --jsDriver-args=-k or --jsDriver-args=--confail to have test failures logged to the console. I'm unclear of the necessity to "ransack" (see remark (*)) the final results file to determine the tests that failed given these command line options. Disregarding where the output is printed (stderr vs stdout), the difference between -k/--confail and the proposed functionality in this patch is that we print a "passed" message for each test that passed. I take it that such "passed" messages are useful?
> 
> (*) I assume that you chose the word "ransack" to emphasize your dislike of the verbose test failure output emitted by jsDriver.pl as opposed to its literal meaning, which implies that the output isn't straightforward to read. If you meant the latter then we should look to clean up the test failure output so that it is straightforward to interpret, including making it straightforward to identify the test(s) that failed.

Yeah. I know the option '-k' which shows detailed information for the failed test. And your thought is generally right. 
Why I propose a this kind option is that I think it's better to provide straightforward and simple test result for an automatic test set.
Actually I believe we need the detailed failures only when we debug a specific failed test, not running full tests. As you told, the current -k|--confail option seems a little bit confusing  so that it is also hard to get a normalized result to be adopted by any automatic testing framework like lava.
For these reasons, I hope to add a new option showing more simple test results for full testing or to revise the -k option to show more straightforward and normalized results like layout test verbose option.
Comment 6 Darin Adler 2014-08-19 09:05:30 PDT
Comment on attachment 181033 [details]
Patch

I don’t think it’s worthwhile to add even more options to the Mozilla test driver unless we have a clear project goal in mind. Up until this point we have just included a copy of this driver script from the Mozilla project as-is with the minimum changes needed to use it and I don’t see the compelling need to add enhancements of our own. review- for now unless we decide we have a project discussion and decide we want this.