In this bug, I want to add an option showing each test result at console window for run-javascriptcore-tests.
Created attachment 181033 [details] Patch
Ccing relevant apple folks
@Eric Seidel Thanks eric!
Comment on attachment 181033 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=181033&action=review > Source/JavaScriptCore/ChangeLog:11 > + By showing test result immediately at console, we can recognize which test is failed > + or not without ransacking the final result file. Notice that you can run run-javascriptcore-tests with --jsDriver-args=-k or --jsDriver-args=--confail to have test failures logged to the console. I'm unclear of the necessity to "ransack" (see remark (*)) the final results file to determine the tests that failed given these command line options. Disregarding where the output is printed (stderr vs stdout), the difference between -k/--confail and the proposed functionality in this patch is that we print a "passed" message for each test that passed. I take it that such "passed" messages are useful? (*) I assume that you chose the word "ransack" to emphasize your dislike of the verbose test failure output emitted by jsDriver.pl as opposed to its literal meaning, which implies that the output isn't straightforward to read. If you meant the latter then we should look to clean up the test failure output so that it is straightforward to interpret, including making it straightforward to identify the test(s) that failed.
(In reply to comment #4) > (From update of attachment 181033 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=181033&action=review > > > Source/JavaScriptCore/ChangeLog:11 > > + By showing test result immediately at console, we can recognize which test is failed > > + or not without ransacking the final result file. > > Notice that you can run run-javascriptcore-tests with --jsDriver-args=-k or --jsDriver-args=--confail to have test failures logged to the console. I'm unclear of the necessity to "ransack" (see remark (*)) the final results file to determine the tests that failed given these command line options. Disregarding where the output is printed (stderr vs stdout), the difference between -k/--confail and the proposed functionality in this patch is that we print a "passed" message for each test that passed. I take it that such "passed" messages are useful? > > (*) I assume that you chose the word "ransack" to emphasize your dislike of the verbose test failure output emitted by jsDriver.pl as opposed to its literal meaning, which implies that the output isn't straightforward to read. If you meant the latter then we should look to clean up the test failure output so that it is straightforward to interpret, including making it straightforward to identify the test(s) that failed. Yeah. I know the option '-k' which shows detailed information for the failed test. And your thought is generally right. Why I propose a this kind option is that I think it's better to provide straightforward and simple test result for an automatic test set. Actually I believe we need the detailed failures only when we debug a specific failed test, not running full tests. As you told, the current -k|--confail option seems a little bit confusing so that it is also hard to get a normalized result to be adopted by any automatic testing framework like lava. For these reasons, I hope to add a new option showing more simple test results for full testing or to revise the -k option to show more straightforward and normalized results like layout test verbose option.
Comment on attachment 181033 [details] Patch I don’t think it’s worthwhile to add even more options to the Mozilla test driver unless we have a clear project goal in mind. Up until this point we have just included a copy of this driver script from the Mozilla project as-is with the minimum changes needed to use it and I don’t see the compelling need to add enhancements of our own. review- for now unless we decide we have a project discussion and decide we want this.