When investigating flaky failures, I usually do it with "old-run-webkit-tests -v". The reason is that it runs tests sequentially, and prints each test out (so I can easily see which tests ran prior to failing one, and are the likely culprits). This is more difficult with NRWT, because (in the order of decreasing importance): 1. its verbose output is way more verbose, to the degree that it's not easy to hunt down where test list is. 2. Retrying failing tests wastes time, and further confuses output. 3. Need to pass extra options (--num-processes) is understandable, but inconvenient nonetheless.
(In reply to comment #0) > When investigating flaky failures, I usually do it with "old-run-webkit-tests -v". The reason is that it runs tests sequentially, and prints each test out (so I can easily see which tests ran prior to failing one, and are the likely culprits). > > This is more difficult with NRWT, because (in the order of decreasing importance): > > 1. its verbose output is way more verbose, to the degree that it's not easy to hunt down where test list is. > I quite agree with this. I have been meaning to implement a simpler format for some time. I have two candidates in mind, and I'd be curious to get your feedback on them. The first is something along the lines of the difference between test-webkitpy and test-webkitpy -v. The first gives the "progress update" like you get today; the latter prints one line per test with the name of the test and passed/failed (this is also similar to how ninja's output works, and I've been meaning to converge with ninja here). The second option is more verbose: use new-run-webkit-tests --print trace-everything. That prints out the expected result (in the TestExpectations sense), the location of the baselines that will be used, the actual result, and the time the test took. I think both forms of output are useful. WDYT? > 2. Retrying failing tests wastes time, and further confuses output. You can disable this today with --no-retry-failures, in case you didn't know that. > 3. Need to pass extra options (--num-processes) is understandable, but inconvenient nonetheless. Yeah. I have no particular objection to adding another command line option that combines some or all o f these things, if that's generally useful. I personally recommend people create shell functions or aliases or other wrappers, though, since most people often find they want a particular combination of things (my most common aliases do disable retries, for example) and it's hard to please everyone without making the help even more unmanageably longer.
This will actually be implemented by the patch posted to bug 92432.
Created attachment 156219 [details] Patch
Comment on attachment 156219 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=156219&action=review rubber stamp. > Tools/ChangeLog:11 > + --debug-rwt-logging to get the full debug stream (aka old ORWT > + --verbose). You mean old NRTW --verbose?
(In reply to comment #4) > (From update of attachment 156219 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=156219&action=review > > rubber stamp. > > > Tools/ChangeLog:11 > > + --debug-rwt-logging to get the full debug stream (aka old ORWT > > + --verbose). > > You mean old NRTW --verbose? er, right.
Created attachment 157082 [details] patch for landing
Committed r124967: <http://trac.webkit.org/changeset/124967>
(In reply to comment #7) > Committed r124967: <http://trac.webkit.org/changeset/124967> It caused a regression. See https://bugs.webkit.org/show_bug.cgi?id=93434 for details.