Bug 186115 – test262-runner should run in verbose mode in automation

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Ryan Haddad]

test262-runner should run in verbose mode in automation so that:

1. It provides output so that we can get a better idea what is going wrong when something like this happens: https://build.webkit.org/builders/Apple%20Sierra%20Debug%20Test262%20%28Tests%29/builds/2594/steps/test262-test/logs/stdio

2. It doesn't hit the '1200 seconds without output' timeout if it is legitimately running tests, but taking longer than 1200 seconds.

[Leo Balter]

Thanks for this feedback, Ryan. I've been wondering if we should have verbose mode in the build bot for a while, now you confirmed my thoughts.

I have one extra question: should we report all the test files in the verbose mode? Right now it's limited to report all the failures (including the known failures).

I believe if we print a line - or even a dot - for each passing test will support the '1200 seconds without output' better

[Ryan Haddad]

(In reply to Leo Balter from comment #1)
> Thanks for this feedback, Ryan. I've been wondering if we should have
> verbose mode in the build bot for a while, now you confirmed my thoughts.
> 
> I have one extra question: should we report all the test files in the
> verbose mode? Right now it's limited to report all the failures (including
> the known failures).

I think it is fine to print everything in verbose mode, including known failures. We do this with 'run-webkit-tests' in automation (i.e. https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20(Tests)/builds/9702/steps/layout-test/logs/stdio).

> I believe if we print a line - or even a dot - for each passing test will
> support the '1200 seconds without output' better

Yes, any output should prevent the step from hitting the buildbot timeout.

[Leo Balter]

Created attachment 341633 [details]
Patch

[Leo Balter]

Ryan, I uploaded a patch with the verbose mode toggled for the build bot + dots for known passing tests. It still prints a full "NEW PASS $filename" for tests the were previously failing.

I decided to try the dots as I believe this still makes some readable output.

I'm also not printing anything the skipped tests, I don't think this will be a problem, but it's easy to also print anything for these cases.

[Leo Balter]

Created attachment 341636 [details]
Patch

[Leo Balter]

I gave a better look to the wpt report and I believe I was being unnecessarily conservative on printing the passing and skipped files.

The new update just prints everything, it's nice to tell what is being surfaced by the runner.

[Yusuke Suzuki]

Comment on attachment 341636 [details]
Patch

Let's land it and watch on the bot results!

[WebKit Commit Bot]

Comment on attachment 341636 [details]
Patch

Clearing flags on attachment: 341636

Committed r232343: <https://trac.webkit.org/changeset/232343>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/40680506>