test262-runner should run in verbose mode in automation so that: 1. It provides output so that we can get a better idea what is going wrong when something like this happens: https://build.webkit.org/builders/Apple%20Sierra%20Debug%20Test262%20%28Tests%29/builds/2594/steps/test262-test/logs/stdio 2. It doesn't hit the '1200 seconds without output' timeout if it is legitimately running tests, but taking longer than 1200 seconds.
Thanks for this feedback, Ryan. I've been wondering if we should have verbose mode in the build bot for a while, now you confirmed my thoughts. I have one extra question: should we report all the test files in the verbose mode? Right now it's limited to report all the failures (including the known failures). I believe if we print a line - or even a dot - for each passing test will support the '1200 seconds without output' better
(In reply to Leo Balter from comment #1) > Thanks for this feedback, Ryan. I've been wondering if we should have > verbose mode in the build bot for a while, now you confirmed my thoughts. > > I have one extra question: should we report all the test files in the > verbose mode? Right now it's limited to report all the failures (including > the known failures). I think it is fine to print everything in verbose mode, including known failures. We do this with 'run-webkit-tests' in automation (i.e. https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20(Tests)/builds/9702/steps/layout-test/logs/stdio). > I believe if we print a line - or even a dot - for each passing test will > support the '1200 seconds without output' better Yes, any output should prevent the step from hitting the buildbot timeout.
Created attachment 341633 [details] Patch
Ryan, I uploaded a patch with the verbose mode toggled for the build bot + dots for known passing tests. It still prints a full "NEW PASS $filename" for tests the were previously failing. I decided to try the dots as I believe this still makes some readable output. I'm also not printing anything the skipped tests, I don't think this will be a problem, but it's easy to also print anything for these cases.
Created attachment 341636 [details] Patch
I gave a better look to the wpt report and I believe I was being unnecessarily conservative on printing the passing and skipped files. The new update just prints everything, it's nice to tell what is being surfaced by the runner.
Comment on attachment 341636 [details] Patch Let's land it and watch on the bot results!
Comment on attachment 341636 [details] Patch Clearing flags on attachment: 341636 Committed r232343: <https://trac.webkit.org/changeset/232343>
All reviewed patches have been landed. Closing bug.
<rdar://problem/40680506>