Bug 186527 – [test262-runner] Test output should summarize tests that are unexpectedly passing/failing

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Ryan Haddad]

The build linked below includes 2 tests that are unexpectedly failing, but it isn't clear which ones they are:

https://build.webkit.org/builders/Apple%20Sierra%20Debug%20Test262%20%28Tests%29/builds/2802

[Leo Balter]

yes, it makes sense when running from verbose mode. It was not necessary on the default run (without verbose) but while I don't have a patch available, the best to check for these are searching for "! NEW"...

Out of curiosity, the respective failures in that build are:

! NEW FAIL test/built-ins/Array/prototype/sort/comparefn-nonfunction-call-throws.js (strict mode)
! NEW FAIL test/built-ins/Array/prototype/sort/comparefn-nonfunction-call-throws.js (default)

[Ryan Haddad]

(In reply to Leo Balter from comment #1)
> yes, it makes sense when running from verbose mode. It was not necessary on
> the default run (without verbose) but while I don't have a patch available,
> the best to check for these are searching for "! NEW"...
> 
> Out of curiosity, the respective failures in that build are:
> 
> ! NEW FAIL
> test/built-ins/Array/prototype/sort/comparefn-nonfunction-call-throws.js
> (strict mode)
> ! NEW FAIL
> test/built-ins/Array/prototype/sort/comparefn-nonfunction-call-throws.js
> (default)
Thank you!

[valerie]

Created attachment 342583 [details]
Patch

[valerie]

In vebose mode, the PASS or FAIL result of every test is printed. This patch adds an additional summary at the end of the output when you are in verbose mode, listing the newly passing and newly failing tests all in one place:

---------------NEW FAILING TESTS SUMMARY---------------

FAIL test/annexB/language/eval-code/direct/func-block-decl-eval-func-skip-early-err-block.js (default)
Test262Error: An initialized binding is not created prior to evaluation Expected a ReferenceError to be thrown but no exception was thrown at all

FAIL test/annexB/language/eval-code/direct/func-block-decl-eval-func-skip-early-err-for-in.js (default)
Test262Error: An initialized binding is not created prior to evaluation Expected a ReferenceError to be thrown but no exception was thrown at all

---------------NEW PASSING TESTS SUMMARY---------------

PASS test/annexB/language/eval-code/direct/func-block-decl-eval-func-init.js (default)

---------------------------------------------------------

[EWS Watchlist]

Comment on attachment 342583 [details]
Patch

Attachment 342583 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/8152168

New failing tests:
accessibility/smart-invert-reference.html

[EWS Watchlist]

Created attachment 342604 [details]
Archive of layout-test-results from ews104 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6

[Michael Saboff]

Comment on attachment 342583 [details]
Patch

r=me

[WebKit Commit Bot]

Comment on attachment 342583 [details]
Patch

Clearing flags on attachment: 342583

Committed r232841: <https://trac.webkit.org/changeset/232841>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/41130656>