Bug 186527 - [test262-runner] Test output should summarize tests that are unexpectedly passing/failing
Summary: [test262-runner] Test output should summarize tests that are unexpectedly pas...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2018-06-11 10:18 PDT by Ryan Haddad
Modified: 2018-06-14 10:50 PDT (History)
9 users (show)

See Also:


Attachments
Patch (3.11 KB, patch)
2018-06-12 12:53 PDT, valerie
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews104 for mac-sierra-wk2 (3.12 MB, application/zip)
2018-06-12 15:31 PDT, EWS Watchlist
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ryan Haddad 2018-06-11 10:18:19 PDT
The build linked below includes 2 tests that are unexpectedly failing, but it isn't clear which ones they are:

https://build.webkit.org/builders/Apple%20Sierra%20Debug%20Test262%20%28Tests%29/builds/2802
Comment 1 Leo Balter 2018-06-11 10:32:31 PDT
yes, it makes sense when running from verbose mode. It was not necessary on the default run (without verbose) but while I don't have a patch available, the best to check for these are searching for "! NEW"...

Out of curiosity, the respective failures in that build are:

! NEW FAIL test/built-ins/Array/prototype/sort/comparefn-nonfunction-call-throws.js (strict mode)
! NEW FAIL test/built-ins/Array/prototype/sort/comparefn-nonfunction-call-throws.js (default)
Comment 2 Ryan Haddad 2018-06-11 10:37:50 PDT
(In reply to Leo Balter from comment #1)
> yes, it makes sense when running from verbose mode. It was not necessary on
> the default run (without verbose) but while I don't have a patch available,
> the best to check for these are searching for "! NEW"...
> 
> Out of curiosity, the respective failures in that build are:
> 
> ! NEW FAIL
> test/built-ins/Array/prototype/sort/comparefn-nonfunction-call-throws.js
> (strict mode)
> ! NEW FAIL
> test/built-ins/Array/prototype/sort/comparefn-nonfunction-call-throws.js
> (default)
Thank you!
Comment 3 valerie 2018-06-12 12:53:38 PDT
Created attachment 342583 [details]
Patch
Comment 4 valerie 2018-06-12 12:55:19 PDT
In vebose mode, the PASS or FAIL result of every test is printed. This patch adds an additional summary at the end of the output when you are in verbose mode, listing the newly passing and newly failing tests all in one place:

---------------NEW FAILING TESTS SUMMARY---------------

FAIL test/annexB/language/eval-code/direct/func-block-decl-eval-func-skip-early-err-block.js (default)
Test262Error: An initialized binding is not created prior to evaluation Expected a ReferenceError to be thrown but no exception was thrown at all

FAIL test/annexB/language/eval-code/direct/func-block-decl-eval-func-skip-early-err-for-in.js (default)
Test262Error: An initialized binding is not created prior to evaluation Expected a ReferenceError to be thrown but no exception was thrown at all

---------------NEW PASSING TESTS SUMMARY---------------

PASS test/annexB/language/eval-code/direct/func-block-decl-eval-func-init.js (default)

---------------------------------------------------------
Comment 5 EWS Watchlist 2018-06-12 15:31:16 PDT
Comment on attachment 342583 [details]
Patch

Attachment 342583 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/8152168

New failing tests:
accessibility/smart-invert-reference.html
Comment 6 EWS Watchlist 2018-06-12 15:31:18 PDT
Created attachment 342604 [details]
Archive of layout-test-results from ews104 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6
Comment 7 Michael Saboff 2018-06-14 10:22:39 PDT
Comment on attachment 342583 [details]
Patch

r=me
Comment 8 WebKit Commit Bot 2018-06-14 10:49:07 PDT
Comment on attachment 342583 [details]
Patch

Clearing flags on attachment: 342583

Committed r232841: <https://trac.webkit.org/changeset/232841>
Comment 9 WebKit Commit Bot 2018-06-14 10:49:09 PDT
All reviewed patches have been landed.  Closing bug.
Comment 10 Radar WebKit Bug Importer 2018-06-14 10:50:27 PDT
<rdar://problem/41130656>