Bug 155476 - run-api-tests doesn't print test name when the test crashes
Summary: run-api-tests doesn't print test name when the test crashes
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Safari 9
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alexey Proskuryakov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-14 17:13 PDT by Alexey Proskuryakov
Modified: 2016-03-15 15:04 PDT (History)
5 users (show)

See Also:


Attachments
proposed fix (2.09 KB, patch)
2016-03-14 17:17 PDT, Alexey Proskuryakov
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews114 for mac-yosemite (1.10 MB, application/zip)
2016-03-14 18:21 PDT, Build Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2016-03-14 17:13:28 PDT
When run-api-tests crashes, the test name is not printed. Even though the test is counted as a failing one and printed at the end, this makes it unnecessarily hard to find assertion stack traces in run-api-tests output.
Comment 1 Alexey Proskuryakov 2016-03-14 17:17:06 PDT
Created attachment 274057 [details]
proposed fix
Comment 2 Build Bot 2016-03-14 18:21:34 PDT
Comment on attachment 274057 [details]
proposed fix

Attachment 274057 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/979670

New failing tests:
js/function-call-aliased.html
Comment 3 Build Bot 2016-03-14 18:21:36 PDT
Created attachment 274061 [details]
Archive of layout-test-results from ews114 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews114  Port: mac-yosemite  Platform: Mac OS X 10.10.5
Comment 4 Daniel Bates 2016-03-14 20:23:04 PDT
Comment on attachment 274057 [details]
proposed fix

View in context: https://bugs.webkit.org/attachment.cgi?id=274057&action=review

> Tools/Scripts/run-api-tests:237
> +            if (!$timedOut && index("@testOutput", $test) == -1) {
> +                print STDOUT possiblyColored("bold red", "UNEXPECTEDLY EXITED"), " $test\n";
> +            }

This is a circuitous way of determining that $test is not in the array @testOutput. We should make use of grep [1] to write this more directly:

if (!$timedOut && !grep { $_ eq $test } @testOutput) {
    print STDOUT possiblyColored("bold red", "UNEXPECTEDLY EXITED"), " $test\n";
}

[1] <http://perldoc.perl.org/functions/grep.html>
Comment 5 Alexey Proskuryakov 2016-03-14 23:55:20 PDT
Is that indeed the case? I believe that it's an array of lines, and the test name doesn't take the whole line.
Comment 6 Daniel Bates 2016-03-15 13:45:48 PDT
(In reply to comment #5)
> Is that indeed the case? I believe that it's an array of lines, and the test
> name doesn't take the whole line.

Notice that grep can we used with a regular expression. Using the \Q and \E specifiers to disable integration of regular expression metacharacters, can we write something of the form:

if (!$timedOut && !grep { /\Q$test\E/ } @testOutput) {
    print STDOUT possiblyColored("bold red", "UNEXPECTEDLY EXITED"), " $test\n";
}

We should look to strengthen this regular expression if possible.
Comment 7 Daniel Bates 2016-03-15 13:50:03 PDT
(In reply to comment #6)
> (In reply to comment #5)
> > Is that indeed the case? I believe that it's an array of lines, and the test
> > name doesn't take the whole line.
> 
> Notice that grep can we used with a regular expression. Using the \Q and \E
> specifiers to disable integration of regular expression metacharacters 

I meant to write:

Notice that grep can be used with a regular expression. Using the metaquoting escape sequence \Q...\E (*) to disable interpretation of regular expression metacharacters, can we write something ...

(*) See <http://perldoc.perl.org/perlre.html#Quoting-metacharacters> for more details.
Comment 8 Alexey Proskuryakov 2016-03-15 14:14:00 PDT
Comment on attachment 274057 [details]
proposed fix

Marking cq+ per discussion with Dan - the potential improvement does not seem to be worth the effort.
Comment 9 WebKit Commit Bot 2016-03-15 15:04:22 PDT
Comment on attachment 274057 [details]
proposed fix

Clearing flags on attachment: 274057

Committed r198231: <http://trac.webkit.org/changeset/198231>
Comment 10 WebKit Commit Bot 2016-03-15 15:04:27 PDT
All reviewed patches have been landed.  Closing bug.