API tests should output json results

[Jonathan Bedard]

Requested by Aakash Jain for adding API tests to EWS.

[Jonathan Bedard]

<rdar://problem/43615652>

[Jonathan Bedard]

Created attachment 347876 [details]
Patch

[Jonathan Bedard]

Created attachment 347879 [details]
Patch

[Aakash Jain]

Comment on attachment 347879 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=347879&action=review

Thanks for the patch. Looks good, one comment below.

> Tools/Scripts/webkitpy/api_tests/manager.py:208
> +                result_dictionary[test] = {'status': 'Skipped', 'output': None}

Do we need to add skipped tests in the json file? EWS doesn't need this, but I understand some other tools might use this information.

If so, we should consider dividing the tests by keys. e.g.:

{
    "skipped": [ {"test_name1": <test_log1>}, {"test_name2": <test_log2>} ],
    "failed": [ {"test_name3": <test_log3>} ],
    "crashed": [ {"test_name4": <test_log4>} ],
    "timedout": [ {"test_name5": <test_log5>} ]
}

[Jonathan Bedard]

Comment on attachment 347879 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=347879&action=review

>> Tools/Scripts/webkitpy/api_tests/manager.py:208
>> +                result_dictionary[test] = {'status': 'Skipped', 'output': None}
> 
> Do we need to add skipped tests in the json file? EWS doesn't need this, but I understand some other tools might use this information.
> 
> If so, we should consider dividing the tests by keys. e.g.:
> 
> {
>     "skipped": [ {"test_name1": <test_log1>}, {"test_name2": <test_log2>} ],
>     "failed": [ {"test_name3": <test_log3>} ],
>     "crashed": [ {"test_name4": <test_log4>} ],
>     "timedout": [ {"test_name5": <test_log5>} ]
> }

We do not, but, for API tests, 'SKIPPED' does not mean 'DISABLED', it means that a signal was received forcing the program to exit early. In practice, this means that EWS should never see SKIPPED tests and if it did, something has gone terribly wrong.

[Jonathan Bedard]

Created attachment 347952 [details]
Patch

[Jonathan Bedard]

Created attachment 347956 [details]
Patch

[Jonathan Bedard]

Created attachment 347960 [details]
Patch for landing

[WebKit Commit Bot]

Comment on attachment 347960 [details]
Patch for landing

Clearing flags on attachment: 347960

Committed r235252: <https://trac.webkit.org/changeset/235252>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/43662254>