Bug 45408 – Include detailed test modifiers (e.g. FLAKY) in results.json for non-layout tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Kinuko Yasuda]

Include detailed test modifiers (e.g. FLAKY) in results.json for failing non-layout tests.
Currently the results.json only includes FAIL or PASS information.

[Kinuko Yasuda]

Created attachment 66935 [details]
Patch

[Ojan Vafai]

Comment on attachment 66935 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=66935&action=prettypatch

> WebKitTools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:64
> +            # Shouldn't happen
If this shouldn't happen, lets just leave out the try/except clause. Otherwise, if this starts happening, we'll never know. I'd be OK with the try/except clause if we logged the error (i.e. a warning level log). But, we shouldn't just silently ignore it.

> WebKitTools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:820
>                                     individual_test_timings)
Indentation is off by a space.

[Kinuko Yasuda]

Created attachment 67775 [details]
Patch

[Kinuko Yasuda]

Sorry it took for me so long to update this again.

Basically mostly same with the previous patch, except that I changed how to record FIXABLE_COUNT.
Now it records 'F' for tests with FAILS modifier, 'X' for disabled tests, 'L' for tests with FLAKY modifier and 'P' for others.  (In an assumption that the number of FAILS_ tests is the one we should focus on to  reduce failures - as we discussed.)

(In reply to comment #2)
> (From update of attachment 66935 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=66935&action=prettypatch
> 
> > WebKitTools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:64
> > +            # Shouldn't happen
> If this shouldn't happen, lets just leave out the try/except clause. Otherwise, if this starts happening, we'll never know. I'd be OK with the try/except clause if we logged the error (i.e. a warning level log). But, we shouldn't just silently ignore it.

Added logging.warn in the except part.

> > WebKitTools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:820
> >                                     individual_test_timings)
> Indentation is off by a space.

Fixed.

[Ojan Vafai]

Comment on attachment 67775 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=67775&action=prettypatch

> WebKitTools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:65
> +            pass

don't need the pass anymore.

[Kinuko Yasuda]

Committed r67692: <http://trac.webkit.org/changeset/67692>