Include detailed test modifiers (e.g. FLAKY) in results.json for failing non-layout tests. Currently the results.json only includes FAIL or PASS information.
Created attachment 66935 [details] Patch
Comment on attachment 66935 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=66935&action=prettypatch > WebKitTools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:64 > + # Shouldn't happen If this shouldn't happen, lets just leave out the try/except clause. Otherwise, if this starts happening, we'll never know. I'd be OK with the try/except clause if we logged the error (i.e. a warning level log). But, we shouldn't just silently ignore it. > WebKitTools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:820 > individual_test_timings) Indentation is off by a space.
Created attachment 67775 [details] Patch
Sorry it took for me so long to update this again. Basically mostly same with the previous patch, except that I changed how to record FIXABLE_COUNT. Now it records 'F' for tests with FAILS modifier, 'X' for disabled tests, 'L' for tests with FLAKY modifier and 'P' for others. (In an assumption that the number of FAILS_ tests is the one we should focus on to reduce failures - as we discussed.) (In reply to comment #2) > (From update of attachment 66935 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=66935&action=prettypatch > > > WebKitTools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:64 > > + # Shouldn't happen > If this shouldn't happen, lets just leave out the try/except clause. Otherwise, if this starts happening, we'll never know. I'd be OK with the try/except clause if we logged the error (i.e. a warning level log). But, we shouldn't just silently ignore it. Added logging.warn in the except part. > > WebKitTools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:820 > > individual_test_timings) > Indentation is off by a space. Fixed.
Comment on attachment 67775 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=67775&action=prettypatch > WebKitTools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:65 > + pass don't need the pass anymore.
Committed r67692: <http://trac.webkit.org/changeset/67692>