Bug 72080 – Layout tests marked as SKIP should not be overridable

• Status: NEW
• Component: Tools / Tests

[Andrew Scherkus]

Split off from https://bugs.webkit.org/show_bug.cgi?id=72001

"""
One possible option would be to make any directories marked as SKIP override expected results for specific files inside the directory; I am not aware of any case in practice where we want to skip all but X files in a dir, and if we did, we could support that by requiring the user to explicitly list all of the tests to skip.
"""

The canonical example is with GPU tests. Most of the layout tests don't need to run so there are expectations such as:
WONTFIX SKIP GPU GPU-CG LEOPARD : media = PASS TIMEOUT FAIL

However individual tests might start running again if a more specific expectation is added later:
BUGWK70748 GPU : media/audio-controls-rendering.html = IMAGE

This leads to increased test cycle time and generates noise on the flakiness dashboard as tests that were previously not generating results are now showing up with results.

As discussed in bug 72001, the current way GPU tests handle this is by manually adding/removing directories in Tools/Scripts/webkitpy/layout_tests/port/chromium_gpu.py

The chromium_gpu.py method isn't ideal either, and perhaps the non-overridable SKIP isn't either so perhaps we can think of something better here.