Bug 256895 – TestExpectations should support some form of inheritance

• Component: Tools / Tests
• Status: NEW

[Anne van Kesteren]

As discussed in https://github.com/WebKit/WebKit/pull/13746#discussion_r1191059364 it would be great if

   imported/w3c/web-platform-tests/websockets [DumpJSConsoleLogInStdErr]
   imported/w3c/web-platform-tests/websockets/test.html [Pass Failure]

worked so we could keep all the external logic around tests in a single (set of) file(s).

[Fujii Hironori]

How about this case?

editing/mac [ Pass ]
editing/mac/dictionary-lookup/dictionary-lookup-input.html [ Crash ]

Pass won't be inherited?

[Radar WebKit Bug Importer]

<rdar://problem/109774162>

[Amanda Falke]

(In reply to Fujii Hironori from comment #1)
> How about this case?
> 
> editing/mac [ Pass ]
> editing/mac/dictionary-lookup/dictionary-lookup-input.html [ Crash ]
> 
> Pass won't be inherited?

Hi Fujii, great point and I thought this as well. According to this comment (https://github.com/WebKit/WebKit/pull/13746#discussion_r1192279547) that I found on the PR, inheritance won't work this way currently. It would be nice if it did

[Sam Sneddon [:gsnedders]]

(In reply to Fujii Hironori from comment #1)
> How about this case?
> 
> editing/mac [ Pass ]
> editing/mac/dictionary-lookup/dictionary-lookup-input.html [ Crash ]
> 
> Pass won't be inherited?

The problem here is we have a number of different things we put in TestExpectations; some of these—the actual expectations—are mutually exclusive ( Pass / Failure / Crash / Timeout / etc.), but some of these are modifiers that adjust how (or if) the test is run: Skip / WontFix / Slow / DumpJSConsoleLogInStdErr / Rebaseline.

At least to me, it seems sensible to allow the modifiers to be inherited separately to the expectations, though I believe we also rely on them _not_ being inherited today (Pass overriding Skip, for example).

If we _do_ inherit modifiers separately we probably need some way to negate them and go back to the default state.

[Jonathan Bedard]

I think the only time we could have expectations inherited as described is when those expectations are Slow or DumpJSConsoleLogInStdErr.

'Skip' should be overridden by 'Pass' (and that's often exactly why we're skipping entire directories, because a more specific platform will run those tests). Likewise, Fail/Crash/Timeout should all override Pass, and Pass should override Fail/Crash/Timeout. It's a bit weird for Fail/Crash/Timeout to override Skip, but it's hard to justify any other behavior, in my mind, since Fail/Crash/Timeout and Skip all override and are overridden by Pass.