Bug 174504 - Marking a directory as [ Pass ] or [ Failure ] or [ Crash ] should not override test-specific expectations
Summary: Marking a directory as [ Pass ] or [ Failure ] or [ Crash ] should not overri...
Status: RESOLVED WONTFIX
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2017-07-14 09:10 PDT by Jonathan Bedard
Modified: 2017-08-28 09:05 PDT (History)
3 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Bedard 2017-07-14 09:10:29 PDT 
Currently, when we mark an entire directory as something, we simply recursively mark every test in that directory as Pass/Failure/Crash/Skip.  We should be smarter about this, particularly with 'Pass.'  For example:

TestExpectations: edit/mac [ Skip ]
platform/wk2/TestExpectations: edit/mac/some-test.html [ Failure ] # XYZ is not implemented in WK2
platform/mac/TestExpectations: edit/mac [ Pass ] # Now edit/mac/some-test.html is marked as passing

This results in 1) duplicated expectations, 2) bugs separated across expectation files and 3) it being difficult to garden tests since platforms frequently mark entire directories as [ Pass ] when those platforms support a feature which their more general parent platform does not.  (ie: Mac generally may not support a given feature, but Sierra might).
Comment 1 Radar WebKit Bug Importer 2017-07-14 09:11:11 PDT 
<rdar://problem/33317289>
Comment 2 Jonathan Bedard 2017-08-28 09:05:42 PDT 
I spoke with Ryan Haddad and Dan Bates about this proposal on 8/25.  It seems that what I'm suggesting would not be an improvement over the current behavior.  Closing this bug.