Currently, mac expectations do not conform to expectations on other platforms. Modify default_baseline_search_path so that mac expectations work like iOS, GTK and WPE test expectations. That is to say, expectations should look like this: {port}-{version}-wk{1/2} -> {port}-{version} -> {port}-wk{1-2} -> {port} -> wk2{optional} -> generic
<rdar://problem/33297388>
Created attachment 315375 [details] Patch
Talked to Dave Kilzer about this. This will likely cause test failures, at least in it's current form. I'm letting EWS chew on it to see exactly how bad this is.
Comment on attachment 315375 [details] Patch Attachment 315375 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/4115247 New failing tests: editing/mac/input/kotoeri-enter-to-confirm-and-newline.html editing/mac/input/hangul-enter-confirms-and-sends-keypress.html editing/mac/input/firstrectforcharacterrange-vertical.html editing/mac/input/firstrectforcharacterrange-plain.html editing/mac/input/caret-primary-bidi.html accessibility/mac/search-predicate.html accessibility/mac/html-slider-indicator.html editing/mac/input/wrapped-line-char-rect.html editing/mac/input/range-for-empty-document.html editing/mac/input/NSBackgroundColor-transparent.html editing/mac/input/5576619.html editing/mac/input/text-input-controller.html editing/mac/pasteboard/paste-and-match-style-selector-event.html editing/mac/input/firstrectforcharacterrange-caret-in-br.html editing/mac/input/firstrectforcharacterrange-styled.html editing/mac/input/secure-input.html editing/mac/selection/25228.html editing/mac/input/replace-invalid-range.html accessibility/mac/listbox-hit-test.html editing/mac/input/bold-node.html editing/mac/input/text-input-controller-no-editable-no-crash.html accessibility/mac/iframe-with-title-correct-hierarchy.html editing/mac/input/insert-delete-smp-symbol.html
Created attachment 315376 [details] Archive of layout-test-results from ews104 for mac-elcapitan-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews104 Port: mac-elcapitan-wk2 Platform: Mac OS X 10.11.6
Created attachment 315424 [details] Patch
The tests that failed because of attachment 315375 [details] shouldn't have. Adding some logging to try and see why they were failing.
Created attachment 315429 [details] Patch
Looked at the failures in attachment 315375 [details] with our bot watcher, Matt Lewis. Turns out, this is a result of an issue I've mentioned previously, although had not yet filed a bug over (Filed now, <https://bugs.webkit.org/show_bug.cgi?id=174504>). In short, when we pass an entire directory, that pass will override a more specific failure expectation. I think that this confirms this approach, given that the only tests which this change breaks is ones broken by another issue.
Comment on attachment 315429 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=315429&action=review r=me as long as there aren't a lot of test failures after this change lands (that Ryan & Matt aren't willing to clean up). > Tools/ChangeLog:10 > + The override order for Mac test expectations does not conform to the pattern used by iOS, > + GTK, Windows and WPE. It would be good to list the "old way" and the "new way" directly in the changelog here: What the current structure would be for sierra wk2 expectations and test results (once highsierra changes are landed): platform/mac-wk2 platform/wk2 platform/mac-sierra platform/mac-highsierra generic Proposed structure for sierra wk2 expectations and test results (once highsierra changes are landed): platform/mac-sierra-wk2 platform/mac-sierra platform/mac-wk2 platform/wk2 generic Maybe also give a before/after wk1 example as well (since "-wk1" suffix is added in some places).
Committed r219507: <http://trac.webkit.org/changeset/219507>