Bug 174472 – Bring mac expectations in-line with other platforms

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Jonathan Bedard]

Currently, mac expectations do not conform to expectations on other platforms.  Modify default_baseline_search_path so that mac expectations work like iOS, GTK and WPE test expectations.  That is to say, expectations should look like this:

{port}-{version}-wk{1/2} -> {port}-{version} -> {port}-wk{1-2} -> {port} -> wk2{optional} -> generic

[Radar WebKit Bug Importer]

<rdar://problem/33297388>

[Jonathan Bedard]

Created attachment 315375 [details]
Patch

[Jonathan Bedard]

Talked to Dave Kilzer about this.

This will likely cause test failures, at least in it's current form.  I'm letting EWS chew on it to see exactly how bad this is.

[Build Bot]

Comment on attachment 315375 [details]
Patch

Attachment 315375 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/4115247

New failing tests:
editing/mac/input/kotoeri-enter-to-confirm-and-newline.html
editing/mac/input/hangul-enter-confirms-and-sends-keypress.html
editing/mac/input/firstrectforcharacterrange-vertical.html
editing/mac/input/firstrectforcharacterrange-plain.html
editing/mac/input/caret-primary-bidi.html
accessibility/mac/search-predicate.html
accessibility/mac/html-slider-indicator.html
editing/mac/input/wrapped-line-char-rect.html
editing/mac/input/range-for-empty-document.html
editing/mac/input/NSBackgroundColor-transparent.html
editing/mac/input/5576619.html
editing/mac/input/text-input-controller.html
editing/mac/pasteboard/paste-and-match-style-selector-event.html
editing/mac/input/firstrectforcharacterrange-caret-in-br.html
editing/mac/input/firstrectforcharacterrange-styled.html
editing/mac/input/secure-input.html
editing/mac/selection/25228.html
editing/mac/input/replace-invalid-range.html
accessibility/mac/listbox-hit-test.html
editing/mac/input/bold-node.html
editing/mac/input/text-input-controller-no-editable-no-crash.html
accessibility/mac/iframe-with-title-correct-hierarchy.html
editing/mac/input/insert-delete-smp-symbol.html

[Build Bot]

Created attachment 315376 [details]
Archive of layout-test-results from ews104 for mac-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-elcapitan-wk2  Platform: Mac OS X 10.11.6

[Jonathan Bedard]

Created attachment 315424 [details]
Patch

[Jonathan Bedard]

The tests that failed because of attachment 315375 [details] shouldn't have.  Adding some logging to try and see why they were failing.

[Jonathan Bedard]

Created attachment 315429 [details]
Patch

[Jonathan Bedard]

Looked at the failures in attachment 315375 [details] with our bot watcher, Matt Lewis.  Turns out, this is a result of an issue I've mentioned previously, although had not yet filed a bug over (Filed now, <https://bugs.webkit.org/show_bug.cgi?id=174504>).  In short, when we pass an entire directory, that pass will override a more specific failure expectation. 

I think that this confirms this approach, given that the only tests which this change breaks is ones broken by another issue.

[David Kilzer (:ddkilzer)]

Comment on attachment 315429 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=315429&action=review

r=me as long as there aren't a lot of test failures after this change lands (that Ryan & Matt aren't willing to clean up).

> Tools/ChangeLog:10
> +        The override order for Mac test expectations does not conform to the pattern used by iOS,
> +        GTK, Windows and WPE.

It would be good to list the "old way" and the "new way" directly in the changelog here:

What the current structure would be for sierra wk2 expectations and test results (once highsierra changes are landed):

platform/mac-wk2
platform/wk2
platform/mac-sierra
platform/mac-highsierra
generic

Proposed structure for sierra wk2 expectations and test results (once highsierra changes are landed):

platform/mac-sierra-wk2
platform/mac-sierra
platform/mac-wk2
platform/wk2
generic

Maybe also give a before/after wk1 example as well (since "-wk1" suffix is added in some places).

[Jonathan Bedard]

Committed r219507: <http://trac.webkit.org/changeset/219507>