Bug 174472 - Bring mac expectations in-line with other platforms
Summary: Bring mac expectations in-line with other platforms
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Jonathan Bedard
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2017-07-13 12:28 PDT by Jonathan Bedard
Modified: 2017-07-14 10:15 PDT (History)
9 users (show)

See Also:


Attachments
Patch (15.07 KB, patch)
2017-07-13 14:39 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews104 for mac-elcapitan-wk2 (1.37 MB, application/zip)
2017-07-13 15:22 PDT, Build Bot
no flags Details
Patch (15.35 KB, patch)
2017-07-14 08:48 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (20.43 KB, patch)
2017-07-14 09:14 PDT, Jonathan Bedard
ddkilzer: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Bedard 2017-07-13 12:28:08 PDT
Currently, mac expectations do not conform to expectations on other platforms.  Modify default_baseline_search_path so that mac expectations work like iOS, GTK and WPE test expectations.  That is to say, expectations should look like this:

{port}-{version}-wk{1/2} -> {port}-{version} -> {port}-wk{1-2} -> {port} -> wk2{optional} -> generic
Comment 1 Radar WebKit Bug Importer 2017-07-13 12:28:38 PDT
<rdar://problem/33297388>
Comment 2 Jonathan Bedard 2017-07-13 14:39:39 PDT
Created attachment 315375 [details]
Patch
Comment 3 Jonathan Bedard 2017-07-13 15:18:59 PDT
Talked to Dave Kilzer about this.

This will likely cause test failures, at least in it's current form.  I'm letting EWS chew on it to see exactly how bad this is.
Comment 4 Build Bot 2017-07-13 15:22:57 PDT
Comment on attachment 315375 [details]
Patch

Attachment 315375 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/4115247

New failing tests:
editing/mac/input/kotoeri-enter-to-confirm-and-newline.html
editing/mac/input/hangul-enter-confirms-and-sends-keypress.html
editing/mac/input/firstrectforcharacterrange-vertical.html
editing/mac/input/firstrectforcharacterrange-plain.html
editing/mac/input/caret-primary-bidi.html
accessibility/mac/search-predicate.html
accessibility/mac/html-slider-indicator.html
editing/mac/input/wrapped-line-char-rect.html
editing/mac/input/range-for-empty-document.html
editing/mac/input/NSBackgroundColor-transparent.html
editing/mac/input/5576619.html
editing/mac/input/text-input-controller.html
editing/mac/pasteboard/paste-and-match-style-selector-event.html
editing/mac/input/firstrectforcharacterrange-caret-in-br.html
editing/mac/input/firstrectforcharacterrange-styled.html
editing/mac/input/secure-input.html
editing/mac/selection/25228.html
editing/mac/input/replace-invalid-range.html
accessibility/mac/listbox-hit-test.html
editing/mac/input/bold-node.html
editing/mac/input/text-input-controller-no-editable-no-crash.html
accessibility/mac/iframe-with-title-correct-hierarchy.html
editing/mac/input/insert-delete-smp-symbol.html
Comment 5 Build Bot 2017-07-13 15:22:58 PDT
Created attachment 315376 [details]
Archive of layout-test-results from ews104 for mac-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-elcapitan-wk2  Platform: Mac OS X 10.11.6
Comment 6 Jonathan Bedard 2017-07-14 08:48:12 PDT
Created attachment 315424 [details]
Patch
Comment 7 Jonathan Bedard 2017-07-14 08:49:18 PDT
The tests that failed because of attachment 315375 [details] shouldn't have.  Adding some logging to try and see why they were failing.
Comment 8 Jonathan Bedard 2017-07-14 09:14:14 PDT
Created attachment 315429 [details]
Patch
Comment 9 Jonathan Bedard 2017-07-14 09:18:29 PDT
Looked at the failures in attachment 315375 [details] with our bot watcher, Matt Lewis.  Turns out, this is a result of an issue I've mentioned previously, although had not yet filed a bug over (Filed now, <https://bugs.webkit.org/show_bug.cgi?id=174504>).  In short, when we pass an entire directory, that pass will override a more specific failure expectation. 

I think that this confirms this approach, given that the only tests which this change breaks is ones broken by another issue.
Comment 10 David Kilzer (:ddkilzer) 2017-07-14 09:48:29 PDT
Comment on attachment 315429 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=315429&action=review

r=me as long as there aren't a lot of test failures after this change lands (that Ryan & Matt aren't willing to clean up).

> Tools/ChangeLog:10
> +        The override order for Mac test expectations does not conform to the pattern used by iOS,
> +        GTK, Windows and WPE.

It would be good to list the "old way" and the "new way" directly in the changelog here:

What the current structure would be for sierra wk2 expectations and test results (once highsierra changes are landed):

platform/mac-wk2
platform/wk2
platform/mac-sierra
platform/mac-highsierra
generic

Proposed structure for sierra wk2 expectations and test results (once highsierra changes are landed):

platform/mac-sierra-wk2
platform/mac-sierra
platform/mac-wk2
platform/wk2
generic

Maybe also give a before/after wk1 example as well (since "-wk1" suffix is added in some places).
Comment 11 Jonathan Bedard 2017-07-14 10:15:04 PDT
Committed r219507: <http://trac.webkit.org/changeset/219507>