Bug 192162 - webkitpy: Implement device type specific expected results
Summary: webkitpy: Implement device type specific expected results
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Jonathan Bedard
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2018-11-29 10:04 PST by Jonathan Bedard
Modified: 2019-01-15 09:46 PST (History)
5 users (show)

See Also:


Attachments
Patch (27.48 KB, patch)
2018-12-10 08:27 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews123 for ios-simulator-wk2 (248.70 KB, application/zip)
2018-12-10 09:31 PST, Build Bot
no flags Details
Directory Approach (84.99 KB, patch)
2018-12-11 09:20 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (72.26 KB, patch)
2018-12-12 09:57 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (72.26 KB, patch)
2018-12-12 10:10 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews126 for ios-simulator-wk2 (2.52 MB, application/zip)
2018-12-12 12:11 PST, Build Bot
no flags Details
Patch (72.86 KB, patch)
2018-12-12 15:01 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (74.11 KB, patch)
2018-12-12 16:37 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews204 for win-future (12.83 MB, application/zip)
2018-12-12 19:33 PST, Build Bot
no flags Details
Patch (80.58 KB, patch)
2018-12-13 08:34 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (81.34 KB, patch)
2018-12-13 17:35 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Patch (33.17 KB, patch)
2019-01-14 15:56 PST, Jonathan Bedard
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Bedard 2018-11-29 10:04:11 PST
It is often the case that a test will have expectations which differ based on the device type the test is being run on. We should optionally break up test expectations into folders with device specific expectations contained within.
Comment 1 Radar WebKit Bug Importer 2018-11-29 10:04:51 PST
<rdar://problem/46345449>
Comment 2 Jonathan Bedard 2018-12-10 08:27:25 PST
Created attachment 356960 [details]
Patch
Comment 3 Build Bot 2018-12-10 09:31:43 PST
Comment on attachment 356960 [details]
Patch

Attachment 356960 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: https://webkit-queues.webkit.org/results/10338667

Number of test failures exceeded the failure limit.
Comment 4 Build Bot 2018-12-10 09:31:44 PST
Created attachment 356967 [details]
Archive of layout-test-results from ews123 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews123  Port: ios-simulator-wk2  Platform: Mac OS X 10.13.6
Comment 5 Jonathan Bedard 2018-12-11 09:20:39 PST
Created attachment 357055 [details]
Directory Approach
Comment 6 Jonathan Bedard 2018-12-12 09:57:06 PST
Created attachment 357148 [details]
Patch
Comment 7 Jonathan Bedard 2018-12-12 10:10:29 PST
Created attachment 357149 [details]
Patch
Comment 8 Build Bot 2018-12-12 12:11:29 PST
Comment on attachment 357149 [details]
Patch

Attachment 357149 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: https://webkit-queues.webkit.org/results/10370859

New failing tests:
fast/text/ipad/bold-tall-body-text-style.html
fast/text-autosizing/ios/ipad/text-size-adjust-inline-style-expected-ipad.html
fast/text/ipad/bold-tall-body-text-style-expected-mismatch-ipad.html
Comment 9 Build Bot 2018-12-12 12:11:31 PST
Created attachment 357154 [details]
Archive of layout-test-results from ews126 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews126  Port: ios-simulator-wk2  Platform: Mac OS X 10.13.6
Comment 10 Jonathan Bedard 2018-12-12 15:01:32 PST
Created attachment 357170 [details]
Patch
Comment 11 Jonathan Bedard 2018-12-12 16:37:24 PST
Created attachment 357194 [details]
Patch
Comment 12 Build Bot 2018-12-12 19:33:10 PST
Comment on attachment 357194 [details]
Patch

Attachment 357194 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/10376839

New failing tests:
ietestcenter/css3/namespaces/prefix-010.xml
ietestcenter/css3/namespaces/prefix-011.xml
ietestcenter/css3/namespaces/prefix-007.xml
Comment 13 Build Bot 2018-12-12 19:33:22 PST
Created attachment 357205 [details]
Archive of layout-test-results from ews204 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews204  Port: win-future  Platform: CYGWIN_NT-6.1-2.10.0-0.325-5-3-x86_64-64bit
Comment 14 Jonathan Bedard 2018-12-13 08:34:36 PST
Created attachment 357231 [details]
Patch
Comment 15 Jonathan Bedard 2018-12-13 13:33:57 PST
I think it's worth sharing some results from local testing here: device type specific expectations will definitely be a rarity. I ran all our iOS tests on an iPad instead of an iPhone SE, and encountered 96 failures. Assuming all these failures are a result of running on iPad instead of iPhone (which is doubtful, but it gives us a pretty good estimate). This means that, including our current device specific tests, we have 118 tests currently checked in that require device specific expected results.
Comment 16 Jonathan Bedard 2018-12-13 17:35:13 PST
Created attachment 357275 [details]
Patch
Comment 17 Jonathan Bedard 2019-01-10 16:35:52 PST
After discussions, the approach this patch is taking is going to change quite dramatically. Instead of introducing device type specific expected results, the baseline search path will be mutated based on the device type used.
Comment 18 Jonathan Bedard 2019-01-14 15:56:42 PST
Created attachment 359090 [details]
Patch
Comment 19 Build Bot 2019-01-14 15:59:27 PST
Attachment 359090 [details] did not pass style-queue:


ERROR: Tools/Scripts/webkitpy/port/base.py:1204:  [Port.expectations_files] An attribute affected in webkitpy.port.base_unittest line 190 hide this method  [pylint/E0202] [5]
Total errors found: 1 in 16 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 20 Jonathan Bedard 2019-01-14 16:12:21 PST
(In reply to Build Bot from comment #19)
> Attachment 359090 [details] did not pass style-queue:
> 
> 
> ERROR: Tools/Scripts/webkitpy/port/base.py:1204:  [Port.expectations_files]
> An attribute affected in webkitpy.port.base_unittest line 190 hide this
> method  [pylint/E0202] [5]
> Total errors found: 1 in 16 files
> 
> 
> If any of these errors are false positives, please file a bug against
> check-webkit-style.

Normally this message would be helpful. In this case, Python is complaining about a function being overwritten during testing.
Comment 21 WebKit Commit Bot 2019-01-15 09:46:37 PST
Comment on attachment 359090 [details]
Patch

Clearing flags on attachment: 359090

Committed r239989: <https://trac.webkit.org/changeset/239989>
Comment 22 WebKit Commit Bot 2019-01-15 09:46:39 PST
All reviewed patches have been landed.  Closing bug.