Bug 192161 – webkitpy: Sort tests by associated device type

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Jonathan Bedard]

There are a number of circumstances where we want to run a specific test on a specific device type. Our current technique for doing this requires each port defining CUSTOM_DEVICE_CLASSES. This approach is problematic because it doesn't work if a simulator is already booted or when running on-device testing. Instead, we should check which device types are available.

[Radar WebKit Bug Importer]

<rdar://problem/46345392>

[Jonathan Bedard]

My original description of this bug isn't accurate.

I think what makes more sense, given the goals of <https://bugs.webkit.org/show_bug.cgi?id=192162>, is to generate a mapping of device-types and test lists then iterate through each device type, checking if the current port supports that device type. Once <https://bugs.webkit.org/show_bug.cgi?id=192162> lands, we'll be able to get ride of the CUSTOM_DEVICE_CLASSES.

[Jonathan Bedard]

(In reply to Jonathan Bedard from comment #2)
> ...
> <https://bugs.webkit.org/show_bug.cgi?id=192162> lands, we'll be able to get
> ride of the CUSTOM_DEVICE_CLASSES.

...be able to get RID of the CUSTOM_DEVICE_CLASSES.

[Jonathan Bedard]

Created attachment 356439 [details]
Patch

[Jonathan Bedard]

Comment on attachment 356439 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=356439&action=review

> Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:259
> +            while index < len(device_type_order):

Consider the example: [iPhone SE, iPhone 7, iPhone, iPad]. In this example, we don't want 'iPhone' tests to boot their own simulator if they don't have to. What this code will do is run the iPhone tests with the iPhone SE, because iPhone matches iPhone SE, then remove both iPhone SE and iPhone from the list. [iPhone 7, iPad] will be left. Then, we will boot an iPhone 7. iPad does not match iPhone 7, so after the iPhone 7 tests, we will boot an iPad.

[Jonathan Bedard]

Created attachment 356514 [details]
Patch

[Jonathan Bedard]

Created attachment 356550 [details]
Patch

[EWS Watchlist]

Comment on attachment 356550 [details]
Patch

Attachment 356550 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/10275026

New failing tests:
http/tests/misc/resource-timing-resolution.html

[EWS Watchlist]

Created attachment 356587 [details]
Archive of layout-test-results from ews101 for mac-sierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-sierra  Platform: Mac OS X 10.12.6

[Jonathan Bedard]

Created attachment 356608 [details]
Patch

[Lucas Forschler]

Comment on attachment 356608 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=356608&action=review

> Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:269
> +                    # Remove devices types from device_type_order once they have been fulfilled

to me, fulfilled implies completed, with no work to be done... but isn't this logic adding it to the list/queue?

I think it might be better to replace fulfilled with 'claimed | queued | recorded | scheduled?

[Jonathan Bedard]

(In reply to Lucas Forschler from comment #11)
> Comment on attachment 356608 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=356608&action=review
> 
> > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:269
> > +                    # Remove devices types from device_type_order once they have been fulfilled
> 
> to me, fulfilled implies completed, with no work to be done... but isn't
> this logic adding it to the list/queue?
> 
> I think it might be better to replace fulfilled with 'claimed | queued |
> recorded | scheduled?

Going with:

# Remove devices types from device_type_order once tests associated with that type have been claimed.

[Jonathan Bedard]

Created attachment 356618 [details]
Patch for landing

[WebKit Commit Bot]

Comment on attachment 356618 [details]
Patch for landing

Clearing flags on attachment: 356618

Committed r238903: <https://trac.webkit.org/changeset/238903>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.