Summary: | run-webkit-tests emits line about number of tests run for each booted simulator regardless of whether any tests were run | ||
---|---|---|---|
Product: | WebKit | Reporter: | Daniel Bates <dbates> |
Component: | Tools / Tests | Assignee: | Jonathan Bedard <jbedard> |
Status: | NEW --- | ||
Severity: | Enhancement | CC: | lforschler, webkit-bug-importer |
Priority: | P2 | Keywords: | InRadar |
Version: | WebKit Local Build | ||
Hardware: | iPhone / iPad | ||
OS: | iOS 12 | ||
See Also: |
https://bugs.webkit.org/show_bug.cgi?id=194133 https://bugs.webkit.org/show_bug.cgi?id=194158 |
Description
Daniel Bates
2019-01-25 16:21:53 PST
Never mind about the more confusing part and the overwriting. It just always emits "Running 0 tests for iPhone SE running None" when it ran the test using iPad. *** Bug 193856 has been marked as a duplicate of this bug. *** Was there a booted iPhone SE and a booted iPad when this was was observed? If they are both booted, I'm not sure that this is the wrong behavior. If only one was booted, this is definitely a problem. (In reply to Jonathan Bedard from comment #4) > Was there a booted iPhone SE and a booted iPad when this was was observed? > If they are both booted, I'm not sure that this is the wrong behavior. If > only one was booted, this is definitely a problem. I had both booted as implied by my comment "it runs it in iPad - why? I didn't say force and I thought the policy was to prefer iPhone SE?". If both are booted why does it only run the test in one of the simulators AND why did it pick iPad to run the test in? (In reply to Daniel Bates from comment #5) > (In reply to Jonathan Bedard from comment #4) > > Was there a booted iPhone SE and a booted iPad when this was was observed? > > If they are both booted, I'm not sure that this is the wrong behavior. If > > only one was booted, this is definitely a problem. > > I had both booted as implied by my comment "it runs it in iPad - why? I > didn't say force and I thought the policy was to prefer iPhone SE?". If both > are booted why does it only run the test in one of the simulators AND why > did it pick iPad to run the test in? The 'policy of preferring iPhone SE' at the moment is in a bit of a weird state, since all of these tests should also be gardened for iPad (not that they are yet, but they should be, see <https://bugs.webkit.org/show_bug.cgi?id=193767>). That being said, there is a pretty good argument to be made about predictably ordering devices in the event a user has multiple booted, right now the order would be determined by the output of 'xcrun simctl list', and even then, I'm not 100% certain which booted device would come first. I'm going to stand by the 'Running 0 tests for xxx' behavior because it makes it clear that there was another device that could have been used, although was not. We only run the tests on one device because after discussing the techniques we could use to run a device multiple times in a single test run, the general consensus was that such an approach was both complicated and confusing and not worth the effort. (In reply to Jonathan Bedard from comment #6) > I'm going to stand by the 'Running 0 tests for xxx' behavior because it > makes it clear that there was another device that could have been used, > although was not. > Great! Why should I care? (In reply to Daniel Bates from comment #7) > (In reply to Jonathan Bedard from comment #6) > > ... > > Great! Why should I care? Because it is not obvious which tests will be run on which devices. Even if we establish a more predictable device precedence order, the inverse of your problem (ie, why did we pick an iPhone SE instead of an iPad) is just as likely. When tests start failing, the user needs to care what device type was used because although tests should be device agnostic, plenty are not. (In reply to Jonathan Bedard from comment #8) > (In reply to Daniel Bates from comment #7) > > (In reply to Jonathan Bedard from comment #6) > > > ... > > > > Great! Why should I care? > > Because it is not obvious which tests will be run on which devices. Even if > we establish a more predictable device precedence order, the inverse of your > problem (ie, why did we pick an iPhone SE instead of an iPad) is just as > likely. > > When tests start failing, the user needs to care what device type was used > because although tests should be device agnostic, plenty are not. Why should I care about what devices were *not* used? (In reply to Daniel Bates from comment #9) > (In reply to Jonathan Bedard from comment #8) > > ... > > Why should I care about what devices were *not* used? The devices listed were valid targets for the specified tests and configuration. Since it's pretty easy to lose track of which simulators you have booted (I think it was Xcode 9 that simulators could be headlessly booted), this lets users see that a device they booted is available and a valid target, even if it was not used. Actually, I think this bug demonstrates why such logging can be useful. In this case, you expected the test to be run on an iPhone SE. It was not, but that device was both available and a valid target. Just from the log, we know that 2 devices were booted and matched the configuration, that information seems useful to me. (In reply to Jonathan Bedard from comment #10) > (In reply to Daniel Bates from comment #9) > > (In reply to Jonathan Bedard from comment #8) > > > ... > > > > Why should I care about what devices were *not* used? > > The devices listed were valid targets for the specified tests and > configuration. Since it's pretty easy to lose track of which simulators you > have booted (I think it was Xcode 9 that simulators could be headlessly > booted), this lets users see that a device they booted is available and a > valid target, even if it was not used. Not useful. I know what simulators I booted. I guess I would appreciate this output more if I had alzheimer's :/ If that's the audience you're going for then success! This is a feature! Or if you're looking for affirmation that the code you wrote is working and that it didn't run the test twice then I can see the benefit of keeping this behavior and closing this bug. I'll be reminded of this each time I run a test and have multiple simulators open and then become confused as to why didn't run-webkit-tests not run the test in all open simulators. > > Actually, I think this bug demonstrates why such logging can be useful. In > this case, you expected the test to be run on an iPhone SE. I got this info just by reading the "Running 1 test for iPad (6th generation) running None" line. I didn't need to see "Running 0 tests for iPhone SE running None". > It was not, but > that device was both available and a valid target. Just from the log, we > know that 2 devices were booted and matched the configuration, that > information seems useful to me. I think there is a saying, something like "less is more". This applies to this bug. Shout out to the "...running None" line. (In reply to Daniel Bates from comment #11) > (In reply to Jonathan Bedard from comment #10) > > (In reply to Daniel Bates from comment #9) > > > (In reply to Jonathan Bedard from comment #8) > > > > ... > > > > > > Why should I care about what devices were *not* used? > > > > The devices listed were valid targets for the specified tests and > > configuration. Since it's pretty easy to lose track of which simulators you > > have booted (I think it was Xcode 9 that simulators could be headlessly > > booted), this lets users see that a device they booted is available and a > > valid target, even if it was not used. > > Not useful. I know what simulators I booted. I guess I would appreciate this > output more if I had alzheimer's :/ If that's the audience you're going for > then success! This is a feature! Or if you're looking for affirmation that > the code you wrote is working and that it didn't run the test twice then I > can see the benefit of keeping this behavior and closing this bug. I'll be > reminded of this each time I run a test and have multiple simulators open > and then become confused as to why didn't run-webkit-tests not run the test > in all open simulators. > > > > > Actually, I think this bug demonstrates why such logging can be useful. In > > this case, you expected the test to be run on an iPhone SE. > > I got this info just by reading the "Running 1 test for iPad (6th > generation) running None" line. I didn't need to see "Running 0 tests for > iPhone SE running None". > > > It was not, but > > that device was both available and a valid target. Just from the log, we > > know that 2 devices were booted and matched the configuration, that > > information seems useful to me. > > I think there is a saying, something like "less is more". This applies to > this bug. Reading this back, the tone sounds negative. I didn't mean to come across like that. I just find the behavior of this bug silly. I can understand if the verbosity is useful to other people. So, this isn't a regression per se as much as it is an intended behavior change and hence this bug represents an enhancement request. Updating bug title and fields to reflect this. (In reply to Daniel Bates from comment #12) > Shout out to the "...running None" line. Filed bug #194133 for this. I filed <https://bugs.webkit.org/show_bug.cgi?id=194158> because it seems that this bug is now entirely about logging concerns. |