Bug 148501 – iOS Simulator API tests fails as Simulator is not running

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

see rdar://problem/22447525

[Aakash Jain]

Created attachment 260008 [details]
Proposed patch

Launching the iOS Simulator before starting the iOS Simulator API tests and quitting it after finishing the tests.

[David Kilzer (:ddkilzer)]

Comment on attachment 260008 [details]
Proposed patch

View in context: https://bugs.webkit.org/attachment.cgi?id=260008&action=review

r- to address issue with simulator not quitting when running dumpTestsBySuite().  If dumpTestsBySuite() needs the simulator, then we need another call to quitIOSSimulator().

> Tools/Scripts/run-api-tests:100
> +relaunchIOSSimulator(findOrCreateSimulatorForIOSDevice(SIMULATOR_DEVICE_SUFFIX_FOR_WEBKIT_DEVELOPMENT)) if willUseIOSSimulatorSDKWhenBuilding();

The simulator is not stopped if dumpTestsBySuite() is called below.

I think this line should move down to just before the 'exit' line, or into runTestsBySuite() if dumpTestsBySuite() does not need the simulator running.

[Daniel Bates]

Comment on attachment 260008 [details]
Proposed patch

View in context: https://bugs.webkit.org/attachment.cgi?id=260008&action=review

> Tools/ChangeLog:10
> +        * Scripts/run-api-tests: Launch the iOS Simulator before starting the iOS Simulator API tests and quit it after finishing the tests.
> +        * Scripts/webkitdirs.pm: Store string "For WebKit Development" to constant SIMULATOR_DEVICE_SUFFIX_FOR_WEBKIT_DEVELOPMENT to make it consistently reusable.

These lines are long. We tend to wrap lines longer than 100 characters in a ChangeLog entry.

>> Tools/Scripts/run-api-tests:100
>> +relaunchIOSSimulator(findOrCreateSimulatorForIOSDevice(SIMULATOR_DEVICE_SUFFIX_FOR_WEBKIT_DEVELOPMENT)) if willUseIOSSimulatorSDKWhenBuilding();
> 
> The simulator is not stopped if dumpTestsBySuite() is called below.
> 
> I think this line should move down to just before the 'exit' line, or into runTestsBySuite() if dumpTestsBySuite() does not need the simulator running.

We may want to consider adding a FIXME comment above this that explains that we are launch/quit the iOS Simulator as a workaround for <rdar://problem/22388812>.

> Tools/Scripts/run-api-tests:162
> +    quitIOSSimulator() if willUseIOSSimulatorSDKWhenBuilding();

We should put this logic inside an END block to ensure that we always run this logic on program termination regardless of whether termination is caused by calling die() (which may be invoked in runTest()). We may also want to consider adding a FIXME comment above this that explains that we launch/quit the iOS Simulator as a workaround for <rdar://problem/22388812>.

You can read more about the END block at <http://perldoc.perl.org/perlmod.html#BEGIN%2c-UNITCHECK%2c-CHECK%2c-INIT-and-END>.

> Tools/Scripts/webkitdirs.pm:84
>         USE_OPEN_COMMAND
> +       SIMULATOR_DEVICE_SUFFIX_FOR_WEBKIT_DEVELOPMENT

Please swap the order of these lines so that this list is in sorted order according to the Unix sort command.

[Aakash Jain]

Created attachment 260090 [details]
Proposed patch

@David, moved quitIOSSimulator() to END block so that it is always executed. Also moved the code little bit down, can't move it below runTestsBySuite() as it uses simulator.

@Daniel, modified the ChangeLog, added FIXME comments, added the END block and swapped the order of lines.

[Daniel Bates]

> Created attachment 260090 [details]
> Proposed patch
> 
> @David, moved quitIOSSimulator() to END block so that it is always executed.
> Also moved the code little bit down, can't move it below runTestsBySuite()
> as it uses simulator.

From talking with Aakash today in person, listAllTests() also invokes TestWebKitAPI to get a listing of tests. According to Aakash, invoking TestWebKitAPI to get the list of tests errors out with an error message (@Aakash: can you post the error message?). Aakash noticed this does not seem to happen when the iOS Simulator is not running (why?). So, we launch/relaunch the iOS Simulator before calling listAllTests(). We should further investigate this issue.

[WebKit Commit Bot]

Comment on attachment 260090 [details]
Proposed patch

Clearing flags on attachment: 260090

Committed r189078: <http://trac.webkit.org/changeset/189078>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Aakash Jain]

It was a warning message and appeared only when I was running in verbose mode. The message was:
/Applications/Xcode.app/Contents/Developer/usr/bin/sim: No simulator devices appear to be running.  Setting data directories to /var/empty

It seems to be coming from "xcrun sim" command inside listAllTests(), when the Simulator is not running. That's the reason we are starting simulator before calling listAllTests().

On investigating further it seems that if the simulator is not running, (and executing in Verbose mode) every test gives this warning.

We should continue debugging it as part of <rdar://problem/22388812>.