Bug 203686 – 8 iOS API test failures on Catalina

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: Tools / Tests

[Aakash Jain]

We updated our iOS bots host OS from Mojave to Catalina yesterday, and since then ~8 API tests are Timing out consistently.
e.g.: https://ews-build.webkit.org/#/builders/9/builds/11207

Found 8 pre-existing API test failures: 
- TestWebKitAPI.ServiceWorkers.ProcessPerSession
- TestWebKitAPI.WebAuthenticationPanel.SubFrameDestructionHidCancel
- TestWebKitAPI.WebKit.ServerTrust, WebKit.QuotaDelegate
- TestWebKitAPI.WebAuthenticationPanel.SubFrameChangeLocationHidCancel
- TestWebKitAPI.ServiceWorkers.ProcessPerSite
- TestWebKitAPI.ServiceWorkers.ParallelProcessLaunch
- TestWebKitAPI.WebKit.FastServerTrust


These failures are currently happening on ews155 and ews157, and these two are updated to Catalina. These failures are not currently happening on ews158 and ews159, and they are still on Mojave.

Looking at ews157 history in https://ews-build.webkit.org/#/workers/23, it passed the API-tests yesterday while it was on Mojave in https://ews-build.webkit.org/#/builders/9/builds/11168, the first build on Catalina (https://ews-build.webkit.org/#/builders/9/builds/11185) failed with these 8 API test failures.

[Radar WebKit Bug Importer]

<rdar://problem/56792095>

[Aakash Jain]

These tests seems to pass on build.webkit.org, and the next obvious question is what's the difference between EWS and build.webkit.org configuration. I compared the configs and the difference is:

EWS passes: --ios-simulator flag
build.webkit.org passes: --iphone-simulator flag 

example:
EWS: https://ews-build.webkit.org/#/builders/9/builds/11295/steps/9/logs/stdio
build.webkit.org: https://build.webkit.org/builders/Apple%20iOS%2013%20Simulator%20Release%20WK2%20%28Tests%29/builds/822/steps/run-api-tests/logs/stdio)

[Aakash Jain]

Note that both EWS and build.webkit.org used to pass --ios-simulator flag earlier, until build.webkit.org config changed a month ago in http://trac.webkit.org/changeset/250746/webkit, and RunAPITests started using new appendCustomTestingFlags() method instead of old appendCustomBuildFlags().

[Jonathan Bedard]

We can make this change for API tests, but it’s unlikely to be the culprit since iOS Simulators are used when iOS is specified.

Do we know what sort of hardware post-commit testing is using? My bet is that EWS machines are just faster than our post-commit testing machines.

[Alexey Proskuryakov]

Are these reproducible when you screen share into the machine, and run the tests manually?

My gut reaction is that such things are usually related either to auto-login, or to (not) having a display dongle.

> Do we know what sort of hardware post-commit testing is using? My bet is that EWS machines are just faster than our post-commit testing machines.

Same hardware.

[Aakash Jain]

(In reply to Alexey Proskuryakov from comment #5)
> Are these reproducible when you screen share into the machine, and run the tests manually?
Yes, I just screen shared into ews157 and run the tests manually, and same 8 tests TimedOut.

Also, I rebooted ews156, and in the next build same tests still failed (in https://ews-build.webkit.org/#/builders/9/builds/11383).

[Aakash Jain]

(In reply to Alexey Proskuryakov from comment #5)
> My gut reaction is that such things are usually related either to auto-login, or to (not) having a display dongle.
auto-login seems to be working fine on ews156. I believe Truitt ensured on yesterday that these bots have dongle.

[Aakash Jain]

One difference between EWS and build.webkit.org bots seems to be OS version.

EWS bots (eww156 and ews157) are on 10.15.1 (19B88), while build.webkit.org bots (bot651 and bot652) are on 10.15 (19A603).


Respective builders: 
https://build.webkit.org/builders/Apple%20iOS%2013%20Simulator%20Release%20WK2%20%28Tests%29
https://ews-build.webkit.org/#/builders/9

[Alexey Proskuryakov]

> I believe Truitt ensured on yesterday that these bots have dongle. 

"system_profiler SPDisplaysDataType" on ews156 doesn't confirm.

[Aakash Jain]

I moved bot651 from build.webkit.org to EWS, and these 8 failures do not reproduce on bot651, build: https://ews-build.webkit.org/#/builders/9/builds/11400

So, it seems like there is some difference between these bots causing these failures. Can be either OS version or display dongle.

[Alexey Proskuryakov]

On the bot, I can 100% reproduce even with one test:

./Tools/Scripts/run-api-tests --no-build --release --verbose --iphone-simulator TestWebKitAPI.ServiceWorkers.ParallelProcessLaunch --no-timeout

But I cannot reproduce on my machine that has the same OS and Xcode.

[Aakash Jain]

Alexey figured out that this was due to a configuration issue on newly upgraded bots. These tests are passing now, e.g.: https://ews-build.webkit.org/#/builders/9/builds/11796