Bug 203686 - 8 iOS API test failures on Catalina
Summary: 8 iOS API test failures on Catalina
Status: RESOLVED CONFIGURATION CHANGED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on: 203950 203951 203952 203953 203954 203955
Blocks:
  Show dependency treegraph
 
Reported: 2019-10-31 10:38 PDT by Aakash Jain
Modified: 2019-11-10 07:34 PST (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Aakash Jain 2019-10-31 10:38:21 PDT
We updated our iOS bots host OS from Mojave to Catalina yesterday, and since then ~8 API tests are Timing out consistently.
e.g.: https://ews-build.webkit.org/#/builders/9/builds/11207

Found 8 pre-existing API test failures: 
- TestWebKitAPI.ServiceWorkers.ProcessPerSession
- TestWebKitAPI.WebAuthenticationPanel.SubFrameDestructionHidCancel
- TestWebKitAPI.WebKit.ServerTrust, WebKit.QuotaDelegate
- TestWebKitAPI.WebAuthenticationPanel.SubFrameChangeLocationHidCancel
- TestWebKitAPI.ServiceWorkers.ProcessPerSite
- TestWebKitAPI.ServiceWorkers.ParallelProcessLaunch
- TestWebKitAPI.WebKit.FastServerTrust


These failures are currently happening on ews155 and ews157, and these two are updated to Catalina. These failures are not currently happening on ews158 and ews159, and they are still on Mojave.

Looking at ews157 history in https://ews-build.webkit.org/#/workers/23, it passed the API-tests yesterday while it was on Mojave in https://ews-build.webkit.org/#/builders/9/builds/11168, the first build on Catalina (https://ews-build.webkit.org/#/builders/9/builds/11185) failed with these 8 API test failures.
Comment 1 Radar WebKit Bug Importer 2019-10-31 13:05:04 PDT
<rdar://problem/56792095>
Comment 2 Aakash Jain 2019-11-01 08:10:02 PDT
These tests seems to pass on build.webkit.org, and the next obvious question is what's the difference between EWS and build.webkit.org configuration. I compared the configs and the difference is:

EWS passes: --ios-simulator flag
build.webkit.org passes: --iphone-simulator flag 

example:
EWS: https://ews-build.webkit.org/#/builders/9/builds/11295/steps/9/logs/stdio
build.webkit.org: https://build.webkit.org/builders/Apple%20iOS%2013%20Simulator%20Release%20WK2%20%28Tests%29/builds/822/steps/run-api-tests/logs/stdio)
Comment 3 Aakash Jain 2019-11-01 08:10:09 PDT
Note that both EWS and build.webkit.org used to pass --ios-simulator flag earlier, until build.webkit.org config changed a month ago in http://trac.webkit.org/changeset/250746/webkit, and RunAPITests started using new appendCustomTestingFlags() method instead of old appendCustomBuildFlags().
Comment 4 Jonathan Bedard 2019-11-01 08:30:07 PDT
We can make this change for API tests, but it’s unlikely to be the culprit since iOS Simulators are used when iOS is specified.

Do we know what sort of hardware post-commit testing is using? My bet is that EWS machines are just faster than our post-commit testing machines.
Comment 5 Alexey Proskuryakov 2019-11-01 10:47:34 PDT
Are these reproducible when you screen share into the machine, and run the tests manually?

My gut reaction is that such things are usually related either to auto-login, or to (not) having a display dongle.

> Do we know what sort of hardware post-commit testing is using? My bet is that EWS machines are just faster than our post-commit testing machines.

Same hardware.
Comment 6 Aakash Jain 2019-11-02 18:24:16 PDT
(In reply to Alexey Proskuryakov from comment #5)
> Are these reproducible when you screen share into the machine, and run the tests manually?
Yes, I just screen shared into ews157 and run the tests manually, and same 8 tests TimedOut.

Also, I rebooted ews156, and in the next build same tests still failed (in https://ews-build.webkit.org/#/builders/9/builds/11383).
Comment 7 Aakash Jain 2019-11-02 18:24:29 PDT
(In reply to Alexey Proskuryakov from comment #5)
> My gut reaction is that such things are usually related either to auto-login, or to (not) having a display dongle.
auto-login seems to be working fine on ews156. I believe Truitt ensured on yesterday that these bots have dongle.
Comment 8 Aakash Jain 2019-11-02 18:26:19 PDT
One difference between EWS and build.webkit.org bots seems to be OS version.

EWS bots (eww156 and ews157) are on 10.15.1 (19B88), while build.webkit.org bots (bot651 and bot652) are on 10.15 (19A603).


Respective builders: 
https://build.webkit.org/builders/Apple%20iOS%2013%20Simulator%20Release%20WK2%20%28Tests%29
https://ews-build.webkit.org/#/builders/9
Comment 9 Alexey Proskuryakov 2019-11-02 18:58:16 PDT
> I believe Truitt ensured on yesterday that these bots have dongle. 

"system_profiler SPDisplaysDataType" on ews156 doesn't confirm.
Comment 10 Aakash Jain 2019-11-03 05:56:12 PST
I moved bot651 from build.webkit.org to EWS, and these 8 failures do not reproduce on bot651, build: https://ews-build.webkit.org/#/builders/9/builds/11400

So, it seems like there is some difference between these bots causing these failures. Can be either OS version or display dongle.
Comment 11 Alexey Proskuryakov 2019-11-07 12:53:32 PST
On the bot, I can 100% reproduce even with one test:

./Tools/Scripts/run-api-tests --no-build --release --verbose --iphone-simulator TestWebKitAPI.ServiceWorkers.ParallelProcessLaunch --no-timeout

But I cannot reproduce on my machine that has the same OS and Xcode.
Comment 12 Aakash Jain 2019-11-10 07:24:07 PST
Alexey figured out that this was due to a configuration issue on newly upgraded bots. These tests are passing now, e.g.: https://ews-build.webkit.org/#/builders/9/builds/11796