Bug 251267 – [ews] Regression: git steps in ews builds have started taking long time

• Status: ASSIGNED
• Component: Tools / Tests

[Aakash Jain]

Regression: git steps in ews builds have started taking long time. They used to take 20-30 seconds earlier. But now, they typically takes 2-3 minutes. This slows down each build and queue. The queues which used to be instanteneous have also started to get backlog of pending build at times. Slow queues have gotten slower because of this. We should look into it and see if we can speed it up.

e.g.: 
git operations took ~3 minutes (178 seconds) in recent build https://ews-build.webkit.org/#/builders/6/builds/104972  (clean-up-git-repo: 1 min 43s, checkout-source: 27s, update-working-directory: 11s, checkout-pull-request: 22s, show-identifier: 14s, fetch-branch-references: 1s).

[Aakash Jain]

Comparison of time spend in major git operation, over time:

On Jan 1, 2023: 178s in https://ews-build.webkit.org/#/builders/6/builds/101700 (clean-up-git-repo: 106s, clean-and-update-working-directory: 22s, update-working-directory: 7s, checkout-pull-request: 25s, show-identifier: 17s, fetch-branch-references: 1s)

On Dec 1, 2022 118s in https://ews-build.webkit.org/#/builders/6/builds/98400 (clean-up-git-repo: 39s, clean-and-update-working-directory: 26s, update-working-directory: 6s, checkout-pull-request: 27s, show-identifier: 15s, fetch-branch-references: 5s)

On Oct 1, 2022: 72s in https://ews-build.webkit.org/#/builders/6/builds/91200 (clean-up-git-repo: 18s, clean-and-update-working-directory: 29s, update-working-directory: 3s, checkout-pull-request: 17s, show-identifier: 3s, fetch-branch-references: 2s)

On July 1, 2022: 80s in https://ews-build.webkit.org/#/builders/6/builds/81000  (clean-up-git-repo: 28s, clean-and-update-working-directory: 22s, update-working-directory: 5s, checkout-pull-request: 10s, show-identifier: 13s, fetch-branch-references: 2s)

On January 1, 2022: 26s in https://ews-build.webkit.org/#/builders/6/builds/65050 (clean-and-update-working-directory: 20s, update-working-directory: 2s, show-identifier: 3s, fetch-branch-references: 1s)

On January 1, 2021: 18s in https://ews-build.webkit.org/#/builders/6/builds/36979 (clean-and-update-working-directory: 13s, update-working-directory: 1s)

On January 1, 2020: 14s in https://ews-build.webkit.org/#/builders/6/builds/10000 (clean-and-update-working-directory: 17s, update-working-directory: 1s)


(Note that time for hidden steps like show-identifier was fetched using buildbot api, e.g.: https://ews-build.webkit.org/api/v2/builders/6/builds/65050/steps).

[Radar WebKit Bug Importer]

<rdar://problem/104744000>

[Aakash Jain]

257496@main added "git prune" command in "clean-up-git-repo" step and it increased the run-time of that step by ~1 minute.

e.g.: https://ews-build.webkit.org/#/builders/6/builds/101700/steps/4/logs/stdio shows that git prune took 67 seconds
Similarly, in https://ews-build.webkit.org/#/builders/6/builds/104972/steps/4/logs/stdio it took 69 seconds.

[Aakash Jain]

One idea is to increase kern.maxvnodes value to make git operations faster (e.g.: sudo sysctl kern.maxvnodes=$((512*1024)))