Bug 252864 – [ews] upload-built-product frequently takes long time

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

upload-built-product step on ews frequently takes long time (probably when server is under heavy load).

e.g.:
macOS-AppleSilicon-Ventura-Debug-Build-EWS queue:
44 minutes in https://ews-build.webkit.org/#/builders/85/builds/9620

macOS-BigSur-Release-Build-EWS queue:
46 minutes in https://ews-build.webkit.org/#/builders/43/builds/42040

iOS-16-Simulator-Build-EWS queue:
24 minutes in https://ews-build.webkit.org/#/builders/76/builds/19785

WPE queue: 
18 minutes in https://ews-build.webkit.org/#/builders/87/builds/770

GTK queue:
7 minutes in https://ews-build.webkit.org/#/builders/36/builds/101510

Some builds on same queues are very fast, e.g.: 49 seconds in https://ews-build.webkit.org/#/builders/43/builds/42039 (even though next build on same queue took 46 minutes to upload). So, this issue is not consistent.

[Radar WebKit Bug Importer]

<rdar://problem/106158090>

[Aakash Jain]

macOS-BigSur-Release-Build-EWS
~40 minutes in upload-built-product step in these recent builds:
https://ews-build.webkit.org/#/builders/43/builds/43104
https://ews-build.webkit.org/#/builders/43/builds/43105
https://ews-build.webkit.org/#/builders/43/builds/43106
https://ews-build.webkit.org/#/builders/43/builds/43107
https://ews-build.webkit.org/#/builders/43/builds/43108

macOS-AppleSilicon-Ventura-Debug-Build-EWS:
~50 minutes in these recent builds:
https://ews-build.webkit.org/#/builders/85/builds/10747
https://ews-build.webkit.org/#/builders/85/builds/10748
https://ews-build.webkit.org/#/builders/85/builds/10749
https://ews-build.webkit.org/#/builders/85/builds/10750

[Carlos Alberto Lopez Perez]

I think this is causing that the jobs dispatched from the master to the workers to have to compete with the upload data.

Right now loading the UI of https://ews-build.webkit.org/#/builders/86 is really slow. Showing the build list takes around 5-10 minutes and it seems buildbot may be having issues dispatch new jobs on time: there are both workers idle and jobs in the queue.


Quoting from one of the main Buildbot developers at https://github.com/buildbot/buildbot/issues/3709

> Problem with FileUpload is that it uses the only worker/master tcp stream so when a big files is being transfered, if you happen to have other builds on the same machine at the same time, the updates may stall during the transfer, up to the point you could have the 20min no update timeout for shell commands. (I experienced that some time ago)

That sounds terrible

I will try to move the sftp upload code from generate-bundle script to a new script ASAP so we can do some tests in order to replace Buildbot FileUpload  with it.

[[e-mail]]

I suspect there were some database network issues. Made some tuning changes earlier at 9:30pm PST which seemed to have reliefed the bottleneck. But I have reverted the change for the time beingit for the time being to make sure it wasn't a coincidence.

If things slow down again, I will try again.

[Aakash Jain]

(In reply to Carlos Alberto Lopez Perez from comment #3)
> I think this is causing that the jobs dispatched from the master to the workers to have to compete with the upload data.
Agree

> Right now loading the UI of https://ews-build.webkit.org/#/builders/86 is really slow.
Yes, buildot seems to be under heavy load, which is causing these issues.

> I will try to move the sftp upload code from generate-bundle script to a new script ASAP so we can do some tests in order to replace Buildbot FileUpload with it.
Yeah, as discussed offline, let's try to do that ASAP.

[Carlos Alberto Lopez Perez]

(In reply to Aakash Jain from comment #5)
> > I will try to move the sftp upload code from generate-bundle script to a new script ASAP so we can do some tests in order to replace Buildbot FileUpload with it.
> Yeah, as discussed offline, let's try to do that ASAP.

Changes landed in 261448@main

Includes a howto about configuring the server: https://github.com/WebKit/WebKit/blob/main/Tools/CISupport/Shared/HOWTO_config_SFTP_uploads.md

And an example class of how the upload step could look (unused for now) was added in build-webkit-org/steps.py with name class UploadBuiltProductViaSftp()

[Carlos Alberto Lopez Perez]

(In reply to [e-mail] from comment #4)
> I suspect there were some database network issues. Made some tuning changes
> earlier at 9:30pm PST which seemed to have reliefed the bottleneck. But I
> have reverted the change for the time beingit for the time being to make
> sure it wasn't a coincidence.
> 
> If things slow down again, I will try again.

Things are really slow now, so much that is not usable: I can't browse the list of current and past builds because it takes too much time to load. For example here: https://ews-build.webkit.org/#/builders/34

[Aakash Jain]

This issue is happening again. e.g.:
Upload from bot to buildbot-server took 30+ minutes in https://ews-build.webkit.org/#/builders/14/builds/1199, https://ews-build.webkit.org/#/builders/14/builds/1198

[Aakash Jain]

Fixed in https://bugs.webkit.org/show_bug.cgi?id=255339