RESOLVED FIXED 252864
[ews] upload-built-product frequently takes long time 
https://bugs.webkit.org/show_bug.cgi?id=252864
Summary [ews] upload-built-product frequently takes long time
Aakash Jain
Reported 2023-02-23 14:45:33 PST
upload-built-product step on ews frequently takes long time (probably when server is under heavy load). e.g.: macOS-AppleSilicon-Ventura-Debug-Build-EWS queue: 44 minutes in https://ews-build.webkit.org/#/builders/85/builds/9620 macOS-BigSur-Release-Build-EWS queue: 46 minutes in https://ews-build.webkit.org/#/builders/43/builds/42040 iOS-16-Simulator-Build-EWS queue: 24 minutes in https://ews-build.webkit.org/#/builders/76/builds/19785 WPE queue: 18 minutes in https://ews-build.webkit.org/#/builders/87/builds/770 GTK queue: 7 minutes in https://ews-build.webkit.org/#/builders/36/builds/101510 Some builds on same queues are very fast, e.g.: 49 seconds in https://ews-build.webkit.org/#/builders/43/builds/42039 (even though next build on same queue took 46 minutes to upload). So, this issue is not consistent.
Attachments
Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2023-03-02 12:35:51 PST
<rdar://problem/106158090>
Carlos Alberto Lopez Perez
Comment 3 2023-03-02 19:59:01 PST
I think this is causing that the jobs dispatched from the master to the workers to have to compete with the upload data. Right now loading the UI of https://ews-build.webkit.org/#/builders/86 is really slow. Showing the build list takes around 5-10 minutes and it seems buildbot may be having issues dispatch new jobs on time: there are both workers idle and jobs in the queue. Quoting from one of the main Buildbot developers at https://github.com/buildbot/buildbot/issues/3709 > Problem with FileUpload is that it uses the only worker/master tcp stream so when a big files is being transfered, if you happen to have other builds on the same machine at the same time, the updates may stall during the transfer, up to the point you could have the 20min no update timeout for shell commands. (I experienced that some time ago) That sounds terrible I will try to move the sftp upload code from generate-bundle script to a new script ASAP so we can do some tests in order to replace Buildbot FileUpload with it.
lingho@apple.com
Comment 4 2023-03-02 22:00:28 PST
I suspect there were some database network issues. Made some tuning changes earlier at 9:30pm PST which seemed to have reliefed the bottleneck. But I have reverted the change for the time beingit for the time being to make sure it wasn't a coincidence. If things slow down again, I will try again.
Aakash Jain
Comment 5 2023-03-03 04:23:05 PST
(In reply to Carlos Alberto Lopez Perez from comment #3) > I think this is causing that the jobs dispatched from the master to the workers to have to compete with the upload data. Agree > Right now loading the UI of https://ews-build.webkit.org/#/builders/86 is really slow. Yes, buildot seems to be under heavy load, which is causing these issues. > I will try to move the sftp upload code from generate-bundle script to a new script ASAP so we can do some tests in order to replace Buildbot FileUpload with it. Yeah, as discussed offline, let's try to do that ASAP.
Carlos Alberto Lopez Perez
Comment 6 2023-03-09 10:50:20 PST
(In reply to Aakash Jain from comment #5) > > I will try to move the sftp upload code from generate-bundle script to a new script ASAP so we can do some tests in order to replace Buildbot FileUpload with it. > Yeah, as discussed offline, let's try to do that ASAP. Changes landed in 261448@main Includes a howto about configuring the server: https://github.com/WebKit/WebKit/blob/main/Tools/CISupport/Shared/HOWTO_config_SFTP_uploads.md And an example class of how the upload step could look (unused for now) was added in build-webkit-org/steps.py with name class UploadBuiltProductViaSftp()
Carlos Alberto Lopez Perez
Comment 7 2023-03-21 06:33:49 PDT
(In reply to lingho@apple.com from comment #4) > I suspect there were some database network issues. Made some tuning changes > earlier at 9:30pm PST which seemed to have reliefed the bottleneck. But I > have reverted the change for the time beingit for the time being to make > sure it wasn't a coincidence. > > If things slow down again, I will try again. Things are really slow now, so much that is not usable: I can't browse the list of current and past builds because it takes too much time to load. For example here: https://ews-build.webkit.org/#/builders/34
Aakash Jain
Comment 8 2023-03-29 13:21:24 PDT
This issue is happening again. e.g.: Upload from bot to buildbot-server took 30+ minutes in https://ews-build.webkit.org/#/builders/14/builds/1199, https://ews-build.webkit.org/#/builders/14/builds/1198
Aakash Jain
Comment 9 2023-04-17 08:43:51 PDT
Fixed in https://bugs.webkit.org/show_bug.cgi?id=255339
Note You need to log in before you can comment on or make changes to this bug.