Bug 204007

Summary: [ews] Prioritize builders over testers when a bot is shared over multiple queues
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, ap, jbedard, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch jbedard: review+

Description Aakash Jain 2019-11-08 09:49:21 PST
In our EWS configuration, we share bots over multiple queues to improve the efficiency. Sometimes a bot might be connected over builder and tester queues. If there is pending build-requests over both the queues, we want to prioritize builder queue over tester.

"By default, buildbot will attempt to start builds on builders in order, beginning with the builder with the oldest pending request."
https://docs.buildbot.net/current/manual/configuration/global.html#prioritizing-builders


For e.g.: 'macOS-High-Sierra-Debug-Build-EWS' and 'macOS-High-Sierra-Debug-WK1-Tests-EWS' share bots ews113 - ews117. Quite often it happens that there are pending build-requests on both the queues (especially when there are test failures on trunk and tester queue is backlogged). By default Buildbot prioritize the queue having older build-request. Because of that, most of the bots keep processing tester queue, and builder queue takes low priority.

However we want builder queues to be prioritized over tester queues. This will also enable to us to do more bot sharing between builder and tester queues, and have higher utilization of our bots.
Comment 1 Aakash Jain 2019-11-08 09:51:42 PST
Created attachment 383135 [details]
Patch
Comment 2 Aakash Jain 2019-11-08 09:52:45 PST
Reference: https://docs.buildbot.net/current/manual/customization.html#builder-priority-functions

Tested on UAT instance using webkit-misc bot.
Comment 3 Jonathan Bedard 2019-11-08 10:23:28 PST
Comment on attachment 383135 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=383135&action=review

> Tools/BuildSlaveSupport/ews-build/loadConfig.py:80
> +    c['prioritizeBuilders'] = prioritizeBuilders

Won't this mean that any build request will pre-empt a test request? Is that actually what we want?
Comment 4 Aakash Jain 2019-11-08 10:38:25 PST
(In reply to Jonathan Bedard from comment #3)
> Won't this mean that any build request will pre-empt a test request?
Yes, but only for bots which are shared on multiple queues. Currently each queue has several dedicated bots. Some of the bots are shared between builder and tester queue.

Currently reverse is happening with the shared bots, especially on 'macOS-High-Sierra-Debug-WK1-Tests-EWS' which shares a lot of bots with 'macOS-High-Sierra-Debug-Build-EWS'.

> Is that actually what we want?
Yes, I believe that is desirable. Also note that builders are usually much faster than testers, since builders do incremental builds.
Comment 5 Jonathan Bedard 2019-11-08 10:49:06 PST
(In reply to Aakash Jain from comment #4)
> (In reply to Jonathan Bedard from comment #3)
> > Won't this mean that any build request will pre-empt a test request?
> Yes, but only for bots which are shared on multiple queues. Currently each
> queue has several dedicated bots. Some of the bots are shared between
> builder and tester queue.
> 
> Currently reverse is happening with the shared bots, especially on
> 'macOS-High-Sierra-Debug-WK1-Tests-EWS' which shares a lot of bots with
> 'macOS-High-Sierra-Debug-Build-EWS'.
> 
> > Is that actually what we want?
> Yes, I believe that is desirable. Also note that builders are usually much
> faster than testers, since builders do incremental builds.

Seems reasonable, but we should keep an eye on this, it seems like a potential source of issues
Comment 6 Aakash Jain 2019-11-08 10:53:51 PST
Committed r252244: <https://trac.webkit.org/changeset/252244>
Comment 7 Radar WebKit Bug Importer 2019-11-08 10:54:20 PST
<rdar://problem/57027388>