Bug 199417

Summary: [EWS] Multiple builds are triggered for one patch sometimes in new EWS
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, ap, commit-queue, dbates, jbedard, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=203903
Attachments:
Description Flags
Patch none

Description Aakash Jain 2019-07-02 12:33:56 PDT
Noticed following duplicate builds for Patch 373348 in https://bugs.webkit.org/show_bug.cgi?id=199412 on all EWS queues:
mac:
https://ews-build.webkit.org/#/builders/7/builds/4506
https://ews-build.webkit.org/#/builders/7/builds/4507

ios:
https://ews-build.webkit.org/#/builders/2/builds/455
https://ews-build.webkit.org/#/builders/2/builds/456

webkitperl:
https://ews-build.webkit.org/#/builders/19/builds/4463
https://ews-build.webkit.org/#/builders/19/builds/4464


It seems like the 'Submit to new EWS' button was pressed almost at the same time when new EWS automatically fetched this patch. The code which submits the patch to Buildbot isn't completely thread-safe and it submitted the patch to EWS twice.
Comment 1 Aakash Jain 2019-07-02 12:34:50 PDT
Logs from ews-app:
2019-07-02 11:48:55,938 - Saved patch in database, id: 373348
2019-07-02 11:48:55,941 - 1 r? patches, 1 patches need to be sent to Buildbot.
2019-07-02 11:48:55,942 - Starting new HTTPS connection (1): bugs.webkit.org
2019-07-02 11:48:56,307 - 1 r? patches, 1 patches need to be sent to Buildbot.
2019-07-02 11:48:56,308 - Starting new HTTPS connection (1): bugs.webkit.org
2019-07-02 11:48:57,218 - Saved build 1_28168 in database for patch_id: 373348
2019-07-02 11:48:57,288 - Saved build 1_28169 in database for patch_id: 373348
2019-07-02 11:48:57,393 - Saved build 1_28170 in database for patch_id: 373348
2019-07-02 11:48:57,446 - Patch 373348 already has bug id 199412 set.
2019-07-02 11:48:57,449 - Patch 373348 has already been sent to Buildbot.
Comment 2 Aakash Jain 2019-08-02 10:00:22 PDT
*** Bug 200336 has been marked as a duplicate of this bug. ***
Comment 4 Aakash Jain 2019-08-02 10:06:07 PDT
This issue also happens when someone clicks 'Submit to new EWS' button multiple times (before the status-bubbles loads).
Comment 5 Aakash Jain 2019-08-02 10:19:34 PDT
I added additional logging to djnago app (ews-app).

'buildbot try' command (to send patch from djnago to buildbot) is taking 15-20 seconds, which is much higher than expected. Due to this after clicking 'Submit to new ews', it takes a while to load the status-bubbles. During this time, if someone clicks the button again, the patch is submitted again. 

Few logs:
2019-08-02 08:14:27,906 - took 15292ms to send patch 375409 to buildbot
2019-08-02 08:21:53,255 - took 16408ms to send patch 375410 to buildbot
2019-08-02 08:22:09,649 - took 27792ms to send patch 375411 to buildbot
2019-08-02 09:16:44,736 - took 15001ms to send patch 375412 to buildbot
2019-08-02 09:27:09,028 - took 16327ms to send patch 375308 to buildbot
2019-08-02 09:52:22,490 - took 15235ms to send patch 375414 to buildbot
2019-08-02 09:53:08,165 - took 17649ms to send patch 375415 to buildbot
2019-08-02 09:57:21,861 - took 15566ms to send patch 375416 to buildbot
Comment 6 Aakash Jain 2019-08-02 10:24:11 PDT
The sent_to_buildbot flag is checked at the very beginning, and we set this flag after sending is successful. If user clicks twice, both threads/processes see sent_to_buildbot as false and send the patch to buildbot, and then both set sent_to_buildbot to True.
Comment 7 Radar WebKit Bug Importer 2019-09-03 10:40:00 PDT
<rdar://problem/54980936>
Comment 8 Aakash Jain 2019-10-23 13:07:44 PDT
Created attachment 381715 [details]
Patch
Comment 9 Jonathan Bedard 2019-10-23 13:30:42 PDT
Comment on attachment 381715 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=381715&action=review

> Tools/BuildSlaveSupport/ews-app/ews/fetcher.py:63
> +        _log.info('{} r? patches, {} patches need to be sent to Buildbot: {}'.format(len(patch_ids), len(patches_to_send), patches_to_send))

What are 'patches_to_send' and what do they look like when printed?
Comment 10 Aakash Jain 2019-10-23 14:28:51 PDT
Comment on attachment 381715 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=381715&action=review

>> Tools/BuildSlaveSupport/ews-app/ews/fetcher.py:63
>> +        _log.info('{} r? patches, {} patches need to be sent to Buildbot: {}'.format(len(patch_ids), len(patches_to_send), patches_to_send))
> 
> What are 'patches_to_send' and what do they look like when printed?

patches_to_send is a list of r? patch_ids which needs EWS processing. 

This is just a log improvement. This is how the log looks like after this change:
29 r? patches, 2 patches need to be sent to Buildbot: [381699, 381700]
Comment 11 Jonathan Bedard 2019-10-23 15:18:23 PDT
Comment on attachment 381715 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=381715&action=review

>>> Tools/BuildSlaveSupport/ews-app/ews/fetcher.py:63
>>> +        _log.info('{} r? patches, {} patches need to be sent to Buildbot: {}'.format(len(patch_ids), len(patches_to_send), patches_to_send))
>> 
>> What are 'patches_to_send' and what do they look like when printed?
> 
> patches_to_send is a list of r? patch_ids which needs EWS processing. 
> 
> This is just a log improvement. This is how the log looks like after this change:
> 29 r? patches, 2 patches need to be sent to Buildbot: [381699, 381700]

Got it, so when we print patches, its just the patch IDs.
Comment 12 WebKit Commit Bot 2019-10-23 17:18:22 PDT
Comment on attachment 381715 [details]
Patch

Clearing flags on attachment: 381715

Committed r251516: <https://trac.webkit.org/changeset/251516>
Comment 13 WebKit Commit Bot 2019-10-23 17:18:24 PDT
All reviewed patches have been landed.  Closing bug.