199417 2019-07-02 12:33:56 -0700 [EWS] Multiple builds are triggered for one patch sometimes in new EWS 2019-11-06 11:48:09 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=203903 InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap commit-queue dbates jbedard webkit-bug-importer oldest_to_newest 1549676 0 aakash_jain 2019-07-02 12:33:56 -0700 Noticed following duplicate builds for Patch 373348 in https://bugs.webkit.org/show_bug.cgi?id=199412 on all EWS queues: mac: https://ews-build.webkit.org/#/builders/7/builds/4506 https://ews-build.webkit.org/#/builders/7/builds/4507 ios: https://ews-build.webkit.org/#/builders/2/builds/455 https://ews-build.webkit.org/#/builders/2/builds/456 webkitperl: https://ews-build.webkit.org/#/builders/19/builds/4463 https://ews-build.webkit.org/#/builders/19/builds/4464 It seems like the 'Submit to new EWS' button was pressed almost at the same time when new EWS automatically fetched this patch. The code which submits the patch to Buildbot isn't completely thread-safe and it submitted the patch to EWS twice. 1549677 1 aakash_jain 2019-07-02 12:34:50 -0700 Logs from ews-app: 2019-07-02 11:48:55,938 - Saved patch in database, id: 373348 2019-07-02 11:48:55,941 - 1 r? patches, 1 patches need to be sent to Buildbot. 2019-07-02 11:48:55,942 - Starting new HTTPS connection (1): bugs.webkit.org 2019-07-02 11:48:56,307 - 1 r? patches, 1 patches need to be sent to Buildbot. 2019-07-02 11:48:56,308 - Starting new HTTPS connection (1): bugs.webkit.org 2019-07-02 11:48:57,218 - Saved build 1_28168 in database for patch_id: 373348 2019-07-02 11:48:57,288 - Saved build 1_28169 in database for patch_id: 373348 2019-07-02 11:48:57,393 - Saved build 1_28170 in database for patch_id: 373348 2019-07-02 11:48:57,446 - Patch 373348 already has bug id 199412 set. 2019-07-02 11:48:57,449 - Patch 373348 has already been sent to Buildbot. 1557956 2 aakash_jain 2019-08-02 10:00:22 -0700 *** Bug 200336 has been marked as a duplicate of this bug. *** 1557957 3 aakash_jain 2019-08-02 10:01:44 -0700 Dan reported that same issue also happened for https://bugs.webkit.org/show_bug.cgi?id=200324 ((attachment 375282) <https://ews-build.webkit.org/#/builders/15/builds/172> and <https://ews-build.webkit.org/#/builders/15/builds/173> 1557961 4 aakash_jain 2019-08-02 10:06:07 -0700 This issue also happens when someone clicks 'Submit to new EWS' button multiple times (before the status-bubbles loads). 1557968 5 aakash_jain 2019-08-02 10:19:34 -0700 I added additional logging to djnago app (ews-app). 'buildbot try' command (to send patch from djnago to buildbot) is taking 15-20 seconds, which is much higher than expected. Due to this after clicking 'Submit to new ews', it takes a while to load the status-bubbles. During this time, if someone clicks the button again, the patch is submitted again. Few logs: 2019-08-02 08:14:27,906 - took 15292ms to send patch 375409 to buildbot 2019-08-02 08:21:53,255 - took 16408ms to send patch 375410 to buildbot 2019-08-02 08:22:09,649 - took 27792ms to send patch 375411 to buildbot 2019-08-02 09:16:44,736 - took 15001ms to send patch 375412 to buildbot 2019-08-02 09:27:09,028 - took 16327ms to send patch 375308 to buildbot 2019-08-02 09:52:22,490 - took 15235ms to send patch 375414 to buildbot 2019-08-02 09:53:08,165 - took 17649ms to send patch 375415 to buildbot 2019-08-02 09:57:21,861 - took 15566ms to send patch 375416 to buildbot 1557969 6 aakash_jain 2019-08-02 10:24:11 -0700 The sent_to_buildbot flag is checked at the very beginning, and we set this flag after sending is successful. If user clicks twice, both threads/processes see sent_to_buildbot as false and send the patch to buildbot, and then both set sent_to_buildbot to True. 1567427 7 webkit-bug-importer 2019-09-03 10:40:00 -0700 <rdar://problem/54980936> 1582969 8 381715 aakash_jain 2019-10-23 13:07:44 -0700 Created attachment 381715 Patch 1582991 9 381715 jbedard 2019-10-23 13:30:42 -0700 Comment on attachment 381715 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=381715&action=review > Tools/BuildSlaveSupport/ews-app/ews/fetcher.py:63 > + _log.info('{} r? patches, {} patches need to be sent to Buildbot: {}'.format(len(patch_ids), len(patches_to_send), patches_to_send)) What are 'patches_to_send' and what do they look like when printed? 1583029 10 381715 aakash_jain 2019-10-23 14:28:51 -0700 Comment on attachment 381715 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=381715&action=review >> Tools/BuildSlaveSupport/ews-app/ews/fetcher.py:63 >> + _log.info('{} r? patches, {} patches need to be sent to Buildbot: {}'.format(len(patch_ids), len(patches_to_send), patches_to_send)) > > What are 'patches_to_send' and what do they look like when printed? patches_to_send is a list of r? patch_ids which needs EWS processing. This is just a log improvement. This is how the log looks like after this change: 29 r? patches, 2 patches need to be sent to Buildbot: [381699, 381700] 1583074 11 381715 jbedard 2019-10-23 15:18:23 -0700 Comment on attachment 381715 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=381715&action=review >>> Tools/BuildSlaveSupport/ews-app/ews/fetcher.py:63 >>> + _log.info('{} r? patches, {} patches need to be sent to Buildbot: {}'.format(len(patch_ids), len(patches_to_send), patches_to_send)) >> >> What are 'patches_to_send' and what do they look like when printed? > > patches_to_send is a list of r? patch_ids which needs EWS processing. > > This is just a log improvement. This is how the log looks like after this change: > 29 r? patches, 2 patches need to be sent to Buildbot: [381699, 381700] Got it, so when we print patches, its just the patch IDs. 1583164 12 381715 commit-queue 2019-10-23 17:18:22 -0700 Comment on attachment 381715 Patch Clearing flags on attachment: 381715 Committed r251516: <https://trac.webkit.org/changeset/251516> 1583165 13 commit-queue 2019-10-23 17:18:24 -0700 All reviewed patches have been landed. Closing bug. 381715 2019-10-23 13:07:44 -0700 2019-10-23 17:18:22 -0700 Patch bug-199417-20191023160743.patch text/plain 5119 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1MTQ5MCkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE3IEBACisyMDE5LTEwLTIzICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBbRVdTXSBNdWx0aXBsZSBidWlsZHMgYXJlIHRyaWdnZXJlZCBmb3Igb25l IHBhdGNoIHNvbWV0aW1lcyBpbiBuZXcgRVdTCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQu b3JnL3Nob3dfYnVnLmNnaT9pZD0xOTk0MTcKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkg KE9PUFMhKS4KKworICAgICAgICAqIEJ1aWxkU2xhdmVTdXBwb3J0L2V3cy1hcHAvZXdzL2ZldGNo ZXIucHk6CisgICAgICAgIChCdWd6aWxsYVBhdGNoRmV0Y2hlci5mZXRjaCk6IFNldCB0aGUgZmxh ZyBiZWZvcmUgc2VuZGluZyB0aGUgcGF0Y2ggdG8gYnVpbGRib3QuIFVuc2V0IGl0IGluIGNhc2Ug b2YgZmFpbHVyZS4gCisgICAgICAgIEFsc28gYWRkZWQgYWRkaXRpb25hbCBjaGVjayBmb3Igc2Vu dF90b19idWlsZGJvdCBmbGFnIGJlZm9yZSBwcm9jZXNzaW5nIHBhdGNoLgorICAgICAgICAqIEJ1 aWxkU2xhdmVTdXBwb3J0L2V3cy1hcHAvZXdzL21vZGVscy9wYXRjaC5weToKKyAgICAgICAgKFBh dGNoLnNhdmVfcGF0Y2gpOiBJbXByb3ZlZCBsb2dnaW5nLgorICAgICAgICAoUGF0Y2guc2V0X3Nl bnRfdG9fYnVpbGRib3QpOiBNb2RpZmllZCB0byBhY2NlcHQgc2VudF90b19idWlsZGJvdCBwYXJh bWF0ZXIsIGFuZCBzZXQgdGhlIHZhbHVlIGFjY29yZGluZ2x5LgorCiAyMDE5LTEwLTIyICBKaWV3 ZW4gVGFuICA8amlld2VuX3RhbkBhcHBsZS5jb20+CiAKICAgICAgICAgW1dlYkF1dGhuXSBTdXBw bHkgRnJhbWVJbmZvIGluIC1bV0tVSURlbGVnYXRlUHJpdmF0ZSBfd2ViVmlldzpydW5XZWJBdXRo ZW50aWNhdGlvblBhbmVsOmluaXRpYXRlZEJ5RnJhbWU6Y29tcGxldGlvbkhhbmRsZXI6XQpJbmRl eDogVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWFwcC9ld3MvZmV0Y2hlci5weQo9PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09Ci0tLSBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYXBwL2V3cy9mZXRjaGVyLnB5CShy ZXZpc2lvbiAyNTE0OTApCisrKyBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYXBwL2V3cy9m ZXRjaGVyLnB5CSh3b3JraW5nIGNvcHkpCkBAIC02MCw5ICs2MCwxMyBAQCBjbGFzcyBCdWd6aWxs YVBhdGNoRmV0Y2hlcigpOgogICAgICAgICBfbG9nLmRlYnVnKCdyPyBwYXRjaGVzOiB7fScuZm9y bWF0KHBhdGNoX2lkcykpCiAgICAgICAgIFBhdGNoLnNhdmVfcGF0Y2hlcyhwYXRjaF9pZHMpCiAg ICAgICAgIHBhdGNoZXNfdG9fc2VuZCA9IHNlbGYucGF0Y2hlc190b19zZW5kX3RvX2J1aWxkYm90 KHBhdGNoX2lkcykKLSAgICAgICAgX2xvZy5pbmZvKCd7fSByPyBwYXRjaGVzLCB7fSBwYXRjaGVz IG5lZWQgdG8gYmUgc2VudCB0byBCdWlsZGJvdC4nLmZvcm1hdChsZW4ocGF0Y2hfaWRzKSwgbGVu KHBhdGNoZXNfdG9fc2VuZCkpKQorICAgICAgICBfbG9nLmluZm8oJ3t9IHI/IHBhdGNoZXMsIHt9 IHBhdGNoZXMgbmVlZCB0byBiZSBzZW50IHRvIEJ1aWxkYm90OiB7fScuZm9ybWF0KGxlbihwYXRj aF9pZHMpLCBsZW4ocGF0Y2hlc190b19zZW5kKSwgcGF0Y2hlc190b19zZW5kKSkKIAogICAgICAg ICBmb3IgcGF0Y2hfaWQgaW4gcGF0Y2hlc190b19zZW5kOgorICAgICAgICAgICAgaWYgUGF0Y2gu aXNfcGF0Y2hfc2VudF90b19idWlsZGJvdChwYXRjaF9pZCk6CisgICAgICAgICAgICAgICAgX2xv Zy5lcnJvcignUGF0Y2gge30gaXMgYWxyZWFkeSBzZW50IHRvIGJ1aWxkYm90LicuZm9ybWF0KHBh dGNoX2lkKSkKKyAgICAgICAgICAgICAgICBjb250aW51ZQorICAgICAgICAgICAgUGF0Y2guc2V0 X3NlbnRfdG9fYnVpbGRib3QocGF0Y2hfaWQsIFRydWUpCiAgICAgICAgICAgICBiel9wYXRjaCA9 IEJ1Z3ppbGxhLnJldHJpZXZlX2F0dGFjaG1lbnQocGF0Y2hfaWQpCiAgICAgICAgICAgICBpZiBu b3QgYnpfcGF0Y2ggb3IgYnpfcGF0Y2hbJ2lkJ10gIT0gcGF0Y2hfaWQ6CiAgICAgICAgICAgICAg ICAgX2xvZy5lcnJvcignVW5hYmxlIHRvIHJldHJpdmUgcGF0Y2ggInt9IicuZm9ybWF0KHBhdGNo X2lkKSkKQEAgLTc1LDkgKzc5LDkgQEAgY2xhc3MgQnVnemlsbGFQYXRjaEZldGNoZXIoKToKICAg ICAgICAgICAgICAgICAgICAgIHByb3BlcnRpZXM9WydwYXRjaF9pZD17fScuZm9ybWF0KHBhdGNo X2lkKSwgJ2J1Z19pZD17fScuZm9ybWF0KGJ6X3BhdGNoWydidWdfaWQnXSksICdvd25lcj17fScu Zm9ybWF0KGJ6X3BhdGNoLmdldCgnY3JlYXRvcicsICcnKSldKQogICAgICAgICAgICAgaWYgcmMg PT0gMDoKICAgICAgICAgICAgICAgICBQYXRjaC5zZXRfYnVnX2lkKHBhdGNoX2lkLCBiel9wYXRj aFsnYnVnX2lkJ10pCi0gICAgICAgICAgICAgICAgUGF0Y2guc2V0X3NlbnRfdG9fYnVpbGRib3Qo cGF0Y2hfaWQpCiAgICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgIF9sb2cuZXJyb3Io J0ZhaWxlZCB0byBzZW5kIHBhdGNoIHRvIGJ1aWxkYm90LicpCisgICAgICAgICAgICAgICAgUGF0 Y2guc2V0X3NlbnRfdG9fYnVpbGRib3QocGF0Y2hfaWQsIEZhbHNlKQogICAgICAgICAgICAgICAg ICNGSVhNRTogc2VuZCBhbiBlbWFpbCBmb3IgdGhpcyBmYWlsdXJlCiAgICAgICAgIHJldHVybiBw YXRjaF9pZHMKIApJbmRleDogVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWFwcC9ld3MvbW9k ZWxzL3BhdGNoLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1h cHAvZXdzL21vZGVscy9wYXRjaC5weQkocmV2aXNpb24gMjUxNDkwKQorKysgVG9vbHMvQnVpbGRT bGF2ZVN1cHBvcnQvZXdzLWFwcC9ld3MvbW9kZWxzL3BhdGNoLnB5CSh3b3JraW5nIGNvcHkpCkBA IC00NSwxMCArNDUsMTEgQEAgY2xhc3MgUGF0Y2gobW9kZWxzLk1vZGVsKToKICAgICBAY2xhc3Nt ZXRob2QKICAgICBkZWYgc2F2ZV9wYXRjaChjbHMsIHBhdGNoX2lkLCBidWdfaWQ9LTEsIG9ic29s ZXRlPUZhbHNlLCBzZW50X3RvX2J1aWxkYm90PUZhbHNlKToKICAgICAgICAgaWYgbm90IFBhdGNo LmlzX3ZhbGlkX3BhdGNoX2lkKHBhdGNoX2lkKToKKyAgICAgICAgICAgIF9sb2cud2FybignUGF0 Y2ggaWQge30gaW4gaW52YWxpZC4gU2tpcHBlZCBzYXZpbmcuJy5mb3JtYXQocGF0Y2hfaWQpKQog ICAgICAgICAgICAgcmV0dXJuIEVSUl9JTlZBTElEX1BBVENIX0lECiAKICAgICAgICAgaWYgUGF0 Y2guaXNfZXhpc3RpbmdfcGF0Y2hfaWQocGF0Y2hfaWQpOgotICAgICAgICAgICAgX2xvZy5kZWJ1 ZygiUGF0Y2ggaWQge30gYWxyZWFkeSBleGlzdHMgaW4gZGF0YWJhc2UuIFNraXBwZWQgc2F2aW5n LiIuZm9ybWF0KHBhdGNoX2lkKSkKKyAgICAgICAgICAgIF9sb2cuZGVidWcoJ1BhdGNoIGlkIHt9 IGFscmVhZHkgZXhpc3RzIGluIGRhdGFiYXNlLiBTa2lwcGVkIHNhdmluZy4nLmZvcm1hdChwYXRj aF9pZCkpCiAgICAgICAgICAgICByZXR1cm4gRVJSX0VYSVNUSU5HX1BBVENICiAgICAgICAgIFBh dGNoKHBhdGNoX2lkLCBidWdfaWQsIG9ic29sZXRlLCBzZW50X3RvX2J1aWxkYm90KS5zYXZlKCkK ICAgICAgICAgX2xvZy5pbmZvKCdTYXZlZCBwYXRjaCBpbiBkYXRhYmFzZSwgaWQ6IHt9Jy5mb3Jt YXQocGF0Y2hfaWQpKQpAQCAtODIsMTggKzgzLDIwIEBAIGNsYXNzIFBhdGNoKG1vZGVscy5Nb2Rl bCk6CiAgICAgICAgICAgICByZXR1cm4gTm9uZQogCiAgICAgQGNsYXNzbWV0aG9kCi0gICAgZGVm IHNldF9zZW50X3RvX2J1aWxkYm90KGNscywgcGF0Y2hfaWQpOgorICAgIGRlZiBzZXRfc2VudF90 b19idWlsZGJvdChjbHMsIHBhdGNoX2lkLCBzZW50X3RvX2J1aWxkYm90PVRydWUpOgogICAgICAg ICBpZiBub3QgUGF0Y2guaXNfZXhpc3RpbmdfcGF0Y2hfaWQocGF0Y2hfaWQpOgotICAgICAgICAg ICAgcmV0dXJuIEVSUl9OT05fRVhJU1RJTkdfUEFUQ0gKKyAgICAgICAgICAgIFBhdGNoLnNhdmVf cGF0Y2gocGF0Y2hfaWQ9cGF0Y2hfaWQsIHNlbnRfdG9fYnVpbGRib3Q9c2VudF90b19idWlsZGJv dCkKKyAgICAgICAgICAgIF9sb2cuaW5mbygnUGF0Y2gge30gc2F2ZWQgdG8gZGF0YWJhc2Ugd2l0 aCBzZW50X3RvX2J1aWxkYm90PXt9Jy5mb3JtYXQocGF0Y2hfaWQsIHNlbnRfdG9fYnVpbGRib3Qp KQorICAgICAgICAgICAgcmV0dXJuIFNVQ0NFU1MKIAogICAgICAgICBwYXRjaCA9IFBhdGNoLm9i amVjdHMuZ2V0KHBrPXBhdGNoX2lkKQotICAgICAgICBpZiBwYXRjaC5zZW50X3RvX2J1aWxkYm90 OgotICAgICAgICAgICAgX2xvZy53YXJuKCdQYXRjaCB7fSBoYXMgYWxyZWFkeSBiZWVuIHNlbnQg dG8gQnVpbGRib3QuJy5mb3JtYXQocGF0Y2hfaWQpKQorICAgICAgICBpZiBwYXRjaC5zZW50X3Rv X2J1aWxkYm90ID09IHNlbnRfdG9fYnVpbGRib3Q6CisgICAgICAgICAgICBfbG9nLndhcm4oJ1Bh dGNoIHt9IGFscmVhZHkgaGFzIHNlbnRfdG9fYnVpbGRib3Q9e30nLmZvcm1hdChwYXRjaF9pZCwg c2VudF90b19idWlsZGJvdCkpCiAgICAgICAgICAgICByZXR1cm4gU1VDQ0VTUwogCi0gICAgICAg IHBhdGNoLnNlbnRfdG9fYnVpbGRib3QgPSBUcnVlCisgICAgICAgIHBhdGNoLnNlbnRfdG9fYnVp bGRib3QgPSBzZW50X3RvX2J1aWxkYm90CiAgICAgICAgIHBhdGNoLnNhdmUoKQotICAgICAgICBf bG9nLmRlYnVnKCdVcGRhdGVkIHBhdGNoIHt9IHdpdGggc2VudF90b19idWlsZGJvdD1UcnVlJy5m b3JtYXQocGF0Y2hfaWQpKQorICAgICAgICBfbG9nLmluZm8oJ1VwZGF0ZWQgcGF0Y2gge30gd2l0 aCBzZW50X3RvX2J1aWxkYm90PXt9Jy5mb3JtYXQocGF0Y2hfaWQsIHNlbnRfdG9fYnVpbGRib3Qp KQogICAgICAgICByZXR1cm4gU1VDQ0VTUwogCiAgICAgQGNsYXNzbWV0aG9kCg==