203858 2019-11-05 11:27:53 -0800 EWS should retry build in case of kill-old-processes failure 2020-02-20 08:06:56 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=208003 InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap commit-queue jbedard webkit-bug-importer oldest_to_newest 1587479 0 aakash_jain 2019-11-05 11:27:53 -0800 Sometime a bot gets in a bad state and kill-old-processes start failing. When that happens, that bot process (and fails) the build very quickly. Because of this all the pending builds end-up failing. For example 2 weeks back ews121 went into bad state and many builds failed, e.g: https://ews-build.webkit.org/#/builders/24/builds/2671 https://ews-build.webkit.org/#/builders/24/builds/2672 https://ews-build.webkit.org/#/builders/24/builds/2673 https://ews-build.webkit.org/#/builders/24/builds/2674 https://ews-build.webkit.org/#/builders/24/builds/2675 https://ews-build.webkit.org/#/builders/24/builds/2677 https://ews-build.webkit.org/#/builders/24/builds/2680 https://ews-build.webkit.org/#/builders/24/builds/2684 https://ews-build.webkit.org/#/builders/24/builds/2690 https://ews-build.webkit.org/#/builders/24/builds/2691 https://ews-build.webkit.org/#/builders/24/builds/2693 https://ews-build.webkit.org/#/builders/24/builds/2694 https://ews-build.webkit.org/#/builders/24/builds/2697 We should retry the build, in case of kill-old-processes failure, so that that bot do not burn through all the pending builds. The build will keep on retrying, until a different bot picks it up. This will make EWS robust against this kind of infrastructure failure. 1587481 1 382837 aakash_jain 2019-11-05 11:33:54 -0800 Created attachment 382837 Patch 1587482 2 aakash_jain 2019-11-05 11:34:57 -0800 Sample run: https://ews-build.webkit-uat.org/#/builders/3/builds/227 1587918 3 jbedard 2019-11-06 14:23:07 -0800 ....if kill-old-prcesses fails, we should force reboot the machine. Do we have any evidence that a retry will actually help? 1589096 4 aakash_jain 2019-11-10 05:29:40 -0800 It just happened again in https://ews-build.webkit.org/#/builders/24/builds/4562 and https://ews-build.webkit.org/#/builders/24/builds/4566 > ....if kill-old-prcesses fails, we should force reboot the machine. Yes, rebooting the machine is a better idea. However, it will take me a while to implement and test that. Meanwhile can we land this (maybe with a FIXME), since this is clearly an improvement. > Do we have any evidence that a retry will actually help? Yes, we have already seen it working many times for RETRY on checkout failure (https://trac.webkit.org/changeset/247364/webkit). For example when bot igalia1-gtk-wk2-ews went out of space in https://ews-build.webkit.org/#/builders/4/builds/6684, instead of simply failing, build was retried, and picked up by different bot and passed in https://ews-build.webkit.org/#/builders/4/builds/6685 1589192 5 jbedard 2019-11-11 07:41:21 -0800 (In reply to Aakash Jain from comment #4) > It just happened again in > https://ews-build.webkit.org/#/builders/24/builds/4562 and > https://ews-build.webkit.org/#/builders/24/builds/4566 > > > ....if kill-old-prcesses fails, we should force reboot the machine. > Yes, rebooting the machine is a better idea. However, it will take me a > while to implement and test that. Meanwhile can we land this (maybe with a > FIXME), since this is clearly an improvement. You've convinced me this is an improvement often enough to be worth landing, although I remain somewhat skeptical of our ability to trust machines which fail kill-old-processes. I'm actually not sure that rebooting takes much testing or additional code. I don't think we need to put effort into being delicate, I don't see a world where machines are failing to kill-old-processes frequently enough to find themselves in a crash loop, and if we're really worried about that, we can just refuse to reboot unless a bot has been up longer than some amount of time (I'd say an hour, but that's sort of arbitrary) > > ... 1589201 6 382837 commit-queue 2019-11-11 08:23:33 -0800 Comment on attachment 382837 Patch Clearing flags on attachment: 382837 Committed r252324: <https://trac.webkit.org/changeset/252324> 1589202 7 commit-queue 2019-11-11 08:23:34 -0800 All reviewed patches have been landed. Closing bug. 1589377 8 webkit-bug-importer 2019-11-11 16:57:21 -0800 <rdar://problem/57099318> 1621178 9 aakash_jain 2020-02-20 06:03:42 -0800 This change seems to be working fine. Few examples where this helped: https://ews-build.webkit.org/#/builders/22/builds/10002 https://ews-build.webkit.org/#/builders/3/builds/16167 https://ews-build.webkit.org/#/builders/3/builds/16230 https://ews-build.webkit.org/#/builders/9/builds/17233 382837 2019-11-05 11:33:54 -0800 2019-11-11 08:23:33 -0800 Patch bug-203858-20191105143353.patch text/plain 2386 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1MjA2NCkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE1IEBACisyMDE5LTExLTA1ICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBFV1Mgc2hvdWxkIHJldHJ5IGJ1aWxkIGluIGNhc2Ugb2Yga2lsbC1vbGQt cHJvY2Vzc2VzIGZhaWx1cmUKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19i dWcuY2dpP2lkPTIwMzg1OAorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgor CisgICAgICAgICogQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzLnB5OgorICAgICAg ICAoS2lsbE9sZFByb2Nlc3Nlcy5ldmFsdWF0ZUNvbW1hbmQpOiBSZXRyeSB0aGUgYnVpbGQgaW4g Y2FzZSBvZiBmYWlsdXJlLgorICAgICAgICAoS2lsbE9sZFByb2Nlc3Nlcy5nZXRSZXN1bHRTdW1t YXJ5KTogVXBkYXRlIHRoZSBidWlsZC1zdGVwIHN1bW1hcnkgc3RyaW5nLgorICAgICAgICAqIEJ1 aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVwc191bml0dGVzdC5weTogVXBkYXRlZCB1bml0 LXRlc3RzLgorCiAyMDE5LTExLTA1ICBXZW5zb24gSHNpZWggIDx3ZW5zb25faHNpZWhAYXBwbGUu Y29tPgogCiAgICAgICAgIE5hdGl2ZSB0ZXh0IHN1YnN0aXR1dGlvbnMgaW50ZXJmZXJlIHdpdGgg SFRNTCA8ZGF0YWxpc3Q+IG9wdGlvbnMgcmVzdWx0aW5nIGluIGNyYXNoCkluZGV4OiBUb29scy9C dWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3RlcHMucHkKPT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gVG9vbHMv QnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzLnB5CShyZXZpc2lvbiAyNTIwNjQpCisr KyBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3RlcHMucHkJKHdvcmtpbmcgY29w eSkKQEAgLTk4NCw2ICs5ODQsMTYgQEAgY2xhc3MgS2lsbE9sZFByb2Nlc3NlcyhzaGVsbC5Db21w aWxlKToKICAgICBkZWYgX19pbml0X18oc2VsZiwgKiprd2FyZ3MpOgogICAgICAgICBzdXBlcihL aWxsT2xkUHJvY2Vzc2VzLCBzZWxmKS5fX2luaXRfXyh0aW1lb3V0PTYwLCBsb2dFbnZpcm9uPUZh bHNlLCAqKmt3YXJncykKIAorICAgIGRlZiBldmFsdWF0ZUNvbW1hbmQoc2VsZiwgY21kKToKKyAg ICAgICAgaWYgY21kLmRpZEZhaWwoKToKKyAgICAgICAgICAgIHNlbGYuYnVpbGQuYnVpbGRGaW5p c2hlZChbJ0ZhaWxlZCB0byBraWxsIG9sZCBwcm9jZXNzZXMsIHJldHJ5aW5nIGJ1aWxkJ10sIFJF VFJZKQorICAgICAgICByZXR1cm4gc2hlbGwuQ29tcGlsZS5ldmFsdWF0ZUNvbW1hbmQoc2VsZiwg Y21kKQorCisgICAgZGVmIGdldFJlc3VsdFN1bW1hcnkoc2VsZik6CisgICAgICAgIGlmIHNlbGYu cmVzdWx0cyA9PSBGQUlMVVJFOgorICAgICAgICAgICAgcmV0dXJuIHt1J3N0ZXAnOiB1J0ZhaWxl ZCB0byBraWxsIG9sZCBwcm9jZXNzZXMnfQorICAgICAgICByZXR1cm4gc2hlbGwuQ29tcGlsZS5n ZXRSZXN1bHRTdW1tYXJ5KHNlbGYpCisKIAogY2xhc3MgUnVuV2ViS2l0VGVzdHMoc2hlbGwuVGVz dCk6CiAgICAgbmFtZSA9ICdsYXlvdXQtdGVzdHMnCkluZGV4OiBUb29scy9CdWlsZFNsYXZlU3Vw cG9ydC9ld3MtYnVpbGQvc3RlcHNfdW5pdHRlc3QucHkKPT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gVG9vbHMvQnVp bGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzX3VuaXR0ZXN0LnB5CShyZXZpc2lvbiAyNTIw NjQpCisrKyBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3RlcHNfdW5pdHRlc3Qu cHkJKHdvcmtpbmcgY29weSkKQEAgLTU5MSw3ICs1OTEsNyBAQCBjbGFzcyBUZXN0S2lsbE9sZFBy b2Nlc3NlcyhCdWlsZFN0ZXBNaXhpCiAgICAgICAgICAgICArIEV4cGVjdFNoZWxsLmxvZygnc3Rk aW8nLCBzdGRvdXQ9J1VuZXhwZWN0ZWQgZXJyb3IuJykKICAgICAgICAgICAgICsgMiwKICAgICAg ICAgKQotICAgICAgICBzZWxmLmV4cGVjdE91dGNvbWUocmVzdWx0PUZBSUxVUkUsIHN0YXRlX3N0 cmluZz0nS2lsbGVkIG9sZCBwcm9jZXNzZXMgKGZhaWx1cmUpJykKKyAgICAgICAgc2VsZi5leHBl Y3RPdXRjb21lKHJlc3VsdD1GQUlMVVJFLCBzdGF0ZV9zdHJpbmc9J0ZhaWxlZCB0byBraWxsIG9s ZCBwcm9jZXNzZXMnKQogICAgICAgICByZXR1cm4gc2VsZi5ydW5TdGVwKCkKIAogCg==