197205 2019-04-23 11:49:31 -0700 webkitpy: Kill unresponsive worker processes 2019-04-23 16:52:50 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 jbedard jbedard ap commit-queue ews-watchlist glenn lforschler ryanhaddad simon.fraser webkit-bug-importer oldest_to_newest 1529674 0 jbedard 2019-04-23 11:49:31 -0700 It's possible for us to get into a state, most common in EWS, where workers are deadlocked waiting on the message queue. In most cases, sending a termination to all workers before joining any workers is sufficient to fix the problem. In some cases, we might need to explicitly kill unresponsive workers. This isn't a great practice, but it's very important when run-webkit-tests has been interrupted, either with a SIGTERM or because the maximum number of failed tests has been hit. 1529675 1 jbedard 2019-04-23 11:52:59 -0700 <rdar://problem/50104388> 1529676 2 368048 jbedard 2019-04-23 11:56:11 -0700 Created attachment 368048 Patch 1529679 3 368049 jbedard 2019-04-23 12:05:24 -0700 Created attachment 368049 Patch 1529686 4 368048 lforschler 2019-04-23 12:28:24 -0700 Comment on attachment 368048 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=368048&action=review I'd like to think on this more, but here are my thoughts: Why are we running into this issue now, and never before? There must be a root cause which gets us in a bad place. Fixing it with a workaround to kill stuck processes seems like we'll just be accepting that things are not great. What does this mean moving forward? Will we start marking these tests as crashing so they don't make the tree red? Will we lose any testing coverage? Will this compound and get even worse moving forward, and cause us to miss things which are really wrong? I'd argue we should find the root cause and determine why our workers are hanging. > Tools/Scripts/webkitpy/common/message_pool.py:143 > + deadline = time.time() + 30 I'm not sure we want to hardcode a 30 second deadline... do we store this timeout value elsewhere we could use here? 1529710 5 jbedard 2019-04-23 13:20:26 -0700 (In reply to Lucas Forschler from comment #4) > Comment on attachment 368048 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=368048&action=review > > I'd like to think on this more, but here are my thoughts: > > Why are we running into this issue now, and never before? There must be a > root cause which gets us in a bad place. Fixing it with a workaround to kill > stuck processes seems like we'll just be accepting that things are not > great. What does this mean moving forward? Will we start marking these tests > as crashing so they don't make the tree red? Will we lose any testing > coverage? Will this compound and get even worse moving forward, and cause us > to miss things which are really wrong? > > I'd argue we should find the root cause and determine why our workers are > hanging. A few responses to this; 1) Why now: If my theory about workers being deadlocked on the message queue is correct (and stack traces corroborate, but do not confirm this) this entire situation is a race condition. I suspect it might be tied to specific kinds of tests, but ultimately, it's not the sort of thing that we're going to be able to find a regression point for. 2) Will we start marking these tests as crashing so they don't make the tree red That's not a great way to address this sort of problem. We aren't actually seeing this in trunk infrastructure anyways (those tests are mostly green), thus far, we've only seen this on EWS. The thing is, EWS is sort of expected to take sub-par code and 'work,' mostly. We know there are some tooling changes that can hose EWS if you send them through, but the sorts of changes we've seen triggering this behavior are the kind of changes EWS is meant to test. Basically, EWS should be able to coherantly handle 30 crashing tests. 3) 'I'd argue we should find the root cause and determine why our workers are hanging' I'm pretty sure that we have addressed the root cause here, at least as far as the hanging goes. This change does 2 separate things, although it's not totally obvious that they are actually distinct. The first thing this change does is make sure that all workers are terminated before we try and join any of them. This may seem like a trivial change, but from my testing on ews106, I'm pretty sure that THIS is actually the root cause. It's also possible that <https://trac.webkit.org/changeset/244064> manifested this bug by trying to log something on the termination handler, which would cause the workers to be posting a message to the manager on the termination handler. The second thing this change does is guarantee that if a worker is hung, we kill it (this seems to be the part of the patch that is contentious). My rationale for this change is essentially this: Once the manager determines that worker is hung, what is the rational thing to do? Never mind the fact that the worker should never get hung (I agree, it shouldn't, hence the first part of this change), once it has already hung, what can be done? The previous code does nothing and waits forever. This is a horrible way to address the problem, especially in EWS. Even if the logging returned by EWS isn't entirely accurate, at least it's present. And at least run-webkit-tests returns with a non-zero exit code since some tests did not pass. It's not a perfect solution, but I'm also not sure it's a case we would encounter after this patch lands. I've applied this patch to ews106 and have been testing for the past hour. While I have doctored the directory so we have more than 30 failures, I have yet to see a worker be killed: all have terminated gracefully because of the 1st part of this change. > > > Tools/Scripts/webkitpy/common/message_pool.py:143 > > + deadline = time.time() + 30 > > I'm not sure we want to hardcode a 30 second deadline... do we store this > timeout value elsewhere we could use here? We don't have an idea of timeouts in the message pool, although we could add it. 1529782 6 368049 lforschler 2019-04-23 15:15:00 -0700 Comment on attachment 368049 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=368049&action=review > Tools/Scripts/webkitpy/common/message_pool.py:146 > + worker.join(timeout=deadline - time.time()) in our chat, you discovered this should probably be join(max(deadline - time.time(), 0)) it might be worth seeing what join will do with a negative timeout... maybe it behaves in an ok way. 1529783 7 368070 jbedard 2019-04-23 15:18:04 -0700 Created attachment 368070 Patch 1529785 8 jbedard 2019-04-23 15:21:03 -0700 (In reply to Lucas Forschler from comment #6) > Comment on attachment 368049 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=368049&action=review > > > Tools/Scripts/webkitpy/common/message_pool.py:146 > > + worker.join(timeout=deadline - time.time()) > > in our chat, you discovered this should probably be join(max(deadline - > time.time(), 0)) > it might be worth seeing what join will do with a negative timeout... maybe > it behaves in an ok way. According to <https://bugs.python.org/issue14753>, it should be treated as a 0...but I dislike relying on confusing behavior like that, let's use max(...). 1529820 9 368070 commit-queue 2019-04-23 16:52:48 -0700 Comment on attachment 368070 Patch Clearing flags on attachment: 368070 Committed r244571: <https://trac.webkit.org/changeset/244571> 1529821 10 commit-queue 2019-04-23 16:52:50 -0700 All reviewed patches have been landed. Closing bug. 368048 2019-04-23 11:56:11 -0700 2019-04-23 12:58:07 -0700 Patch bug-197205-20190423115610.patch text/plain 1816 jbedard SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI0NDU1MykKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE0IEBACisyMDE5LTA0LTIzICBKb25hdGhhbiBCZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNv bT4KKworICAgICAgICB3ZWJraXRweTogS2lsbCB1bnJlc3BvbnNpdmUgd29ya2VyIHByb2Nlc3Nl cworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTk3MjA1 CisgICAgICAgIDxyZGFyOi8vcHJvYmxlbS81MDEwNDM4OD4KKworICAgICAgICBSZXZpZXdlZCBi eSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvY29tbW9uL21l c3NhZ2VfcG9vbC5weToKKyAgICAgICAgKF9NZXNzYWdlUG9vbC5fY2xvc2UpOiBTZW5kIHRlcm1p bmF0aW9uIGJlZm9yZSBqb2luaW5nIHdvcmtlcnMsIGtpbGwgdW5yZXNwb25zaXZlIHdvcmtlcnMu CisKIDIwMTktMDQtMjMgIEFha2FzaCBKYWluICA8YWFrYXNoX2phaW5AYXBwbGUuY29tPgogCiAg ICAgICAgIFtld3MtYnVpbGRdIEFkZCB1bml0IHRlc3RzIGZvciBVbkFwcGx5UGF0Y2hJZlJlcXVp cmVkCkluZGV4OiBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9tZXNzYWdlX3Bvb2wucHkK PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PQotLS0gVG9vbHMvU2NyaXB0cy93ZWJraXRweS9jb21tb24vbWVzc2FnZV9wb29s LnB5CShyZXZpc2lvbiAyNDQwOTQpCisrKyBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9t ZXNzYWdlX3Bvb2wucHkJKHdvcmtpbmcgY29weSkKQEAgLTQzLDcgKzQzLDkgQEAgaW50ZWFkLgog aW1wb3J0IGNQaWNrbGUKIGltcG9ydCBsb2dnaW5nCiBpbXBvcnQgbXVsdGlwcm9jZXNzaW5nCitp bXBvcnQgb3MKIGltcG9ydCBRdWV1ZQoraW1wb3J0IHNpZ25hbAogaW1wb3J0IHN5cwogaW1wb3J0 IHRpbWUKIGltcG9ydCB0cmFjZWJhY2sKQEAgLTEzNyw3ICsxMzksMTYgQEAgY2xhc3MgX01lc3Nh Z2VQb29sKG9iamVjdCk6CiAgICAgICAgIGZvciB3b3JrZXIgaW4gc2VsZi5fd29ya2VyczoKICAg ICAgICAgICAgIGlmIHdvcmtlci5pc19hbGl2ZSgpOgogICAgICAgICAgICAgICAgIHdvcmtlci50 ZXJtaW5hdGUoKQotICAgICAgICAgICAgICAgIHdvcmtlci5qb2luKCkKKworICAgICAgICBkZWFk bGluZSA9IHRpbWUudGltZSgpICsgMzAKKyAgICAgICAgZm9yIHdvcmtlciBpbiBzZWxmLl93b3Jr ZXJzOgorICAgICAgICAgICAgd29ya2VyLmpvaW4odGltZW91dD1kZWFkbGluZSAtIHRpbWUudGlt ZSgpKQorCisgICAgICAgIGZvciB3b3JrZXIgaW4gc2VsZi5fd29ya2VyczoKKyAgICAgICAgICAg IGlmIHdvcmtlci5pc19hbGl2ZSgpOgorICAgICAgICAgICAgICAgIF9sb2cuZXJyb3IoJ3t9IGZh aWxlZCB0byB0ZXJtaW5hdGUsIGtpbGxpbmcgaXQnLmZvcm1hdCh3b3JrZXIubmFtZSkpCisgICAg ICAgICAgICAgICAgb3Mua2lsbCh3b3JrZXIuaWRlbnQsIHNpZ25hbC5TSUdLSUxMKQorCiAgICAg ICAgIHNlbGYuX3dvcmtlcnMgPSBbXQogICAgICAgICBpZiBub3Qgc2VsZi5fcnVubmluZ19pbmxp bmU6CiAgICAgICAgICAgICAjIEZJWE1FOiBUaGlzIGlzIGEgaGFjayB0byBnZXQgbXVsdGlwcm9j ZXNzaW5nIHRvIG5vdCBsb2cgdHJhY2ViYWNrcyBkdXJpbmcgc2h1dGRvd24gOiguCg== 368049 2019-04-23 12:05:24 -0700 2019-04-23 15:18:03 -0700 Patch bug-197205-20190423120524.patch text/plain 1855 jbedard SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI0NDU1MykKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE0IEBACisyMDE5LTA0LTIzICBKb25hdGhhbiBCZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNv bT4KKworICAgICAgICB3ZWJraXRweTogS2lsbCB1bnJlc3BvbnNpdmUgd29ya2VyIHByb2Nlc3Nl cworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTk3MjA1 CisgICAgICAgIDxyZGFyOi8vcHJvYmxlbS81MDEwNDM4OD4KKworICAgICAgICBSZXZpZXdlZCBi eSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvY29tbW9uL21l c3NhZ2VfcG9vbC5weToKKyAgICAgICAgKF9NZXNzYWdlUG9vbC5fY2xvc2UpOiBTZW5kIHRlcm1p bmF0aW9uIGJlZm9yZSBqb2luaW5nIHdvcmtlcnMsIGtpbGwgdW5yZXNwb25zaXZlIHdvcmtlcnMu CisKIDIwMTktMDQtMjMgIEFha2FzaCBKYWluICA8YWFrYXNoX2phaW5AYXBwbGUuY29tPgogCiAg ICAgICAgIFtld3MtYnVpbGRdIEFkZCB1bml0IHRlc3RzIGZvciBVbkFwcGx5UGF0Y2hJZlJlcXVp cmVkCkluZGV4OiBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9tZXNzYWdlX3Bvb2wucHkK PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PQotLS0gVG9vbHMvU2NyaXB0cy93ZWJraXRweS9jb21tb24vbWVzc2FnZV9wb29s LnB5CShyZXZpc2lvbiAyNDQwOTQpCisrKyBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9t ZXNzYWdlX3Bvb2wucHkJKHdvcmtpbmcgY29weSkKQEAgLTQzLDcgKzQzLDkgQEAgaW50ZWFkLgog aW1wb3J0IGNQaWNrbGUKIGltcG9ydCBsb2dnaW5nCiBpbXBvcnQgbXVsdGlwcm9jZXNzaW5nCitp bXBvcnQgb3MKIGltcG9ydCBRdWV1ZQoraW1wb3J0IHNpZ25hbAogaW1wb3J0IHN5cwogaW1wb3J0 IHRpbWUKIGltcG9ydCB0cmFjZWJhY2sKQEAgLTEzNyw3ICsxMzksMTcgQEAgY2xhc3MgX01lc3Nh Z2VQb29sKG9iamVjdCk6CiAgICAgICAgIGZvciB3b3JrZXIgaW4gc2VsZi5fd29ya2VyczoKICAg ICAgICAgICAgIGlmIHdvcmtlci5pc19hbGl2ZSgpOgogICAgICAgICAgICAgICAgIHdvcmtlci50 ZXJtaW5hdGUoKQotICAgICAgICAgICAgICAgIHdvcmtlci5qb2luKCkKKworICAgICAgICBkZWFk bGluZSA9IHRpbWUudGltZSgpICsgMzAKKyAgICAgICAgZm9yIHdvcmtlciBpbiBzZWxmLl93b3Jr ZXJzOgorICAgICAgICAgICAgaWYgd29ya2VyLmlzX2FsaXZlKCk6CisgICAgICAgICAgICAgICAg d29ya2VyLmpvaW4odGltZW91dD1kZWFkbGluZSAtIHRpbWUudGltZSgpKQorCisgICAgICAgIGZv ciB3b3JrZXIgaW4gc2VsZi5fd29ya2VyczoKKyAgICAgICAgICAgIGlmIHdvcmtlci5pc19hbGl2 ZSgpOgorICAgICAgICAgICAgICAgIF9sb2cuZXJyb3IoJ3t9IGZhaWxlZCB0byB0ZXJtaW5hdGUs IGtpbGxpbmcgaXQnLmZvcm1hdCh3b3JrZXIubmFtZSkpCisgICAgICAgICAgICAgICAgb3Mua2ls bCh3b3JrZXIuaWRlbnQsIHNpZ25hbC5TSUdLSUxMKQorCiAgICAgICAgIHNlbGYuX3dvcmtlcnMg PSBbXQogICAgICAgICBpZiBub3Qgc2VsZi5fcnVubmluZ19pbmxpbmU6CiAgICAgICAgICAgICAj IEZJWE1FOiBUaGlzIGlzIGEgaGFjayB0byBnZXQgbXVsdGlwcm9jZXNzaW5nIHRvIG5vdCBsb2cg dHJhY2ViYWNrcyBkdXJpbmcgc2h1dGRvd24gOiguCg== 368070 2019-04-23 15:18:04 -0700 2019-04-23 16:52:48 -0700 Patch bug-197205-20190423151804.patch text/plain 2656 jbedard SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI0NDU1MykKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE1IEBACisyMDE5LTA0LTIzICBKb25hdGhhbiBCZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNv bT4KKworICAgICAgICB3ZWJraXRweTogS2lsbCB1bnJlc3BvbnNpdmUgd29ya2VyIHByb2Nlc3Nl cworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTk3MjA1 CisgICAgICAgIDxyZGFyOi8vcHJvYmxlbS81MDEwNDM4OD4KKworICAgICAgICBSZXZpZXdlZCBi eSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvY29tbW9uL21l c3NhZ2VfcG9vbC5weToKKyAgICAgICAgKF9NZXNzYWdlUG9vbC5fX2luaXRfXyk6IEFkZCBjbG9z ZSB0aW1lb3V0LgorICAgICAgICAoX01lc3NhZ2VQb29sLl9jbG9zZSk6IFNlbmQgdGVybWluYXRp b24gYmVmb3JlIGpvaW5pbmcgd29ya2Vycywga2lsbCB1bnJlc3BvbnNpdmUgd29ya2Vycy4KKwog MjAxOS0wNC0yMyAgQWFrYXNoIEphaW4gIDxhYWthc2hfamFpbkBhcHBsZS5jb20+CiAKICAgICAg ICAgW2V3cy1idWlsZF0gQWRkIHVuaXQgdGVzdHMgZm9yIFVuQXBwbHlQYXRjaElmUmVxdWlyZWQK SW5kZXg6IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvY29tbW9uL21lc3NhZ2VfcG9vbC5weQo9PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09Ci0tLSBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9tZXNzYWdlX3Bvb2wucHkJ KHJldmlzaW9uIDI0NDA5NCkKKysrIFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvY29tbW9uL21lc3Nh Z2VfcG9vbC5weQkod29ya2luZyBjb3B5KQpAQCAtNDMsNyArNDMsOSBAQCBpbnRlYWQuCiBpbXBv cnQgY1BpY2tsZQogaW1wb3J0IGxvZ2dpbmcKIGltcG9ydCBtdWx0aXByb2Nlc3NpbmcKK2ltcG9y dCBvcwogaW1wb3J0IFF1ZXVlCitpbXBvcnQgc2lnbmFsCiBpbXBvcnQgc3lzCiBpbXBvcnQgdGlt ZQogaW1wb3J0IHRyYWNlYmFjawpAQCAtNjIsNyArNjQsNyBAQCBkZWYgZ2V0KGNhbGxlciwgd29y a2VyX2ZhY3RvcnksIG51bV93b3JrCiAKIAogY2xhc3MgX01lc3NhZ2VQb29sKG9iamVjdCk6Ci0g ICAgZGVmIF9faW5pdF9fKHNlbGYsIGNhbGxlciwgd29ya2VyX2ZhY3RvcnksIG51bV93b3JrZXJz LCB3b3JrZXJfc3RhcnR1cF9kZWxheV9zZWNzPTAuMCwgaG9zdD1Ob25lKToKKyAgICBkZWYgX19p bml0X18oc2VsZiwgY2FsbGVyLCB3b3JrZXJfZmFjdG9yeSwgbnVtX3dvcmtlcnMsIHdvcmtlcl9z dGFydHVwX2RlbGF5X3NlY3M9MC4wLCBob3N0PU5vbmUsIHRpbWVvdXQ9MzApOgogICAgICAgICBz ZWxmLl9jYWxsZXIgPSBjYWxsZXIKICAgICAgICAgc2VsZi5fd29ya2VyX2ZhY3RvcnkgPSB3b3Jr ZXJfZmFjdG9yeQogICAgICAgICBzZWxmLl9udW1fd29ya2VycyA9IG51bV93b3JrZXJzCkBAIC03 Miw2ICs3NCw3IEBAIGNsYXNzIF9NZXNzYWdlUG9vbChvYmplY3QpOgogICAgICAgICBzZWxmLl9o b3N0ID0gaG9zdAogICAgICAgICBzZWxmLl9uYW1lID0gJ21hbmFnZXInCiAgICAgICAgIHNlbGYu X3J1bm5pbmdfaW5saW5lID0gKHNlbGYuX251bV93b3JrZXJzID09IDEpCisgICAgICAgIHNlbGYu X3RpbWVvdXQgPSB0aW1lb3V0CiAgICAgICAgIGlmIHNlbGYuX3J1bm5pbmdfaW5saW5lOgogICAg ICAgICAgICAgc2VsZi5fbWVzc2FnZXNfdG9fd29ya2VyID0gUXVldWUuUXVldWUoKQogICAgICAg ICAgICAgc2VsZi5fbWVzc2FnZXNfdG9fbWFuYWdlciA9IFF1ZXVlLlF1ZXVlKCkKQEAgLTEzNyw3 ICsxNDAsMTUgQEAgY2xhc3MgX01lc3NhZ2VQb29sKG9iamVjdCk6CiAgICAgICAgIGZvciB3b3Jr ZXIgaW4gc2VsZi5fd29ya2VyczoKICAgICAgICAgICAgIGlmIHdvcmtlci5pc19hbGl2ZSgpOgog ICAgICAgICAgICAgICAgIHdvcmtlci50ZXJtaW5hdGUoKQotICAgICAgICAgICAgICAgIHdvcmtl ci5qb2luKCkKKworICAgICAgICBkZWFkbGluZSA9IHRpbWUudGltZSgpICsgc2VsZi5fdGltZW91 dAorICAgICAgICBmb3Igd29ya2VyIGluIHNlbGYuX3dvcmtlcnM6CisgICAgICAgICAgICBpZiB3 b3JrZXIuaXNfYWxpdmUoKToKKyAgICAgICAgICAgICAgICB3b3JrZXIuam9pbih0aW1lb3V0PW1h eChkZWFkbGluZSAtIHRpbWUudGltZSgpLCAwKSkKKyAgICAgICAgICAgIGlmIHdvcmtlci5pc19h bGl2ZSgpOgorICAgICAgICAgICAgICAgIF9sb2cuZXJyb3IoJ3t9IGZhaWxlZCB0byB0ZXJtaW5h dGUsIGtpbGxpbmcgaXQnLmZvcm1hdCh3b3JrZXIubmFtZSkpCisgICAgICAgICAgICAgICAgb3Mu a2lsbCh3b3JrZXIuaWRlbnQsIHNpZ25hbC5TSUdLSUxMKQorCiAgICAgICAgIHNlbGYuX3dvcmtl cnMgPSBbXQogICAgICAgICBpZiBub3Qgc2VsZi5fcnVubmluZ19pbmxpbmU6CiAgICAgICAgICAg ICAjIEZJWE1FOiBUaGlzIGlzIGEgaGFjayB0byBnZXQgbXVsdGlwcm9jZXNzaW5nIHRvIG5vdCBs b2cgdHJhY2ViYWNrcyBkdXJpbmcgc2h1dGRvd24gOiguCg==