95906 2012-09-05 15:45:48 -0700 WKTR often reports an unresponsive WebProcess on Mac bots 2017-07-18 08:29:42 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified NEW InRadar, MakingBotsRed P2 Normal --- 1 thorton slewis abarth dpranke eric jberlin mitz mrobinson simon.fraser slewis webkit-bug-importer webkit.review.bot zan oldest_to_newest 712606 0 thorton 2012-09-05 15:45:48 -0700 Some output from after my logging improvement yesterday (http://trac.webkit.org/changeset/127547): http://build.webkit.org/results/Apple%20MountainLion%20Debug%20WK2%20(Tests)/r127647%20(647)/fast/js/dfg-int32array-crash-log.txt > No crash log found for WebProcess:16532. > Process failed to become responsive before timing out. > > stdout: > Timed out waiting for final message from web process > > stderr: > #PROCESS UNRESPONSIVE - WebProcess (pid 16532) > #EOF http://build.webkit.org/results/Apple%20Lion%20Release%20WK2%20(Tests)/r127645%20(2940)/svg/animations/mozilla/animateMotion-by-1-crash-log.txt > No crash log found for WebProcess:33091. > Process failed to become responsive before timing out. > > stdout: > Failed to reset to consistent state before the first test > stderr: > #PROCESS UNRESPONSIVE - WebProcess (pid 33091) > #EOF There are dozens upon dozens of examples of this happening, on Lion and Mountain Lion, across many machines. It seems to come in bunches - 20 tests from different shards will fail nearly simultaneously, and the harness will then give up. I've never reproduced this locally, but it is one of the most significant issues blocking us from greenness on the Mac WK2 test bots, and has been for some time now. Questions I have: Is this the same (effectively) as a timeout for a test that doesn't call waitUntilDone (i.e. is this what you get if you have a non-waitUntilDone test, but then it hangs for some reason (infinite loop in JS, WebCore bug, whatever)?)? If so, why don't we treat it as a timeout instead of a crash? (not that that would really help) Could this be caused simply by slow/overloaded machines? (this is something I want to test locally) Why does it come in batches, across many processes? (see http://build.webkit.org/results/Apple%20Lion%20Release%20WK2%20(Tests)/r127645%20(2940)/results.html) 712633 1 slewis 2012-09-05 16:08:59 -0700 I took a blind stab at fixing this by doubling the timeout length. https://bugs.webkit.org/show_bug.cgi?id=87720. It didn't have any effect on the frequency, so I think this is more complicated than slow machines. I can reproduce this locally by running with a lot of parallel processes, more than my machine is supposed to support. Whether that is just reproducing the symptom instead of the root cause I don't know. One theory I had was that this was related to plugin process crashes which aren't caught by the harness. Another theory I have is that this is some kind of disk contention. It could be that resetting the state is holding a lock on a file and that when multiple processes try to access it they time out. Currently when WKTR notices the WebProcess didn't come back from this call it kills the WebProcess. If we wanted to treat this as a timeout WKTR would need to modified to not kill the WebProcess. If we did that restructuring we could get samples which might help diagnose the issue. 712643 2 thorton 2012-09-05 16:18:00 -0700 (In reply to comment #1) > I took a blind stab at fixing this by doubling the timeout length. https://bugs.webkit.org/show_bug.cgi?id=87720. It didn't have any effect on the frequency, so I think this is more complicated than slow machines. > > I can reproduce this locally by running with a lot of parallel processes, more than my machine is supposed to support. Whether that is just reproducing the symptom instead of the root cause I don't know. Ah, I'll have to try that. > One theory I had was that this was related to plugin process crashes which aren't caught by the harness. PluginProcess crashes shouldn't hang the WebProcess. We don't see that normally, do we? > Another theory I have is that this is some kind of disk contention. It could be that resetting the state is holding a lock on a file and that when multiple processes try to access it they time out. Most of the timeouts are not during the reset settings to consistent state bit, they're mostly "during the test" i.e. after the page load started, but before didFinishLoad. > Currently when WKTR notices the WebProcess didn't come back from this call it kills the WebProcess. If we wanted to treat this as a timeout WKTR would need to modified to not kill the WebProcess. If we did that restructuring we could get samples which might help diagnose the issue. It ... has to kill the WebProcess, the WebProcess is *not responding*. (at least if by "not kill", you mean "use it for subsequent tests"). But yes, if we could sample it first that would be nice. 712693 3 slewis 2012-09-05 17:13:57 -0700 (In reply to comment #2) > (In reply to comment #1) > > I took a blind stab at fixing this by doubling the timeout length. https://bugs.webkit.org/show_bug.cgi?id=87720. It didn't have any effect on the frequency, so I think this is more complicated than slow machines. > > > > I can reproduce this locally by running with a lot of parallel processes, more than my machine is supposed to support. Whether that is just reproducing the symptom instead of the root cause I don't know. > > Ah, I'll have to try that. > > > One theory I had was that this was related to plugin process crashes which aren't caught by the harness. > > PluginProcess crashes shouldn't hang the WebProcess. We don't see that normally, do we? No, that's the whole point of the plugin process. I just noticed that a few of the tests where this happened matched up to tests where the plugin process was crashing. > > > Another theory I have is that this is some kind of disk contention. It could be that resetting the state is holding a lock on a file and that when multiple processes try to access it they time out. > > Most of the timeouts are not during the reset settings to consistent state bit, they're mostly "during the test" i.e. after the page load started, but before didFinishLoad. OK, that's more information than I had. > > > Currently when WKTR notices the WebProcess didn't come back from this call it kills the WebProcess. If we wanted to treat this as a timeout WKTR would need to modified to not kill the WebProcess. If we did that restructuring we could get samples which might help diagnose the issue. > > It ... has to kill the WebProcess, the WebProcess is *not responding*. (at least if by "not kill", you mean "use it for subsequent tests"). But yes, if we could sample it first that would be nice. Yes, I meant to keep it alive long enough to sample, because currently by the time we get the message the WebProcess has already been killed. The next step I was going to take was to try to get samples. My next bot watching cycle comes around next week so hopefully I'll be able to take a look at it then. 713516 4 thorton 2012-09-06 11:42:10 -0700 I just sampled, it looks like the bots might be getting stuck doing I/O? So maybe you weren't far off with your disk contention theory! 713625 5 webkit-bug-importer 2012-09-06 13:25:57 -0700 <rdar://problem/12248711> 713631 6 slewis 2012-09-06 13:30:06 -0700 *** Bug 87720 has been marked as a duplicate of this bug. *** 714136 7 thorton 2012-09-06 22:50:23 -0700 News: 1. Yes, the bots are extremely strained for resources (I/O mostly), though it's not totally clear why. Having a billion WebProcesses, WebKitTestRunners, and Python processes probably isn't helping, though. 2. r110382 made WKTR no longer wait for LongTimeout while waiting for the final message from the WebProcess (except in the case of waitUntilDone tests). This probably made the situation a lot worse. I need to understand why this change was made, and what we can do to reintroduce a timeout there without breaking run-perf-tests. 714146 8 thorton 2012-09-06 22:57:12 -0700 (In reply to comment #7) > News: > > 1. Yes, the bots are extremely strained for resources (I/O mostly), though it's not totally clear why. Having a billion WebProcesses, WebKitTestRunners, and Python processes probably isn't helping, though. > > 2. r110382 made WKTR no longer wait for LongTimeout while waiting for the final message from the WebProcess (except in the case of waitUntilDone tests). This probably made the situation a lot worse. I need to understand why this change was made, and what we can do to reintroduce a timeout there without breaking run-perf-tests. Oh, hey. The intent of "NoTimeout" is that we wait forever, not that we wait for no time. However, there's a FIXME that notes that this isn't implemented on Mac, and is instead causing it to wait for no time (because the timeout is -1). I'm not sure we want to wait forever (what if the WebProcess gets stuck forever!?), but this also explains what they were going for. 714148 9 thorton 2012-09-06 22:58:28 -0700 (In reply to comment #8) Oh no! I misunderstood m_useWaitToDumpWatchdogTimer, I don't think this is relevant at all :( 716205 10 thorton 2012-09-10 10:07:16 -0700 In an attempt to work around this, Stephanie doubled the shortTimeout duration (https://bugs.webkit.org/show_bug.cgi?id=87720). However, the failures we are seeing are generally against longTimeout runUntils. I'm going to post a patch to double longTimeout (to 60 seconds); we should see what happens with that. 716210 11 thorton 2012-09-10 10:10:02 -0700 (In reply to comment #10) > In an attempt to work around this, Stephanie doubled the shortTimeout duration (https://bugs.webkit.org/show_bug.cgi?id=87720). However, the failures we are seeing are generally against longTimeout runUntils. I'm going to post a patch to double longTimeout (to 60 seconds); we should see what happens with that. Ugh, can't double it because of: // defaultLongTimeout + defaultShortTimeout should be less than 50, // the default timeout value of the test harness so we can detect an // unresponsive web process. (unless we also crank the harness timeout... or maybe it was already increased? I'll check.) 716329 12 163174 thorton 2012-09-10 11:46:27 -0700 Created attachment 163174 increase all the WKTR timeouts This is a total shot in the dark (and we should look at fixing the hardware), but it would be interesting to see what happens. 716498 13 thorton 2012-09-10 14:10:30 -0700 Rate of timeouts decreased but they're still happening. 716627 14 dpranke 2012-09-10 15:55:54 -0700 I think it would probably be better to reduce the number of children you're running instead of increasing the timeout. In my experience, the slowdown from thrashing and timeout is usually much worse than the slowdown from decreased parallelism. At any rate, I think it's probably at least worth trying fewer children to get a data point. FWIW, I will also see if I can reproduce this locally, since I have a mac pro w/ a regular hard drive ... 718866 15 163731 slewis 2012-09-12 16:31:40 -0700 Created attachment 163731 reduces the number wk2 testers running 718967 16 slewis 2012-09-12 18:34:19 -0700 Committed http://trac.webkit.org/projects/webkit/changeset/128391 718982 17 slewis 2012-09-12 19:07:15 -0700 16 -> 12 didn't cut it. I'm going to wait and get a couple more data points. If the bottleneck is the disk then the number of acceptable processes is probably not a function of the number of available cores. http://build.webkit.org/results/Apple%20Lion%20Release%20WK2%20(Tests)/r128393%20(3199)/results.html 800034 18 eric 2013-01-04 00:53:01 -0800 Attachment 163731 was posted by a committer and has review+, assigning to Stephanie Lewis for commit. 800605 19 slewis 2013-01-04 14:06:00 -0800 All patches submitted, issue still occurs 845286 20 163174 eric 2013-03-01 10:38:18 -0800 Comment on attachment 163174 increase all the WKTR timeouts Cleared Dan Bernstein's review+ from obsolete attachment 163174 so that this bug does not appear in http://webkit.org/pending-commit. 845287 21 163731 eric 2013-03-01 10:38:21 -0800 Comment on attachment 163731 reduces the number wk2 testers running Cleared Dirk Pranke's review+ from obsolete attachment 163731 so that this bug does not appear in http://webkit.org/pending-commit. 163174 2012-09-10 11:46:27 -0700 2013-03-01 10:38:18 -0800 increase all the WKTR timeouts timeouts.diff text/plain 2279 thorton ZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBiL1Rvb2xzL0NoYW5nZUxvZwppbmRleCA1YTY2 MDQxLi5mYWE1YjhkIDEwMDY0NAotLS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hh bmdlTG9nCkBAIC0xLDMgKzEsMTggQEAKKzIwMTItMDktMTAgIFRpbSBIb3J0b24gIDx0aW1vdGh5 X2hvcnRvbkBhcHBsZS5jb20+CisKKyAgICAgICAgV0tUUiBvZnRlbiByZXBvcnRzIGFuIHVucmVz cG9uc2l2ZSBXZWJQcm9jZXNzIG9uIE1hYyBib3RzCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJr aXQub3JnL3Nob3dfYnVnLmNnaT9pZD05NTkwNgorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9E WSAoT09QUyEpLgorCisgICAgICAgIEF0dGVtcHQgdG8gd29yayBhcm91bmQgcmVzb3VyY2UgY29u dGVudGlvbiBpc3N1ZXMgYnkgaW5jcmVhc2luZyB0aGUgTlJXVAorICAgICAgICB0aW1lb3V0IHdo ZW4gcnVubmluZyBhZ2FpbnN0IFdLVFIsIGFzIHdlbGwgYXMgV0tUUidzIG93biBXZWJQcm9jZXNz IHRpbWVvdXQuCisKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0 L2Jhc2UucHk6CisgICAgICAgIChQb3J0LmRlZmF1bHRfdGltZW91dF9tcyk6IERlZmF1bHQgdGlt ZW91dCB3aGVuIHVzaW5nIFdLVFIgLT4gODAgc2Vjb25kcy4KKyAgICAgICAgKiBXZWJLaXRUZXN0 UnVubmVyL1Rlc3RDb250cm9sbGVyLmNwcDoKKyAgICAgICAgKFdUUik6IExvbmdUaW1lb3V0ICh1 c2VkIHdoZW4gd2FpdGluZyBmb3IgV2ViUHJvY2VzcyB0byBmaW5pc2gpIC0+IDYwIHNlY29uZHMu CisKIDIwMTItMDktMTAgIFRvbW15IFdpZGVuZmx5Y2h0ICA8dG9tbXl3QGdvb2dsZS5jb20+CiAK ICAgICAgICAgW2Nocm9taXVtXSBNZWRpYVN0cmVhbSBBUEk6IFJlbW92ZSB0aGUgRGVzY3JpcHRv ciBwb3N0Zml4CmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0 cy9wb3J0L2Jhc2UucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0 L2Jhc2UucHkKaW5kZXggODY4NjQzNy4uODk5N2E1ZiAxMDA3NTUKLS0tIGEvVG9vbHMvU2NyaXB0 cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMv d2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvYmFzZS5weQpAQCAtMTU2LDcgKzE1Niw3IEBAIGNs YXNzIFBvcnQob2JqZWN0KToKICAgICAgICAgaWYgc2VsZi5nZXRfb3B0aW9uKCd3ZWJraXRfdGVz dF9ydW5uZXInKToKICAgICAgICAgICAgICMgQWRkIHNvbWUgbW9yZSB0aW1lIHRvIFdlYktpdFRl c3RSdW5uZXIgYmVjYXVzZSBpdCBuZWVkcyB0byBzeW5jcm9uaXNlIHRoZSBzdGF0ZQogICAgICAg ICAgICAgIyB3aXRoIHRoZSB3ZWIgcHJvY2VzcyBhbmQgd2Ugd2FudCB0byBkZXRlY3QgaWYgdGhl cmUgaXMgYSBwcm9ibGVtIHdpdGggdGhhdCBpbiB0aGUgZHJpdmVyLgotICAgICAgICAgICAgcmV0 dXJuIDUwICogMTAwMAorICAgICAgICAgICAgcmV0dXJuIDgwICogMTAwMAogICAgICAgICByZXR1 cm4gMzUgKiAxMDAwCiAKICAgICBkZWYgZHJpdmVyX3N0b3BfdGltZW91dChzZWxmKToKZGlmZiAt LWdpdCBhL1Rvb2xzL1dlYktpdFRlc3RSdW5uZXIvVGVzdENvbnRyb2xsZXIuY3BwIGIvVG9vbHMv V2ViS2l0VGVzdFJ1bm5lci9UZXN0Q29udHJvbGxlci5jcHAKaW5kZXggYWY4ZjcwMi4uMDE3OTUy MiAxMDA2NDQKLS0tIGEvVG9vbHMvV2ViS2l0VGVzdFJ1bm5lci9UZXN0Q29udHJvbGxlci5jcHAK KysrIGIvVG9vbHMvV2ViS2l0VGVzdFJ1bm5lci9UZXN0Q29udHJvbGxlci5jcHAKQEAgLTU1LDEw ICs1NSwxMCBAQAogCiBuYW1lc3BhY2UgV1RSIHsKIAotLy8gZGVmYXVsdExvbmdUaW1lb3V0ICsg ZGVmYXVsdFNob3J0VGltZW91dCBzaG91bGQgYmUgbGVzcyB0aGFuIDUwLAorLy8gZGVmYXVsdExv bmdUaW1lb3V0ICsgZGVmYXVsdFNob3J0VGltZW91dCBzaG91bGQgYmUgbGVzcyB0aGFuIDgwLAog Ly8gdGhlIGRlZmF1bHQgdGltZW91dCB2YWx1ZSBvZiB0aGUgdGVzdCBoYXJuZXNzIHNvIHdlIGNh biBkZXRlY3QgYW4KIC8vIHVucmVzcG9uc2l2ZSB3ZWIgcHJvY2Vzcy4KLXN0YXRpYyBjb25zdCBk b3VibGUgZGVmYXVsdExvbmdUaW1lb3V0ID0gMzA7CitzdGF0aWMgY29uc3QgZG91YmxlIGRlZmF1 bHRMb25nVGltZW91dCA9IDYwOwogc3RhdGljIGNvbnN0IGRvdWJsZSBkZWZhdWx0U2hvcnRUaW1l b3V0ID0gMTU7CiBzdGF0aWMgY29uc3QgZG91YmxlIGRlZmF1bHROb1RpbWVvdXQgPSAtMTsKIAo= 163731 2012-09-12 16:31:40 -0700 2013-03-01 10:38:21 -0800 reduces the number wk2 testers running 0001-Reduce-parallism-on-the-wk2-testers.patch text/plain 2667 slewis RnJvbSA5N2QzNjU1ZTEzMzY4YTRkMzhjOWYxYjQzMTE0MTExZjA3YTVjZGRhIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBTdGVwaGFuaWUgTGV3aXMgPHNsZXdpc0BhcHBsZS5jb20+CkRh dGU6IFdlZCwgMTIgU2VwIDIwMTIgMTY6Mjk6NDkgLTA3MDAKU3ViamVjdDogW1BBVENIXSBSZWR1 Y2UgcGFyYWxsaXNtIG9uIHRoZSB3azIgdGVzdGVycy4KIGh0dHBzOi8vYnVncy53ZWJraXQub3Jn L3Nob3dfYnVnLmNnaT9pZD05NTkwNgoKUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCgpUaGUg d2syIHRlc3RlcnMgb24gTW91bnRhaW4gTGlvbiBhcmUgZ2V0dGluZyBzdHVjayBhbmQgdGltaW5n IG91dC4gIFRoZSBwcm9ibGVtIGFwcGVhcnMgdG8gYmUgZHVlIHRvCnJlc291cmNlIGNvbnRlbnRp b24uICBSZWR1c2luZyB0aGUgbnVtYmVyIG9mIHByb2Nlc3NlcyBhbGxldmlhdGVzIHRoZSBpc3N1 ZS4KClN0YXJ0aW5nIGJ5IHJlZHVjaW5nIDI1JS4KCiogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRf dGVzdHMvcG9ydC9tYWMucHk6CihNYWNQb3J0LmRlZmF1bHRfY2hpbGRfcHJvY2Vzc2VzKToKLS0t CiBUb29scy9DaGFuZ2VMb2cgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgMTUg KysrKysrKysrKysrKysrCiBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0 L21hYy5weSB8ICAgIDYgKysrKysrCiAyIGZpbGVzIGNoYW5nZWQsIDIxIGluc2VydGlvbnMoKykK CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cgYi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggOTIy YmFlZC4uOGQxOWZjMyAxMDA2NDQKLS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0No YW5nZUxvZwpAQCAtMSwzICsxLDE4IEBACisyMDEyLTA5LTEyICBTdGVwaGFuaWUgTGV3aXMgIDxz bGV3aXNAYXBwbGUuY29tPgorCisgICAgICAgIFJlZHVjZSBwYXJhbGxpc20gb24gdGhlIHdrMiB0 ZXN0ZXJzLgorICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9 OTU5MDYKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBU aGUgd2syIHRlc3RlcnMgb24gTW91bnRhaW4gTGlvbiBhcmUgZ2V0dGluZyBzdHVjayBhbmQgdGlt aW5nIG91dC4gIFRoZSBwcm9ibGVtIGFwcGVhcnMgdG8gYmUgZHVlIHRvCisgICAgICAgIHJlc291 cmNlIGNvbnRlbnRpb24uICBSZWR1c2luZyB0aGUgbnVtYmVyIG9mIHByb2Nlc3NlcyBhbGxldmlh dGVzIHRoZSBpc3N1ZS4gIAorCisgICAgICAgIFN0YXJ0aW5nIGJ5IHJlZHVjaW5nIDI1JS4KKwor ICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvbWFjLnB5OgorICAg ICAgICAoTWFjUG9ydC5kZWZhdWx0X2NoaWxkX3Byb2Nlc3Nlcyk6CisKIDIwMTItMDktMTIgIEtl bm5ldGggUm9oZGUgQ2hyaXN0aWFuc2VuICA8a2VubmV0aEB3ZWJraXQub3JnPgogCiAgICAgICAg IFJlc3BlY3QgV0VCS0lUT1VUUFVURElSIHdoZW4gcnVubmluZyBFRkwgdGVzdHMKZGlmZiAtLWdp dCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvbWFjLnB5IGIvVG9v bHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9tYWMucHkKaW5kZXggNDAwOTU5 Yy4uYmRmNTllMSAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvcG9ydC9tYWMucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMv cG9ydC9tYWMucHkKQEAgLTExNSw2ICsxMTUsMTIgQEAgY2xhc3MgTWFjUG9ydChBcHBsZVBvcnQp OgogCiAgICAgICAgIGRlZmF1bHRfY291bnQgPSBzdXBlcihNYWNQb3J0LCBzZWxmKS5kZWZhdWx0 X2NoaWxkX3Byb2Nlc3NlcygpCiAKKyAgICAgICAgIyBGSVhNRTogaHR0cHM6Ly9idWdzLndlYmtp dC5vcmcvc2hvd19idWcuY2dpP2lkPTk1OTA2ICBXaXRoIHRvbyBtYW55IFdlYlByb2Nlc3MgV0sy IHRlc3RzIGdldCBzdHVjayBpbiByZXNvdXJjZSBjb250ZW50aW9uLgorICAgICAgICAjIFRvIGFs bGV2aWF0ZSB0aGUgaXNzdWUgcmVkdWNlIHRoZSBudW1iZXIgb2YgcnVubmluZyBwcm9jZXNzZXMK KyAgICAgICAgIyBBbmVjZG90YWwgZXZpZGVuY2Ugc3VnZ2VzdHMgdGhhdCBhIDQgY29yZS84IGNv cmUgbG9naWNhbCBtYWNoaW5lIG1heSBydW4gaW50byB0aGlzLCBidXQgdGhhdCBhIDIgY29yZS80 IGNvcmUgbG9naWNhbCBtYWNoaW5lIGRvZXMgbm90LgorICAgICAgICBpZiBzZWxmLl92ZXJzaW9u ID09ICJtb3VudGFpbmxpb24iIGFuZCBzZWxmLmdldF9vcHRpb24oJ3dlYmtpdF90ZXN0X3J1bm5l cicpIGFuZCBkZWZhdWx0X2NvdW50ID4gNDoKKyAgICAgICAgICAgIGRlZmF1bHRfY291bnQgPSBp bnQoLjc1ICogZGVmYXVsdF9jb3VudCkKKwogICAgICAgICAjIE1ha2Ugc3VyZSB3ZSBoYXZlIGVu b3VnaCByYW0gdG8gc3VwcG9ydCB0aGF0IG1hbnkgaW5zdGFuY2VzOgogICAgICAgICB0b3RhbF9t ZW1vcnkgPSBzZWxmLmhvc3QucGxhdGZvcm0udG90YWxfYnl0ZXNfbWVtb3J5KCkKICAgICAgICAg Ynl0ZXNfcGVyX2RydCA9IDI1NiAqIDEwMjQgKiAxMDI0ICAjIEFzc3VtZSBlYWNoIERSVCBuZWVk cyAyNTZNQiB0byBydW4uCi0tIAoxLjcuMTAuMiAoQXBwbGUgR2l0LTMzKQoK