83076 2012-04-03 14:49:33 -0700 [Chromium] Lots of timeouts causing Mac10.6 to exit early. 2012-05-18 15:10:43 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED WORKSFORME P2 Normal --- 1 ojan dpranke abarth apavlov dglazkov dpranke hans hclam rniwa tony webkit.review.bot zmo oldest_to_newest 594919 0 ojan 2012-04-03 14:49:33 -0700 http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6/builds/14439/steps/webkit_tests/logs/stdio Exiting early after 0 crashes and 20 timeouts. 21784 tests run. Regressions: Unexpected tests timed out : (20) animations/cross-fade-background-image.html = TIMEOUT compositing/geometry/empty-embed-rects.html = TIMEOUT compositing/self-painting-layers.html = TIMEOUT compositing/transitions/scale-transition-no-start.html = TIMEOUT css1/basic/class_as_selector.html = TIMEOUT css1/box_properties/acid_test.html = TIMEOUT css1/cascade/cascade_order.html = TIMEOUT css1/classification/display.html = TIMEOUT css1/color_and_background/background.html = TIMEOUT css1/conformance/forward_compatible_parsing.html = TIMEOUT css1/font_properties/font.html = TIMEOUT css1/pseudo/anchor.html = TIMEOUT fast/forms/search-rtl.html = TIMEOUT fast/images/embed-does-not-propagate-dimensions-to-object-ancestor.html = TIMEOUT fast/loader/local-CSS-from-local.html = TIMEOUT fast/table/invisible-cell-background.html = TIMEOUT fast/text/international/plane2.html = TIMEOUT fast/text/justify-ideograph-complex.html = TIMEOUT fast/workers/storage/interrupt-database.html = TIMEOUT http/tests/appcache/remove-cache.html = TIMEOUT 594933 1 dpranke 2012-04-03 15:01:34 -0700 I'm on this one ... 595099 2 135477 dpranke 2012-04-03 18:21:47 -0700 Created attachment 135477 Patch 595100 3 135477 eric 2012-04-03 18:23:43 -0700 Comment on attachment 135477 Patch Bleh. Why not just up the amount of ram we expect child processes to take? That seems like a less gross hack. 595105 4 135478 dpranke 2012-04-03 18:29:52 -0700 Created attachment 135478 add Changelog, port same logic to apple mac 595108 5 dpranke 2012-04-03 18:32:54 -0700 (In reply to comment #3) > (From update of attachment 135477 [details]) > Bleh. Why not just up the amount of ram we expect child processes to take? That seems like a less gross hack. At the moment, at least on the Chromium SL bot, it doesn't look ram-related. It looks like we're thrashing on something else, but have plenty of RAM free. 595109 6 dpranke 2012-04-03 18:35:01 -0700 I'm going to land this as-is, so that I can get the bot back online and we can get more data. Unfortunately, it's been flaky and aborting early for so long that I can't easily reproduce things or figure debug it (I've tried rolling back builds on that bot and run into a sordid list of issues that is stopping me that I need to work through in parallel). 595110 7 dpranke 2012-04-03 18:35:24 -0700 I'll be happy to roll this out if there are other issues or if we really think this is the wrong thing to do. 595117 8 dpranke 2012-04-03 18:40:54 -0700 Committed r113122: <http://trac.webkit.org/changeset/113122> 595118 9 dpranke 2012-04-03 18:41:27 -0700 re-opening, I don't consider this fixed yet. 615858 10 dpranke 2012-05-03 18:21:12 -0700 Note that we're seeing this quite a bit lately, even after the patch (see, e.g., http://build.chromium.org/p/chromium.webkit/waterfall?builder=Webkit%20Mac10.6&last_time=1336069767 ) ... it's possible that r115490 has made things worse, but I don't know what else might be contributing. 616486 11 dpranke 2012-05-04 11:58:19 -0700 It seems like we're frequently seeing many of the same tests timing out this week, so I'm going to start marking them as flaky timeouts here and we'll see if this contains the problem, or if we're seeing systemic flakiness. Here's the first batch: compositing/geometry/outline-change.html css3/selectors3/xml/css3-modsel-161.xml css3/selectors3/xml/css3-modsel-166.xml css3/selectors3/xml/css3-modsel-166a.xml editing/deleting/delete-3857753-fix.html editing/deleting/delete-3865854-fix.html editing/deleting/delete-3928305-fix.html editing/execCommand/4747450.html editing/execCommand/4786404-1.html editing/execCommand/4786404-2.html editing/execCommand/4916235.html editing/input/caret-at-the-edge-of-input.html editing/execCommand/format-block-with-trailing-br.html editing/execCommand/format-block-without-body-crash.html editing/execCommand/format-block.html editing/execCommand/forward-delete-no-scroll.html editing/execCommand/hilitecolor.html editing/input/emacs-ctrl-o.html editing/input/div-first-child-rule-input.html editing/input/div-first-child-rule-textarea.html editing/input/ime-composition-clearpreedit.html editing/input/insert-wrapping-space-in-textarea.html editing/input/option-page-up-down.html editing/input/page-up-down-scrolls.html editing/inserting/12882.html editing/inserting/4278698.html http/tests/history/back-with-fragment-change.php http/tests/history/cross-origin-replace-history-object.html http/tests/history/history-navigations-set-referrer.html http/tests/history/popstate-fires-with-pending-requests.html http/tests/history/redirect-200-refresh-0-seconds.pl http/tests/history/redirect-200-refresh-2-seconds.pl http/tests/history/redirect-301.html 616487 12 dpranke 2012-05-04 12:00:12 -0700 rniwa - it looks like maybe these editing tests started being flaky earlier this week. Can you take a look? 616493 13 rniwa 2012-05-04 12:05:41 -0700 Are you sure they're really timing out? Aren't they just slow? I don't see any changes that can cause things to timeout: http://trac.webkit.org/log/trunk/Source/WebCore/editing 616500 14 dpranke 2012-05-04 12:11:09 -0700 (In reply to comment #13) > Are you sure they're really timing out? Aren't they just slow? Well, by definition they're timing out, but it could be because they're slow and should just be marked as slow :). If you think we should try marking them as slow instead that's fine. > > I don't see any changes that can cause things to timeout: > http://trac.webkit.org/log/trunk/Source/WebCore/editing Yeah, I didn't either, but I don't tend to like to mark tests as slow unless I'm familiar with them and would expect them to take a while to run. 616501 15 rniwa 2012-05-04 12:12:18 -0700 (In reply to comment #14) > (In reply to comment #13) > > Are you sure they're really timing out? Aren't they just slow? > > Well, by definition they're timing out, but it could be because they're slow and should just be marked as slow :). If you think we should try marking them as slow instead that's fine. I don't mind marking the entire "editing" directory as "slow" for that matter. Many of editing tests are integration tests and take a long time to run. 616505 16 dpranke 2012-05-04 12:14:29 -0700 (In reply to comment #15) > (In reply to comment #14) > > (In reply to comment #13) > > > Are you sure they're really timing out? Aren't they just slow? > > > > Well, by definition they're timing out, but it could be because they're slow and should just be marked as slow :). If you think we should try marking them as slow instead that's fine. > > I don't mind marking the entire "editing" directory as "slow" for that matter. Many of editing tests are integration tests and take a long time to run. Okay, I'll update the expectations for editing tests. Thanks! 616537 17 dpranke 2012-05-04 12:45:11 -0700 Here's some more ... I'm not filled with confidence in this approach: fast/workers/storage/multiple-databases-garbage-collection.html = TIMEOUT fast/workers/storage/multiple-transactions-on-different-handles-sync.html = TIMEOUT http/tests/history/redirect-302.html = TIMEOUT http/tests/history/redirect-303.html = TIMEOUT http/tests/misc/object-embedding-svg-delayed-size-negotiation.xhtml = TIMEOUT platform/chromium/virtual/gpu/canvas/philip/tests/2d.text-custom-font-load-crash.html = TIMEOUT platform/chromium/virtual/gpu/fast/canvas/2d.text.draw.fill.maxWidth.gradient.html = TIMEOUT 616547 18 rniwa 2012-05-04 12:53:54 -0700 Maybe something in webkitpy is affecting the timing? 616552 19 dpranke 2012-05-04 12:59:45 -0700 (In reply to comment #18) > Maybe something in webkitpy is affecting the timing? It's possible, but I don't know what it would be. I will probably let this approach go for the afternoon or so to get more data on the flakiness, and if it doesn't clear up I will try going back to --test-shell mode. As I've noted elsewhere, one aspect of using DRT mode is that NRWT itself enforces the timeout and kills DRT when the test times out; maybe this is leaving something in an unhappy state w/ the O/S, or we're leaving things locked somewhere, and that's causing things to go downhill. 616585 20 tony 2012-05-04 13:32:25 -0700 I feel like we're playing whack-a-mole and even if we find the culprit, tests we mark as timeout/slow now will be forgotten. I would feel better about reverting changes until the bots improve. Once the bots improve, we can reland patches (maybe with speculative fixes) to isolate the cause. I.e., I would handle unknown flakiness the same way we handle perf regressions. 616588 21 dpranke 2012-05-04 13:35:58 -0700 (In reply to comment #20) > I feel like we're playing whack-a-mole and even if we find the culprit, tests we mark as timeout/slow now will be forgotten. > This is a valid concern. > I would feel better about reverting changes until the bots improve. Once the bots improve, we can reland patches (maybe with speculative fixes) to isolate the cause. I.e., I would handle unknown flakiness the same way we handle perf regressions. Apart from the one python change -- which I'm already planning to revert to see if it help -- any suggestions for what other changes to revert? 616600 22 dpranke 2012-05-04 13:46:12 -0700 Okay, I've switched back to "test shell" mode on SL in http://trac.webkit.org/changeset/116161 . Let's see what happens now. 616648 23 tony 2012-05-04 14:17:47 -0700 Looking at the waterfall, it looks like the set of failing tests isn't at all consistent. I doubt adding suppressions will green the tree. Here's the first set of timeouts I see. It's from the beginning of Wednesday. http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6/builds/15522/steps/webkit_tests/logs/stdio But zmo said the flakiness started earlier, maybe last Friday? Here are NRWT changes that touch NRWT code around that time: 115377 115452 115490 115729? None of the changes look that suspect, but I don't know of any other way to determine the cause of the regression. 616667 24 dpranke 2012-05-04 14:28:18 -0700 When I(In reply to comment #23) > Looking at the waterfall, it looks like the set of failing tests isn't at all consistent. I doubt adding suppressions will green the tree. > > Here's the first set of timeouts I see. It's from the beginning of Wednesday. > http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6/builds/15522/steps/webkit_tests/logs/stdio > There are definitely timeouts earlier, e.g.: http://build.chromium.org/p/chromium.webkit/waterfall?last_time=1335833009&show=Webkit%20Mac10.6 > But zmo said the flakiness started earlier, maybe last Friday? Here are NRWT changes that touch NRWT code around that time: > > 115377 > 115452 > 115490 > 115729? > > None of the changes look that suspect, but I don't know of any other way to determine the cause of the regression. Well, 115490 is definitely suspicious (and already disabled, so now we're just waiting). You can see a marked uptick in flakiness in the first build after that changes: http://build.chromium.org/p/chromium.webkit/waterfall?force=true&last_time=1335569069&show=Webkit%20Mac10.6 (see build 15326, in particular). 628135 25 dpranke 2012-05-18 15:10:43 -0700 closing this as WORKSFORME (the status is debatable; it probably could be WONTFIX or FIXED as well). For whatever reason, our old Xserves appear to be flaky in the release build. Since we haven't seen this issue anywhere else, and we're migrated off of the Xserves, we're gonna ignore this. 135477 2012-04-03 18:21:47 -0700 2012-04-03 18:29:48 -0700 Patch bug-83076-20120403182146.patch text/plain 1748 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogMTEzMDg3CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggOTM4NTQzZWEwZGJmMjkwNzA4ZTAyNGJmMjgyNWY3OTI4 N2RmN2NjOS4uODE2ODU0MThmZmQwM2ZkYjQzNWRjNDE5ZGNiNWUwMTIzOWJmMDIwNyAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDEz IEBACisyMDEyLTA0LTAzICBEaXJrIFByYW5rZSAgPGRwcmFua2VAY2hyb21pdW0ub3JnPgorCisg ICAgICAgIFtDaHJvbWl1bV0gTG90cyBvZiB0aW1lb3V0cyBjYXVzaW5nIE1hYzEwLjYgdG8gZXhp dCBlYXJseS4KKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lk PTgzMDc2CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAg KiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Nocm9taXVtX21hYy5weToKKyAg ICAgICAgKENocm9taXVtTWFjUG9ydC5kZWZhdWx0X2NoaWxkX3Byb2Nlc3Nlcyk6CisKIDIwMTIt MDQtMDMgIE9qYW4gVmFmYWkgIDxvamFuQGNocm9taXVtLm9yZz4KIAogICAgICAgICBHaXZlIG1v cmUgaHVtYW4gZnJpZW5kbHkgZXJyb3IgbWVzc2FnZSB3aGVuIGJ1aWxkZXJzIGZhaWwgdG8gbG9h ZCBvciBoYXZlIHN0YWxlIGRhdGEuCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5 L2xheW91dF90ZXN0cy9wb3J0L2Nocm9taXVtX21hYy5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0 cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21pdW1fbWFjLnB5CmluZGV4IDMwZjRhYTk5MTRlNWNh OGE2NmMxNTc2MzVmNjIwZDAwNmIyOGFlY2YuLjNhN2M0YTAyZDFhMDM4YjZiYjhjZTVmMzllZGE3 OTcwZDcyYWI4MjIgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rl c3RzL3BvcnQvY2hyb21pdW1fbWFjLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5 b3V0X3Rlc3RzL3BvcnQvY2hyb21pdW1fbWFjLnB5CkBAIC0xMDUsNiArMTA1LDE2IEBAIGNsYXNz IENocm9taXVtTWFjUG9ydChjaHJvbWl1bS5DaHJvbWl1bVBvcnQpOgogICAgIGRlZiBvcGVyYXRp bmdfc3lzdGVtKHNlbGYpOgogICAgICAgICByZXR1cm4gJ21hYycKIAorICAgIGRlZiBkZWZhdWx0 X2NoaWxkX3Byb2Nlc3NlcyhzZWxmKToKKyAgICAgICAgIyBGSVhNRTogQXMgYSB0ZW1wb3Jhcnkg d29ya2Fyb3VuZCB3aGlsZSB3ZSBmaWd1cmUgb3V0IHdoYXQncyBnb2luZworICAgICAgICAjIG9u IHdpdGggaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTgzMDc2LCByZWR1 Y2UgYnkKKyAgICAgICAgIyBoYWxmIHRoZSAjIG9mIHdvcmtlcnMgd2UgcnVuIGJ5IGRlZmF1bHQg b24gYmlnZ2VyIG1hY2hpbmVzLgorICAgICAgICBkZWZhdWx0X2NvdW50ID0gc3VwZXIoQ2hyb21p dW1NYWNQb3J0LCBzZWxmKS5kZWZhdWx0X2NoaWxkX3Byb2Nlc3NlcygpCisgICAgICAgIGlmIGRl ZmF1bHRfY291bnQgPj0gODoKKyAgICAgICAgICAgIGNwdV9jb3VudCA9IHNlbGYuX2V4ZWN1dGl2 ZS5jcHVfY291bnQoKQorICAgICAgICAgICAgcmV0dXJuIG1heCgxLCBtaW4oZGVmYXVsdF9jb3Vu dCwgaW50KGNwdV9jb3VudCAvIDIpKSkKKyAgICAgICAgcmV0dXJuIGRlZmF1bHRfY291bnQKKwog ICAgICMKICAgICAjIFBST1RFQ1RFRCBNRVRIT0RTCiAgICAgIwo= 135478 2012-04-03 18:29:52 -0700 2012-04-03 18:41:35 -0700 add Changelog, port same logic to apple mac bug-83076-20120403182951.patch text/plain 3492 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogMTEzMDg3CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggOTM4NTQzZWEwZGJmMjkwNzA4ZTAyNGJmMjgyNWY3OTI4 N2RmN2NjOS4uNmQzOWM2YTE0MDlkMjExZjg3MGVlNDZkNjczNmJhMjRjYjc0YmZiYSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDI1 IEBACisyMDEyLTA0LTAzICBEaXJrIFByYW5rZSAgPGRwcmFua2VAY2hyb21pdW0ub3JnPgorCisg ICAgICAgIFtDaHJvbWl1bV0gTG90cyBvZiB0aW1lb3V0cyBjYXVzaW5nIE1hYzEwLjYgdG8gZXhp dCBlYXJseS4KKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lk PTgzMDc2CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAg QWRkIGxvZ2ljIHRvIHRoZSBhcHBsZSBtYWMgYW5kIGNocm9taXVtIG1hYyBjb2RlIHRvIG5vdCB1 c2UKKyAgICAgICAgdG9vIG1hbnkgd29ya2VyczsgaXQgbG9va3MgbGlrZSB0aGUgeHNlcnZlcyAo YW5kIHBvc3NpYmx5IG1hYworICAgICAgICBwcm9zKSBjb3VudCBoeXBlcnRocmVhZGVkIGNvcmVz IHdoZW4gdGhleSByZWFsbHkgc2hvdWxkbid0IGFuZAorICAgICAgICB3ZSBlbmQgdXAgdXNpbmcg dG9vIG1hbnkgd29ya2VycyBhdCBhIHRpbWU7IHRoaXMgbGVhZHMgdG8gdGVzdHMKKyAgICAgICAg dGhyYXNoaW5nIGFuZCB0aW1pbmcgb3V0LgorCisgICAgICAgIFRoaXMgY2hhbmdlIGlzIGEgdGVt cG9yYXJ5IGZpeCB0byBtYWtlIHRoZSBib3RzIGhhcHB5IHdoaWxlIEkKKyAgICAgICAgbG9vayBp bnRvIG1vcmUgcHJvZmlsaW5nIGFuZCBvdGhlciBmaXhlcy4KKworICAgICAgICAqIFNjcmlwdHMv d2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvbWFjLnB5OgorICAgICAgICAoQ2hyb21pdW1NYWNQ b3J0LmRlZmF1bHRfY2hpbGRfcHJvY2Vzc2VzKToKKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0 cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21pdW1fbWFjLnB5OgorICAgICAgICAoQ2hyb21pdW1N YWNQb3J0LmRlZmF1bHRfY2hpbGRfcHJvY2Vzc2VzKToKKwogMjAxMi0wNC0wMyAgT2phbiBWYWZh aSAgPG9qYW5AY2hyb21pdW0ub3JnPgogCiAgICAgICAgIEdpdmUgbW9yZSBodW1hbiBmcmllbmRs eSBlcnJvciBtZXNzYWdlIHdoZW4gYnVpbGRlcnMgZmFpbCB0byBsb2FkIG9yIGhhdmUgc3RhbGUg ZGF0YS4KZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3Bv cnQvY2hyb21pdW1fbWFjLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMv cG9ydC9jaHJvbWl1bV9tYWMucHkKaW5kZXggMzBmNGFhOTkxNGU1Y2E4YTY2YzE1NzYzNWY2MjBk MDA2YjI4YWVjZi4uM2E3YzRhMDJkMWEwMzhiNmJiOGNlNWYzOWVkYTc5NzBkNzJhYjgyMiAxMDA2 NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1 bV9tYWMucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9j aHJvbWl1bV9tYWMucHkKQEAgLTEwNSw2ICsxMDUsMTYgQEAgY2xhc3MgQ2hyb21pdW1NYWNQb3J0 KGNocm9taXVtLkNocm9taXVtUG9ydCk6CiAgICAgZGVmIG9wZXJhdGluZ19zeXN0ZW0oc2VsZik6 CiAgICAgICAgIHJldHVybiAnbWFjJwogCisgICAgZGVmIGRlZmF1bHRfY2hpbGRfcHJvY2Vzc2Vz KHNlbGYpOgorICAgICAgICAjIEZJWE1FOiBBcyBhIHRlbXBvcmFyeSB3b3JrYXJvdW5kIHdoaWxl IHdlIGZpZ3VyZSBvdXQgd2hhdCdzIGdvaW5nCisgICAgICAgICMgb24gd2l0aCBodHRwczovL2J1 Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9ODMwNzYsIHJlZHVjZSBieQorICAgICAgICAj IGhhbGYgdGhlICMgb2Ygd29ya2VycyB3ZSBydW4gYnkgZGVmYXVsdCBvbiBiaWdnZXIgbWFjaGlu ZXMuCisgICAgICAgIGRlZmF1bHRfY291bnQgPSBzdXBlcihDaHJvbWl1bU1hY1BvcnQsIHNlbGYp LmRlZmF1bHRfY2hpbGRfcHJvY2Vzc2VzKCkKKyAgICAgICAgaWYgZGVmYXVsdF9jb3VudCA+PSA4 OgorICAgICAgICAgICAgY3B1X2NvdW50ID0gc2VsZi5fZXhlY3V0aXZlLmNwdV9jb3VudCgpCisg ICAgICAgICAgICByZXR1cm4gbWF4KDEsIG1pbihkZWZhdWx0X2NvdW50LCBpbnQoY3B1X2NvdW50 IC8gMikpKQorICAgICAgICByZXR1cm4gZGVmYXVsdF9jb3VudAorCiAgICAgIwogICAgICMgUFJP VEVDVEVEIE1FVEhPRFMKICAgICAjCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5 L2xheW91dF90ZXN0cy9wb3J0L21hYy5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL3BvcnQvbWFjLnB5CmluZGV4IDZjNTU3MTIxZDJlZjMyNGVmYzZiMzBkZDM2NmYwYjMz ZTc0MzYxYjkuLjk0ZjcwMTdhMmEzNDhkODc1OGU5YTU2NGY4YTI3ZDY4YjVjY2E5NWUgMTAwNjQ0 Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvbWFjLnB5Cisr KyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvbWFjLnB5CkBAIC0x MDcsNyArMTA3LDE1IEBAIGNsYXNzIE1hY1BvcnQoQXBwbGVQb3J0KToKICAgICAgICAgaWYgc2Vs Zi5pc19zbm93bGVvcGFyZCgpOgogICAgICAgICAgICAgX2xvZy53YXJuKCJDYW5ub3QgcnVuIHRl c3RzIGluIHBhcmFsbGVsIG9uIFNub3cgTGVvcGFyZCBkdWUgdG8gcmRhcjovL3Byb2JsZW0vMTA2 MjE1MjUuIikKICAgICAgICAgICAgIHJldHVybiAxCi0gICAgICAgIHJldHVybiBzdXBlcihNYWNQ b3J0LCBzZWxmKS5kZWZhdWx0X2NoaWxkX3Byb2Nlc3NlcygpCisKKyAgICAgICAgIyBGSVhNRTog QXMgYSB0ZW1wb3Jhcnkgd29ya2Fyb3VuZCB3aGlsZSB3ZSBmaWd1cmUgb3V0IHdoYXQncyBnb2lu ZworICAgICAgICAjIG9uIHdpdGggaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dp P2lkPTgzMDc2LCByZWR1Y2UgYnkKKyAgICAgICAgIyBoYWxmIHRoZSAjIG9mIHdvcmtlcnMgd2Ug cnVuIGJ5IGRlZmF1bHQgb24gYmlnZ2VyIG1hY2hpbmVzLgorICAgICAgICBkZWZhdWx0X2NvdW50 ID0gc3VwZXIoTWFjUG9ydCwgc2VsZikuZGVmYXVsdF9jaGlsZF9wcm9jZXNzZXMoKQorICAgICAg ICBpZiBkZWZhdWx0X2NvdW50ID49IDg6CisgICAgICAgICAgICBjcHVfY291bnQgPSBzZWxmLl9l eGVjdXRpdmUuY3B1X2NvdW50KCkKKyAgICAgICAgICAgIHJldHVybiBtYXgoMSwgbWluKGRlZmF1 bHRfY291bnQsIGludChjcHVfY291bnQgLyAyKSkpCisgICAgICAgIHJldHVybiBkZWZhdWx0X2Nv dW50CiAKICAgICBkZWYgX2J1aWxkX2phdmFfdGVzdF9zdXBwb3J0KHNlbGYpOgogICAgICAgICBq YXZhX3Rlc3RzX3BhdGggPSBzZWxmLl9maWxlc3lzdGVtLmpvaW4oc2VsZi5sYXlvdXRfdGVzdHNf ZGlyKCksICJqYXZhIikK