98562 2012-10-05 15:21:22 -0700 nrwt: [chromium] run http tests in parallel on bigger machines 2012-10-09 12:31:33 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 dpranke dpranke abarth chase jchaffraix ojan rniwa tony webkit.review.bot oldest_to_newest 736028 0 dpranke 2012-10-05 15:21:22 -0700 nrwt: [chromium] run http tests in parallel on bigger machines 736037 1 167396 dpranke 2012-10-05 15:31:29 -0700 Created attachment 167396 Patch 736045 2 dpranke 2012-10-05 15:37:24 -0700 specifically, this should only affect the Mac 10.7 bot (which has 8 workers, so we'll run 2 http tests in parallel) and the Linux dbg bot (which runs 24 works, so 6 http tests in parallel -> the debug bot is a beast). 736046 3 dpranke 2012-10-05 15:38:01 -0700 if we see any increased flakiness on those bots, we can back the change out, otherwise we can look at adding more workers to the other debug bots. 736058 4 tony 2012-10-05 15:47:01 -0700 A couple weeks ago, I was testing setting --max-locked-shards=2 on my Z620 and found that it made the perf tests flaky, since they run in the same shard. Do you see any flakiness in the perf tests on your machine? 736067 5 dpranke 2012-10-05 15:58:21 -0700 I am running some tests to see. In reality I think we need to separate out the concepts of "test needs the lock" from "test should not be run while other tests are running", and we should probably strive to not have any tests fall in the latter category (although they might be necessary for some periods of time). Otherwise we'll always have bottlenecks in the cycle time. 736074 6 167396 scheib 2012-10-05 16:21:26 -0700 Comment on attachment 167396 Patch Dirk made me! 736076 7 167396 scheib 2012-10-05 16:21:58 -0700 Comment on attachment 167396 Patch Test complete. 736078 8 dpranke 2012-10-05 16:23:25 -0700 (we were testing if a non-reviewer can r+ bugs ...) 736090 9 dpranke 2012-10-05 16:37:09 -0700 So, ironically, on Linux on my z600 in a release build, I see flakiness in http and storage, but not perf (and I always get flakiness in http and storage). I don't see any flakiness on the Mac at all (again in Release). 736874 10 167396 eric 2012-10-08 11:50:58 -0700 Comment on attachment 167396 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=167396&action=review LGTM. > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:120 > + def default_max_locked_shards(self): > + """Return the number of "locked" shards to run in parallel (like the http tests).""" > + max_locked_shards = int(self.default_child_processes()) / 4 > + if not max_locked_shards: > + return 1 > + return max_locked_shards I assume the plan is to move this logic down once tested in the field? 736882 11 dpranke 2012-10-08 11:53:25 -0700 (In reply to comment #10) > (From update of attachment 167396 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=167396&action=review > > LGTM. > > > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:120 > > + def default_max_locked_shards(self): > > + """Return the number of "locked" shards to run in parallel (like the http tests).""" > > + max_locked_shards = int(self.default_child_processes()) / 4 > > + if not max_locked_shards: > > + return 1 > > + return max_locked_shards > > I assume the plan is to move this logic down once tested in the field? If by "down" you mean into base.py and "tested in the field" you mean "works stably on other ports", then yes :). 737098 12 dpranke 2012-10-08 15:06:00 -0700 Committed r130690: <http://trac.webkit.org/changeset/130690> 737219 13 ojan 2012-10-08 16:47:48 -0700 Looks like a couple perf tests started consistently failing after this. http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=%5Eperf 737268 14 dpranke 2012-10-08 17:46:50 -0700 (In reply to comment #13) > Looks like a couple perf tests started consistently failing after this. > > http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=%5Eperf I'm not sure exactly what you're seeing here, so could you confirm you're seeing what I'm seeing? Namely, it looks like maybe perf/mouse-event started failing consistently on Linux (dbg), but it appears to be failing the same way that it failed on Linux and Linux 32, so maybe it should be marked as SLOW. In the other cases, it just looks to me like the tests have been flaky, period, and are mostly marked as PASS FAIL. So, I'm not sure why we're even running these? 737298 15 ojan 2012-10-08 18:18:41 -0700 (In reply to comment #14) > (In reply to comment #13) > > Looks like a couple perf tests started consistently failing after this. > > > > http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=%5Eperf > > I'm not sure exactly what you're seeing here, so could you confirm you're seeing what I'm seeing? Namely, it looks like maybe perf/mouse-event started failing consistently on Linux (dbg), but it appears to be failing the same way that it failed on Linux and Linux 32, so maybe it should be marked as SLOW. They're not timing out, so marking them slow won't help. perf/mouse-event.html was consistently passing before this patch. The way these tests work is to run at different magnitudes and try to compare to see if they're constant, linear, polynomial, etc. So, if they don't have a whole core to themselves, they're likely to be more flaky. > In the other cases, it just looks to me like the tests have been flaky, period, and are mostly marked as PASS FAIL. > > So, I'm not sure why we're even running these? It's a little confusing because I recently committed a couple patches to reduce flakiness. So, you can see that some of them haven't been flaky for the last couple dozen runs (e.g. perf/typing-at-end-of-line.html). I've been waiting for it to stabilize before removing things from TestExpectations. In general, I'm not sure how I feel about the LayoutTests/perf tests. Some of them are never flaky. But, ~30% or so are always flaky. It does give a way to add regression tests when making order of magnitude performance improvements. 737332 16 dpranke 2012-10-08 18:48:12 -0700 (In reply to comment #15) > (In reply to comment #14) > > (In reply to comment #13) > > > Looks like a couple perf tests started consistently failing after this. > > > > > > http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=%5Eperf > > > > I'm not sure exactly what you're seeing here, so could you confirm you're seeing what I'm seeing? Namely, it looks like maybe perf/mouse-event started failing consistently on Linux (dbg), but it appears to be failing the same way that it failed on Linux and Linux 32, so maybe it should be marked as SLOW. > > They're not timing out, so marking them slow won't help. perf/mouse-event.html was consistently passing before this patch. Good point. Okay, so there weren't any other tests that started failing or became flakier? It only looked like the one to me. > The way these tests work is to run at different magnitudes and try to compare to see if they're constant, linear, polynomial, etc. So, if they don't have a whole core to themselves, they're likely to be more flaky. > Even without this change, assuming these tests had a dedicated core is a dangerous assumption. As I said in comment #5, I would rather approach this problem by splitting this out into "tests that are load-sensitive" vs. "tests that need the server lock" and try to solve the problem that way. It would not be hard to implement at least a coarse "run these tests by themselves" mechanism and I think it would be generally useful. Do you think we should revert this change in the mean time, or suppress the new failure in the meantime? 738061 17 dpranke 2012-10-09 12:24:37 -0700 (In reply to comment #16) > (In reply to comment #15) > > (In reply to comment #14) > > > (In reply to comment #13) > > > > Looks like a couple perf tests started consistently failing after this. > > > > > > > > http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=%5Eperf > > > > > > I'm not sure exactly what you're seeing here, so could you confirm you're seeing what I'm seeing? Namely, it looks like maybe perf/mouse-event started failing consistently on Linux (dbg), but it appears to be failing the same way that it failed on Linux and Linux 32, so maybe it should be marked as SLOW. > > > > They're not timing out, so marking them slow won't help. perf/mouse-event.html was consistently passing before this patch. As an aside, we normally skip the perf tests in debug, but this one was getting run due to an overriding expectation of flakiness on Release that wasn't properly scoped. I've fixed the expectation in http://trac.webkit.org/changeset/130793 , but that doesn't change the general nature of the problem. 738066 18 ojan 2012-10-09 12:26:36 -0700 I didn't notice that this was a debug bot. Nevermind. We can ignore it. I should probably update TestExpectations. 738076 19 dpranke 2012-10-09 12:31:33 -0700 (In reply to comment #18) > I didn't notice that this was a debug bot. Nevermind. We can ignore it. I should probably update TestExpectations. I did already. See above :). 167396 2012-10-05 15:31:29 -0700 2012-10-08 11:50:58 -0700 Patch bug-98562-20121005153037.patch text/plain 8587 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogMTMwNTUxCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZTQ3NjI4YjJmOTg1ZGJhZWM2MTM0NzQ1NTE5NjE0NWNl MTc5NzMwYy4uNTA1MmQ2ZmYxY2VjNjk0MmQyNjc4ZjI4NDRhMjhiMzAwMTUzM2Q0MCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDM1 IEBACisyMDEyLTEwLTA1ICBEaXJrIFByYW5rZSAgPGRwcmFua2VAY2hyb21pdW0ub3JnPgorCisg ICAgICAgIG5yd3Q6IFtjaHJvbWl1bV0gcnVuIGh0dHAgdGVzdHMgaW4gcGFyYWxsZWwgb24gYmln Z2VyIG1hY2hpbmVzCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNn aT9pZD05ODU2MgorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAg ICAgIFRoZSAibG9ja2VkIHRlc3RzIiBzaGFyZCAod2hpY2ggaW5jbHVkZXMgdGhlIGh0dHAgdGVz dHMgYW5kIHRoZQorICAgICAgICBwZXJmIHRlc3RzKSBpcyB0aGUgbG9uZyBwb2xlIG9uIG1hY2hp bmVzIHdoZXJlIHdlIGhhdmUgNCBvciBtb3JlCisgICAgICAgIHdvcmtlcnMsIHNvIHdlIHNob3Vs ZCBzdGFydCBtYWtpbmcgc3VyZSB0aGF0CisgICAgICAgIHdlIGNhbiBydW4gaHR0cCB0ZXN0cyBp biBwYXJhbGxlbCwgZm9sbG93aW5nIHRoZSBub3JtYWwgc2hhcmRpbmcKKyAgICAgICAgcnVsZXMg KGFsbCB0ZXN0cyBpbiB0aGUgc2FtZSBkaXJlY3RvcnkgYXJlIGluIHRoZSBzYW1lIHNoYXJkIGJ5 CisgICAgICAgIGRlZmF1bHQpLiBXZSBzaG91bGQgc3RpbGwgcHJvYmFibHkgbGltaXQgdGhlIG51 bWJlciBvZiB3b3JrZXJzCisgICAgICAgIGhpdHRpbmcgdGhlIHdlYiBzZXJ2ZXIgaW4gcGFyYWxs ZWwgd2hlcmUgd2UgY2FuOyBhIGhldXJpc3RpYyBvZgorICAgICAgICBubyBtb3JlIHRoYW4gMjUl IG9mIHRoZW0gc2VlbXMgb2theSBmb3IgYSBzdGFydC4gVGhpcyB3aWxsCisgICAgICAgIGxpa2Vs eSBvbmx5IGFmZmVjdCBkZXZlbG9wZXIgd29ya3N0YXRpb25zIGFuZCBhIGNvdXBsZSBvZiBib3Rz CisgICAgICAgIGF0IGZpcnN0LCBzbyBzaG91bGQgYmUgbG93IHJpc2sgYW5kIGEgZ29vZCByZXdh cmQuCisKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jhc2Uu cHk6CisgICAgICAgIChQb3J0LmRlZmF1bHRfbWF4X2xvY2tlZF9zaGFyZHMpOgorICAgICAgICAq IFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21pdW0ucHk6CisgICAgICAg IChDaHJvbWl1bVBvcnQuZGVmYXVsdF9tYXhfbG9ja2VkX3NoYXJkcyk6CisgICAgICAgICogU2Ny aXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1bV9wb3J0X3Rlc3RjYXNlLnB5 OgorICAgICAgICAoQ2hyb21pdW1Qb3J0VGVzdENhc2UudGVzdF9kZWZhdWx0X21heF9sb2NrZWRf c2hhcmRzKToKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L3Bv cnRfdGVzdGNhc2UucHk6CisgICAgICAgIChQb3J0VGVzdENhc2UudGVzdF9kZWZhdWx0X21heF9s b2NrZWRfc2hhcmRzKToKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9y dW5fd2Via2l0X3Rlc3RzLnB5OgorICAgICAgICAoX3NldF91cF9kZXJpdmVkX29wdGlvbnMpOgor ICAgICAgICAocGFyc2VfYXJncyk6CisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRf dGVzdHMvcnVuX3dlYmtpdF90ZXN0c19pbnRlZ3JhdGlvbnRlc3QucHk6CisgICAgICAgIChNYWlu VGVzdC50ZXN0X21heF9sb2NrZWRfc2hhcmRzKToKKwogMjAxMi0xMC0wNSAgWmFuIERvYmVyc2Vr ICA8emFuZG9iZXJzZWtAZ21haWwuY29tPgogCiAgICAgICAgIFtHdGtdIGZhc3QveHNsL3hzbHQt bWlzc2luZy1uYW1lc3BhY2UtaW4teHNsdC54bWwgaXMgZmFpbGluZyBvbiB0aGUgNjQtYml0IERl YnVnIGJ1aWxkZXIKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rl c3RzL3BvcnQvYmFzZS5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3Bv cnQvYmFzZS5weQppbmRleCA5ZjY5MWFlM2EzMGE0ZjgyYjZhZjI2ZWVhNGY2YmZhMzVjMDU5YWNi Li5hZTU1YzY4NGRkYzZhMjk4ODcyMjIyN2YyYTgwYTI5ZmVhNDcxNTg0IDEwMDc1NQotLS0gYS9U b29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jhc2UucHkKKysrIGIvVG9v bHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlLnB5CkBAIC0xODIsNiAr MTgyLDEwIEBAIGNsYXNzIFBvcnQob2JqZWN0KToKICAgICAgICAgIiIiUmV0dXJuIHRoZSBudW1i ZXIgb2YgRHVtcFJlbmRlclRyZWUgaW5zdGFuY2VzIHRvIHVzZSBmb3IgdGhpcyBwb3J0LiIiIgog ICAgICAgICByZXR1cm4gc2VsZi5fZXhlY3V0aXZlLmNwdV9jb3VudCgpCiAKKyAgICBkZWYgZGVm YXVsdF9tYXhfbG9ja2VkX3NoYXJkcyhzZWxmKToKKyAgICAgICAgIiIiUmV0dXJuIHRoZSBudW1i ZXIgb2YgImxvY2tlZCIgc2hhcmRzIHRvIHJ1biBpbiBwYXJhbGxlbCAobGlrZSB0aGUgaHR0cCB0 ZXN0cykuIiIiCisgICAgICAgIHJldHVybiAxCisKICAgICBkZWYgd29ya2VyX3N0YXJ0dXBfZGVs YXlfc2VjcyhzZWxmKToKICAgICAgICAgIyBGSVhNRTogSWYgd2Ugc3RhcnQgd29ya2VycyB1cCB0 b28gcXVpY2tseSwgRHVtcFJlbmRlclRyZWUgYXBwZWFycwogICAgICAgICAjIHRvIHRocmFzaCBv biBzb21ldGhpbmcgYW5kIHRpbWUgb3V0IGl0cyBmaXJzdCBmZXcgdGVzdHMuIFVudGlsCmRpZmYg LS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Nocm9taXVt LnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1bS5w eQppbmRleCBkNjhiZDQ2YmU5YzM4NWQ2MDdlZGQzYzJkMTk2M2FlOGZjZDUwODZiLi5kNzI5MDA3 ODQwZDQ2M2M1ZTE2MzQxNWY4YzhmNmE1MTRlNmZhZjU5IDEwMDc1NQotLS0gYS9Ub29scy9TY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Nocm9taXVtLnB5CisrKyBiL1Rvb2xzL1Nj cmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21pdW0ucHkKQEAgLTExMiw2ICsx MTIsMTMgQEAgY2xhc3MgQ2hyb21pdW1Qb3J0KFBvcnQpOgogICAgIGRlZiBpc19jaHJvbWl1bShz ZWxmKToKICAgICAgICAgcmV0dXJuIFRydWUKIAorICAgIGRlZiBkZWZhdWx0X21heF9sb2NrZWRf c2hhcmRzKHNlbGYpOgorICAgICAgICAiIiJSZXR1cm4gdGhlIG51bWJlciBvZiAibG9ja2VkIiBz aGFyZHMgdG8gcnVuIGluIHBhcmFsbGVsIChsaWtlIHRoZSBodHRwIHRlc3RzKS4iIiIKKyAgICAg ICAgbWF4X2xvY2tlZF9zaGFyZHMgPSBpbnQoc2VsZi5kZWZhdWx0X2NoaWxkX3Byb2Nlc3Nlcygp KSAvIDQKKyAgICAgICAgaWYgbm90IG1heF9sb2NrZWRfc2hhcmRzOgorICAgICAgICAgICAgcmV0 dXJuIDEKKyAgICAgICAgcmV0dXJuIG1heF9sb2NrZWRfc2hhcmRzCisKICAgICBkZWYgZGVmYXVs dF9waXhlbF90ZXN0cyhzZWxmKToKICAgICAgICAgcmV0dXJuIFRydWUKIApkaWZmIC0tZ2l0IGEv VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1bV9wb3J0X3Rl c3RjYXNlLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJv bWl1bV9wb3J0X3Rlc3RjYXNlLnB5CmluZGV4IGNjNTBjNjRlYjE3ZDgyMDQ5ZDIwMzFmOWMwMWYz NTlkZDIzM2MyY2YuLmEwODJhMTMxYzgxMDFjOTAyNmE3NGJkYmNkNzE3YzRiMDZkZmQ2OGIgMTAw NjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21p dW1fcG9ydF90ZXN0Y2FzZS5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90 ZXN0cy9wb3J0L2Nocm9taXVtX3BvcnRfdGVzdGNhc2UucHkKQEAgLTQ4LDYgKzQ4LDEzIEBAIGNs YXNzIENocm9taXVtUG9ydFRlc3RDYXNlKHBvcnRfdGVzdGNhc2UuUG9ydFRlc3RDYXNlKToKICAg ICAgICAgcG9ydCA9IHNlbGYubWFrZV9wb3J0KCkKICAgICAgICAgcG9ydC5jaGVja19idWlsZChu ZWVkc19odHRwPVRydWUpCiAKKyAgICBkZWYgdGVzdF9kZWZhdWx0X21heF9sb2NrZWRfc2hhcmRz KHNlbGYpOgorICAgICAgICBwb3J0ID0gc2VsZi5tYWtlX3BvcnQoKQorICAgICAgICBwb3J0LmRl ZmF1bHRfY2hpbGRfcHJvY2Vzc2VzID0gbGFtYmRhOiAxNgorICAgICAgICBzZWxmLmFzc2VydEVx dWFscyhwb3J0LmRlZmF1bHRfbWF4X2xvY2tlZF9zaGFyZHMoKSwgNCkKKyAgICAgICAgcG9ydC5k ZWZhdWx0X2NoaWxkX3Byb2Nlc3NlcyA9IGxhbWJkYTogMgorICAgICAgICBzZWxmLmFzc2VydEVx dWFscyhwb3J0LmRlZmF1bHRfbWF4X2xvY2tlZF9zaGFyZHMoKSwgMSkKKwogICAgIGRlZiB0ZXN0 X2RlZmF1bHRfdGltZW91dF9tcyhzZWxmKToKICAgICAgICAgc2VsZi5hc3NlcnRFcXVhbHMoc2Vs Zi5tYWtlX3BvcnQob3B0aW9ucz1Nb2NrT3B0aW9ucyhjb25maWd1cmF0aW9uPSdSZWxlYXNlJykp LmRlZmF1bHRfdGltZW91dF9tcygpLCA2MDAwKQogICAgICAgICBzZWxmLmFzc2VydEVxdWFscyhz ZWxmLm1ha2VfcG9ydChvcHRpb25zPU1vY2tPcHRpb25zKGNvbmZpZ3VyYXRpb249J0RlYnVnJykp LmRlZmF1bHRfdGltZW91dF9tcygpLCAxMjAwMCkKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMv d2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvcG9ydF90ZXN0Y2FzZS5weSBiL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvcG9ydF90ZXN0Y2FzZS5weQppbmRleCA0NzQz NDcwZDFmNDJlYzU1OGZjOGIyMTUyNzNjYTgzZjFiMjUyNzQ3Li5mNzA0YTdhMTMxNTFjZjM1MzQ2 ODhlZDEyYThhMDU0NWQ1Mjc4MDM0IDEwMDc1NQotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5 L2xheW91dF90ZXN0cy9wb3J0L3BvcnRfdGVzdGNhc2UucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93 ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9wb3J0X3Rlc3RjYXNlLnB5CkBAIC05MCw2ICs5MCwx MyBAQCBjbGFzcyBQb3J0VGVzdENhc2UodW5pdHRlc3QuVGVzdENhc2UpOgogICAgICAgICBwb3J0 X25hbWUgPSBzZWxmLnBvcnRfbWFrZXIuZGV0ZXJtaW5lX2Z1bGxfcG9ydF9uYW1lKGhvc3QsIG9w dGlvbnMsIHBvcnRfbmFtZSkKICAgICAgICAgcmV0dXJuIHNlbGYucG9ydF9tYWtlcihob3N0LCBw b3J0X25hbWUsIG9wdGlvbnM9b3B0aW9ucywgY29uZmlnPWNvbmZpZywgKiprd2FyZ3MpCiAKKyAg ICBkZWYgdGVzdF9kZWZhdWx0X21heF9sb2NrZWRfc2hhcmRzKHNlbGYpOgorICAgICAgICBwb3J0 ID0gc2VsZi5tYWtlX3BvcnQoKQorICAgICAgICBwb3J0LmRlZmF1bHRfY2hpbGRfcHJvY2Vzc2Vz ID0gbGFtYmRhOiAxNgorICAgICAgICBzZWxmLmFzc2VydEVxdWFscyhwb3J0LmRlZmF1bHRfbWF4 X2xvY2tlZF9zaGFyZHMoKSwgMSkKKyAgICAgICAgcG9ydC5kZWZhdWx0X2NoaWxkX3Byb2Nlc3Nl cyA9IGxhbWJkYTogMgorICAgICAgICBzZWxmLmFzc2VydEVxdWFscyhwb3J0LmRlZmF1bHRfbWF4 X2xvY2tlZF9zaGFyZHMoKSwgMSkKKwogICAgIGRlZiB0ZXN0X2RlZmF1bHRfdGltZW91dF9tcyhz ZWxmKToKICAgICAgICAgc2VsZi5hc3NlcnRFcXVhbHMoc2VsZi5tYWtlX3BvcnQob3B0aW9ucz1N b2NrT3B0aW9ucyhjb25maWd1cmF0aW9uPSdSZWxlYXNlJykpLmRlZmF1bHRfdGltZW91dF9tcygp LCAzNTAwMCkKICAgICAgICAgc2VsZi5hc3NlcnRFcXVhbHMoc2VsZi5tYWtlX3BvcnQob3B0aW9u cz1Nb2NrT3B0aW9ucyhjb25maWd1cmF0aW9uPSdEZWJ1ZycpKS5kZWZhdWx0X3RpbWVvdXRfbXMo KSwgMzUwMDApCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0 cy9ydW5fd2Via2l0X3Rlc3RzLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvcnVuX3dlYmtpdF90ZXN0cy5weQppbmRleCBhMjIwMzI5ZDVlOTRmNWU2MWJmOTNjOTFmMWZj NDlhOGYwNWU0YjhiLi44OTUyMjA3OWM1MDRlOWNiZWJjYTQ2YjI5NTlkZTZkOThhYzg3ZDU1IDEw MDc1NQotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0 X3Rlc3RzLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93 ZWJraXRfdGVzdHMucHkKQEAgLTEyOSw2ICsxMjksOSBAQCBkZWYgX3NldF91cF9kZXJpdmVkX29w dGlvbnMocG9ydCwgb3B0aW9ucyk6CiAgICAgaWYgbm90IG9wdGlvbnMuY2hpbGRfcHJvY2Vzc2Vz OgogICAgICAgICBvcHRpb25zLmNoaWxkX3Byb2Nlc3NlcyA9IG9zLmVudmlyb24uZ2V0KCJXRUJL SVRfVEVTVF9DSElMRF9QUk9DRVNTRVMiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHN0cihwb3J0LmRlZmF1bHRfY2hpbGRfcHJvY2Vzc2VzKCkpKQor ICAgIGlmIG5vdCBvcHRpb25zLm1heF9sb2NrZWRfc2hhcmRzOgorICAgICAgICBvcHRpb25zLm1h eF9sb2NrZWRfc2hhcmRzID0gaW50KG9zLmVudmlyb24uZ2V0KCJXRUJLSVRfVEVTVF9NQVhfTE9D S0VEX1NIQVJEUyIsCisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgc3RyKHBvcnQuZGVmYXVsdF9tYXhfbG9ja2VkX3NoYXJkcygpKSkpCiAKICAg ICBpZiBub3Qgb3B0aW9ucy5jb25maWd1cmF0aW9uOgogICAgICAgICBvcHRpb25zLmNvbmZpZ3Vy YXRpb24gPSBwb3J0LmRlZmF1bHRfY29uZmlndXJhdGlvbigpCkBAIC00MTYsNyArNDE5LDcgQEAg ZGVmIHBhcnNlX2FyZ3MoYXJncz1Ob25lKToKICAgICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24o Ii0tbm8tcmV0cnktZmFpbHVyZXMiLCBhY3Rpb249InN0b3JlX2ZhbHNlIiwKICAgICAgICAgICAg IGRlc3Q9InJldHJ5X2ZhaWx1cmVzIiwKICAgICAgICAgICAgIGhlbHA9IkRvbid0IHJlLXRyeSBh bnkgdGVzdHMgdGhhdCBwcm9kdWNlIHVuZXhwZWN0ZWQgcmVzdWx0cy4iKSwKLSAgICAgICAgb3B0 cGFyc2UubWFrZV9vcHRpb24oIi0tbWF4LWxvY2tlZC1zaGFyZHMiLCB0eXBlPSJpbnQiLCBkZWZh dWx0PTEsCisgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCItLW1heC1sb2NrZWQtc2hhcmRz IiwgdHlwZT0iaW50IiwgZGVmYXVsdD0wLAogICAgICAgICAgICAgaGVscD0iU2V0IHRoZSBtYXhp bXVtIG51bWJlciBvZiBsb2NrZWQgc2hhcmRzIiksCiAgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0 aW9uKCItLWFkZGl0aW9uYWwtZW52LXZhciIsIHR5cGU9InN0cmluZyIsIGFjdGlvbj0iYXBwZW5k IiwgZGVmYXVsdD1bXSwKICAgICAgICAgICAgIGhlbHA9IlBhc3NlcyB0aGF0IGVudmlyb25tZW50 IHZhcmlhYmxlIHRvIHRoZSB0ZXN0cyAoLS1hZGRpdGlvbmFsLWVudi12YXI9TkFNRT1WQUxVRSki KSwKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93 ZWJraXRfdGVzdHNfaW50ZWdyYXRpb250ZXN0LnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9s YXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0c19pbnRlZ3JhdGlvbnRlc3QucHkKaW5kZXggMjRi YTE1YTQ3YTA0ZGViNjZhOWZhYjYyODY3ODcxNmQ3ZWVkNmYxOS4uZTEwNDkyYTFhZDFhNTQ1ODEy NjgxNzliMmRjZjUyZjYxM2RjNjYwZSAxMDA3NTUKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRw eS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0c19pbnRlZ3JhdGlvbnRlc3QucHkKKysrIGIv VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0c19pbnRl Z3JhdGlvbnRlc3QucHkKQEAgLTMwMSw2ICszMDEsMTIgQEAgY2xhc3MgTWFpblRlc3QodW5pdHRl c3QuVGVzdENhc2UsIFN0cmVhbVRlc3RpbmdNaXhpbik6CiAgICAgICAgIGZvciBiYXRjaCBpbiBi YXRjaF90ZXN0c19ydW46CiAgICAgICAgICAgICBzZWxmLmFzc2VydFRydWUobGVuKGJhdGNoKSA8 PSAyLCAnJXMgaGFkIHRvbyBtYW55IHRlc3RzJyAlICcsICcuam9pbihiYXRjaCkpCiAKKyAgICBk ZWYgdGVzdF9tYXhfbG9ja2VkX3NoYXJkcyhzZWxmKToKKyAgICAgICAgaWYgbm90IHNlbGYuc2hv dWxkX3Rlc3RfcHJvY2Vzc2VzOgorICAgICAgICAgICAgcmV0dXJuCisgICAgICAgIF8sIF8sIHJl Z3VsYXJfb3V0cHV0LCBfID0gbG9nZ2luZ19ydW4oWyctLWRlYnVnLXJ3dC1sb2dnaW5nJywgJy0t Y2hpbGQtcHJvY2Vzc2VzJywgJzInXSwgc2hhcmVkX3BvcnQ9RmFsc2UpCisgICAgICAgIHNlbGYu YXNzZXJ0VHJ1ZShhbnkoWycoMSBsb2NrZWQpJyBpbiBsaW5lIGZvciBsaW5lIGluIHJlZ3VsYXJf b3V0cHV0LmJ1Zmxpc3RdKSkKKwogICAgIGRlZiB0ZXN0X2NoaWxkX3Byb2Nlc3Nlc18yKHNlbGYp OgogICAgICAgICBpZiBzZWxmLnNob3VsZF90ZXN0X3Byb2Nlc3NlczoKICAgICAgICAgICAgIF8s IF8sIHJlZ3VsYXJfb3V0cHV0LCBfID0gbG9nZ2luZ19ydW4oCg==