112890 2013-03-21 02:26:04 -0700 NRWT: --force should run tests as if they're expected to pass 2013-10-31 11:11:27 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 123167 1 rniwa webkit-unassigned abarth alancutter ap commit-queue dpranke eric.carlson eric glenn jberlin ossy rego+ews roger_fong simon.fraser xan.lopez oldest_to_newest 860159 0 rniwa 2013-03-21 02:26:04 -0700 --force appears to be broken at least on Mac WK1. 860165 1 eric 2013-03-21 02:38:21 -0700 --force Run all tests, even those marked SKIP in the test list (same as --skipped=ignore) 860473 2 rniwa 2013-03-21 11:05:46 -0700 (In reply to comment #1) > --force Run all tests, even those marked SKIP in the test list (same as --skipped=ignore) Hm... maybe it's doing the right thing? What I would expect --force to do is to treat all tests to have Pass expectation. 860541 3 dpranke 2013-03-21 12:06:14 -0700 Yup, that's not what it does (nor is it what it's ever done). You're suggesting a reasonable feature addition, though. 860648 4 dpranke 2013-03-21 13:52:36 -0700 To clarify, if a test is marked as Skip (only), then when it is run, it will be expected to pass. You're suggesting that --force should do something else, and also change the expectations for tests that are otherwise normally run but expected to fail one way or another? 860651 5 rniwa 2013-03-21 13:54:20 -0700 (In reply to comment #4) > To clarify, if a test is marked as Skip (only), then when it is run, it will be expected to pass. > > You're suggesting that --force should do something else, and also change the expectations for tests that are otherwise normally run but expected to fail one way or another? I'm expecting that Failure, ImageOnlyFailure, Crash, etc… test expectations to be ignored, and all tests to be expected to Pass. 860652 6 dpranke 2013-03-21 13:55:39 -0700 Ok. I'm not opposed to this idea, but I'm not sure if we should use --force for that. It might confuse people used to the existing behavior (or break scripts using it). On the other hand, it does seem like a reasonable name, and we do have --skipped= for the other behavior. 860659 7 rniwa 2013-03-21 14:01:46 -0700 (In reply to comment #6) > Ok. I'm not opposed to this idea, but I'm not sure if we should use --force for that. It might confuse people used to the existing behavior (or break scripts using it). > > On the other hand, it does seem like a reasonable name, and we do have --skipped= for the other behavior. Do you know if there are any tools that depend on this behavior? I feel like the current --force should be renamed to --ignore-skips. 860661 8 rniwa 2013-03-21 14:02:16 -0700 Ugh… I guess ignoring skips might sound ambiguous. Maybe --run-skipped? 860680 9 dpranke 2013-03-21 14:09:20 -0700 --force is currently a synonym for --skipped=ignore . If we can get people to change at all, they should just change to that, rather than adding yet another --run-skipped or --ignore-skipped flag. I do not know if any scripts currently use this. I think Ossy has done something along these lines in the past, so I'm cc'ing him. 860684 10 rniwa 2013-03-21 14:12:23 -0700 (In reply to comment #9) > --force is currently a synonym for --skipped=ignore . If we can get people to change at all, they should just change to that, rather than adding yet another --run-skipped or --ignore-skipped flag. Yeah, I agree. We already have waaay too many options in new-run-webkit-tests. 941832 11 214736 tgergely.u-szeged 2013-10-21 07:07:19 -0700 Created attachment 214736 Modifies the behaviour of nrwt --force switch. 941938 12 dpranke 2013-10-21 11:42:37 -0700 The code looks correct. However, this patch appears to ignore much of the discussion in the bug ... are you sure that you want to change the semantics of the existing flag? Insofar as I'm not working on the code in WebKit much these days, I'll let someone else (rniwa?) decide if they want to r+ this or not. 942130 13 214736 commit-queue 2013-10-21 19:16:01 -0700 Comment on attachment 214736 Modifies the behaviour of nrwt --force switch. Clearing flags on attachment: 214736 Committed r157774: <http://trac.webkit.org/changeset/157774> 942131 14 commit-queue 2013-10-21 19:16:04 -0700 All reviewed patches have been landed. Closing bug. 942368 15 roger_fong 2013-10-22 11:56:01 -0700 This caused 2 python tests to fail on all platforms: http://build.webkit.org/builders/Apple%20MountainLion%20Release%20WK2%20%28Tests%29/builds/12800/steps/webkitpy-test/logs/stdio 942619 16 214736 ossy 2013-10-23 01:53:28 -0700 Comment on attachment 214736 Modifies the behaviour of nrwt --force switch. View in context: https://bugs.webkit.org/attachment.cgi?id=214736&action=review > Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:341 > + if options.force and options.skipped not in ('ignore', 'default'): > + _log.warning("--force overrides --skipped=%s" % (options.skipped)) > + options.skipped = 'ignore' > + It seems the bug is here. After this change --force makes NRWT run [ Failure ] marked tests if they marked as [ Pass ], but it doesn't run [ Skip ] and [ WontFix ] marked tests as they pass. The problem is that options.skipped = 'default' by default, but it should be 'ignore' if options.force is true. Tamás, please run Tools/Scripts/test-webkitpy unit tests next time you touches webkitpy codes. I thought CQ bots run webkitpy tests ... but it seems they don't. I found this: "# FIXME: We should teach the commit-queue and the EWS how to run these tests." in Tools/Scripts/webkitpy/tool/steps/runtests.py 945483 17 215642 tgergely.u-szeged 2013-10-31 04:13:21 -0700 Created attachment 215642 Modifies the behaviour of nrwt --force switch. Corrected, now --force really implies --skipped=ignore 945593 18 215642 commit-queue 2013-10-31 11:11:22 -0700 Comment on attachment 215642 Modifies the behaviour of nrwt --force switch. Clearing flags on attachment: 215642 Committed r158373: <http://trac.webkit.org/changeset/158373> 945594 19 commit-queue 2013-10-31 11:11:27 -0700 All reviewed patches have been landed. Closing bug. 214736 2013-10-21 07:07:19 -0700 2013-10-31 04:13:21 -0700 Modifies the behaviour of nrwt --force switch. nrwt-force-switch-implies-pass-expectations.patch text/plain 5476 tgergely.u-szeged ZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBiL1Rvb2xzL0NoYW5nZUxvZwppbmRleCA0MGE2 N2M2Li44MjZhYWM0IDEwMDY0NAotLS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hh bmdlTG9nCkBAIC0xLDMgKzEsMjIgQEAKKzIwMTMtMTAtMTUgIFRhbWFzIEdlcmdlbHkgIDxnZXJ0 b21AaW5mLnUtc3plZ2VkLmh1PgorCisgICAgICAgIFJ1biB0ZXN0cyBhcyBpZiB0aGV5IGFyZSBl eHBlY3RlZCB0byBwYXNzIHdoZW4gLS1mb3JjZSBpcyBnaXZlbi4KKyAgICAgICAgaHR0cHM6Ly9i dWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTExMjg5MAorCisgICAgICAgIFJldmlld2Vk IGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFRoZSAtLWZvcmNlIHN3aXRjaCBvZiBuZXct cnVuLXdlYmtpdC10ZXN0cyBpcyBub3cgZm9yY2VzIGFsbCB0ZXN0cyB0bworICAgICAgICBiZSBy dW4gd2l0aCAnUEFTUycgYXMgZXhwZWN0ZWQgcmVzdWx0LgorCisgICAgICAgICogU2NyaXB0cy93 ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbWFuYWdlci5weToKKyAgICAgICAgKE1h bmFnZXIucnVuKToKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2Rl bHMvdGVzdF9leHBlY3RhdGlvbnMucHk6CisgICAgICAgIChUZXN0RXhwZWN0YXRpb25zLl9faW5p dF9fKToKKyAgICAgICAgKFRlc3RFeHBlY3RhdGlvbnMuX2FkZF9leHBlY3RhdGlvbnMpOgorICAg ICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHk6 CisgICAgICAgIChwYXJzZV9hcmdzKToKKyAgICAgICAgKF9zZXRfdXBfZGVyaXZlZF9vcHRpb25z KToKKwogMjAxMy0xMC0wNCAgQnJlbnQgRnVsZ2hhbSAgPGJmdWxnaGFtQGFwcGxlLmNvbT4KIAog ICAgICAgICBbV2luZG93c10gQ29ycmVjdCBydW4tcGF0aCB3aGVuIHVzaW5nIDY0LWJpdCBiaW5h cmllcy4KZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2Nv bnRyb2xsZXJzL21hbmFnZXIucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0 cy9jb250cm9sbGVycy9tYW5hZ2VyLnB5CmluZGV4IDZiZGQ3ZTkuLjdlODU1ZWMgMTAwNjQ0Ci0t LSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL21hbmFn ZXIucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxl cnMvbWFuYWdlci5weQpAQCAtMTc5LDcgKzE3OSw3IEBAIGNsYXNzIE1hbmFnZXIob2JqZWN0KToK ICAgICAgICAgICAgIHJldHVybiB0ZXN0X3J1bl9yZXN1bHRzLlJ1bkRldGFpbHMoZXhpdF9jb2Rl PS0xKQogCiAgICAgICAgIHNlbGYuX3ByaW50ZXIud3JpdGVfdXBkYXRlKCJQYXJzaW5nIGV4cGVj dGF0aW9ucyAuLi4iKQotICAgICAgICBzZWxmLl9leHBlY3RhdGlvbnMgPSB0ZXN0X2V4cGVjdGF0 aW9ucy5UZXN0RXhwZWN0YXRpb25zKHNlbGYuX3BvcnQsIHRlc3RfbmFtZXMpCisgICAgICAgIHNl bGYuX2V4cGVjdGF0aW9ucyA9IHRlc3RfZXhwZWN0YXRpb25zLlRlc3RFeHBlY3RhdGlvbnMoc2Vs Zi5fcG9ydCwgdGVzdF9uYW1lcywgZm9yY2VfZXhwZWN0YXRpb25zX3Bhc3M9c2VsZi5fb3B0aW9u cy5mb3JjZSkKIAogICAgICAgICB0ZXN0c190b19ydW4sIHRlc3RzX3RvX3NraXAgPSBzZWxmLl9w cmVwYXJlX2xpc3RzKHBhdGhzLCB0ZXN0X25hbWVzKQogICAgICAgICBzZWxmLl9wcmludGVyLnBy aW50X2ZvdW5kKGxlbih0ZXN0X25hbWVzKSwgbGVuKHRlc3RzX3RvX3J1biksIHNlbGYuX29wdGlv bnMucmVwZWF0X2VhY2gsIHNlbGYuX29wdGlvbnMuaXRlcmF0aW9ucykKZGlmZiAtLWdpdCBhL1Rv b2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL21vZGVscy90ZXN0X2V4cGVjdGF0aW9u cy5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL21vZGVscy90ZXN0X2V4 cGVjdGF0aW9ucy5weQppbmRleCAyNmZkOWRkLi40MDYwZTQzIDEwMDY0NAotLS0gYS9Ub29scy9T Y3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9leHBlY3RhdGlvbnMucHkK KysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbW9kZWxzL3Rlc3RfZXhw ZWN0YXRpb25zLnB5CkBAIC04MzMsNyArODMzLDcgQEAgY2xhc3MgVGVzdEV4cGVjdGF0aW9ucyhv YmplY3QpOgogICAgICMgRklYTUU6IFRoaXMgY29uc3RydWN0b3IgZG9lcyB0b28gbXVjaCB3b3Jr LiBXZSBzaG91bGQgbW92ZSB0aGUgYWN0dWFsIHBhcnNpbmcgb2YKICAgICAjIHRoZSBleHBlY3Rh dGlvbnMgaW50byBzZXBhcmF0ZSByb3V0aW5lcyBzbyB0aGF0IGxpbnRpbmcgYW5kIGhhbmRsaW5n IG92ZXJyaWRlcwogICAgICMgY2FuIGJlIGNvbnRyb2xsZWQgc2VwYXJhdGVseSwgYW5kIHRoZSBj b25zdHJ1Y3RvciBjYW4gYmUgbW9yZSBvZiBhIG5vLW9wLgotICAgIGRlZiBfX2luaXRfXyhzZWxm LCBwb3J0LCB0ZXN0cz1Ob25lLCBpbmNsdWRlX2dlbmVyaWM9VHJ1ZSwgaW5jbHVkZV9vdmVycmlk ZXM9VHJ1ZSwgZXhwZWN0YXRpb25zX3RvX2xpbnQ9Tm9uZSk6CisgICAgZGVmIF9faW5pdF9fKHNl bGYsIHBvcnQsIHRlc3RzPU5vbmUsIGluY2x1ZGVfZ2VuZXJpYz1UcnVlLCBpbmNsdWRlX292ZXJy aWRlcz1UcnVlLCBleHBlY3RhdGlvbnNfdG9fbGludD1Ob25lLCBmb3JjZV9leHBlY3RhdGlvbnNf cGFzcz1GYWxzZSk6CiAgICAgICAgIHNlbGYuX2Z1bGxfdGVzdF9saXN0ID0gdGVzdHMKICAgICAg ICAgc2VsZi5fdGVzdF9jb25maWcgPSBwb3J0LnRlc3RfY29uZmlndXJhdGlvbigpCiAgICAgICAg IHNlbGYuX2lzX2xpbnRfbW9kZSA9IGV4cGVjdGF0aW9uc190b19saW50IGlzIG5vdCBOb25lCkBA IC04NDIsNiArODQyLDcgQEAgY2xhc3MgVGVzdEV4cGVjdGF0aW9ucyhvYmplY3QpOgogICAgICAg ICBzZWxmLl9wb3J0ID0gcG9ydAogICAgICAgICBzZWxmLl9za2lwcGVkX3Rlc3RzX3dhcm5pbmdz ID0gW10KICAgICAgICAgc2VsZi5fZXhwZWN0YXRpb25zID0gW10KKyAgICAgICAgc2VsZi5fZm9y Y2VfZXhwZWN0YXRpb25zX3Bhc3MgPSBmb3JjZV9leHBlY3RhdGlvbnNfcGFzcwogCiAgICAgICAg IGV4cGVjdGF0aW9uc19kaWN0ID0gZXhwZWN0YXRpb25zX3RvX2xpbnQgb3IgcG9ydC5leHBlY3Rh dGlvbnNfZGljdCgpCiAKQEAgLTk5MCw3ICs5OTEsMTEgQEAgY2xhc3MgVGVzdEV4cGVjdGF0aW9u cyhvYmplY3QpOgogCiAgICAgZGVmIF9hZGRfZXhwZWN0YXRpb25zKHNlbGYsIGV4cGVjdGF0aW9u X2xpc3QpOgogICAgICAgICBmb3IgZXhwZWN0YXRpb25fbGluZSBpbiBleHBlY3RhdGlvbl9saXN0 OgotICAgICAgICAgICAgaWYgbm90IGV4cGVjdGF0aW9uX2xpbmUuZXhwZWN0YXRpb25zOgorICAg ICAgICAgICAgaWYgc2VsZi5fZm9yY2VfZXhwZWN0YXRpb25zX3Bhc3M6CisgICAgICAgICAgICAg ICAgZXhwZWN0YXRpb25fbGluZS5leHBlY3RhdGlvbnMgPSBbJ1BBU1MnXQorICAgICAgICAgICAg ICAgIGV4cGVjdGF0aW9uX2xpbmUucGFyc2VkX2V4cGVjdGF0aW9ucyA9IHNldChbUEFTU10pCisK KyAgICAgICAgICAgIGVsaWYgbm90IGV4cGVjdGF0aW9uX2xpbmUuZXhwZWN0YXRpb25zOgogICAg ICAgICAgICAgICAgIGNvbnRpbnVlCiAKICAgICAgICAgICAgIGlmIHNlbGYuX2lzX2xpbnRfbW9k ZSBvciBzZWxmLl90ZXN0X2NvbmZpZyBpbiBleHBlY3RhdGlvbl9saW5lLm1hdGNoaW5nX2NvbmZp Z3VyYXRpb25zOgpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvcnVuX3dlYmtpdF90ZXN0cy5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rl c3RzL3J1bl93ZWJraXRfdGVzdHMucHkKaW5kZXggZDU3NDhkYi4uNTA2N2M4MiAxMDA2NDQKLS0t IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0cy5w eQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rl c3RzLnB5CkBAIC0yMjYsOCArMjI2LDggQEAgZGVmIHBhcnNlX2FyZ3MoYXJncyk6CiAgICAgICAg ICAgICAgICAgICInaWdub3JlJyA9PSBSdW4gdGhlbSBhbnl3YXksICIKICAgICAgICAgICAgICAg ICAgIidvbmx5JyA9PSBvbmx5IHJ1biB0aGUgU0tJUCB0ZXN0cywgIgogICAgICAgICAgICAgICAg ICAiJ2Fsd2F5cycgPT0gYWx3YXlzIHNraXAsIGV2ZW4gaWYgbGlzdGVkIG9uIHRoZSBjb21tYW5k IGxpbmUuIikpLAotICAgICAgICBvcHRwYXJzZS5tYWtlX29wdGlvbigiLS1mb3JjZSIsIGRlc3Q9 InNraXBwZWQiLCBhY3Rpb249InN0b3JlX2NvbnN0IiwgY29uc3Q9J2lnbm9yZScsCi0gICAgICAg ICAgICBoZWxwPSJSdW4gYWxsIHRlc3RzLCBldmVuIHRob3NlIG1hcmtlZCBTS0lQIGluIHRoZSB0 ZXN0IGxpc3QgKHNhbWUgYXMgLS1za2lwcGVkPWlnbm9yZSkiKSwKKyAgICAgICAgb3B0cGFyc2Uu bWFrZV9vcHRpb24oIi0tZm9yY2UiLCBhY3Rpb249InN0b3JlX3RydWUiLCBkZWZhdWx0PUZhbHNl LAorICAgICAgICAgICAgaGVscD0iUnVuIGFsbCB0ZXN0cyB3aXRoIFBBU1MgYXMgZXhwZWN0ZWQg cmVzdWx0LCBldmVuIHRob3NlIG1hcmtlZCBTS0lQIGluIHRoZSB0ZXN0IGxpc3QgKGltcGxpZXMg LS1za2lwcGVkPWlnbm9yZSkiKSwKICAgICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24oIi0tdGlt ZS1vdXQtbXMiLAogICAgICAgICAgICAgaGVscD0iU2V0IHRoZSB0aW1lb3V0IGZvciBlYWNoIHRl c3QiKSwKICAgICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24oIi0tb3JkZXIiLCBhY3Rpb249InN0 b3JlIiwgZGVmYXVsdD0ibmF0dXJhbCIsCkBAIC0zMzUsNiArMzM1LDEwIEBAIGRlZiBfc2V0X3Vw X2Rlcml2ZWRfb3B0aW9ucyhwb3J0LCBvcHRpb25zKToKICAgICAgICAgICAgIGFkZGl0aW9uYWxf cGxhdGZvcm1fZGlyZWN0b3JpZXMuYXBwZW5kKHBvcnQuaG9zdC5maWxlc3lzdGVtLmFic3BhdGgo cGF0aCkpCiAgICAgICAgIG9wdGlvbnMuYWRkaXRpb25hbF9wbGF0Zm9ybV9kaXJlY3RvcnkgPSBh ZGRpdGlvbmFsX3BsYXRmb3JtX2RpcmVjdG9yaWVzCiAKKyAgICBpZiBvcHRpb25zLmZvcmNlIGFu ZCBvcHRpb25zLnNraXBwZWQgbm90IGluICgnaWdub3JlJywgJ2RlZmF1bHQnKToKKyAgICAgICAg X2xvZy53YXJuaW5nKCItLWZvcmNlIG92ZXJyaWRlcyAtLXNraXBwZWQ9JXMiICUgKG9wdGlvbnMu c2tpcHBlZCkpCisgICAgICAgIG9wdGlvbnMuc2tpcHBlZCA9ICdpZ25vcmUnCisKICAgICBpZiBu b3Qgb3B0aW9ucy5odHRwIGFuZCBvcHRpb25zLnNraXBwZWQgaW4gKCdpZ25vcmUnLCAnb25seScp OgogICAgICAgICBfbG9nLndhcm5pbmcoIi0tZm9yY2UvLS1za2lwcGVkPSVzIG92ZXJyaWRlcyAt LW5vLWh0dHAuIiAlIChvcHRpb25zLnNraXBwZWQpKQogICAgICAgICBvcHRpb25zLmh0dHAgPSBU cnVlCg== 215642 2013-10-31 04:13:21 -0700 2013-10-31 11:11:22 -0700 Modifies the behaviour of nrwt --force switch. nrwt-force-switch-implies-pass-expectations.patch text/plain 5475 tgergely.u-szeged ZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBiL1Rvb2xzL0NoYW5nZUxvZwppbmRleCAxOGI0 YTA2Li5lYmMzZTEyIDEwMDY0NAotLS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hh bmdlTG9nCkBAIC0xLDMgKzEsMjIgQEAKKzIwMTMtMTAtMzEgIFRhbWFzIEdlcmdlbHkgIDxnZXJ0 b21AaW5mLnUtc3plZ2VkLmh1PgorCisgICAgICAgIFJ1biB0ZXN0cyBhcyBpZiB0aGV5IGFyZSBl eHBlY3RlZCB0byBwYXNzIHdoZW4gLS1mb3JjZSBpcyBnaXZlbi4KKyAgICAgICAgaHR0cHM6Ly9i dWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTExMjg5MAorCisgICAgICAgIFJldmlld2Vk IGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFRoZSAtLWZvcmNlIHN3aXRjaCBvZiBuZXct cnVuLXdlYmtpdC10ZXN0cyBpcyBub3cgZm9yY2VzIGFsbCB0ZXN0cyB0bworICAgICAgICBiZSBy dW4gd2l0aCAnUEFTUycgYXMgZXhwZWN0ZWQgcmVzdWx0LgorCisgICAgICAgICogU2NyaXB0cy93 ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbWFuYWdlci5weToKKyAgICAgICAgKE1h bmFnZXIucnVuKToKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2Rl bHMvdGVzdF9leHBlY3RhdGlvbnMucHk6CisgICAgICAgIChUZXN0RXhwZWN0YXRpb25zLl9faW5p dF9fKToKKyAgICAgICAgKFRlc3RFeHBlY3RhdGlvbnMuX2FkZF9leHBlY3RhdGlvbnMpOgorICAg ICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHk6 CisgICAgICAgIChwYXJzZV9hcmdzKToKKyAgICAgICAgKF9zZXRfdXBfZGVyaXZlZF9vcHRpb25z KToKKwogMjAxMy0xMC0zMCAgQWxleGV5IFByb3NrdXJ5YWtvdiAgPGFwQGFwcGxlLmNvbT4KIAog ICAgICAgICBUZWFjaCB0aGUgc3R5bGUgY2hlY2tlciBhYm91dCBFTlVNX0NMQVNTCmRpZmYgLS1n aXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9tYW5h Z2VyLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMv bWFuYWdlci5weQppbmRleCA0YzM5YTM0Li4xNjEyZTg0IDEwMDY0NAotLS0gYS9Ub29scy9TY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9tYW5hZ2VyLnB5CisrKyBiL1Rv b2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL21hbmFnZXIucHkK QEAgLTE3OSw3ICsxNzksNyBAQCBjbGFzcyBNYW5hZ2VyKG9iamVjdCk6CiAgICAgICAgICAgICBy ZXR1cm4gdGVzdF9ydW5fcmVzdWx0cy5SdW5EZXRhaWxzKGV4aXRfY29kZT0tMSkKIAogICAgICAg ICBzZWxmLl9wcmludGVyLndyaXRlX3VwZGF0ZSgiUGFyc2luZyBleHBlY3RhdGlvbnMgLi4uIikK LSAgICAgICAgc2VsZi5fZXhwZWN0YXRpb25zID0gdGVzdF9leHBlY3RhdGlvbnMuVGVzdEV4cGVj dGF0aW9ucyhzZWxmLl9wb3J0LCB0ZXN0X25hbWVzKQorICAgICAgICBzZWxmLl9leHBlY3RhdGlv bnMgPSB0ZXN0X2V4cGVjdGF0aW9ucy5UZXN0RXhwZWN0YXRpb25zKHNlbGYuX3BvcnQsIHRlc3Rf bmFtZXMsIGZvcmNlX2V4cGVjdGF0aW9uc19wYXNzPXNlbGYuX29wdGlvbnMuZm9yY2UpCiAKICAg ICAgICAgdGVzdHNfdG9fcnVuLCB0ZXN0c190b19za2lwID0gc2VsZi5fcHJlcGFyZV9saXN0cyhw YXRocywgdGVzdF9uYW1lcykKICAgICAgICAgc2VsZi5fcHJpbnRlci5wcmludF9mb3VuZChsZW4o dGVzdF9uYW1lcyksIGxlbih0ZXN0c190b19ydW4pLCBzZWxmLl9vcHRpb25zLnJlcGVhdF9lYWNo LCBzZWxmLl9vcHRpb25zLml0ZXJhdGlvbnMpCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dl YmtpdHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9leHBlY3RhdGlvbnMucHkgYi9Ub29scy9T Y3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9leHBlY3RhdGlvbnMucHkK aW5kZXggYzhlYTMyMi4uNDgzZDcyMCAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRw eS9sYXlvdXRfdGVzdHMvbW9kZWxzL3Rlc3RfZXhwZWN0YXRpb25zLnB5CisrKyBiL1Rvb2xzL1Nj cmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL21vZGVscy90ZXN0X2V4cGVjdGF0aW9ucy5weQpA QCAtODMzLDcgKzgzMyw3IEBAIGNsYXNzIFRlc3RFeHBlY3RhdGlvbnMob2JqZWN0KToKICAgICAj IEZJWE1FOiBUaGlzIGNvbnN0cnVjdG9yIGRvZXMgdG9vIG11Y2ggd29yay4gV2Ugc2hvdWxkIG1v dmUgdGhlIGFjdHVhbCBwYXJzaW5nIG9mCiAgICAgIyB0aGUgZXhwZWN0YXRpb25zIGludG8gc2Vw YXJhdGUgcm91dGluZXMgc28gdGhhdCBsaW50aW5nIGFuZCBoYW5kbGluZyBvdmVycmlkZXMKICAg ICAjIGNhbiBiZSBjb250cm9sbGVkIHNlcGFyYXRlbHksIGFuZCB0aGUgY29uc3RydWN0b3IgY2Fu IGJlIG1vcmUgb2YgYSBuby1vcC4KLSAgICBkZWYgX19pbml0X18oc2VsZiwgcG9ydCwgdGVzdHM9 Tm9uZSwgaW5jbHVkZV9nZW5lcmljPVRydWUsIGluY2x1ZGVfb3ZlcnJpZGVzPVRydWUsIGV4cGVj dGF0aW9uc190b19saW50PU5vbmUpOgorICAgIGRlZiBfX2luaXRfXyhzZWxmLCBwb3J0LCB0ZXN0 cz1Ob25lLCBpbmNsdWRlX2dlbmVyaWM9VHJ1ZSwgaW5jbHVkZV9vdmVycmlkZXM9VHJ1ZSwgZXhw ZWN0YXRpb25zX3RvX2xpbnQ9Tm9uZSwgZm9yY2VfZXhwZWN0YXRpb25zX3Bhc3M9RmFsc2UpOgog ICAgICAgICBzZWxmLl9mdWxsX3Rlc3RfbGlzdCA9IHRlc3RzCiAgICAgICAgIHNlbGYuX3Rlc3Rf Y29uZmlnID0gcG9ydC50ZXN0X2NvbmZpZ3VyYXRpb24oKQogICAgICAgICBzZWxmLl9pc19saW50 X21vZGUgPSBleHBlY3RhdGlvbnNfdG9fbGludCBpcyBub3QgTm9uZQpAQCAtODQyLDYgKzg0Miw3 IEBAIGNsYXNzIFRlc3RFeHBlY3RhdGlvbnMob2JqZWN0KToKICAgICAgICAgc2VsZi5fcG9ydCA9 IHBvcnQKICAgICAgICAgc2VsZi5fc2tpcHBlZF90ZXN0c193YXJuaW5ncyA9IFtdCiAgICAgICAg IHNlbGYuX2V4cGVjdGF0aW9ucyA9IFtdCisgICAgICAgIHNlbGYuX2ZvcmNlX2V4cGVjdGF0aW9u c19wYXNzID0gZm9yY2VfZXhwZWN0YXRpb25zX3Bhc3MKIAogICAgICAgICBleHBlY3RhdGlvbnNf ZGljdCA9IGV4cGVjdGF0aW9uc190b19saW50IG9yIHBvcnQuZXhwZWN0YXRpb25zX2RpY3QoKQog CkBAIC05OTAsNyArOTkxLDExIEBAIGNsYXNzIFRlc3RFeHBlY3RhdGlvbnMob2JqZWN0KToKIAog ICAgIGRlZiBfYWRkX2V4cGVjdGF0aW9ucyhzZWxmLCBleHBlY3RhdGlvbl9saXN0KToKICAgICAg ICAgZm9yIGV4cGVjdGF0aW9uX2xpbmUgaW4gZXhwZWN0YXRpb25fbGlzdDoKLSAgICAgICAgICAg IGlmIG5vdCBleHBlY3RhdGlvbl9saW5lLmV4cGVjdGF0aW9uczoKKyAgICAgICAgICAgIGlmIHNl bGYuX2ZvcmNlX2V4cGVjdGF0aW9uc19wYXNzOgorICAgICAgICAgICAgICAgIGV4cGVjdGF0aW9u X2xpbmUuZXhwZWN0YXRpb25zID0gWydQQVNTJ10KKyAgICAgICAgICAgICAgICBleHBlY3RhdGlv bl9saW5lLnBhcnNlZF9leHBlY3RhdGlvbnMgPSBzZXQoW1BBU1NdKQorCisgICAgICAgICAgICBl bGlmIG5vdCBleHBlY3RhdGlvbl9saW5lLmV4cGVjdGF0aW9uczoKICAgICAgICAgICAgICAgICBj b250aW51ZQogCiAgICAgICAgICAgICBpZiBzZWxmLl9pc19saW50X21vZGUgb3Igc2VsZi5fdGVz dF9jb25maWcgaW4gZXhwZWN0YXRpb25fbGluZS5tYXRjaGluZ19jb25maWd1cmF0aW9uczoKZGlm ZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRf dGVzdHMucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0 X3Rlc3RzLnB5CmluZGV4IGZkYzIwY2IuLjk2MDM4ZmYgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHkKKysrIGIvVG9vbHMv U2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0cy5weQpAQCAtMjI2 LDggKzIyNiw4IEBAIGRlZiBwYXJzZV9hcmdzKGFyZ3MpOgogICAgICAgICAgICAgICAgICAiJ2ln bm9yZScgPT0gUnVuIHRoZW0gYW55d2F5LCAiCiAgICAgICAgICAgICAgICAgICInb25seScgPT0g b25seSBydW4gdGhlIFNLSVAgdGVzdHMsICIKICAgICAgICAgICAgICAgICAgIidhbHdheXMnID09 IGFsd2F5cyBza2lwLCBldmVuIGlmIGxpc3RlZCBvbiB0aGUgY29tbWFuZCBsaW5lLiIpKSwKLSAg ICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24oIi0tZm9yY2UiLCBkZXN0PSJza2lwcGVkIiwgYWN0 aW9uPSJzdG9yZV9jb25zdCIsIGNvbnN0PSdpZ25vcmUnLAotICAgICAgICAgICAgaGVscD0iUnVu IGFsbCB0ZXN0cywgZXZlbiB0aG9zZSBtYXJrZWQgU0tJUCBpbiB0aGUgdGVzdCBsaXN0IChzYW1l IGFzIC0tc2tpcHBlZD1pZ25vcmUpIiksCisgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCIt LWZvcmNlIiwgYWN0aW9uPSJzdG9yZV90cnVlIiwgZGVmYXVsdD1GYWxzZSwKKyAgICAgICAgICAg IGhlbHA9IlJ1biBhbGwgdGVzdHMgd2l0aCBQQVNTIGFzIGV4cGVjdGVkIHJlc3VsdCwgZXZlbiB0 aG9zZSBtYXJrZWQgU0tJUCBpbiB0aGUgdGVzdCBsaXN0IChpbXBsaWVzIC0tc2tpcHBlZD1pZ25v cmUpIiksCiAgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCItLXRpbWUtb3V0LW1zIiwKICAg ICAgICAgICAgIGhlbHA9IlNldCB0aGUgdGltZW91dCBmb3IgZWFjaCB0ZXN0IiksCiAgICAgICAg IG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCItLW9yZGVyIiwgYWN0aW9uPSJzdG9yZSIsIGRlZmF1bHQ9 Im5hdHVyYWwiLApAQCAtMzM3LDYgKzMzNywxMSBAQCBkZWYgX3NldF91cF9kZXJpdmVkX29wdGlv bnMocG9ydCwgb3B0aW9ucyk6CiAgICAgICAgICAgICBhZGRpdGlvbmFsX3BsYXRmb3JtX2RpcmVj dG9yaWVzLmFwcGVuZChwb3J0Lmhvc3QuZmlsZXN5c3RlbS5hYnNwYXRoKHBhdGgpKQogICAgICAg ICBvcHRpb25zLmFkZGl0aW9uYWxfcGxhdGZvcm1fZGlyZWN0b3J5ID0gYWRkaXRpb25hbF9wbGF0 Zm9ybV9kaXJlY3RvcmllcwogCisgICAgaWYgb3B0aW9ucy5mb3JjZToKKyAgICAgICAgaWYgb3B0 aW9ucy5za2lwcGVkIG5vdCBpbiAoJ2lnbm9yZScsICdkZWZhdWx0Jyk6CisgICAgICAgICAgICBf bG9nLndhcm5pbmcoIi0tZm9yY2Ugb3ZlcnJpZGVzIC0tc2tpcHBlZD0lcyIgJSAob3B0aW9ucy5z a2lwcGVkKSkKKyAgICAgICAgb3B0aW9ucy5za2lwcGVkID0gJ2lnbm9yZScKKwogICAgIGlmIG5v dCBvcHRpb25zLmh0dHAgYW5kIG9wdGlvbnMuc2tpcHBlZCBpbiAoJ2lnbm9yZScsICdvbmx5Jyk6 CiAgICAgICAgIF9sb2cud2FybmluZygiLS1mb3JjZS8tLXNraXBwZWQ9JXMgb3ZlcnJpZGVzIC0t bm8taHR0cC4iICUgKG9wdGlvbnMuc2tpcHBlZCkpCiAgICAgICAgIG9wdGlvbnMuaHR0cCA9IFRy dWUK