226273 2021-05-26 08:26:59 -0700 Layout tests should exit early if stress mode detects a failure 2022-01-25 12:46:44 -0800 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified ASSIGNED https://bugs.webkit.org/show_bug.cgi?id=226097 InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap dewei_zhu jbedard ryanhaddad webkit-bug-importer wilander oldest_to_newest 1763923 0 aakash_jain 2021-05-26 08:26:59 -0700 Layout test build should exit early if stress mode detects a failure, i.e.: we do not need to run the complete layout-test suite (few times) if stress mode indicates that the patch being tested introduced test failure. This would reduce the time to declare failure from few hours to few minutes for the patches which breaks layout-tests while adding/modifying those layout-tests. Stress mode was added on various layout-test queues in https://bugs.webkit.org/show_bug.cgi?id=226097 1763925 1 429762 aakash_jain 2021-05-26 08:33:32 -0700 Created attachment 429762 Patch 1763929 2 aakash_jain 2021-05-26 08:41:37 -0700 Examples: Patch with failure in stress mode (with this early exit): https://ews-build.webkit-uat.org/#/builders/34/builds/31444 (build completed in less than a minute) Patch with failure in non-stress mode: https://ews-build.webkit-uat.org/#/builders/34/builds/31455 (build took ~2 hours), https://ews-build.webkit-uat.org/#/builders/34/builds/31462 (build took ~4 hours) 1763956 3 429762 ap 2021-05-26 10:13:29 -0700 Comment on attachment 429762 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=429762&action=review > Tools/ChangeLog:3 > + Layout tests should exit early if stress mode detects a failure The downside to this change is that for patches that fail stress test AND break existing tests, EWS won't tell people anything about the latter. This will increase the number of times they will have to upload the patch for EWS testing. 1763963 4 aakash_jain 2021-05-26 10:24:40 -0700 That's true. Although they will get the results much faster. So it's a small trade-off. Also, there would still be one queue where we run complete layout-test suite without the stress mode, which is Windows (since it builds and test in same build, and we haven't modified that queue yet). So, if the patch breaks existing tests on windows as well, then the EWS will tell about those failures. 1764128 5 429762 ap 2021-05-26 17:01:58 -0700 Comment on attachment 429762 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=429762&action=review Seems OK to make this change, although ideally, this would produce a red bubble and mark the patch cq-, then run the regular tests anyway. I don't feel entirely qualified to r+ buildbot steps code. > Tools/CISupport/ews-build/factories.py:138 > + self.addStep(RunWebKitTestsInStressMode(num_iterations=10, stopBuildOnFailure=True)) Will TriggerCrashLogSubmission and SetBuildSummary still run if this fails? 1764428 6 aakash_jain 2021-05-27 11:29:32 -0700 (In reply to Alexey Proskuryakov from comment #5) > Seems OK to make this change, although ideally, this would produce a red bubble and mark the patch cq-, then run the regular tests anyway. The purpose of this patch is to save build time by exiting early, which wouldn't be achieved in that case. > Will TriggerCrashLogSubmission and SetBuildSummary still run if this fails? TriggerCrashLogSubmission is not run when layout-tests fails (irrespective of this patch), just filed Bug 226348 for it. SetBuildSummary wouldn't be run, but similar action would be done by the SetCommitQueueMinusFlagOnPatch step. 1764429 7 aakash_jain 2021-05-27 11:33:06 -0700 Another example where this would have saved significant build time (build would have taken 1-2 minutes instead of 3-4 hours, with same output): https://ews-build.webkit.org/#/builders/56/builds/8700, https://ews-build.webkit.org/#/builders/51/builds/14407 1764431 8 jbedard 2021-05-27 11:38:59 -0700 Code looks good, although I'm a bit torn if this is actually the best behavior. I can see this being good in some cases, bad in others, although the efficiency wins probably out-weigh the badness 1764462 9 ryanhaddad 2021-05-27 12:59:56 -0700 (In reply to Aakash Jain from comment #7) > Another example where this would have saved significant build time (build > would have taken 1-2 minutes instead of 3-4 hours, with same output): > https://ews-build.webkit.org/#/builders/56/builds/8700, > https://ews-build.webkit.org/#/builders/51/builds/14407 I share the same concerns about whether or not this will cause additional churn for EWS if the developer sees a failure with the new test, fixes it, then sees failures with existing tests on the next run. These are great examples of the potential for time savings though.. Maybe we can announce our intent to enable this on webkit-dev and see if there are enough concerns to warrant holding off? (In reply to Alexey Proskuryakov from comment #5) > Comment on attachment 429762 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=429762&action=review > > Seems OK to make this change, although ideally, this would produce a red > bubble and mark the patch cq-, then run the regular tests anyway. How difficult would it be to implement this kind of functionality? 1764503 10 ap 2021-05-27 14:23:57 -0700 (In reply to Aakash Jain from comment #6) > (In reply to Alexey Proskuryakov from comment #5) > > Seems OK to make this change, although ideally, this would produce a red bubble and mark the patch cq-, then run the regular tests anyway. > The purpose of this patch is to save build time by exiting early, which > wouldn't be achieved in that case. This is a tradeoff between using bot time and providing all necessary data to the engineer. I guess I didn't fully understand the goal, but to me, the ideal behavior would prioritize engineering time, by providing some results quickly, and providing complete results a little slower. It's similar to how we don't stop all EWSes when one fails - we COULD stop everything if one platform doesn't build, or even if style check fails, but we try to paint a complete picture. 1764506 11 jbedard 2021-05-27 14:35:50 -0700 What if we opted out of retires if stress tests failed? Wouldn't that speed things up considerably? 1766002 12 webkit-bug-importer 2021-06-02 08:27:18 -0700 <rdar://problem/78764585> 1833820 13 aakash_jain 2022-01-25 12:46:36 -0800 (In reply to Ryan Haddad from comment #9) > How difficult would it be to implement this kind of functionality? It's not difficult, although that would defeat the purpose of this patch which was to save time by exiting early. > What if we opted out of retries if stress tests failed? Wouldn't that speed things up considerably? If we go that route, we would still need to do run-layout-tests-without-patch, we might save one run (re-run-layout-tests), but the results might be more flaky. 429762 2021-05-26 08:33:32 -0700 2021-05-27 11:37:41 -0700 Patch bug-226273-20210526113331.patch text/plain 11291 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI3ODA4MykKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE5IEBACisyMDIxLTA1LTI2ICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBMYXlvdXQgdGVzdHMgc2hvdWxkIGV4aXQgZWFybHkgaWYgc3RyZXNzIG1v ZGUgZGV0ZWN0cyBhIGZhaWx1cmUKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hv d19idWcuY2dpP2lkPTIyNjI3MworCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEp LgorCisgICAgICAgICogQ0lTdXBwb3J0L2V3cy1idWlsZC9zdGVwcy5weToKKyAgICAgICAgKFJ1 bldlYktpdFRlc3RzSW5TdHJlc3NNb2RlLl9faW5pdF9fKToKKyAgICAgICAgKFJ1bldlYktpdFRl c3RzSW5TdHJlc3NNb2RlLmV2YWx1YXRlQ29tbWFuZCk6IEFkZGVkIEFuYWx5emVMYXlvdXRUZXN0 c1Jlc3VsdHMgc3RlcCBpbiBjYXNlIG9mIGZhaWx1cmUKKyAgICAgICAgc28gYXMgdG8gZWFybHkg dGhlIGJ1aWxkIGVhcmx5LgorICAgICAgICAoQW5hbHl6ZUxheW91dFRlc3RzUmVzdWx0cy5zdGFy dCk6CisgICAgICAgICogQ0lTdXBwb3J0L2V3cy1idWlsZC9zdGVwc191bml0dGVzdC5weTogQWRk ZWQgYW5kIHVwZGF0ZWQgdW5pdC10ZXN0cy4KKyAgICAgICAgKiBDSVN1cHBvcnQvZXdzLWJ1aWxk L2ZhY3Rvcmllcy5weToKKyAgICAgICAgKiBDSVN1cHBvcnQvZXdzLWJ1aWxkL2ZhY3Rvcmllc191 bml0dGVzdC5weTogVXBkYXRlZCB1bml0LXRlc3RzLgorCiAyMDIxLTA1LTI2ICBKb25hdGhhbiBC ZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNvbT4KIAogICAgICAgICBbdGVzdC1sbGRiLXdlYmtpdF0g VXBkYXRlIGZvciBYY29kZSAxMi41CkluZGV4OiBUb29scy9DSVN1cHBvcnQvZXdzLWJ1aWxkL2Zh Y3Rvcmllcy5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DSVN1cHBvcnQvZXdzLWJ1aWxkL2ZhY3Rv cmllcy5weQkocmV2aXNpb24gMjc4MDgzKQorKysgVG9vbHMvQ0lTdXBwb3J0L2V3cy1idWlsZC9m YWN0b3JpZXMucHkJKHdvcmtpbmcgY29weSkKQEAgLTEzNSw3ICsxMzUsNyBAQCBjbGFzcyBUZXN0 RmFjdG9yeShGYWN0b3J5KToKICAgICAgICAgc2VsZi5hZGRTdGVwKEtpbGxPbGRQcm9jZXNzZXMo KSkKICAgICAgICAgaWYgc2VsZi5MYXlvdXRUZXN0Q2xhc3M6CiAgICAgICAgICAgICBzZWxmLmFk ZFN0ZXAoRmluZE1vZGlmaWVkTGF5b3V0VGVzdHMoc2tpcEJ1aWxkSWZOb1Jlc3VsdD1GYWxzZSkp Ci0gICAgICAgICAgICBzZWxmLmFkZFN0ZXAoUnVuV2ViS2l0VGVzdHNJblN0cmVzc01vZGUobnVt X2l0ZXJhdGlvbnM9MTApKQorICAgICAgICAgICAgc2VsZi5hZGRTdGVwKFJ1bldlYktpdFRlc3Rz SW5TdHJlc3NNb2RlKG51bV9pdGVyYXRpb25zPTEwLCBzdG9wQnVpbGRPbkZhaWx1cmU9VHJ1ZSkp CiAgICAgICAgICAgICBzZWxmLmFkZFN0ZXAoc2VsZi5MYXlvdXRUZXN0Q2xhc3MoKSkKICAgICAg ICAgaWYgc2VsZi5BUElUZXN0Q2xhc3M6CiAgICAgICAgICAgICBzZWxmLmFkZFN0ZXAoc2VsZi5B UElUZXN0Q2xhc3MoKSkKSW5kZXg6IFRvb2xzL0NJU3VwcG9ydC9ld3MtYnVpbGQvZmFjdG9yaWVz X3VuaXR0ZXN0LnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL0NJU3VwcG9ydC9ld3MtYnVpbGQvZmFj dG9yaWVzX3VuaXR0ZXN0LnB5CShyZXZpc2lvbiAyNzgwODMpCisrKyBUb29scy9DSVN1cHBvcnQv ZXdzLWJ1aWxkL2ZhY3Rvcmllc191bml0dGVzdC5weQkod29ya2luZyBjb3B5KQpAQCAtNDU3LDcg KzQ1Nyw3IEBAIGNsYXNzIFRlc3RMYXlvdXRUZXN0c0ZhY3RvcnkoVGVzdENhc2UpOgogICAgICAg ICAgICAgX0J1aWxkU3RlcEZhY3Rvcnkoc3RlcHMuV2FpdEZvckNyYXNoQ29sbGVjdGlvbiksCiAg ICAgICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShzdGVwcy5LaWxsT2xkUHJvY2Vzc2VzKSwKICAg ICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0ZXBzLkZpbmRNb2RpZmllZExheW91dFRlc3Rz LCBza2lwQnVpbGRJZk5vUmVzdWx0PUZhbHNlKSwKLSAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0 b3J5KHN0ZXBzLlJ1bldlYktpdFRlc3RzSW5TdHJlc3NNb2RlLCBudW1faXRlcmF0aW9ucz0xMCks CisgICAgICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShzdGVwcy5SdW5XZWJLaXRUZXN0c0luU3Ry ZXNzTW9kZSwgbnVtX2l0ZXJhdGlvbnM9MTAsIHN0b3BCdWlsZE9uRmFpbHVyZT1UcnVlKSwKICAg ICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0ZXBzLlJ1bldlYktpdDFUZXN0cyksCiAgICAg ICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShzdGVwcy5UcmlnZ2VyQ3Jhc2hMb2dTdWJtaXNzaW9u KSwKICAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0ZXBzLlNldEJ1aWxkU3VtbWFyeSks CkBAIC00ODEsNyArNDgxLDcgQEAgY2xhc3MgVGVzdExheW91dFRlc3RzRmFjdG9yeShUZXN0Q2Fz ZSk6CiAgICAgICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShzdGVwcy5XYWl0Rm9yQ3Jhc2hDb2xs ZWN0aW9uKSwKICAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0ZXBzLktpbGxPbGRQcm9j ZXNzZXMpLAogICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rvcnkoc3RlcHMuRmluZE1vZGlmaWVk TGF5b3V0VGVzdHMsIHNraXBCdWlsZElmTm9SZXN1bHQ9RmFsc2UpLAotICAgICAgICAgICAgX0J1 aWxkU3RlcEZhY3Rvcnkoc3RlcHMuUnVuV2ViS2l0VGVzdHNJblN0cmVzc01vZGUsIG51bV9pdGVy YXRpb25zPTEwKSwKKyAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0ZXBzLlJ1bldlYktp dFRlc3RzSW5TdHJlc3NNb2RlLCBudW1faXRlcmF0aW9ucz0xMCwgc3RvcEJ1aWxkT25GYWlsdXJl PVRydWUpLAogICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rvcnkoc3RlcHMuUnVuV2ViS2l0MVRl c3RzKSwKICAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0ZXBzLlRyaWdnZXJDcmFzaExv Z1N1Ym1pc3Npb24pLAogICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rvcnkoc3RlcHMuU2V0QnVp bGRTdW1tYXJ5KSwKQEAgLTUwNCw3ICs1MDQsNyBAQCBjbGFzcyBUZXN0TGF5b3V0VGVzdHNGYWN0 b3J5KFRlc3RDYXNlKToKICAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0ZXBzLldhaXRG b3JDcmFzaENvbGxlY3Rpb24pLAogICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rvcnkoc3RlcHMu S2lsbE9sZFByb2Nlc3NlcyksCiAgICAgICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShzdGVwcy5G aW5kTW9kaWZpZWRMYXlvdXRUZXN0cywgc2tpcEJ1aWxkSWZOb1Jlc3VsdD1GYWxzZSksCi0gICAg ICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShzdGVwcy5SdW5XZWJLaXRUZXN0c0luU3RyZXNzTW9k ZSwgbnVtX2l0ZXJhdGlvbnM9MTApLAorICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rvcnkoc3Rl cHMuUnVuV2ViS2l0VGVzdHNJblN0cmVzc01vZGUsIG51bV9pdGVyYXRpb25zPTEwLCBzdG9wQnVp bGRPbkZhaWx1cmU9VHJ1ZSksCiAgICAgICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShzdGVwcy5S dW5XZWJLaXRUZXN0cyksCiAgICAgICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShzdGVwcy5Ucmln Z2VyQ3Jhc2hMb2dTdWJtaXNzaW9uKSwKICAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0 ZXBzLlNldEJ1aWxkU3VtbWFyeSksCkBAIC01MjcsNyArNTI3LDcgQEAgY2xhc3MgVGVzdExheW91 dFRlc3RzRmFjdG9yeShUZXN0Q2FzZSk6CiAgICAgICAgICAgICBfQnVpbGRTdGVwRmFjdG9yeShz dGVwcy5XYWl0Rm9yQ3Jhc2hDb2xsZWN0aW9uKSwKICAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0 b3J5KHN0ZXBzLktpbGxPbGRQcm9jZXNzZXMpLAogICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rv cnkoc3RlcHMuRmluZE1vZGlmaWVkTGF5b3V0VGVzdHMsIHNraXBCdWlsZElmTm9SZXN1bHQ9RmFs c2UpLAotICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rvcnkoc3RlcHMuUnVuV2ViS2l0VGVzdHNJ blN0cmVzc01vZGUsIG51bV9pdGVyYXRpb25zPTEwKSwKKyAgICAgICAgICAgIF9CdWlsZFN0ZXBG YWN0b3J5KHN0ZXBzLlJ1bldlYktpdFRlc3RzSW5TdHJlc3NNb2RlLCBudW1faXRlcmF0aW9ucz0x MCwgc3RvcEJ1aWxkT25GYWlsdXJlPVRydWUpLAogICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rv cnkoc3RlcHMuUnVuV2ViS2l0VGVzdHMpLAogICAgICAgICAgICAgX0J1aWxkU3RlcEZhY3Rvcnko c3RlcHMuVHJpZ2dlckNyYXNoTG9nU3VibWlzc2lvbiksCiAgICAgICAgICAgICBfQnVpbGRTdGVw RmFjdG9yeShzdGVwcy5TZXRCdWlsZFN1bW1hcnkpLApAQCAtNTUwLDcgKzU1MCw3IEBAIGNsYXNz IFRlc3RMYXlvdXRUZXN0c0ZhY3RvcnkoVGVzdENhc2UpOgogICAgICAgICAgICAgX0J1aWxkU3Rl cEZhY3Rvcnkoc3RlcHMuRXh0cmFjdEJ1aWx0UHJvZHVjdCksCiAgICAgICAgICAgICBfQnVpbGRT dGVwRmFjdG9yeShzdGVwcy5LaWxsT2xkUHJvY2Vzc2VzKSwKICAgICAgICAgICAgIF9CdWlsZFN0 ZXBGYWN0b3J5KHN0ZXBzLkZpbmRNb2RpZmllZExheW91dFRlc3RzLCBza2lwQnVpbGRJZk5vUmVz dWx0PUZhbHNlKSwKLSAgICAgICAgICAgIF9CdWlsZFN0ZXBGYWN0b3J5KHN0ZXBzLlJ1bldlYktp dFRlc3RzSW5TdHJlc3NNb2RlLCBudW1faXRlcmF0aW9ucz0xMCksCisgICAgICAgICAgICBfQnVp bGRTdGVwRmFjdG9yeShzdGVwcy5SdW5XZWJLaXRUZXN0c0luU3RyZXNzTW9kZSwgbnVtX2l0ZXJh dGlvbnM9MTAsIHN0b3BCdWlsZE9uRmFpbHVyZT1UcnVlKSwKICAgICAgICAgICAgIF9CdWlsZFN0 ZXBGYWN0b3J5KHN0ZXBzLlJ1bldlYktpdFRlc3RzKSwKICAgICAgICAgICAgIF9CdWlsZFN0ZXBG YWN0b3J5KHN0ZXBzLlNldEJ1aWxkU3VtbWFyeSksCiAgICAgICAgIF0pCkluZGV4OiBUb29scy9D SVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL0NJU3VwcG9y dC9ld3MtYnVpbGQvc3RlcHMucHkJKHJldmlzaW9uIDI3ODA4MykKKysrIFRvb2xzL0NJU3VwcG9y dC9ld3MtYnVpbGQvc3RlcHMucHkJKHdvcmtpbmcgY29weSkKQEAgLTIyOTQsOCArMjI5NCw5IEBA IGNsYXNzIFJ1bldlYktpdFRlc3RzSW5TdHJlc3NNb2RlKFJ1bldlYksKICAgICBzdWZmaXggPSAn c3RyZXNzLW1vZGUnCiAgICAgRVhJVF9BRlRFUl9GQUlMVVJFUyA9ICcxMCcKIAotICAgIGRlZiBf X2luaXRfXyhzZWxmLCBudW1faXRlcmF0aW9ucz0xMDApOgorICAgIGRlZiBfX2luaXRfXyhzZWxm LCBudW1faXRlcmF0aW9ucz0xMDAsIHN0b3BCdWlsZE9uRmFpbHVyZT1GYWxzZSk6CiAgICAgICAg IHNlbGYubnVtX2l0ZXJhdGlvbnMgPSBudW1faXRlcmF0aW9ucworICAgICAgICBzZWxmLnN0b3BC dWlsZE9uRmFpbHVyZSA9IHN0b3BCdWlsZE9uRmFpbHVyZQogICAgICAgICBzdXBlcihSdW5XZWJL aXRUZXN0c0luU3RyZXNzTW9kZSwgc2VsZikuX19pbml0X18oKQogCiAgICAgZGVmIHNldExheW91 dFRlc3RDb21tYW5kKHNlbGYpOgpAQCAtMjMxNSwxMSArMjMxNiwxMSBAQCBjbGFzcyBSdW5XZWJL aXRUZXN0c0luU3RyZXNzTW9kZShSdW5XZWJLCiAgICAgICAgICAgICBzZWxmLnNldFByb3BlcnR5 KCdidWlsZF9zdW1tYXJ5JywgbWVzc2FnZSkKICAgICAgICAgZWxzZToKICAgICAgICAgICAgIHNl bGYuc2V0UHJvcGVydHkoJ2J1aWxkX3N1bW1hcnknLCAnRm91bmQgdGVzdCBmYWlsdXJlcycpCi0g ICAgICAgICAgICBzZWxmLmJ1aWxkLmFkZFN0ZXBzQWZ0ZXJDdXJyZW50U3RlcChbCi0gICAgICAg ICAgICAgICAgQXJjaGl2ZVRlc3RSZXN1bHRzKCksCi0gICAgICAgICAgICAgICAgVXBsb2FkVGVz dFJlc3VsdHMoaWRlbnRpZmllcj1zZWxmLnN1ZmZpeCksCi0gICAgICAgICAgICAgICAgRXh0cmFj dFRlc3RSZXN1bHRzKGlkZW50aWZpZXI9c2VsZi5zdWZmaXgpLAotICAgICAgICAgICAgXSkKKyAg ICAgICAgICAgIHNlbGYuc2V0UHJvcGVydHkoJ3N0cmVzc19tb2RlX2ZhaWx1cmVzJywgVHJ1ZSkK KyAgICAgICAgICAgIHN0ZXBzX3RvX2FkZCA9IFtBcmNoaXZlVGVzdFJlc3VsdHMoKSwgVXBsb2Fk VGVzdFJlc3VsdHMoaWRlbnRpZmllcj1zZWxmLnN1ZmZpeCksIEV4dHJhY3RUZXN0UmVzdWx0cyhp ZGVudGlmaWVyPXNlbGYuc3VmZml4KV0KKyAgICAgICAgICAgIGlmIHNlbGYuc3RvcEJ1aWxkT25G YWlsdXJlOgorICAgICAgICAgICAgICAgIHN0ZXBzX3RvX2FkZC5hcHBlbmQoQW5hbHl6ZUxheW91 dFRlc3RzUmVzdWx0cygpKQorICAgICAgICAgICAgc2VsZi5idWlsZC5hZGRTdGVwc0FmdGVyQ3Vy cmVudFN0ZXAoc3RlcHNfdG9fYWRkKQogICAgICAgICByZXR1cm4gcmMKIAogICAgIGRlZiBkb1N0 ZXBJZihzZWxmLCBzdGVwKToKQEAgLTI1NjYsNiArMjU2Nyw5IEBAIGNsYXNzIEFuYWx5emVMYXlv dXRUZXN0c1Jlc3VsdHMoYnVpbGRzdGUKICAgICAgICAgY2xlYW5fdHJlZV9yZXN1bHRzX2ZhaWxp bmdfdGVzdHMgPSBzZXQoc2VsZi5nZXRQcm9wZXJ0eSgnY2xlYW5fdHJlZV9ydW5fZmFpbHVyZXMn LCBbXSkpCiAgICAgICAgIGZsYWt5X2ZhaWx1cmVzID0gZmlyc3RfcmVzdWx0c19mYWlsaW5nX3Rl c3RzLnVuaW9uKHNlY29uZF9yZXN1bHRzX2ZhaWxpbmdfdGVzdHMpIC0gZmlyc3RfcmVzdWx0c19m YWlsaW5nX3Rlc3RzLmludGVyc2VjdGlvbihzZWNvbmRfcmVzdWx0c19mYWlsaW5nX3Rlc3RzKQog CisgICAgICAgIGlmIHNlbGYuZ2V0UHJvcGVydHkoJ3N0cmVzc19tb2RlX2ZhaWx1cmVzJywgRmFs c2UpOgorICAgICAgICAgICAgcmV0dXJuIHNlbGYucmVwb3J0X2ZhaWx1cmUoZmlyc3RfcmVzdWx0 c19mYWlsaW5nX3Rlc3RzKQorCiAgICAgICAgIGlmIChub3QgZmlyc3RfcmVzdWx0c19mYWlsaW5n X3Rlc3RzKSBhbmQgKG5vdCBzZWNvbmRfcmVzdWx0c19mYWlsaW5nX3Rlc3RzKToKICAgICAgICAg ICAgICMgSWYgd2UndmUgbWFkZSBpdCBoZXJlLCB0aGVuIGxheW91dC10ZXN0cyBhbmQgcmUtcnVu LWxheW91dC10ZXN0cyBmYWlsZWQsIHdoaWNoIG1lYW5zCiAgICAgICAgICAgICAjIHRoZXJlIHNo b3VsZCBoYXZlIGJlZW4gc29tZSB0ZXN0IGZhaWx1cmVzLiBPdGhlcndpc2UgdGhlcmUgaXMgc29t ZSB1bmV4cGVjdGVkIGlzc3VlLgpJbmRleDogVG9vbHMvQ0lTdXBwb3J0L2V3cy1idWlsZC9zdGVw c191bml0dGVzdC5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DSVN1cHBvcnQvZXdzLWJ1aWxkL3N0 ZXBzX3VuaXR0ZXN0LnB5CShyZXZpc2lvbiAyNzgwODMpCisrKyBUb29scy9DSVN1cHBvcnQvZXdz LWJ1aWxkL3N0ZXBzX3VuaXR0ZXN0LnB5CSh3b3JraW5nIGNvcHkpCkBAIC0xOTQ5LDcgKzE5NDks MTAgQEAgY2xhc3MgVGVzdFJ1bldlYktpdFRlc3RzSW5TdHJlc3NNb2RlKEJ1aQogICAgICAgICAg ICAgKyAwLAogICAgICAgICApCiAgICAgICAgIHNlbGYuZXhwZWN0T3V0Y29tZShyZXN1bHQ9U1VD Q0VTUywgc3RhdGVfc3RyaW5nPSdQYXNzZWQgbGF5b3V0IHRlc3RzJykKLSAgICAgICAgcmV0dXJu IHNlbGYucnVuU3RlcCgpCisgICAgICAgIHJjID0gc2VsZi5ydW5TdGVwKCkKKyAgICAgICAgc2Vs Zi5hc3NlcnRFcXVhbChzZWxmLmdldFByb3BlcnR5KCdidWlsZF9zdW1tYXJ5JyksICdQYXNzZWQg bGF5b3V0IHRlc3RzJykKKyAgICAgICAgc2VsZi5hc3NlcnRFcXVhbChzZWxmLmdldFByb3BlcnR5 KCdzdHJlc3NfbW9kZV9mYWlsdXJlcycpLCBOb25lKQorICAgICAgICByZXR1cm4gcmMKIAogICAg IGRlZiB0ZXN0X2ZhaWx1cmUoc2VsZik6CiAgICAgICAgIHNlbGYuY29uZmlndXJlU3RlcCgpCkBA IC0xOTczLDYgKzE5NzYsNyBAQCBjbGFzcyBUZXN0UnVuV2ViS2l0VGVzdHNJblN0cmVzc01vZGUo QnVpCiAgICAgICAgIHNlbGYuZXhwZWN0T3V0Y29tZShyZXN1bHQ9RkFJTFVSRSwgc3RhdGVfc3Ry aW5nPSdsYXlvdXQtdGVzdHMgKGZhaWx1cmUpJykKICAgICAgICAgcmMgPSBzZWxmLnJ1blN0ZXAo KQogICAgICAgICBzZWxmLmFzc2VydEVxdWFsKHNlbGYuZ2V0UHJvcGVydHkoJ2J1aWxkX3N1bW1h cnknKSwgJ0ZvdW5kIHRlc3QgZmFpbHVyZXMnKQorICAgICAgICBzZWxmLmFzc2VydEVxdWFsKHNl bGYuZ2V0UHJvcGVydHkoJ3N0cmVzc19tb2RlX2ZhaWx1cmVzJyksIFRydWUpCiAgICAgICAgIHJl dHVybiByYwogCiAKQEAgLTIwMDksNyArMjAxMywxMCBAQCBjbGFzcyBUZXN0UnVuV2ViS2l0VGVz dHNJblN0cmVzc0d1YXJkbWFsCiAgICAgICAgICAgICArIDAsCiAgICAgICAgICkKICAgICAgICAg c2VsZi5leHBlY3RPdXRjb21lKHJlc3VsdD1TVUNDRVNTLCBzdGF0ZV9zdHJpbmc9J1Bhc3NlZCBs YXlvdXQgdGVzdHMnKQotICAgICAgICByZXR1cm4gc2VsZi5ydW5TdGVwKCkKKyAgICAgICAgcmMg PSBzZWxmLnJ1blN0ZXAoKQorICAgICAgICBzZWxmLmFzc2VydEVxdWFsKHNlbGYuZ2V0UHJvcGVy dHkoJ2J1aWxkX3N1bW1hcnknKSwgJ1Bhc3NlZCBsYXlvdXQgdGVzdHMnKQorICAgICAgICBzZWxm LmFzc2VydEVxdWFsKHNlbGYuZ2V0UHJvcGVydHkoJ3N0cmVzc19tb2RlX2ZhaWx1cmVzJyksIE5v bmUpCisgICAgICAgIHJldHVybiByYwogCiAgICAgZGVmIHRlc3RfZmFpbHVyZShzZWxmKToKICAg ICAgICAgc2VsZi5jb25maWd1cmVTdGVwKCkKQEAgLTIwMzMsNiArMjA0MCw3IEBAIGNsYXNzIFRl c3RSdW5XZWJLaXRUZXN0c0luU3RyZXNzR3VhcmRtYWwKICAgICAgICAgc2VsZi5leHBlY3RPdXRj b21lKHJlc3VsdD1GQUlMVVJFLCBzdGF0ZV9zdHJpbmc9J2xheW91dC10ZXN0cyAoZmFpbHVyZSkn KQogICAgICAgICByYyA9IHNlbGYucnVuU3RlcCgpCiAgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWwo c2VsZi5nZXRQcm9wZXJ0eSgnYnVpbGRfc3VtbWFyeScpLCAnRm91bmQgdGVzdCBmYWlsdXJlcycp CisgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWwoc2VsZi5nZXRQcm9wZXJ0eSgnc3RyZXNzX21vZGVf ZmFpbHVyZXMnKSwgVHJ1ZSkKICAgICAgICAgcmV0dXJuIHJjCiAKIApAQCAtMjI4NSwxOCArMjI5 MywyOCBAQCBjbGFzcyBUZXN0QW5hbHl6ZUxheW91dFRlc3RzUmVzdWx0cyhCdWlsCiAgICAgICAg IHNlbGYuc2V0UHJvcGVydHkoJ2NsZWFuX3RyZWVfcmVzdWx0c19leGNlZWRfZmFpbHVyZV9saW1p dCcsIEZhbHNlKQogICAgICAgICBzZWxmLnNldFByb3BlcnR5KCdjbGVhbl90cmVlX3J1bl9mYWls dXJlcycsIFtdKQogCisgICAgZGVmIHRlc3Rfc3RyZXNzX21vZGVfZmFpbHVyZShzZWxmKToKKyAg ICAgICAgc2VsZi5jb25maWd1cmVTdGVwKCkKKyAgICAgICAgc2VsZi5zZXRQcm9wZXJ0eSgnc3Ry ZXNzX21vZGVfZmFpbHVyZXMnLCBUcnVlKQorICAgICAgICBzZWxmLnNldFByb3BlcnR5KCdmaXJz dF9ydW5fZmFpbHVyZXMnLCBbJ2pxdWVyeS9vZmZzZXQuaHRtbCddKQorICAgICAgICBzZWxmLmV4 cGVjdE91dGNvbWUocmVzdWx0PUZBSUxVUkUsIHN0YXRlX3N0cmluZz0nRm91bmQgMSBuZXcgdGVz dCBmYWlsdXJlOiBqcXVlcnkvb2Zmc2V0Lmh0bWwgKGZhaWx1cmUpJykKKyAgICAgICAgcmMgPSBz ZWxmLnJ1blN0ZXAoKQorICAgICAgICBzZWxmLmFzc2VydEVxdWFsKHNlbGYuZ2V0UHJvcGVydHko J2J1Z3ppbGxhX2NvbW1lbnRfdGV4dCcpLCBOb25lKQorICAgICAgICBzZWxmLmFzc2VydEVxdWFs KHNlbGYuZ2V0UHJvcGVydHkoJ2J1aWxkX2ZpbmlzaF9zdW1tYXJ5JyksICdGb3VuZCAxIG5ldyB0 ZXN0IGZhaWx1cmU6IGpxdWVyeS9vZmZzZXQuaHRtbCcpCisgICAgICAgIHJldHVybiByYworCiAg ICAgZGVmIHRlc3RfZmFpbHVyZV9pbnRyb2R1Y2VkX2J5X3BhdGNoKHNlbGYpOgogICAgICAgICBz ZWxmLmNvbmZpZ3VyZVN0ZXAoKQotICAgICAgICBzZWxmLnNldFByb3BlcnR5KCdmaXJzdF9ydW5f ZmFpbHVyZXMnLCBbImpxdWVyeS9vZmZzZXQuaHRtbCJdKQotICAgICAgICBzZWxmLnNldFByb3Bl cnR5KCdzZWNvbmRfcnVuX2ZhaWx1cmVzJywgWyJqcXVlcnkvb2Zmc2V0Lmh0bWwiXSkKKyAgICAg ICAgc2VsZi5zZXRQcm9wZXJ0eSgnZmlyc3RfcnVuX2ZhaWx1cmVzJywgWydqcXVlcnkvb2Zmc2V0 Lmh0bWwnXSkKKyAgICAgICAgc2VsZi5zZXRQcm9wZXJ0eSgnc2Vjb25kX3J1bl9mYWlsdXJlcycs IFsnanF1ZXJ5L29mZnNldC5odG1sJ10pCiAgICAgICAgIHNlbGYuZXhwZWN0T3V0Y29tZShyZXN1 bHQ9RkFJTFVSRSwgc3RhdGVfc3RyaW5nPSdGb3VuZCAxIG5ldyB0ZXN0IGZhaWx1cmU6IGpxdWVy eS9vZmZzZXQuaHRtbCAoZmFpbHVyZSknKQogICAgICAgICByZXR1cm4gc2VsZi5ydW5TdGVwKCkK IAogICAgIGRlZiB0ZXN0X2ZhaWx1cmVfb25fY2xlYW5fdHJlZShzZWxmKToKICAgICAgICAgc2Vs Zi5jb25maWd1cmVTdGVwKCkKLSAgICAgICAgc2VsZi5zZXRQcm9wZXJ0eSgnZmlyc3RfcnVuX2Zh aWx1cmVzJywgWyJqcXVlcnkvb2Zmc2V0Lmh0bWwiXSkKLSAgICAgICAgc2VsZi5zZXRQcm9wZXJ0 eSgnc2Vjb25kX3J1bl9mYWlsdXJlcycsIFsianF1ZXJ5L29mZnNldC5odG1sIl0pCi0gICAgICAg IHNlbGYuc2V0UHJvcGVydHkoJ2NsZWFuX3RyZWVfcnVuX2ZhaWx1cmVzJywgWyJqcXVlcnkvb2Zm c2V0Lmh0bWwiXSkKKyAgICAgICAgc2VsZi5zZXRQcm9wZXJ0eSgnZmlyc3RfcnVuX2ZhaWx1cmVz JywgWydqcXVlcnkvb2Zmc2V0Lmh0bWwnXSkKKyAgICAgICAgc2VsZi5zZXRQcm9wZXJ0eSgnc2Vj b25kX3J1bl9mYWlsdXJlcycsIFsnanF1ZXJ5L29mZnNldC5odG1sJ10pCisgICAgICAgIHNlbGYu c2V0UHJvcGVydHkoJ2NsZWFuX3RyZWVfcnVuX2ZhaWx1cmVzJywgWydqcXVlcnkvb2Zmc2V0Lmh0 bWwnXSkKICAgICAgICAgc2VsZi5leHBlY3RPdXRjb21lKHJlc3VsdD1TVUNDRVNTLCBzdGF0ZV9z dHJpbmc9J1Bhc3NlZCBsYXlvdXQgdGVzdHMnKQogICAgICAgICByYyA9IHNlbGYucnVuU3RlcCgp CiAgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWwoc2VsZi5nZXRQcm9wZXJ0eSgnYnVpbGRfc3VtbWFy eScpLCAnRm91bmQgMSBwcmUtZXhpc3RpbmcgdGVzdCBmYWlsdXJlOiBqcXVlcnkvb2Zmc2V0Lmh0 bWwnKQo=