91089 2012-07-12 06:33:11 -0700 [NRWT] runs platform specific tests that it shouldn't with --force 2012-08-07 00:47:44 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED NRWT P2 Normal --- 1 kbalazs kbalazs abarth dglazkov dpranke eric galpeter jesus kkristof ojan ossy tony webkit.review.bot oldest_to_newest 667715 0 kbalazs 2012-07-12 06:33:11 -0700 I'm using this with a Qt build (Qt 5): Tools/Scripts/run-webkit-tests --no-new-test-results --pixel-tests -2 --new-baseline --force LayoutTests/compositing/ And I get results like this: Writing new expected result "platform/qt-5.0-wk2/platform/chromium/compositing/3d-corners-expected.txt Before that, I got the results for the generic tests in compositing as expected, but than it starts to run tests from platform/chromium/compositing. This seems to be a bug in platform detection. Using --additional-platform-directory=my/wk/path/LayoutTests/platform/qt-5.0-wk2 as Ossy adviced to me does not help. 667722 1 kbalazs 2012-07-12 06:37:16 -0700 Hm, that's even worst, it's not just an issue with --new-baseline but generally if I pass LayoutTests/compositing nrwt runs tests from LayoutTests/platform/chromium/compositing. 668509 2 dpranke 2012-07-12 19:08:02 -0700 I think this was probably introduced when I started running the platform-specific additions to the generic directories in bug 37956. Will take a look. 668545 3 dpranke 2012-07-12 20:06:20 -0700 Ah, I see what's going on . The way things are implemented in NRWT, we find *all* of the tests in the tree, and then add SKIP expectations for the tests we don't want to run (as returned by Port.skipped_layout_tests). That means that there is a SKIP expectation for platform/chromium. However, you're running with --force, which says to run all tests, even if they're skipped :). I'm not sure if I consider this a bug so much as an unfortunate interaction of the three features (finding platform-specific versions of tests for a generic directory, implicitly skipping platforms other than your own, and --force). I think you can work around this by -i/--ignore-tests platform/chromium, but --force might actually trump everything, and I'm not sure that that's bad. Thoughts? 668851 4 kbalazs 2012-07-13 04:18:59 -0700 I see, I agree it's actually not a bug. It seems the right behavior of --force to run these tests. However, when using --force to check what can be removed from skip list, this behavior is a bit uncomfortable. I don't think ports want to run tests from LayoutTests/platform other than the ones that belongs to them. Yesterday I made this tiny change locally: diff --git a/Tools/Scripts/webkitpy/layout_tests/port/base.py b/Tools/Scripts/webkitpy/layout_tests/port/base.py index fbf0b93..3d28dbc 100755 --- a/Tools/Scripts/webkitpy/layout_tests/port/base.py +++ b/Tools/Scripts/webkitpy/layout_tests/port/base.py @@ -506,7 +506,7 @@ class Port(object): expanded_paths.append(path) if self.test_isdir(path) and not path.startswith('platform'): for platform_dir in all_platform_dirs: - if fs.isdir(fs.join(platform_dir, path)): + if fs.isdir(fs.join(platform_dir, path)) and platform_dir in self.baseline_search_path(): expanded_paths.append(self.relative_test_filename(fs.join(platform_dir, path))) return expanded_paths With this change I believe we are still be able to run other platform's tests with --additional-platform-directory. So, the question is whether it is worth to mess up the semantic of --force a bit in favor of easier skip list maintenance. I'm curious about other's opinion. 669151 5 dpranke 2012-07-13 09:57:11 -0700 Personally I find the "run the platform-specific versions of the generic directory, too" feature a little goofy, but then that's probably because I spend my time hacking webkitpy and not webkit :). Your suggested change had occurred to me as well; it doesn't actually change what --force does, and I'm fine with it, so if we want to make that change, great! I agree that people aren't likely to complain, since if you really wanted to run platform/chromium/compositing all you'd have to do would be to add it on the command line (or use --additional-platform-dir, as you say). 669253 6 eric 2012-07-13 11:04:46 -0700 We should just get rid of these tests in platform/ Move them to nicely named direcxtories and skip them everywhere they're not needed. having tests in platform/ is a very old solution to this problem, which I think causes more confusion than help. 684848 7 dpranke 2012-08-01 18:27:46 -0700 I'm inclined to WONTFIX this ... any objections? 685050 8 kbalazs 2012-08-02 00:33:38 -0700 (In reply to comment #7) > I'm inclined to WONTFIX this ... any objections? I would like to see this fixed. I couldn't get sort of it yet. Btw, do we have a consensus that we should remove tests from platform directories? It's much more comfortable to add a test to a platform directory than skipping on every other platform, although it is a questionable behavior to add a test for only one platform. 685487 9 dpranke 2012-08-02 09:33:32 -0700 (In reply to comment #8) > (In reply to comment #7) > > I'm inclined to WONTFIX this ... any objections? > > I would like to see this fixed. I couldn't get sort of it yet. Btw, do we have a consensus that we should remove tests from platform directories? It's much more comfortable to add a test to a platform directory than skipping on every other platform, although it is a questionable behavior to add a test for only one platform. Whoops, sorry, I forgot abou the change you suggested in comment #4. I'm still fine with that, if you want to make it. As to getting rid of platform directories, I would leave that for a separate discussion. 685607 10 dpranke 2012-08-02 11:22:39 -0700 Oh, hmm, I thought of something I don't like about your suggestion. The way things are currently implemented, tests() returns the same list independent of the port; that way the statistics we report for total # of files is the same across ports, and just the # of tests that skip varies, and so changing that seems less good. In particular, when you have multiple variants of the same port (e.g., chromium, or qt), it's nice to be able to compare how many tests are being skipped on each variant. Although, re-thinking it, this isn't actually quite true with the addition of virtual test suite, since ports with different lists of virtual suites will have different numbers of tests. Currently, the only port that uses virtual tests is chromium, but at least there all of the variants use the same list of suites, so at least I keep the last sentence true. So, I still have a slight preference for leaving things the way they are, I guess. But if you'd strongly prefer to make the change, I won't be that upset by it. 685744 11 kbalazs 2012-08-02 13:37:10 -0700 (In reply to comment #10) > Oh, hmm, I thought of something I don't like about your suggestion. The way things are currently implemented, tests() returns the same list independent of the port; that way the statistics we report for total # of files is the same across ports, and just the # of tests that skip varies, and so changing that seems less good. In particular, when you have multiple variants of the same port (e.g., chromium, or qt), it's nice to be able to compare how many tests are being skipped on each variant. > > Although, re-thinking it, this isn't actually quite true with the addition of virtual test suite, since ports with different lists of virtual suites will have different numbers of tests. Currently, the only port that uses virtual tests is chromium, but at least there all of the variants use the same list of suites, so at least I keep the last sentence true. > > So, I still have a slight preference for leaving things the way they are, I guess. But if you'd strongly prefer to make the change, I won't be that upset by it. Ok, than I think we should get rid of platform tests instead. 685791 12 dpranke 2012-08-02 14:02:58 -0700 (In reply to comment #11) > (In reply to comment #10) > > Oh, hmm, I thought of something I don't like about your suggestion. The way things are currently implemented, tests() returns the same list independent of the port; that way the statistics we report for total # of files is the same across ports, and just the # of tests that skip varies, and so changing that seems less good. In particular, when you have multiple variants of the same port (e.g., chromium, or qt), it's nice to be able to compare how many tests are being skipped on each variant. > > > > Although, re-thinking it, this isn't actually quite true with the addition of virtual test suite, since ports with different lists of virtual suites will have different numbers of tests. Currently, the only port that uses virtual tests is chromium, but at least there all of the variants use the same list of suites, so at least I keep the last sentence true. > > > > So, I still have a slight preference for leaving things the way they are, I guess. But if you'd strongly prefer to make the change, I won't be that upset by it. > > Ok, than I think we should get rid of platform tests instead. I actually like the platform tests concept, and it feels ugly to have to explicitly list in files tests that should be Skipped by definition (even though that's what Chromium has historically done). If WONTFIXing this isn't acceptable, I'd prefer making the change in comment #4 to having to explicitly move the platform-specific tests into a "not-platform dir but really it's a platform dir" and then skip things. That feels like a bad workaround. 685813 13 kbalazs 2012-08-02 14:21:34 -0700 > I actually like the platform tests concept, and it feels ugly to have to explicitly list in files tests that should be Skipped by definition (even though that's what Chromium has historically done). > > If WONTFIXing this isn't acceptable, I'd prefer making the change in comment #4 to having to explicitly move the platform-specific tests into a "not-platform dir but really it's a platform dir" and then skip things. That feels like a bad workaround. It's the matter of how important it is to make using --force comfortable. I would like to hear other's opinion, so Ossy, Jeez, please share your opinion. I will ping you tomorrow anyway if you don't :) 686673 14 ossy 2012-08-03 07:04:16 -0700 (In reply to comment #7) > I'm inclined to WONTFIX this ... any objections? Yes. This is a valid and annoying bug. Qt platform never should run tests under platform/chromium, platform/mac, ... directory. We don't want to run them with and without --force. I think handling of tests_for_other_platforms() is absolutely incorrect. These tests shouldn't have SKIP expectations, they shouldn't exist for other ports at all. 687321 15 dpranke 2012-08-03 18:35:51 -0700 sigh ... thinking about this again, I realized I forgot that we were talking about which platform directories to run *when you specify a generic directory*. So, this won't affect the stats when running all the tests. I am totally fine with rwt --qt compositing not running platform/chromium/compositing. Let 's do this :). Sorry for the back-and-forth. 688048 16 156670 kbalazs 2012-08-06 05:41:01 -0700 Created attachment 156670 Patch 688198 17 156670 dpranke 2012-08-06 09:31:23 -0700 Comment on attachment 156670 Patch thanks! since this is a functional change in how the tool works, it would be nice to have a test for it in run_webkit_tests_integrationtest.py; can you add a fixme to this or a stubbed out test, at least? 688212 18 kbalazs 2012-08-06 09:40:49 -0700 (In reply to comment #17) > (From update of attachment 156670 [details]) > thanks! since this is a functional change in how the tool works, it would be nice to have a test for it in run_webkit_tests_integrationtest.py; can you add a fixme to this or a stubbed out test, at least? No need to add a new test because it already regress run_webkit_tests_integrationtest.MainTest.test_platform_tests_are_found :) I will look into fixing it. 688235 19 156718 kbalazs 2012-08-06 10:06:17 -0700 Created attachment 156718 Patch 688249 20 156718 webkit.review.bot 2012-08-06 10:26:10 -0700 Comment on attachment 156718 Patch Attachment 156718 did not pass chromium-ews (chromium-xvfb): Output: http://queues.webkit.org/results/13439681 688274 21 156718 dpranke 2012-08-06 10:54:35 -0700 Comment on attachment 156718 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=156718&action=review r+ w/ nits. Feel free to land if you can figure out why the CQ is failing ... > Tools/Scripts/webkitpy/layout_tests/port/test.py:225 > tests.add('platform/test-snow-leopard/websocket/test.html') hm ... this is probably a bug, the baseline_search_path() checks for test-mac-snowleopard but here we have test-snow-leopard. I should fix that. It might be clearer if we added a test/test-win-win7/http/test.html, since win-win7 shouldn't be in the path for either mac-leopard or mac-snowleopard. > Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:899 > + # FIXME: we should not rely on knowleadge about the default port. If you changed line 897 to get_tests_run(['--platform', 'test-mac-leopard', 'http'], ...) you can get rid of the FIXME. 688923 22 kbalazs 2012-08-07 00:47:30 -0700 Landed in http://trac.webkit.org/changeset/124862. 156670 2012-08-06 05:41:01 -0700 2012-08-06 10:06:10 -0700 Patch bug-91089-20120806144030.patch text/plain 1603 kbalazs U3VidmVyc2lvbiBSZXZpc2lvbjogMTI0NzU2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNGI0ZDQwYTlhMzY0ZmE4ZDljMzgxZjRmNjgyNTYzZDIy OWRiODAxMy4uMDc4YzFkNzAzODNlNzA4NzBhYjcwNzY0ZmU5MjFlNjlhNGJlM2RiNCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE1 IEBACisyMDEyLTA4LTA2ICBCYWxhenMgS2VsZW1lbiAgPGtiYWxhenNAd2Via2l0Lm9yZz4KKwor ICAgICAgICBbTlJXVF0gcnVucyBwbGF0Zm9ybSBzcGVjaWZpYyB0ZXN0cyB0aGF0IGl0IHNob3Vs ZG4ndCB3aXRoIC0tZm9yY2UKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19i dWcuY2dpP2lkPTkxMDg5CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisK KyAgICAgICAgSWdub3JlIG90aGVyIHBsYXRmb3JtJ3MgZGlyZWN0b3JpZXMgZnJvbSBwbGF0Zm9y bS8uCisKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jhc2Uu cHk6CisgICAgICAgIChQb3J0Ll9leHBhbmRlZF9wYXRocyk6CisKIDIwMTItMDgtMDYgIE1pa2hh aWwgUG96ZG55YWtvdiAgPG1pa2hhaWwucG96ZG55YWtvdkBpbnRlbC5jb20+CiAKICAgICAgICAg W0VGTF0gRUZMJ3MgTGF5b3V0VGVzdENvbnRyb2xsZXIgbmVlZHMgcmVtb3ZlQWxsVmlzaXRlZExp bmtzIGltcGxlbWVudGF0aW9uCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xh eW91dF90ZXN0cy9wb3J0L2Jhc2UucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90 ZXN0cy9wb3J0L2Jhc2UucHkKaW5kZXggNmE5YWFmNzdjZTYzNmE1NGFjNTdmN2U4ZGJjMzk0NTQ0 MzU4ZWUwZi4uNmMyNDMzZDc5ZmI0M2JlNmEzMmI5MmNjODgxZjUxYWQ5ZmJmOWJiYSAxMDA3NTUK LS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlLnB5Cisr KyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvYmFzZS5weQpAQCAt NTYwLDcgKzU2MCw3IEBAIGNsYXNzIFBvcnQob2JqZWN0KToKICAgICAgICAgICAgIGV4cGFuZGVk X3BhdGhzLmFwcGVuZChwYXRoKQogICAgICAgICAgICAgaWYgc2VsZi50ZXN0X2lzZGlyKHBhdGgp IGFuZCBub3QgcGF0aC5zdGFydHN3aXRoKCdwbGF0Zm9ybScpOgogICAgICAgICAgICAgICAgIGZv ciBwbGF0Zm9ybV9kaXIgaW4gYWxsX3BsYXRmb3JtX2RpcnM6Ci0gICAgICAgICAgICAgICAgICAg IGlmIGZzLmlzZGlyKGZzLmpvaW4ocGxhdGZvcm1fZGlyLCBwYXRoKSk6CisgICAgICAgICAgICAg ICAgICAgIGlmIGZzLmlzZGlyKGZzLmpvaW4ocGxhdGZvcm1fZGlyLCBwYXRoKSkgYW5kIHBsYXRm b3JtX2RpciBpbiBzZWxmLmJhc2VsaW5lX3NlYXJjaF9wYXRoKCk6CiAgICAgICAgICAgICAgICAg ICAgICAgICBleHBhbmRlZF9wYXRocy5hcHBlbmQoc2VsZi5yZWxhdGl2ZV90ZXN0X2ZpbGVuYW1l KGZzLmpvaW4ocGxhdGZvcm1fZGlyLCBwYXRoKSkpCiAKICAgICAgICAgcmV0dXJuIGV4cGFuZGVk X3BhdGhzCg== 156718 2012-08-06 10:06:17 -0700 2012-08-07 00:47:44 -0700 Patch bug-91089-20120806190546.patch text/plain 3841 kbalazs U3VidmVyc2lvbiBSZXZpc2lvbjogMTI0Nzc2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZmRiZTY1M2ZiMThmNWQwYzY5MTE0OTk2ZTE2MTFmYjZh ZTQxNDQ1YS4uYjY4YWQ4OTEyNjVkOGUxYjM5MjhkY2RiNTIxZmVlN2M2ZTNiZWE0NyAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSw1ICsxLDIy IEBACiAyMDEyLTA4LTA2ICBCYWxhenMgS2VsZW1lbiAgPGtiYWxhenNAd2Via2l0Lm9yZz4KIAor ICAgICAgICBbTlJXVF0gcnVucyBwbGF0Zm9ybSBzcGVjaWZpYyB0ZXN0cyB0aGF0IGl0IHNob3Vs ZG4ndCB3aXRoIC0tZm9yY2UKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19i dWcuY2dpP2lkPTkxMDg5CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisK KyAgICAgICAgSWdub3JlIG90aGVyIHBsYXRmb3JtJ3MgZGlyZWN0b3JpZXMgZnJvbSBwbGF0Zm9y bS8uCisKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jhc2Uu cHk6CisgICAgICAgIChQb3J0Ll9leHBhbmRlZF9wYXRocyk6CisgICAgICAgICogU2NyaXB0cy93 ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC90ZXN0LnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Vi a2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHNfaW50ZWdyYXRpb250ZXN0LnB5Ogor ICAgICAgICAoTWFpblRlc3QudGVzdF9wbGF0Zm9ybV90ZXN0c19hcmVfZm91bmQpOgorICAgICAg ICBVcGRhdGVkIGludGVncmF0aW9uIHRlc3QgaW4gYWNjb3JkYW5jZSB0byB0aGUgbmV3IGJlaGF2 aW9yLiBVbmZvcnR1bmF0ZWx5IGl0IGhhcyB0byByZWx5CisgICAgICAgIG9uIHRoZSBuYW1lIG9m IHRoZSBkZWZhdWx0IHBvcnQgYmVjYXVzZSBpdCBpcyBoYXJkY29kZWQgaW4gdGVzdC5weS4gQWRk ZWQgYSBmaXhtZSBmb3IgdGhpcy4KKworMjAxMi0wOC0wNiAgQmFsYXpzIEtlbGVtZW4gIDxrYmFs YXpzQHdlYmtpdC5vcmc+CisKICAgICAgICAgV1RSIHNob3VsZCBiZSBhYmxlIHRvIGxvYWQgZXh0 ZXJuYWwgcmVzb3VyY2VzCiAgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVn LmNnaT9pZD04OTM4MgogCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91 dF90ZXN0cy9wb3J0L2Jhc2UucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0 cy9wb3J0L2Jhc2UucHkKaW5kZXggMzFhOWVkNGUwM2JkZTljMmM5MjU4Y2QzMWNjZGI0YmI3MzRm OTRhMS4uODM5MDhmYTkwMjFmZmFhNWZlYTMxMWM3ZjBkMDgzN2I0NTc3YzY3NiAxMDA3NTUKLS0t IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlLnB5CisrKyBi L1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvYmFzZS5weQpAQCAtNTY0 LDcgKzU2NCw3IEBAIGNsYXNzIFBvcnQob2JqZWN0KToKICAgICAgICAgICAgIGV4cGFuZGVkX3Bh dGhzLmFwcGVuZChwYXRoKQogICAgICAgICAgICAgaWYgc2VsZi50ZXN0X2lzZGlyKHBhdGgpIGFu ZCBub3QgcGF0aC5zdGFydHN3aXRoKCdwbGF0Zm9ybScpOgogICAgICAgICAgICAgICAgIGZvciBw bGF0Zm9ybV9kaXIgaW4gYWxsX3BsYXRmb3JtX2RpcnM6Ci0gICAgICAgICAgICAgICAgICAgIGlm IGZzLmlzZGlyKGZzLmpvaW4ocGxhdGZvcm1fZGlyLCBwYXRoKSk6CisgICAgICAgICAgICAgICAg ICAgIGlmIGZzLmlzZGlyKGZzLmpvaW4ocGxhdGZvcm1fZGlyLCBwYXRoKSkgYW5kIHBsYXRmb3Jt X2RpciBpbiBzZWxmLmJhc2VsaW5lX3NlYXJjaF9wYXRoKCk6CiAgICAgICAgICAgICAgICAgICAg ICAgICBleHBhbmRlZF9wYXRocy5hcHBlbmQoc2VsZi5yZWxhdGl2ZV90ZXN0X2ZpbGVuYW1lKGZz LmpvaW4ocGxhdGZvcm1fZGlyLCBwYXRoKSkpCiAKICAgICAgICAgcmV0dXJuIGV4cGFuZGVkX3Bh dGhzCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0 L3Rlc3QucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L3Rlc3Qu cHkKaW5kZXggMGNkMTcxZjNiY2E2YWJlYmRiMjFiOThlMmE0NWMyMWExNGE5MDExMi4uODI3Y2Vm MzM0MGQxMjIzM2NhMDE2NzhhZWEzOTRkOTI3N2MxMGM1OSAxMDA2NDQKLS0tIGEvVG9vbHMvU2Ny aXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC90ZXN0LnB5CisrKyBiL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvdGVzdC5weQpAQCAtMjIwLDYgKzIyMCw3IEBA IGxheWVyIGF0ICgwLDApIHNpemUgODAweDM0CiAgICAgdGVzdHMuYWRkKCd3ZWJzb2NrZXQvdGVz dHMvcGFzc2VzL3RleHQuaHRtbCcpCiAKICAgICAjIEZvciAtLW5vLWh0dHAgdGVzdHMsIHRlc3Qg dGhhdCBwbGF0Zm9ybSBzcGVjaWZpYyBIVFRQIHRlc3RzIGFyZSBwcm9wZXJseSBza2lwcGVkLgor ICAgIHRlc3RzLmFkZCgncGxhdGZvcm0vdGVzdC1tYWMtbGVvcGFyZC9odHRwL3Rlc3QuaHRtbCcp CiAgICAgdGVzdHMuYWRkKCdwbGF0Zm9ybS90ZXN0LXNub3ctbGVvcGFyZC9odHRwL3Rlc3QuaHRt bCcpCiAgICAgdGVzdHMuYWRkKCdwbGF0Zm9ybS90ZXN0LXNub3ctbGVvcGFyZC93ZWJzb2NrZXQv dGVzdC5odG1sJykKIApkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRf dGVzdHMvcnVuX3dlYmtpdF90ZXN0c19pbnRlZ3JhdGlvbnRlc3QucHkgYi9Ub29scy9TY3JpcHRz L3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX2ludGVncmF0aW9udGVzdC5w eQppbmRleCAwMmU3NWJhZjcyNDM4OTgzNWI3YTc1OGM2MmMxZjBjMWMzN2NlZmQ2Li4wZDg4Zjg3 NWUxZDk4NTQ5YmY2ODliMWE5MjhjNTM3MDEzYzkwNjEyIDEwMDc1NQotLS0gYS9Ub29scy9TY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX2ludGVncmF0aW9udGVz dC5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0 X3Rlc3RzX2ludGVncmF0aW9udGVzdC5weQpAQCAtODk1LDcgKzg5NSwxMCBAQCBjbGFzcyBNYWlu VGVzdCh1bml0dGVzdC5UZXN0Q2FzZSwgU3RyZWFtVGVzdGluZ01peGluKToKIAogICAgIGRlZiB0 ZXN0X3BsYXRmb3JtX3Rlc3RzX2FyZV9mb3VuZChzZWxmKToKICAgICAgICAgdGVzdHNfcnVuID0g Z2V0X3Rlc3RzX3J1bihbJ2h0dHAnXSwgdGVzdHNfaW5jbHVkZWQ9VHJ1ZSwgZmxhdHRlbl9iYXRj aGVzPVRydWUpCi0gICAgICAgIHNlbGYuYXNzZXJ0VHJ1ZSgncGxhdGZvcm0vdGVzdC1zbm93LWxl b3BhcmQvaHR0cC90ZXN0Lmh0bWwnIGluIHRlc3RzX3J1bikKKyAgICAgICAgIyBPbmx5IHRlc3Rz IGZvciB0aGUgYWN0dWFsIHBsYXRmb3JtIHNob3VsZCBiZSBmb3VuZC4KKyAgICAgICAgIyBGSVhN RTogd2Ugc2hvdWxkIG5vdCByZWx5IG9uIGtub3dsZWFkZ2UgYWJvdXQgdGhlIGRlZmF1bHQgcG9y dC4KKyAgICAgICAgc2VsZi5hc3NlcnRUcnVlKCdwbGF0Zm9ybS90ZXN0LW1hYy1sZW9wYXJkL2h0 dHAvdGVzdC5odG1sJyBpbiB0ZXN0c19ydW4pCisgICAgICAgIHNlbGYuYXNzZXJ0RmFsc2UoJ3Bs YXRmb3JtL3Rlc3Qtc25vdy1sZW9wYXJkL2h0dHAvdGVzdC5odG1sJyBpbiB0ZXN0c19ydW4pCiAK ICAgICBkZWYgdGVzdF9vdXRwdXRfZGlmZnMoc2VsZik6CiAgICAgICAgICMgVGVzdCB0byBlbnN1 cmUgdGhhdCB3ZSBkb24ndCBnZW5lcmF0ZSAtd2RpZmYuaHRtbCBvciAtcHJldHR5Lmh0bWwgaWYg d2RpZmYgYW5kIFByZXR0eVBhdGNoCg==