60957 2011-05-17 07:00:05 -0700 Pass flags from run_webkit_tests to ImageDiff, read results from stdout and store in unexpected_results.json 2012-06-19 14:46:29 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) Unspecified Unspecified RESOLVED WONTFIX NRWT P2 Normal --- 0 tomhudson webkit-unassigned dpranke tomhudson oldest_to_newest 405036 0 tomhudson 2011-05-17 07:00:05 -0700 Pass flags from run_webkit_tests to ImageDiff, read results from stdout and store in unexpected_results.json 405037 1 93762 tomhudson 2011-05-17 07:01:45 -0700 Created attachment 93762 Patch 405040 2 tomhudson 2011-05-17 07:04:51 -0700 Draft solution to (1) cache data from ImageDiff on Chromium port object in worker processes and (2) use message-passing and message-handling stubs on port object to get the data back to the manager process, where it is written into JSON file. 409815 3 93762 tomhudson 2011-05-25 12:10:23 -0700 Comment on attachment 93762 Patch Takes input from bug 60569, which is now r+ tony@chromium.org; output is to be read by bug 60964. 410971 4 93762 dpranke 2011-05-26 16:21:15 -0700 Comment on attachment 93762 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=93762&action=review So this patch isn't what I'd like to see as the long-term way to do this, but it's close for an incremental step. Ideally we should do something like modify layout_package/single_test_runner._compare_image and test_result_writer.diff_image so that we can compute your metrics in a single step, and then pass a dict of the metrics through a TestFailure object. However, changing the code to do this is a bit invasive to do all at once and may be more than you're comfortable with (as we've discussed previously). I'll upload a patch that demonstrates the protocol-level changes I have in mind and we can eventually merge that patch with this line of development. However, this batch is basically fine and doesn't need to be completely redone to make those changes. I'm R-'ing it for a bunch of specific things we can tweak somewhat to simplify things, in particular so that we don't need to generate two "finished_test" messages per test. Let me know if you have any questions on my comments. I look forward to getting this feature working and will be glad to help now that I'm not either on vacation or in workshops :) > Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:263 > + default="", I will usually try to omit the default (which is equivalent to default=None) if I want something to be empty by default. > Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py:161 > + port_obj.add_port_test_data(result_type, test, test_dict) Can you add a FIXME here to extract this data from the failure_types list directly instead of having to jump through the handle_port_finished_test / add_port_test_data hoop instead? If it's not clear what I have in mind, I can upload a modified version of this patch that does what I'm talking about. > Tools/Scripts/webkitpy/layout_tests/layout_package/worker.py:143 > self._worker_connection.post_message('finished_test', result, elapsed_time) Can we change this so that instead of sending a port_finished_test message we call into a port routine that returns a dict and send the dict along in finished_test? I don't really want to double the number of messages sent in a test run, and I have no concerns over changing the message format. > Tools/Scripts/webkitpy/layout_tests/port/base.py:686 > + def add_port_test_data(self, result_type, test, out_test_dict): If the intent is for these to be default implementations that don't do anything, can we please add a comment to that effect and a 'pass' statement? On the other hand, if you want these to be virtual methods that must be implemented in subclasses, please add a raise NotImplementedError (as we do in other routines). Also, please add tests for these routines in port/port_testcase.py. > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:162 > + return diff_filename[prefix_len:-suffix_len] + '.html' Ugh. There should be a method on the base class to do something like this, and we shouldn't introduce a dependency on test_result_writer into this file. Can you add a FIXME to fix this in a later patch? > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:196 > + if self.get_option('write_image_diff_metrics'): Is there a reason we shouldn't just always pass the flag and write the metrics? > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:346 > + def _test_name_from_filename(self, diff_filename): There is a port.relative_test_filename() in base.py that does this already. 412864 5 tomhudson 2011-05-31 14:32:26 -0700 > > > Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py:161 > > + port_obj.add_port_test_data(result_type, test, test_dict) > > Can you add a FIXME here to extract this data from the failure_types list directly instead of having to jump through the handle_port_finished_test / add_port_test_data hoop instead? > > If it's not clear what I have in mind, I can upload a modified version of this patch that does what I'm talking about. > > > Tools/Scripts/webkitpy/layout_tests/layout_package/worker.py:143 > > self._worker_connection.post_message('finished_test', result, elapsed_time) > > Can we change this so that instead of sending a port_finished_test message we call into a port routine that returns a dict and send the dict along in finished_test? I don't really want to double the number of messages sent in a test run, and I have no concerns over changing the message format. Part 1 is not 100% clear to me, unless part 2 is tied into it - I should be able to upload what I think you mean tomorrow. It'll be: - new field on TestResult object - new routine on Port called from worker thread to add arbitrary data to TestResult - Port.add_port_test_data replaced by a generic routine that checks for above arbitrary data on the TestResult object and, if present, adds to test_dict - Get rid of the port_finished_test send & receive code. - Get rid of the dependency on test_writer. > > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:196 > > + if self.get_option('write_image_diff_metrics'): > > Is there a reason we shouldn't just always pass the flag and write the metrics? Tony Chang & I talked for a while when he first reviewed 60569, and it was important to him not to slow down NRWT on the bots; especially once we start computing multiple metrics over the images (qv discussion of 60964), that's multiple passes over every pixel of every difference image, and a lot of extra data being written to unexpected_results.json and archived. As long as the only consumer of this data is RebaselineServer, it seemed reasonable to make it argument-driven and only happen on interactive local runs, although now thinking about it you might at times want it from a bot if there's a bug you can't reproduce locally? Tom 412873 6 dpranke 2011-05-31 14:39:17 -0700 (In reply to comment #5) > > > > > Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py:161 > > > + port_obj.add_port_test_data(result_type, test, test_dict) > > > > Can you add a FIXME here to extract this data from the failure_types list directly instead of having to jump through the handle_port_finished_test / add_port_test_data hoop instead? > > > > If it's not clear what I have in mind, I can upload a modified version of this patch that does what I'm talking about. > > > > > Tools/Scripts/webkitpy/layout_tests/layout_package/worker.py:143 > > > self._worker_connection.post_message('finished_test', result, elapsed_time) > > > > Can we change this so that instead of sending a port_finished_test message we call into a port routine that returns a dict and send the dict along in finished_test? I don't really want to double the number of messages sent in a test run, and I have no concerns over changing the message format. > > Part 1 is not 100% clear to me, unless part 2 is tied into it - I should be able to upload what I think you mean tomorrow. It'll be: > - new field on TestResult object > - new routine on Port called from worker thread to add arbitrary data to TestResult > - Port.add_port_test_data replaced by a generic routine that checks for above arbitrary data on the TestResult object and, if present, adds to test_dict > - Get rid of the port_finished_test send & receive code. > - Get rid of the dependency on test_writer. > Sounds pretty close :) > > > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:196 > > > + if self.get_option('write_image_diff_metrics'): > > > > Is there a reason we shouldn't just always pass the flag and write the metrics? > > Tony Chang & I talked for a while when he first reviewed 60569, and it was important to him not to slow down NRWT on the bots; especially once we start computing multiple metrics over the images (qv discussion of 60964), that's multiple passes over every pixel of every difference image, and a lot of extra data being written to unexpected_results.json and archived. > Well, given that files that fail the image check are a very small percentage of the total number of tests we run, I'd be surprised if it made much difference. > As long as the only consumer of this data is RebaselineServer, it seemed reasonable to make it argument-driven and only happen on interactive local runs, although now thinking about it you might at times want it from a bot if there's a bug you can't reproduce locally? That, or eventually we'll build tools where the gardeners (or, any webkit committer looking at failures in the tree) can also pull this data to help them determine how to triage failures. That said, I'm happy to start with a flag and we can get rid of it later. 652844 7 dpranke 2012-06-19 14:46:29 -0700 tom, I'm closing this since AFAIK we have no plans to do anything like this any time soon. Hopefully that's okay; if not, please feel free to reopen and we can discuss what to do with it. 93762 2011-05-17 07:01:45 -0700 2011-05-26 16:21:15 -0700 Patch bug-60957-20110517100144.patch text/plain 11461 tomhudson SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDg2NjcxKQorKysgVG9vbHMvQ2hhbmdlTG9nCSh3b3JraW5nIGNvcHkpCkBAIC0xLDMg KzEsMjMgQEAKKzIwMTEtMDUtMTcgIFRvbSBIdWRzb24gIDx0b21odWRzb25AZ29vZ2xlLmNvbT4K KworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBQYXNzIGZs YWdzIGZyb20gcnVuX3dlYmtpdF90ZXN0cyB0byBJbWFnZURpZmYsIHJlYWQgcmVzdWx0cyBmcm9t IHN0ZG91dCBhbmQgc3RvcmUgaW4gdW5leHBlY3RlZF9yZXN1bHRzLmpzb24KKyAgICAgICAgaHR0 cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTYwOTU3CisKKyAgICAgICAgKiBT Y3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9tYW5hZ2VyLnB5Ogor ICAgICAgICBEZWZlciB0byBQb3J0IG9iamVjdCB3aGVuIHJlY2VpdmluZyAncG9ydF9maW5pc2hl ZF90ZXN0JyBvYmplY3QgYW5kCisgICAgICAgIHdoZW4gd3JpdGluZyBvdXQgZWFjaCB0ZXN0J3Mg ZGF0YSB0byB1bmV4cGVjdGVkX3Jlc3VsdHMuanNvbi4KKyAgICAgICAgKiBTY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS93b3JrZXIucHk6CisgICAgICAgIERlZmVy IHRvIFBvcnQgb2JqZWN0IGJlZm9yZSBzZW5kaW5nICdmaW5pc2hlZF90ZXN0JyBtZXNzYWdlIHRv CisgICAgICAgIHNlbmQgYW55IGltcGxlbWVudGF0aW9uLXNwZWNpZmljICdwb3J0X2ZpbmlzaGVk X3Rlc3QnIG1lc3NhZ2UuCisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMv cG9ydC9iYXNlLnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3Bv cnQvY2hyb21pdW0ucHk6CisgICAgICAgIFNlbmQgYW5kIHJlY2VpdmUgJ3BvcnRfZmluaXNoZWRf dGVzdCc7IHRyYWNrIHBlci10ZXN0IG1ldHJpYworICAgICAgICBkYXRhIG91dHB1dCBieSBJbWFn ZURpZmYuCisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtp dF90ZXN0cy5weToKKyAgICAgICAgQWRkIG5ldyBhcmd1bWVudHMgdG8gYmUgcGFzc2VkIHRocm91 Z2ggdG8gSW1hZ2VEaWZmLgorCiAyMDExLTA1LTE3ICBBZGFtIFJvYmVuICA8YXJvYmVuQGFwcGxl LmNvbT4KIAogICAgICAgICBNYWtlIHJ1bi1hcGktdGVzdHMgd29yayBvbiBXaW5kb3dzIHdoZW4g dGhlcmUgYXJlIHNwYWNlcyBpbiB0aGUgcGF0aApJbmRleDogVG9vbHMvU2NyaXB0cy93ZWJraXRw eS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0cy5weQo9PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9T Y3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzLnB5CShyZXZpc2lv biA4NjU5NykKKysrIFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJr aXRfdGVzdHMucHkJKHdvcmtpbmcgY29weSkKQEAgLTI1NCw2ICsyNTQsMTUgQEAgZGVmIHBhcnNl X2FyZ3MoYXJncz1Ob25lKToKICAgICAgICAgICAgIGFjdGlvbj0ic3RvcmVfdHJ1ZSIsCiAgICAg ICAgICAgICBkZWZhdWx0PUZhbHNlLAogICAgICAgICAgICAgaGVscD0iUnVuIGdyYXBoaWNzIHRl c3RzIG9uIHJlYWwgR1BVIGhhcmR3YXJlIHZzIHNvZnR3YXJlIiksCisgICAgICAgIG9wdHBhcnNl Lm1ha2Vfb3B0aW9uKCItLXdyaXRlLWltYWdlLWRpZmYtbWV0cmljcyIsCisgICAgICAgICAgICBh Y3Rpb249InN0b3JlX3RydWUiLAorICAgICAgICAgICAgZGVmYXVsdD1GYWxzZSwKKyAgICAgICAg ICAgIGhlbHA9IldyaXRlIG91dCBtZXRyaWMgZGF0YSBmcm9tIEltYWdlRGlmZiIpLAorICAgICAg ICBvcHRwYXJzZS5tYWtlX29wdGlvbigiLS1pbWFnZS1kaWZmLW1ldHJpYyIsCisgICAgICAgICAg ICBhY3Rpb249InN0b3JlIiwKKyAgICAgICAgICAgIGRlZmF1bHQ9IiIsCisgICAgICAgICAgICBo ZWxwPSJVc2UgYSBub24tc3RhbmRhcmQgSW1hZ2VEaWZmIG1ldHJpYzsgcnVuIEltYWdlRGlmZiAt LWhlbHAgIgorICAgICAgICAgICAgICAgICAiZm9yIGEgbGlzdCBvZiBzdXBwb3J0ZWQgbWV0cmlj cy4iKSwKICAgICBdCiAKICAgICB3ZWJraXRfb3B0aW9ucyA9IFsKSW5kZXg6IFRvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL21hbmFnZXIucHkKPT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PQotLS0gVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2th Z2UvbWFuYWdlci5weQkocmV2aXNpb24gODY1OTcpCisrKyBUb29scy9TY3JpcHRzL3dlYmtpdHB5 L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9tYW5hZ2VyLnB5CSh3b3JraW5nIGNvcHkpCkBA IC0xNTgsNiArMTU4LDggQEAgZGVmIHN1bW1hcml6ZV9yZXN1bHRzKHBvcnRfb2JqLCBleHBlY3Rh dAogICAgICAgICAjIEZJWE1FOiBTZXQgdGhpcyBjb3JyZWN0bHkgb25jZSBodHRwczovL3dlYmtp dC5vcmcvYi8zNzczOSBpcyBmaXhlZAogICAgICAgICAjIGFuZCBvbmx5IHNldCBpdCBpZiB0aGVy ZSBhY3R1YWxseSBpcyBzdGRlcnIgZGF0YS4KIAorICAgICAgICBwb3J0X29iai5hZGRfcG9ydF90 ZXN0X2RhdGEocmVzdWx0X3R5cGUsIHRlc3QsIHRlc3RfZGljdCkKKwogICAgICAgICBmYWlsdXJl X3R5cGVzID0gW3R5cGUoZikgZm9yIGYgaW4gcmVzdWx0LmZhaWx1cmVzXQogICAgICAgICAjIEZJ WE1FOiBnZXQgcmlkIG9mIGFsbCB0aGlzIGlzXyogdmFsdWVzIG9uY2UgdGhlcmUgaXMgYSAxOjEg bWFwIGJldHdlZW4KICAgICAgICAgIyBUZXN0RmFpbHVyZSB0eXBlIGFuZCB0ZXN0X2V4cGVjdGF0 aW9ucy5FWFBFQ1RBVElPTi4KQEAgLTEzMjcsNiArMTMyOSw5IEBAIGNsYXNzIE1hbmFnZXI6CiAg ICAgICAgIHNlbGYuX2FsbF9yZXN1bHRzLmFwcGVuZChyZXN1bHQpCiAgICAgICAgIHNlbGYuX3Vw ZGF0ZV9zdW1tYXJ5X3dpdGhfcmVzdWx0KHNlbGYuX2N1cnJlbnRfcmVzdWx0X3N1bW1hcnksIHJl c3VsdCkKIAorICAgIGRlZiBoYW5kbGVfcG9ydF9maW5pc2hlZF90ZXN0KHNlbGYsIHNvdXJjZSwg KmFyZ3MpOgorICAgICAgICBzZWxmLl9wb3J0LmhhbmRsZV9wb3J0X2ZpbmlzaGVkX3Rlc3QoKmFy Z3MpCisKIAogZGVmIHJlYWRfdGVzdF9maWxlcyhmcywgZmlsZXMpOgogICAgIHRlc3RzID0gW10K SW5kZXg6IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdl L3dvcmtlci5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90 ZXN0cy9sYXlvdXRfcGFja2FnZS93b3JrZXIucHkJKHJldmlzaW9uIDg2NTk3KQorKysgVG9vbHMv U2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2thZ2Uvd29ya2VyLnB5CSh3 b3JraW5nIGNvcHkpCkBAIC0xMzksNiArMTM5LDcgQEAgY2xhc3MgV29ya2VyKG1hbmFnZXJfd29y a2VyX2Jyb2tlci5BYnN0cgogICAgICAgICByZXN1bHQgPSBzZWxmLnJ1bl90ZXN0X3dpdGhfdGlt ZW91dCh0ZXN0X2lucHV0LCB0ZXN0X3RpbWVvdXRfc2VjKQogCiAgICAgICAgIGVsYXBzZWRfdGlt ZSA9IHRpbWUudGltZSgpIC0gc3RhcnQKKyAgICAgICAgc2VsZi5fcG9ydC5zZW5kX3BvcnRfZmlu aXNoZWRfdGVzdCh0ZXN0X2lucHV0LmZpbGVuYW1lLCBzZWxmLl93b3JrZXJfY29ubmVjdGlvbikK ICAgICAgICAgc2VsZi5fd29ya2VyX2Nvbm5lY3Rpb24ucG9zdF9tZXNzYWdlKCdmaW5pc2hlZF90 ZXN0JywgcmVzdWx0LCBlbGFwc2VkX3RpbWUpCiAKICAgICAgICAgc2VsZi5jbGVhbl91cF9hZnRl cl90ZXN0KHRlc3RfaW5wdXQsIHJlc3VsdCkKSW5kZXg6IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkv bGF5b3V0X3Rlc3RzL3BvcnQvYmFzZS5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9TY3JpcHRzL3dl YmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jhc2UucHkJKHJldmlzaW9uIDg2NTk3KQorKysgVG9v bHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlLnB5CSh3b3JraW5nIGNv cHkpCkBAIC02ODMsNiArNjgzLDE3IEBAIGNsYXNzIFBvcnQob2JqZWN0KToKICAgICAgICAgaWYg c2VsZi5faHR0cF9sb2NrOgogICAgICAgICAgICAgc2VsZi5faHR0cF9sb2NrLmNsZWFudXBfaHR0 cF9sb2NrKCkKIAorICAgIGRlZiBhZGRfcG9ydF90ZXN0X2RhdGEoc2VsZiwgcmVzdWx0X3R5cGUs IHRlc3QsIG91dF90ZXN0X2RpY3QpOgorICAgICAgICAiIiJBZGQgYW55IHBvcnQtc3BlY2lmaWMg ZGF0YSB0byBqc29uIHJlc3VsdHMgZmlsZXMuIiIiCisKKyAgICBkZWYgc2VuZF9wb3J0X2Zpbmlz aGVkX3Rlc3Qoc2VsZiwgdGVzdF9maWxlbmFtZSwgd29ya2VyX2Nvbm5lY3Rpb24pOgorICAgICAg ICAiIiJTZW5kIGFueSBwb3J0LXNwZWNpZmljIG1lc3NhZ2UgZnJvbSB3b3JrZXIgdG8gbWFuYWdl cgorICAgICAgICBhZnRlciBhIHNpbmdsZSB0ZXN0IGhhcyBydW4uIiIiCisKKyAgICBkZWYgaGFu ZGxlX3BvcnRfZmluaXNoZWRfdGVzdChzZWxmLCBhcmdzKToKKyAgICAgICAgIiIiUHJvY2VzcyBh bnkgcG9ydC1zcGVjaWZpYyBtZXNzYWdlIGZyb20gd29ya2VyIHRvIG1hbmFnZXIKKyAgICAgICAg YWZ0ZXIgYSBzaW5nbGUgdGVzdCBoYXMgcnVuLiIiIgorCiAgICAgIwogICAgICMgVEVTVCBFWFBF Q1RBVElPTi1SRUxBVEVEIE1FVEhPRFMKICAgICAjCkluZGV4OiBUb29scy9TY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9wb3J0L2Nocm9taXVtLnB5Cj09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL1Nj cmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21pdW0ucHkJKHJldmlzaW9uIDg2 NTk3KQorKysgVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1 bS5weQkod29ya2luZyBjb3B5KQpAQCAtNDEsOSArNDEsMTEgQEAgaW1wb3J0IHdlYmJyb3dzZXIK IGZyb20gd2Via2l0cHkuY29tbW9uLmNvbmZpZyBpbXBvcnQgdXJscwogZnJvbSB3ZWJraXRweS5j b21tb24ubmV0LnRlc3RvdXRwdXRzZXQgaW1wb3J0IFRlc3RPdXRwdXRTZXQKIGZyb20gd2Via2l0 cHkuY29tbW9uLnN5c3RlbSBpbXBvcnQgZXhlY3V0aXZlCitmcm9tIHdlYmtpdHB5LmNvbW1vbi5z eXN0ZW0gaW1wb3J0IHBhdGgKIGZyb20gd2Via2l0cHkuY29tbW9uLnN5c3RlbS5wYXRoIGltcG9y dCBjeWdwYXRoCiBmcm9tIHdlYmtpdHB5LmNvbW1vbi5zeXN0ZW0uemlwZmlsZXNldCBpbXBvcnQg WmlwRmlsZVNldAogZnJvbSB3ZWJraXRweS5sYXlvdXRfdGVzdHMubGF5b3V0X3BhY2thZ2UgaW1w b3J0IHRlc3RfZXhwZWN0YXRpb25zCitmcm9tIHdlYmtpdHB5LmxheW91dF90ZXN0cy5sYXlvdXRf cGFja2FnZSBpbXBvcnQgdGVzdF9yZXN1bHRfd3JpdGVyCiBmcm9tIHdlYmtpdHB5LmxheW91dF90 ZXN0cy5wb3J0IGltcG9ydCBiYXNlCiBmcm9tIHdlYmtpdHB5LmxheW91dF90ZXN0cy5wb3J0IGlt cG9ydCBidWlsZGVycwogZnJvbSB3ZWJraXRweS5sYXlvdXRfdGVzdHMucG9ydCBpbXBvcnQgaHR0 cF9zZXJ2ZXIKQEAgLTY1LDYgKzY3LDcgQEAgY2xhc3MgQ2hyb21pdW1Qb3J0KGJhc2UuUG9ydCk6 CiAgICAgZGVmIF9faW5pdF9fKHNlbGYsICoqa3dhcmdzKToKICAgICAgICAgYmFzZS5Qb3J0Ll9f aW5pdF9fKHNlbGYsICoqa3dhcmdzKQogICAgICAgICBzZWxmLl9jaHJvbWl1bV9iYXNlX2RpciA9 IE5vbmUKKyAgICAgICAgc2VsZi5faW1hZ2VfZGlmZl9tZXRyaWNfZGljdCA9IHt9CiAKICAgICBk ZWYgX2NoZWNrX2ZpbGVfZXhpc3RzKHNlbGYsIHBhdGhfdG9fZmlsZSwgZmlsZV9kZXNjcmlwdGlv biwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIG92ZXJyaWRlX3N0ZXA9Tm9uZSwgbG9nZ2lu Zz1UcnVlKToKQEAgLTEzNSw2ICsxMzgsNDIgQEAgY2xhc3MgQ2hyb21pdW1Qb3J0KGJhc2UuUG9y dCk6CiAgICAgICAgIHJldHVybiBzZWxmLl9jaGVja19maWxlX2V4aXN0cyhpbWFnZV9kaWZmX3Bh dGgsICdpbWFnZSBkaWZmIGV4ZScsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBvdmVycmlkZV9zdGVwLCBsb2dnaW5nKQogCisgICAgIyBBbGwgd2UgZXhwZWN0IHRvIG5l ZWQgaXMgdGhlIHJldHVybiBjb2RlCisgICAgZGVmIF9kaWZmX3dpdGhfcmV0dXJuX2NvZGUoc2Vs ZiwgY21kKToKKyAgICAgICAgcmVzdWx0ID0gVHJ1ZQorICAgICAgICBleGl0X2NvZGUgPSBzZWxm Ll9leGVjdXRpdmUucnVuX2NvbW1hbmQoY21kLCByZXR1cm5fZXhpdF9jb2RlPVRydWUpCisgICAg ICAgIGlmIGV4aXRfY29kZSA9PSAwOgorICAgICAgICAgICAgIyBUaGUgaW1hZ2VzIGFyZSB0aGUg c2FtZS4KKyAgICAgICAgICAgIHJlc3VsdCA9IEZhbHNlCisgICAgICAgIGVsaWYgZXhpdF9jb2Rl ICE9IDE6CisgICAgICAgICAgICBfbG9nLmVycm9yKCJpbWFnZSBkaWZmIHJldHVybmVkIGFuIGV4 aXQgY29kZSBvZiAiCisgICAgICAgICAgICAgICAgICAgICAgICsgc3RyKGV4aXRfY29kZSkpCisg ICAgICAgICAgICAjIFJldHVybmluZyBGYWxzZSBoZXJlIGNhdXNlcyB0aGUgc2NyaXB0IHRvIHRo aW5rIHRoYXQgd2UKKyAgICAgICAgICAgICMgc3VjY2Vzc2Z1bGx5IGNyZWF0ZWQgdGhlIGRpZmYg ZXZlbiB0aG91Z2ggd2UgZGlkbid0LiAgSWYKKyAgICAgICAgICAgICMgd2UgcmV0dXJuIFRydWUs IHdlIHRoaW5rIHRoYXQgdGhlIGltYWdlcyBtYXRjaCBidXQgdGhlIGhhc2hlcworICAgICAgICAg ICAgIyBkb24ndCBtYXRjaC4KKyAgICAgICAgICAgICMgRklYTUU6IEZpZ3VyZSBvdXQgd2h5IGlt YWdlX2RpZmYgcmV0dXJucyBvdGhlciB2YWx1ZXMuCisgICAgICAgICAgICByZXN1bHQgPSBGYWxz ZQorICAgICAgICByZXR1cm4gcmVzdWx0CisKKyAgICBkZWYgX3Rlc3RfbmFtZV9mcm9tX2RpZmZf ZmlsZW5hbWUoc2VsZiwgZGlmZl9maWxlbmFtZSk6CisgICAgICAgIHByZWZpeF9sZW4gPSBsZW4o c2VsZi5yZXN1bHRzX2RpcmVjdG9yeSgpICsgJy8nKQorICAgICAgICBzdWZmaXhfbGVuID0gbGVu KHRlc3RfcmVzdWx0X3dyaXRlci5UZXN0UmVzdWx0V3JpdGVyLkZJTEVOQU1FX1NVRkZJWF9JTUFH RV9ESUZGKQorICAgICAgICByZXR1cm4gZGlmZl9maWxlbmFtZVtwcmVmaXhfbGVuOi1zdWZmaXhf bGVuXSArICcuaHRtbCcKKworICAgIGRlZiBfZGlmZl9hbmRfY2FwdHVyZV9zdGRvdXQoc2VsZiwg Y21kLCBkaWZmX2ZpbGVuYW1lKToKKyAgICAgICAgcmVzdWx0ID0gRmFsc2UKKyAgICAgICAgcG9y dCA9IHNlbGYKKworICAgICAgICBkZWYgX3N0b3JlX21ldHJpY3Moc2NyaXB0X2Vycm9yKToKKyAg ICAgICAgICAgIGlmIGRpZmZfZmlsZW5hbWUgIT0gTm9uZToKKyAgICAgICAgICAgICAgICB0ZXN0 X25hbWUgPSBwb3J0Ll90ZXN0X25hbWVfZnJvbV9kaWZmX2ZpbGVuYW1lKGRpZmZfZmlsZW5hbWUp CisgICAgICAgICAgICAgICAgcG9ydC5faW1hZ2VfZGlmZl9tZXRyaWNfZGljdFt0ZXN0X25hbWVd ID0gZmxvYXQoc2NyaXB0X2Vycm9yLm91dHB1dCkKKyAgICAgICAgICAgIHJlc3VsdCA9IFRydWUK KworICAgICAgICBzZWxmLl9leGVjdXRpdmUucnVuX2NvbW1hbmQoY21kLCBlcnJvcl9oYW5kbGVy PV9zdG9yZV9tZXRyaWNzKQorICAgICAgICByZXR1cm4gcmVzdWx0CisKICAgICBkZWYgZGlmZl9p bWFnZShzZWxmLCBleHBlY3RlZF9jb250ZW50cywgYWN0dWFsX2NvbnRlbnRzLAogICAgICAgICAg ICAgICAgICAgIGRpZmZfZmlsZW5hbWU9Tm9uZSk6CiAgICAgICAgICMgRklYTUU6IG5lZWQgdW5p dCB0ZXN0cyBmb3IgdGhpcy4KQEAgLTE1MCwyNyArMTg5LDI1IEBAIGNsYXNzIENocm9taXVtUG9y dChiYXNlLlBvcnQpOgogICAgICAgICBzZWxmLl9maWxlc3lzdGVtLndyaXRlX2JpbmFyeV9maWxl KGFjdHVhbF9maWxlbmFtZSwgYWN0dWFsX2NvbnRlbnRzKQogCiAgICAgICAgIGV4ZWN1dGFibGUg PSBzZWxmLl9wYXRoX3RvX2ltYWdlX2RpZmYoKQorICAgICAgICBzaG91bGRfY2FwdHVyZV9kaWZm X3N0ZG91dCA9IEZhbHNlCiAgICAgICAgIGlmIGRpZmZfZmlsZW5hbWU6CiAgICAgICAgICAgICBj bWQgPSBbZXhlY3V0YWJsZSwgJy0tZGlmZicsIGV4cGVjdGVkX2ZpbGVuYW1lLAogICAgICAgICAg ICAgICAgICAgIGFjdHVhbF9maWxlbmFtZSwgZGlmZl9maWxlbmFtZV0KKyAgICAgICAgICAgIGlm IHNlbGYuZ2V0X29wdGlvbignd3JpdGVfaW1hZ2VfZGlmZl9tZXRyaWNzJyk6CisgICAgICAgICAg ICAgICAgY21kLmFwcGVuZCgnLS13cml0ZS1pbWFnZS1kaWZmLW1ldHJpY3MnKQorICAgICAgICAg ICAgICAgIHNob3VsZF9jYXB0dXJlX2RpZmZfc3Rkb3V0ID0gVHJ1ZQogICAgICAgICBlbHNlOgog ICAgICAgICAgICAgY21kID0gW2V4ZWN1dGFibGUsIGV4cGVjdGVkX2ZpbGVuYW1lLCBhY3R1YWxf ZmlsZW5hbWVdCiAKLSAgICAgICAgcmVzdWx0ID0gVHJ1ZQorICAgICAgICBpZiBzZWxmLmdldF9v cHRpb24oJ2ltYWdlX2RpZmZfbWV0cmljJykgIT0gTm9uZSBhbmQgXAorICAgICAgICAgICBzZWxm LmdldF9vcHRpb24oJ2ltYWdlX2RpZmZfbWV0cmljJykgIT0gJyc6CisgICAgICAgICAgICBjbWQu YXBwZW5kKCctLScgKyBzZWxmLmdldF9vcHRpb24oJ2ltYWdlX2RpZmZfbWV0cmljJykpCisKICAg ICAgICAgdHJ5OgotICAgICAgICAgICAgZXhpdF9jb2RlID0gc2VsZi5fZXhlY3V0aXZlLnJ1bl9j b21tYW5kKGNtZCwgcmV0dXJuX2V4aXRfY29kZT1UcnVlKQotICAgICAgICAgICAgaWYgZXhpdF9j b2RlID09IDA6Ci0gICAgICAgICAgICAgICAgIyBUaGUgaW1hZ2VzIGFyZSB0aGUgc2FtZS4KLSAg ICAgICAgICAgICAgICByZXN1bHQgPSBGYWxzZQotICAgICAgICAgICAgZWxpZiBleGl0X2NvZGUg IT0gMToKLSAgICAgICAgICAgICAgICBfbG9nLmVycm9yKCJpbWFnZSBkaWZmIHJldHVybmVkIGFu IGV4aXQgY29kZSBvZiAiCi0gICAgICAgICAgICAgICAgICAgICAgICAgICArIHN0cihleGl0X2Nv ZGUpKQotICAgICAgICAgICAgICAgICMgUmV0dXJuaW5nIEZhbHNlIGhlcmUgY2F1c2VzIHRoZSBz Y3JpcHQgdG8gdGhpbmsgdGhhdCB3ZQotICAgICAgICAgICAgICAgICMgc3VjY2Vzc2Z1bGx5IGNy ZWF0ZWQgdGhlIGRpZmYgZXZlbiB0aG91Z2ggd2UgZGlkbid0LiAgSWYKLSAgICAgICAgICAgICAg ICAjIHdlIHJldHVybiBUcnVlLCB3ZSB0aGluayB0aGF0IHRoZSBpbWFnZXMgbWF0Y2ggYnV0IHRo ZSBoYXNoZXMKLSAgICAgICAgICAgICAgICAjIGRvbid0IG1hdGNoLgotICAgICAgICAgICAgICAg ICMgRklYTUU6IEZpZ3VyZSBvdXQgd2h5IGltYWdlX2RpZmYgcmV0dXJucyBvdGhlciB2YWx1ZXMu Ci0gICAgICAgICAgICAgICAgcmVzdWx0ID0gRmFsc2UKKyAgICAgICAgICAgIGlmIHNob3VsZF9j YXB0dXJlX2RpZmZfc3Rkb3V0OgorICAgICAgICAgICAgICAgIHJlc3VsdCA9IHNlbGYuX2RpZmZf YW5kX2NhcHR1cmVfc3Rkb3V0KGNtZCwgZGlmZl9maWxlbmFtZSkKKyAgICAgICAgICAgIGVsc2U6 CisgICAgICAgICAgICAgICAgcmVzdWx0ID0gc2VsZi5fZGlmZl93aXRoX3JldHVybl9jb2RlKGNt ZCkKICAgICAgICAgZXhjZXB0IE9TRXJyb3IsIGU6CiAgICAgICAgICAgICBpZiBlLmVycm5vID09 IGVycm5vLkVOT0VOVCBvciBlLmVycm5vID09IGVycm5vLkVBQ0NFUzoKICAgICAgICAgICAgICAg ICBfY29tcGFyZV9hdmFpbGFibGUgPSBGYWxzZQpAQCAtMjk0LDYgKzMzMSwzNCBAQCBjbGFzcyBD aHJvbWl1bVBvcnQoYmFzZS5Qb3J0KToKICAgICAgICAgcmVwb3MuYXBwZW5kKCgnY2hyb21lJywg c2VsZi5wYXRoX2Zyb21fY2hyb21pdW1fYmFzZSgpKSkKICAgICAgICAgcmV0dXJuIHJlcG9zCiAK KyAgICBkZWYgYWRkX3BvcnRfdGVzdF9kYXRhKHNlbGYsIHJlc3VsdF90eXBlLCB0ZXN0LCBvdXRf dGVzdF9kaWN0KToKKyAgICAgICAgaWYgcmVzdWx0X3R5cGUgPT0gdGVzdF9leHBlY3RhdGlvbnMu SU1BR0Ugb3IgXAorICAgICAgICAgICByZXN1bHRfdHlwZSA9PSB0ZXN0X2V4cGVjdGF0aW9ucy5J TUFHRV9QTFVTX1RFWFQ6CisgICAgICAgICAgICBpZiB0ZXN0IGluIHNlbGYuX2ltYWdlX2RpZmZf bWV0cmljX2RpY3Q6CisgICAgICAgICAgICAgICAgbWV0cmljID0gc2VsZi5faW1hZ2VfZGlmZl9t ZXRyaWNfZGljdFt0ZXN0XQorICAgICAgICAgICAgICAgIGlmIG1ldHJpYyAhPSBOb25lOgorICAg ICAgICAgICAgICAgICAgICBvdXRfdGVzdF9kaWN0WydtZXRyaWMnXSA9IG1ldHJpYworICAgICAg ICAgICAgICAgIGVsc2U6CisgICAgICAgICAgICAgICAgICAgIF9sb2cuZXJyb3IoJyAgbWV0cmlj IGlzIGVtcHR5JykKKyAgICAgICAgICAgICMgRklYTUUgZmFpbGluZyB0aGlzIGlzIGFuIGVycm9y IGlmZiB0aGV5IHBhc3NlZCBpbgorICAgICAgICAgICAgIyAtLXdyaXRlLWltYWdlLWRpZmYtbWV0 cmljcworCisgICAgZGVmIF90ZXN0X25hbWVfZnJvbV9maWxlbmFtZShzZWxmLCBkaWZmX2ZpbGVu YW1lKToKKyAgICAgICAgcHJlZml4X2xlbiA9IGxlbihzZWxmLmxheW91dF90ZXN0c19kaXIoKSAr ICcvJykKKyAgICAgICAgcmV0dXJuIGRpZmZfZmlsZW5hbWVbcHJlZml4X2xlbjpdCisKKyAgICBk ZWYgc2VuZF9wb3J0X2ZpbmlzaGVkX3Rlc3Qoc2VsZiwgdGVzdF9maWxlbmFtZSwgd29ya2VyX2Nv bm5lY3Rpb24pOgorICAgICAgICB0ZXN0X25hbWUgPSBzZWxmLl90ZXN0X25hbWVfZnJvbV9maWxl bmFtZSh0ZXN0X2ZpbGVuYW1lKQorICAgICAgICBpZiB0ZXN0X25hbWUgaW4gc2VsZi5faW1hZ2Vf ZGlmZl9tZXRyaWNfZGljdDoKKyAgICAgICAgICAgIG1ldHJpYyA9IHNlbGYuX2ltYWdlX2RpZmZf bWV0cmljX2RpY3RbdGVzdF9uYW1lXQorICAgICAgICAgICAgYXJncyA9ICh0ZXN0X25hbWUsIG1l dHJpYykKKyAgICAgICAgICAgIHdvcmtlcl9jb25uZWN0aW9uLnBvc3RfbWVzc2FnZSgncG9ydF9m aW5pc2hlZF90ZXN0JywgYXJncykKKworICAgIGRlZiBoYW5kbGVfcG9ydF9maW5pc2hlZF90ZXN0 KHNlbGYsIGFyZ3MpOgorICAgICAgICB0ZXN0X25hbWUgPSBhcmdzWzBdCisgICAgICAgIG1ldHJp YyA9IGZsb2F0KGFyZ3NbMV0pCisgICAgICAgIHNlbGYuX2ltYWdlX2RpZmZfbWV0cmljX2RpY3Rb dGVzdF9uYW1lXSA9IG1ldHJpYworCiAgICAgIwogICAgICMgUFJPVEVDVEVEIE1FVEhPRFMKICAg ICAjCg==