37262 2010-04-08 02:58:11 -0700 new-run-webkit-tests should estimate test completion time 2010-04-08 15:29:41 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) Other OS X 10.5 RESOLVED WONTFIX P2 Normal --- 1 abarth webkit-unassigned dpranke eric ojan ossy oldest_to_newest 210107 0 abarth 2010-04-08 02:58:11 -0700 new-run-webkit-tests should estimate test completion time 210108 1 52851 abarth 2010-04-08 02:59:39 -0700 Created attachment 52851 Patch 210328 2 dpranke 2010-04-08 12:36:14 -0700 Note that in the perl run-webkit-tests, the HTTP tests were put at the end intentionally because they were slowest. The original chrome run-webkit-tests put HTTP tests at the beginning, since that shard was usually the slowest. But, I would have thought that with Ojan's more recent sharding efforts, it would be more uniform. If you run with --experimental-fully-parallel, then the distribution should become more uniform, and if you run with --randomize-order, that would also make it more (arguably completely) uniform. One or both of these should probably be the default, but they risk making the testing more nondeterministic. (Although I suspect that the timing issues when running in parallel under load do enough to make things nondeterministic anyway that it's moot). Perhaps we should have the script output a seed number when running with --randomize-order (visible with --log config), so that the shuffling could at least be reproduced if necessary. 210331 3 abarth 2010-04-08 12:38:37 -0700 TOTT tells me that we should randomize once a day and record the seed. That way you get lots of order coverage but order-dependent failures show up as red tests for 24hrs so you have a chance of fixing them. 210333 4 eric 2010-04-08 12:43:28 -0700 http tests were put at the end for run-webkit-tests so that it was possible to lock around use of the httpd server was my understanding. Eventually we'll have to fix new-run-webkit-tests to support running more than one copy on the machine at once. 210334 5 52851 abarth 2010-04-08 12:43:42 -0700 Comment on attachment 52851 Patch I had fun writing this patch, but we probably don't need it at the moment. The percent complete thing is helpful to me so I don't have to do math in my head. 210336 6 eric 2010-04-08 12:45:00 -0700 If every failing test would just record the test which were run previous to it on that Driver instance, that would make it super easy to reproduce any failure due to sharding/randomization. 210343 7 52851 dpranke 2010-04-08 12:55:54 -0700 Comment on attachment 52851 Patch > diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/layout_package/eta.py b/WebKitTools/Scripts/webkitpy/layout_tests/layout_package/eta.py > new file mode 100644 > index 0000000..ecdf7b5 > --- /dev/null > +++ b/WebKitTools/Scripts/webkitpy/layout_tests/layout_package/eta.py > +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE > +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > + > +import time I think PEP-8 asks that you put a module-level docstring here. > + > + > +class _Checkpoint(object): > + """A reference point for estimating the remaining time.""" > + def __init__(self, index, total, t): > + self._index = index > + self._total = total > + self._time = t > + > + def estimate(self, index, now): > + elapsed_index = index - self._index > + elapsed_time = now - self._time > + return elapsed_time * (self._total - index) / elapsed_index > + Please add docstrings indicating what index, now, total, and t are. > + > +class ETA(object): > + """A class for estimating the amount of time remaining. > + This class uses an exponentially weighed series of checkpoints to estimate > + how much time remains.""" Can you describe why the algorithm is the way it is (or include a reference to a description of this somewhere, if it's a standard algorithm? I've never seen it before). It's not obvious why you are skipping some checkpoints, or how this thing works overall. In particular, it's not clear why this algorithm will produce better results than a simple (time so far) / ( tests so far / total tests) estimate, which is a lot simpler and more obvious. I'm sure that this algorithm is probably better, or else you wouldn't have bothered to code it, but for anyone not familiar with this type of thing, they'll have no idea how to debug it or change it. As to the correctness of the computation, I will not review that at the moment since I'm hoping you'll explain it to me first :) In run_webkit_tests.py: > > @@ -714,10 +717,16 @@ class TestRunner: > > def _display_one_line_progress(self, result_summary): > """Displays the progress through the test run.""" > - percent_complete = 100 * (result_summary.expected + result_summary.unexpected) / result_summary.total > - self._meter.update("Testing (%d%%): %d ran as expected, %d didn't, %d left" % > - (percent_complete, result_summary.expected, > - result_summary.unexpected, result_summary.remaining)) > + if not self._eta: > + self._eta = eta.ETA(result_summary.total) Should self._eta be created at the start of _run_tests(), rather than here? Otherwise you are discarding the time spent between starting the test run and the first invocation of this call. > + number_ran = result_summary.expected + result_summary.unexpected > + percent_complete = 100 * number_ran / result_summary.total > + eta_estimate = self._eta.estimate(number_ran) > + eta_string = ", %ds remaining" % math.floor(eta_estimate) if eta_estimate else "" I'm not at all a fan of this Python idiom (too Perl-y). Can you rewrite this as a standard if : else: branch? The latter has the advantage of working correctly with the python coverage tool. 210345 8 abarth 2010-04-08 12:59:16 -0700 Thanks for the review. I'm happy to address your comments if we want this patch, but I'm also not that married to getting this feature. :) 210355 9 dpranke 2010-04-08 13:38:48 -0700 (In reply to comment #4) > http tests were put at the end for run-webkit-tests so that it was possible to > lock around use of the httpd server was my understanding. > > Eventually we'll have to fix new-run-webkit-tests to support running more than > one copy on the machine at once. Why? As opposed to just getting it to run as fast as possible over all the cores in the machine and queuing each run? Handling real concurrency would involve supporting multiple HTTP servers, etc, which might be a real hassle. 210356 10 dpranke 2010-04-08 13:39:23 -0700 (In reply to comment #8) > Thanks for the review. I'm happy to address your comments if we want this > patch, but I'm also not that married to getting this feature. :) If you had a reference for the algorithm, that would be interesting. Otherwise, addressing it is up to you. 210400 11 abarth 2010-04-08 14:50:32 -0700 > Why? As opposed to just getting it to run as fast as possible over all the > cores in the machine and queuing each run? I think queuing would be an acceptable solution for our use case, which is running multiple bots (commit-queue, mac-ews) on the same machine. 210401 12 abarth 2010-04-08 14:51:46 -0700 > If you had a reference for the algorithm, that would be interesting. Otherwise, > addressing it is up to you. The algorithm is just simple-exponential-smoothing as explained here: http://www.duke.edu/~rnau/411avg.htm 210403 13 ossy 2010-04-08 14:56:52 -0700 (In reply to comment #4) > http tests were put at the end for run-webkit-tests so that it was possible to > lock around use of the httpd server was my understanding. > > Eventually we'll have to fix new-run-webkit-tests to support running more than > one copy on the machine at once. Yes, you're right, we put http tests at the end because now run-webkit-tests script grab the lock of httpd before the first http test and release the lock when exit. Our first idea was to fix tests and make run-webkit-tests script to be able to run multiple httpd. Fixing run-webkit-tests isn't a big task, but test cases and expected files with hard coded localhost, 127.0.0.1, 8000, 8080, etc. port numbers. You can find how many test cases must be refactored for this in description of related bug: https://bugs.webkit.org/show_bug.cgi?id=33153 Hard coded URLs and port numbers isn't a big problem if you would like to run only one multi-threaded (new-)run-webkit-tests script. But it is a blocker task if you would like to run more script. 210436 14 ossy 2010-04-08 15:29:41 -0700 (In reply to comment #9) > Why? As opposed to just getting it to run as fast as possible over all the > cores in the machine and queuing each run? > > Handling real concurrency would involve supporting multiple HTTP servers, etc, > which might be a real hassle. Queueing can be a good solution, but we should guarantee running in the arrival order to avoid starvation (see dining philosophers). András made a good solution for this in run-webkit-tests implementation (--wait-for-http locking mechanism). The idea is very simple, the script touch a new lock.XXXX file, where XXX is an automatically incremental value, and always runs the owner of least one. 52851 2010-04-08 02:59:39 -0700 2010-04-08 12:55:54 -0700 Patch bug-37262-20100408025937.patch text/plain 6915 abarth ZGlmZiAtLWdpdCBhL1dlYktpdFRvb2xzL0NoYW5nZUxvZyBiL1dlYktpdFRvb2xzL0NoYW5nZUxv ZwppbmRleCAyMGM4YjUyLi5lNjJhYTYwIDEwMDY0NAotLS0gYS9XZWJLaXRUb29scy9DaGFuZ2VM b2cKKysrIGIvV2ViS2l0VG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDUgKzEsMjAgQEAKIDIwMTAtMDQt MDggIEFkYW0gQmFydGggIDxhYmFydGhAd2Via2l0Lm9yZz4KIAorICAgICAgICBSZXZpZXdlZCBi eSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBuZXctcnVuLXdlYmtpdC10ZXN0cyBzaG91bGQg ZXN0aW1hdGUgdGVzdCBjb21wbGV0aW9uIHRpbWUKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtp dC5vcmcvc2hvd19idWcuY2dpP2lkPTM3MjYyCisKKyAgICAgICAgQSBzaW1wbGUgZXN0aW1hdGUg b2YgaG93IG11Y2ggdGltZSByZW1haW5zIHRvIHJ1biB0aGUgdGVzdHMuICBUaGUKKyAgICAgICAg YWN0dWFsIHRlc3QgdGltZSBpcyBoaWdobHkgbm9uLXVuaWZvcm0uICBUaGlzIGFsZ29yaXRobSBk b2VzIGEKKyAgICAgICAgcmVhc29uYWJsZSBqb2IsIGJ1dCBpdCB1bmRlcmVzdGltYXRlcyB0aGUg dGltZSBpbiB0aGUgYmVnaW5uaW5nIGJlY2F1c2UKKyAgICAgICAgdGhlIHRlc3RzIGF0IHRoZSBl bmQgYXJlIG11Y2ggc2xvd2VyIGZvciBzb21lIHJlYXNvbi4KKworICAgICAgICAqIFNjcmlwdHMv d2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL2V0YS5weTogQWRkZWQuCisgICAg ICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0cy5weToK KworMjAxMC0wNC0wOCAgQWRhbSBCYXJ0aCAgPGFiYXJ0aEB3ZWJraXQub3JnPgorCiAgICAgICAg IFJldmlld2VkIGJ5IEVyaWMgU2VpZGVsLgogCiAgICAgICAgIG5ldy1ydW4td2Via2l0LXRlc3Rz IHNob3VsZCBnaXZlIGEgcGVyY2VudCBjb21wbGV0ZSBpbmRpY2F0aW9uCmRpZmYgLS1naXQgYS9X ZWJLaXRUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9l dGEucHkgYi9XZWJLaXRUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRf cGFja2FnZS9ldGEucHkKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMC4uZWNkZjdi NQotLS0gL2Rldi9udWxsCisrKyBiL1dlYktpdFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL2xheW91dF9wYWNrYWdlL2V0YS5weQpAQCAtMCwwICsxLDc1IEBACisjIENvcHlyaWdo dCAoQykgMjAxMCBHb29nbGUgSW5jLiBBbGwgcmlnaHRzIHJlc2VydmVkLgorIworIyBSZWRpc3Ry aWJ1dGlvbiBhbmQgdXNlIGluIHNvdXJjZSBhbmQgYmluYXJ5IGZvcm1zLCB3aXRoIG9yIHdpdGhv dXQKKyMgbW9kaWZpY2F0aW9uLCBhcmUgcGVybWl0dGVkIHByb3ZpZGVkIHRoYXQgdGhlIGZvbGxv d2luZyBjb25kaXRpb25zIGFyZQorIyBtZXQ6CisjCisjICAgICAqIFJlZGlzdHJpYnV0aW9ucyBv ZiBzb3VyY2UgY29kZSBtdXN0IHJldGFpbiB0aGUgYWJvdmUgY29weXJpZ2h0CisjIG5vdGljZSwg dGhpcyBsaXN0IG9mIGNvbmRpdGlvbnMgYW5kIHRoZSBmb2xsb3dpbmcgZGlzY2xhaW1lci4KKyMg ICAgICogUmVkaXN0cmlidXRpb25zIGluIGJpbmFyeSBmb3JtIG11c3QgcmVwcm9kdWNlIHRoZSBh Ym92ZQorIyBjb3B5cmlnaHQgbm90aWNlLCB0aGlzIGxpc3Qgb2YgY29uZGl0aW9ucyBhbmQgdGhl IGZvbGxvd2luZyBkaXNjbGFpbWVyCisjIGluIHRoZSBkb2N1bWVudGF0aW9uIGFuZC9vciBvdGhl ciBtYXRlcmlhbHMgcHJvdmlkZWQgd2l0aCB0aGUKKyMgZGlzdHJpYnV0aW9uLgorIyAgICAgKiBO ZWl0aGVyIHRoZSBuYW1lIG9mIEdvb2dsZSBJbmMuIG5vciB0aGUgbmFtZXMgb2YgaXRzCisjIGNv bnRyaWJ1dG9ycyBtYXkgYmUgdXNlZCB0byBlbmRvcnNlIG9yIHByb21vdGUgcHJvZHVjdHMgZGVy aXZlZCBmcm9tCisjIHRoaXMgc29mdHdhcmUgd2l0aG91dCBzcGVjaWZpYyBwcmlvciB3cml0dGVu IHBlcm1pc3Npb24uCisjCisjIFRISVMgU09GVFdBUkUgSVMgUFJPVklERUQgQlkgVEhFIENPUFlS SUdIVCBIT0xERVJTIEFORCBDT05UUklCVVRPUlMKKyMgIkFTIElTIiBBTkQgQU5ZIEVYUFJFU1Mg T1IgSU1QTElFRCBXQVJSQU5USUVTLCBJTkNMVURJTkcsIEJVVCBOT1QKKyMgTElNSVRFRCBUTywg VEhFIElNUExJRUQgV0FSUkFOVElFUyBPRiBNRVJDSEFOVEFCSUxJVFkgQU5EIEZJVE5FU1MgRk9S CisjIEEgUEFSVElDVUxBUiBQVVJQT1NFIEFSRSBESVNDTEFJTUVELiBJTiBOTyBFVkVOVCBTSEFM TCBUSEUgQ09QWVJJR0hUCisjIE9XTkVSIE9SIENPTlRSSUJVVE9SUyBCRSBMSUFCTEUgRk9SIEFO WSBESVJFQ1QsIElORElSRUNULCBJTkNJREVOVEFMLAorIyBTUEVDSUFMLCBFWEVNUExBUlksIE9S IENPTlNFUVVFTlRJQUwgREFNQUdFUyAoSU5DTFVESU5HLCBCVVQgTk9UCisjIExJTUlURUQgVE8s IFBST0NVUkVNRU5UIE9GIFNVQlNUSVRVVEUgR09PRFMgT1IgU0VSVklDRVM7IExPU1MgT0YgVVNF LAorIyBEQVRBLCBPUiBQUk9GSVRTOyBPUiBCVVNJTkVTUyBJTlRFUlJVUFRJT04pIEhPV0VWRVIg Q0FVU0VEIEFORCBPTiBBTlkKKyMgVEhFT1JZIE9GIExJQUJJTElUWSwgV0hFVEhFUiBJTiBDT05U UkFDVCwgU1RSSUNUIExJQUJJTElUWSwgT1IgVE9SVAorIyAoSU5DTFVESU5HIE5FR0xJR0VOQ0Ug T1IgT1RIRVJXSVNFKSBBUklTSU5HIElOIEFOWSBXQVkgT1VUIE9GIFRIRSBVU0UKKyMgT0YgVEhJ UyBTT0ZUV0FSRSwgRVZFTiBJRiBBRFZJU0VEIE9GIFRIRSBQT1NTSUJJTElUWSBPRiBTVUNIIERB TUFHRS4KKworaW1wb3J0IHRpbWUKKworCitjbGFzcyBfQ2hlY2twb2ludChvYmplY3QpOgorICAg ICIiIkEgcmVmZXJlbmNlIHBvaW50IGZvciBlc3RpbWF0aW5nIHRoZSByZW1haW5pbmcgdGltZS4i IiIKKyAgICBkZWYgX19pbml0X18oc2VsZiwgaW5kZXgsIHRvdGFsLCB0KToKKyAgICAgICAgc2Vs Zi5faW5kZXggPSBpbmRleAorICAgICAgICBzZWxmLl90b3RhbCA9IHRvdGFsCisgICAgICAgIHNl bGYuX3RpbWUgPSB0CisKKyAgICBkZWYgZXN0aW1hdGUoc2VsZiwgaW5kZXgsIG5vdyk6CisgICAg ICAgIGVsYXBzZWRfaW5kZXggPSBpbmRleCAtIHNlbGYuX2luZGV4CisgICAgICAgIGVsYXBzZWRf dGltZSA9IG5vdyAtIHNlbGYuX3RpbWUKKyAgICAgICAgcmV0dXJuIGVsYXBzZWRfdGltZSAqIChz ZWxmLl90b3RhbCAtIGluZGV4KSAvIGVsYXBzZWRfaW5kZXgKKworCitjbGFzcyBFVEEob2JqZWN0 KToKKyAgICAiIiJBIGNsYXNzIGZvciBlc3RpbWF0aW5nIHRoZSBhbW91bnQgb2YgdGltZSByZW1h aW5pbmcuCisgICAgVGhpcyBjbGFzcyB1c2VzIGFuIGV4cG9uZW50aWFsbHkgd2VpZ2hlZCBzZXJp ZXMgb2YgY2hlY2twb2ludHMgdG8gZXN0aW1hdGUKKyAgICBob3cgbXVjaCB0aW1lIHJlbWFpbnMu IiIiCisgICAgX3RvdGFsX2NoZWNrcG9pbnRzID0gMTAwCisgICAgX2NoZWNrcG9pbnRzX3RvX3Nr aXAgPSAxMAorCisgICAgZGVmIF9faW5pdF9fKHNlbGYsIHRvdGFsKToKKyAgICAgICAgc2VsZi5f dG90YWwgPSB0b3RhbAorICAgICAgICBzZWxmLl9jaGVja3BvaW50cyA9IFtdCisKKyAgICBkZWYg ZXN0aW1hdGUoc2VsZiwgaW5kZXgpOgorICAgICAgICAjIFRoaXMgYWxnb3JpdGhtIGlzIHByZXR0 eSBzaW1wbGUgZm9yIG5vdy4gV2UgYWN0dWFsbHkgd2FudCBhIHJvbGxpbmcKKyAgICAgICAgIyB3 aW5kb3csIGJ1dCB3ZSBjYW4gYWRkIHRoYXQgbGF0ZXIuCisgICAgICAgIG5vdyA9IHRpbWUudGlt ZSgpCisgICAgICAgIHNlbGYuX3VwZGF0ZShpbmRleCwgbm93KQorICAgICAgICBjaGVja3BvaW50 X2xlbmd0aCA9IGxlbihzZWxmLl9jaGVja3BvaW50cykKKyAgICAgICAgaWYgY2hlY2twb2ludF9s ZW5ndGggPCBzZWxmLl9jaGVja3BvaW50c190b19za2lwOgorICAgICAgICAgICAgcmV0dXJuIE5v bmUKKyAgICAgICAgZXRhID0gc2VsZi5fY2hlY2twb2ludHNbMF0uZXN0aW1hdGUoaW5kZXgsIG5v dykKKyAgICAgICAgZm9yIGkgaW4gcmFuZ2UoMCwgY2hlY2twb2ludF9sZW5ndGggLSBzZWxmLl9j aGVja3BvaW50c190b19za2lwKToKKyAgICAgICAgICAgIGV0YSA9IChldGEgKyBzZWxmLl9jaGVj a3BvaW50c1tpXS5lc3RpbWF0ZShpbmRleCwgbm93KSkgLyAyCisgICAgICAgIHJldHVybiBldGEK KworICAgIGRlZiBfdXBkYXRlKHNlbGYsIGluZGV4LCBub3cpOgorICAgICAgICAiIiJBZGQgYW5v dGhlciBjaGVja3BvaW50IHRvIG91ciBkYXRhYmFzZSwgaWYgbmVlZGVkLiIiIgorICAgICAgICBp ZiBzZWxmLl90b3RhbCA8IHNlbGYuX3RvdGFsX2NoZWNrcG9pbnRzOgorICAgICAgICAgICAgcmV0 dXJuCisgICAgICAgIGlmIGluZGV4ICUgKHNlbGYuX3RvdGFsIC8gc2VsZi5fdG90YWxfY2hlY2tw b2ludHMpICE9IDA6CisgICAgICAgICAgICByZXR1cm4KKyAgICAgICAgc2VsZi5fY2hlY2twb2lu dHMuYXBwZW5kKF9DaGVja3BvaW50KGluZGV4LCBzZWxmLl90b3RhbCwgbm93KSkKZGlmZiAtLWdp dCBhL1dlYktpdFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRf dGVzdHMucHkgYi9XZWJLaXRUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5f d2Via2l0X3Rlc3RzLnB5CmluZGV4IGQ2ODU0NDIuLjVkZjU0NDQgMTAwNzU1Ci0tLSBhL1dlYktp dFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHkK KysrIGIvV2ViS2l0VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtp dF90ZXN0cy5weQpAQCAtNjAsNiArNjAsNyBAQCBpbXBvcnQgdHJhY2ViYWNrCiAKIGltcG9ydCBz aW1wbGVqc29uCiAKK2Zyb20gbGF5b3V0X3BhY2thZ2UgaW1wb3J0IGV0YQogZnJvbSBsYXlvdXRf cGFja2FnZSBpbXBvcnQgdGVzdF9leHBlY3RhdGlvbnMKIGZyb20gbGF5b3V0X3BhY2thZ2UgaW1w b3J0IGpzb25fbGF5b3V0X3Jlc3VsdHNfZ2VuZXJhdG9yCiBmcm9tIGxheW91dF9wYWNrYWdlIGlt cG9ydCBtZXRlcmVkX3N0cmVhbQpAQCAtMTgwLDYgKzE4MSw3IEBAIGNsYXNzIFRlc3RSdW5uZXI6 CiAgICAgICAgIHNlbGYuX3BvcnQgPSBwb3J0CiAgICAgICAgIHNlbGYuX29wdGlvbnMgPSBvcHRp b25zCiAgICAgICAgIHNlbGYuX21ldGVyID0gbWV0ZXIKKyAgICAgICAgc2VsZi5fZXRhID0gTm9u ZQogCiAgICAgICAgICMgZGlzYWJsZSB3c3Mgc2VydmVyLiBuZWVkIHRvIGluc3RhbGwgcHlPcGVu U1NMIG9uIGJ1aWxkYm90cy4KICAgICAgICAgIyBzZWxmLl93ZWJzb2NrZXRfc2VjdXJlX3NlcnZl ciA9IHdlYnNvY2tldF9zZXJ2ZXIuUHlXZWJTb2NrZXQoCkBAIC01NTcsNiArNTU5LDcgQEAgY2xh c3MgVGVzdFJ1bm5lcjoKICAgICAgICAgICAgIHJlc3VsdF9zdW1tYXJ5OiBzdW1tYXJ5IG9iamVj dCB0byBwb3B1bGF0ZSB3aXRoIHRoZSByZXN1bHRzCiAgICAgICAgICIiIgogICAgICAgICBzZWxm Ll9tZXRlci51cGRhdGUoJ1N0YXJ0aW5nIHRlc3Qgc2hlbGxzIC4uLicpCisgICAgICAgIHNlbGYu X2V0YSA9IE5vbmUKICAgICAgICAgdGhyZWFkcyA9IHNlbGYuX2luc3RhbnRpYXRlX3Rlc3Rfc2hl bGxfdGhyZWFkcyhmaWxlX2xpc3QsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgcmVzdWx0X3N1bW1hcnkpCiAKQEAgLTcxNCwxMCArNzE3LDE2 IEBAIGNsYXNzIFRlc3RSdW5uZXI6CiAKICAgICBkZWYgX2Rpc3BsYXlfb25lX2xpbmVfcHJvZ3Jl c3Moc2VsZiwgcmVzdWx0X3N1bW1hcnkpOgogICAgICAgICAiIiJEaXNwbGF5cyB0aGUgcHJvZ3Jl c3MgdGhyb3VnaCB0aGUgdGVzdCBydW4uIiIiCi0gICAgICAgIHBlcmNlbnRfY29tcGxldGUgPSAx MDAgKiAocmVzdWx0X3N1bW1hcnkuZXhwZWN0ZWQgKyByZXN1bHRfc3VtbWFyeS51bmV4cGVjdGVk KSAvIHJlc3VsdF9zdW1tYXJ5LnRvdGFsCi0gICAgICAgIHNlbGYuX21ldGVyLnVwZGF0ZSgiVGVz dGluZyAoJWQlJSk6ICVkIHJhbiBhcyBleHBlY3RlZCwgJWQgZGlkbid0LCAlZCBsZWZ0IiAlCi0g ICAgICAgICAgICAocGVyY2VudF9jb21wbGV0ZSwgcmVzdWx0X3N1bW1hcnkuZXhwZWN0ZWQsCi0g ICAgICAgICAgICAgcmVzdWx0X3N1bW1hcnkudW5leHBlY3RlZCwgcmVzdWx0X3N1bW1hcnkucmVt YWluaW5nKSkKKyAgICAgICAgaWYgbm90IHNlbGYuX2V0YToKKyAgICAgICAgICAgIHNlbGYuX2V0 YSA9IGV0YS5FVEEocmVzdWx0X3N1bW1hcnkudG90YWwpCisgICAgICAgIG51bWJlcl9yYW4gPSBy ZXN1bHRfc3VtbWFyeS5leHBlY3RlZCArIHJlc3VsdF9zdW1tYXJ5LnVuZXhwZWN0ZWQKKyAgICAg ICAgcGVyY2VudF9jb21wbGV0ZSA9IDEwMCAqIG51bWJlcl9yYW4gLyByZXN1bHRfc3VtbWFyeS50 b3RhbAorICAgICAgICBldGFfZXN0aW1hdGUgPSBzZWxmLl9ldGEuZXN0aW1hdGUobnVtYmVyX3Jh bikKKyAgICAgICAgZXRhX3N0cmluZyA9ICIsICVkcyByZW1haW5pbmciICUgbWF0aC5mbG9vcihl dGFfZXN0aW1hdGUpIGlmIGV0YV9lc3RpbWF0ZSBlbHNlICIiCisgICAgICAgIHNlbGYuX21ldGVy LnVwZGF0ZSgiVGVzdGluZyAoJWQlJSVzKTogJWQgcmFuIGFzIGV4cGVjdGVkLCAlZCBkaWRuJ3Qs ICVkIGxlZnQiICUKKyAgICAgICAgICAgIChwZXJjZW50X2NvbXBsZXRlLCBldGFfc3RyaW5nLAor ICAgICAgICAgICAgIHJlc3VsdF9zdW1tYXJ5LmV4cGVjdGVkLCByZXN1bHRfc3VtbWFyeS51bmV4 cGVjdGVkLAorICAgICAgICAgICAgIHJlc3VsdF9zdW1tYXJ5LnJlbWFpbmluZykpCiAKICAgICBk ZWYgX2Rpc3BsYXlfZGV0YWlsZWRfcHJvZ3Jlc3Moc2VsZiwgcmVzdWx0X3N1bW1hcnkpOgogICAg ICAgICAiIiJEaXNwbGF5IGRldGFpbGVkIHByb2dyZXNzIG91dHB1dCB3aGVyZSB3ZSBwcmludCB0 aGUgZGlyZWN0b3J5IG5hbWUK