90062 2012-06-27 04:32:21 -0700 Output numbers of run-perf-tests should be rounded in accordance with sample standard deviations 2017-07-18 08:30:20 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified REOPENED P2 Normal --- 77037 1 haraken haraken abarth dpranke eric pdr rniwa webkit.review.bot zimmermann zoltan oldest_to_newest 658528 0 haraken 2012-06-27 04:32:21 -0700 Currently the output of performance tests looks like this: DESCRIPTION: This benchmark covers DOM attributes that return small integers. RESULT Bindings: scroll-top= 191.379586433 runs/s median= 191.731644624 runs/s, stdev= 2.56134259787 runs/s, min= 180.412371134 runs/s, max= 192.771084337 runs/s "191.731644624 runs/s" is too verbose. It should be "191.73 runs/s". Whichever units we use (runs/s or ms), rounding off the output to the second decimal place would be enough and preferable. 658530 1 149726 haraken 2012-06-27 04:35:27 -0700 Created attachment 149726 Patch 658661 2 149726 rniwa 2012-06-27 09:31:02 -0700 Comment on attachment 149726 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=149726&action=review > Tools/ChangeLog:16 > + "191.731644624 runs/s" is too verbose. It should be "191.73 runs/s". > + Whichever unit we use (runs/s or ms), rounding off the output to the second > + decimal place would be enough and preferable. No, the whole point of printing out all decimals is so that you can apply arithmetic operations on those numbers and still preserve enough significant figures. e.g. there are tests that only have 0.7 runs/s for example. For those tests, printing 2 decimals points would mean that we'd get only 2 significant digits. That's not acceptable. In order to figure out the right significant figures, we need to first figure out the measurement errors. From measurement errors, we can find out the sufficient significant figures for standard deviations, from which we can find out the right significant figures for the mean, median, etc... Just picking 2 decimal point arbitrarily isn't an acceptable solution here. 658663 3 rniwa 2012-06-27 09:33:22 -0700 +WildFox since he would surely know all about significant figures for he is a physicist :) 658972 4 haraken 2012-06-27 16:51:12 -0700 (In reply to comment #2) > No, the whole point of printing out all decimals is so that you can apply arithmetic operations on those numbers and still preserve enough significant figures. Are the numbers used for arithmetic calculations? I've thought that _log.info() lines are just for human readability, and the arithmetic calculations (e.g. drawing perf-bot graphs etc) are done by using 'results', which contains all decimals. 658981 5 rniwa 2012-06-27 16:55:02 -0700 (In reply to comment #4) > (In reply to comment #2) > > No, the whole point of printing out all decimals is so that you can apply arithmetic operations on those numbers and still preserve enough significant figures. > > Are the numbers used for arithmetic calculations? I've thought that _log.info() lines are just for human readability, and the arithmetic calculations (e.g. drawing perf-bot graphs etc) are done by using 'results', which contains all decimals. Yes, I (humans) compute average between different runs all the time. 658984 6 haraken 2012-06-27 16:59:47 -0700 OK, marking as WONTFIX. 845531 7 rniwa 2013-03-01 14:55:45 -0800 Let's do this properly. 845551 8 191049 rniwa 2013-03-01 15:12:58 -0800 Created attachment 191049 Fixes the bug 845554 9 191049 eric 2013-03-01 15:16:22 -0800 Comment on attachment 191049 Fixes the bug View in context: https://bugs.webkit.org/attachment.cgi?id=191049&action=review > Tools/ChangeLog:12 > + We show one more digit than the most significant digit in the sample standard deviation since it's > + helpful when doing back-of-the-envelop calculations using these numbers. I have no stats background... but it seems like digits are either significant or not, right? If you're including non-significant digits in these calculations, isn't that just noise? > Tools/Scripts/webkitpy/performance_tests/perftest.py:159 > + return round(x, -highest_decimal_place(x) + sig_fig - 1) Confused by the -1? > Tools/Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py:91 > +median= 1488 ms, stdev= 1.02 %, min= 1471 ms, max= 1510 ms Why the change to percent? 845563 10 191049 rniwa 2013-03-01 15:32:13 -0800 Comment on attachment 191049 Fixes the bug View in context: https://bugs.webkit.org/attachment.cgi?id=191049&action=review >> Tools/ChangeLog:12 >> + helpful when doing back-of-the-envelop calculations using these numbers. > > I have no stats background... but it seems like digits are either significant or not, right? If you're including non-significant digits in these calculations, isn't that just noise? "most significant digit" as in http://planetmath.org/MostSignificantDigit.html i.e. left most digit of a number when it's shown in the positional notation: http://en.wikipedia.org/wiki/Positional_notation Maybe there's a better word for it? >> Tools/Scripts/webkitpy/performance_tests/perftest.py:159 >> + return round(x, -highest_decimal_place(x) + sig_fig - 1) > > Confused by the -1? Say we have 1.2345, then to round it at 3rd sig. fig. we do: round(1.2345, 2), not round(1.2345, 3) 845564 11 191049 rniwa 2013-03-01 15:33:21 -0800 Comment on attachment 191049 Fixes the bug View in context: https://bugs.webkit.org/attachment.cgi?id=191049&action=review >> Tools/Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py:91 >> +median= 1488 ms, stdev= 1.02 %, min= 1471 ms, max= 1510 ms > > Why the change to percent? I find that to be more useful maybe maybe not? 845765 12 eric 2013-03-02 00:50:44 -0800 The patch looks reasonable, but you should really ask someone who's taken a stats class to review. :) 845866 13 zoltan 2013-03-02 13:35:39 -0800 (In reply to comment #11) > (From update of attachment 191049 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=191049&action=review > > >> Tools/Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py:91 > >> +median= 1488 ms, stdev= 1.02 %, min= 1471 ms, max= 1510 ms > > > > Why the change to percent? > > I find that to be more useful maybe maybe not? (In reply to comment #12) > The patch looks reasonable, but you should really ask someone who's taken a stats class to review. :) I prefer using the percentage, it's more straightforward. 1099975 14 191049 rniwa 2015-06-05 14:49:08 -0700 Comment on attachment 191049 Fixes the bug This patch is too old. 149726 2012-06-27 04:35:27 -0700 2013-03-01 15:12:55 -0800 Patch bug-90062-20120627203526.patch text/plain 2170 haraken U3VidmVyc2lvbiBSZXZpc2lvbjogMTIxMzM1CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZDc3N2VlM2UzOGJjOTFhMDkxYTVhNjk4YTY2NDQ0YTRi ZDliNWE2Zi4uMTY0NDU0Nzg1ZDQ4NjlmNGE5ZmVmM2VlZDRhMWNkNjE1MThmOGFjOCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDIz IEBACisyMDEyLTA2LTI3ICBLZW50YXJvIEhhcmEgIDxoYXJha2VuQGNocm9taXVtLm9yZz4KKwor ICAgICAgICBPdXRwdXQgbnVtYmVycyBvZiBwZXJmb3JtYW5jZSB0ZXN0cyBhcmUgdG9vIHZlcmJv c2UKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTkwMDYy CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgQ3VycmVu dGx5IHRoZSBvdXRwdXQgb2YgcGVyZm9ybWFuY2UgdGVzdHMgbG9va3MgbGlrZSB0aGlzOgorCisg ICAgICAgICAgREVTQ1JJUFRJT046IFRoaXMgYmVuY2htYXJrIGNvdmVycyBET00gYXR0cmlidXRl cyB0aGF0IHJldHVybiBzbWFsbCBpbnRlZ2Vycy4KKyAgICAgICAgICBSRVNVTFQgQmluZGluZ3M6 IHNjcm9sbC10b3A9IDE5MS4zNzk1ODY0MzMgcnVucy9zCisgICAgICAgICAgbWVkaWFuPSAxOTEu NzMxNjQ0NjI0IHJ1bnMvcywgc3RkZXY9IDIuNTYxMzQyNTk3ODcgcnVucy9zLCBtaW49IDE4MC40 MTIzNzExMzQgcnVucy9zLCBtYXg9IDE5Mi43NzEwODQzMzcgcnVucy9zCisKKyAgICAgICAgIjE5 MS43MzE2NDQ2MjQgcnVucy9zIiBpcyB0b28gdmVyYm9zZS4gSXQgc2hvdWxkIGJlICIxOTEuNzMg cnVucy9zIi4KKyAgICAgICAgV2hpY2hldmVyIHVuaXQgd2UgdXNlIChydW5zL3Mgb3IgbXMpLCBy b3VuZGluZyBvZmYgdGhlIG91dHB1dCB0byB0aGUgc2Vjb25kCisgICAgICAgIGRlY2ltYWwgcGxh Y2Ugd291bGQgYmUgZW5vdWdoIGFuZCBwcmVmZXJhYmxlLgorCisgICAgICAgICogU2NyaXB0cy93 ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdC5weToKKyAgICAgICAgKFBlcmZUZXN0 Lm91dHB1dF9zdGF0aXN0aWNzKToKKwogMjAxMi0wNi0yNyAgWmFuIERvYmVyc2VrICA8emFuZG9i ZXJzZWtAZ21haWwuY29tPgogCiAgICAgICAgIFtHdGtdIEFkZCBzdXBwb3J0IGZvciB0aGUgR2Ft ZXBhZCBBUEkKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvcGVyZm9ybWFuY2Vf dGVzdHMvcGVyZnRlc3QucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rl c3RzL3BlcmZ0ZXN0LnB5CmluZGV4IGY3NmQ4NWIxYjI3ZjM4ZDQwMjJiMDk0YTFjZTQ5NjcxYjNm YTZjNTUuLmM1Yjk3ZWU3MGZkZGQ1NmFjMWE5OWRlNWM1ZmE3M2ZmOTMyYzY1M2EgMTAwNjQ0Ci0t LSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvcGVyZm9ybWFuY2VfdGVzdHMvcGVyZnRlc3QucHkK KysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdC5w eQpAQCAtMTUxLDggKzE1MSw4IEBAIGNsYXNzIFBlcmZUZXN0KG9iamVjdCk6CiAgICAgICAgIHVu aXQgPSByZXN1bHRzWyd1bml0J10KICAgICAgICAgaWYgZGVzY3JpcHRpb25fc3RyaW5nOgogICAg ICAgICAgICAgX2xvZy5pbmZvKCdERVNDUklQVElPTjogJXMnICUgZGVzY3JpcHRpb25fc3RyaW5n KQotICAgICAgICBfbG9nLmluZm8oJ1JFU1VMVCAlcz0gJXMgJXMnICUgKHRlc3RfbmFtZS5yZXBs YWNlKCcvJywgJzogJyksIHJlc3VsdHNbJ2F2ZyddLCB1bml0KSkKLSAgICAgICAgX2xvZy5pbmZv KCcsICcuam9pbihbJyVzPSAlcyAlcycgJSAoa2V5LCByZXN1bHRzW2tleV0sIHVuaXQpIGZvciBr ZXkgaW4gc2VsZi5fc3RhdGlzdGljc19rZXlzWzE6XV0pKQorICAgICAgICBfbG9nLmluZm8oJ1JF U1VMVCAlcz0gJS4ybGYgJXMnICUgKHRlc3RfbmFtZS5yZXBsYWNlKCcvJywgJzogJyksIHJlc3Vs dHNbJ2F2ZyddLCB1bml0KSkKKyAgICAgICAgX2xvZy5pbmZvKCcsICcuam9pbihbJyVzPSAlLjJs ZiAlcycgJSAoa2V5LCByZXN1bHRzW2tleV0sIHVuaXQpIGZvciBrZXkgaW4gc2VsZi5fc3RhdGlz dGljc19rZXlzWzE6XV0pKQogCiAKIGNsYXNzIENocm9taXVtU3R5bGVQZXJmVGVzdChQZXJmVGVz dCk6Cg== 191049 2013-03-01 15:12:58 -0800 2015-06-05 14:49:08 -0700 Fixes the bug bug-90062-20130301150909.patch text/plain 6079 rniwa SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDE0NDUxMCkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDI2IEBACisyMDEzLTAzLTAxICBSeW9zdWtlIE5pd2EgIDxybml3YUB3ZWJraXQub3JnPgor CisgICAgICAgIE91dHB1dCBudW1iZXJzIG9mIHJ1bi1wZXJmLXRlc3RzIHNob3VsZCBiZSByb3Vu ZGVkIGluIGFjY29yZGFuY2Ugd2l0aCBzYW1wbGUgc3RhbmRhcmQgZGV2aWF0aW9ucworICAgICAg ICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9OTAwNjIKKworICAgICAg ICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBSb3VuZCB0aGUgc2FtcGxl IHN0YW5kYXJkIGRldmlhdGlvbiBhdCAzIHNpZ25pZmljYW50IGZpZ3VyZXMgYW5kIHNob3cgaXQg aW4gcGVyY2VudGFnZSB3aXRoIHJlc3BlY3QKKyAgICAgICAgdG8gdGhlIG1lYW4uIFJvdW5kIG90 aGVyIHN0YXRpc3RpY3MgYmFzZWQgb24gdGhlIG1vc3Qgc2lnbmlmaWNhbnQgZGlnaXQgb2YgdGhl IHNhbXBsZSBzdGFuZGFyZCBkZXZpYXRpb24uCisKKyAgICAgICAgV2Ugc2hvdyBvbmUgbW9yZSBk aWdpdCB0aGFuIHRoZSBtb3N0IHNpZ25pZmljYW50IGRpZ2l0IGluIHRoZSBzYW1wbGUgc3RhbmRh cmQgZGV2aWF0aW9uIHNpbmNlIGl0J3MKKyAgICAgICAgaGVscGZ1bCB3aGVuIGRvaW5nIGJhY2st b2YtdGhlLWVudmVsb3AgY2FsY3VsYXRpb25zIHVzaW5nIHRoZXNlIG51bWJlcnMuCisKKyAgICAg ICAgV2UgYWxzbyBvbmx5IHJvdW5kIGRlY2ltYWwgcG9pbnRzIHNpbmNlIGl0J3Mgbm90IG9idmlv dXMgYXQgd2hpY2ggcG9pbnQgbnVtYmVycyBhcmUgcm91bmRlZCB3aGVuIHdlIGhhdmUKKyAgICAg ICAgYnVuY2ggb2YgemVyb3MuIGUuZy4gMTIzMDAwIHZlcnN1cyAxMjM0NTYgd2hlcmUgNDU2IGlz IHRoZSByb3VuZGVkIHBvcnRpb24uCisKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L3BlcmZv cm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0LnB5OgorICAgICAgICAoUGVyZlRlc3QubG9nX3N0YXRpc3Rp Y3MuaGlnaGVzdF9kZWNpbWFsX3BsYWNlKTogQWRkZWQuCisgICAgICAgIChQZXJmVGVzdC5sb2df c3RhdGlzdGljcy5yb3VuZF90b19zaWdfZmlnKTogQWRkZWQuCisgICAgICAgIChQZXJmVGVzdC5s b2dfc3RhdGlzdGljcyk6CisgICAgICAgIChQZXJmVGVzdC5sb2dfc3RhdGlzdGljcy5yb3VuZF90 b19zaWdfZmlnKToKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3Rz L3BlcmZ0ZXN0c3J1bm5lcl9pbnRlZ3JhdGlvbnRlc3QucHk6CisKIDIwMTMtMDMtMDEgIFJ5b3N1 a2UgTml3YSAgPHJuaXdhQHdlYmtpdC5vcmc+CiAKICAgICAgICAgRG9uJ3QgdXNlIGxlZ2FjeSB0 ZXN0IG5hbWVzIHdoZW4gcmV0dXJuaW5nIHJlc3VsdHMgZnJvbSBQZXJmVGVzdC5ydW4KSW5kZXg6 IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvcGVyZm9ybWFuY2VfdGVzdHMvcGVyZnRlc3QucHkKPT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PQotLS0gVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJm dGVzdC5weQkocmV2aXNpb24gMTQ0NTEwKQorKysgVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJm b3JtYW5jZV90ZXN0cy9wZXJmdGVzdC5weQkod29ya2luZyBjb3B5KQpAQCAtMSw0ICsxLDUgQEAK ICMgQ29weXJpZ2h0IChDKSAyMDEyIEdvb2dsZSBJbmMuIEFsbCByaWdodHMgcmVzZXJ2ZWQuCisj IENvcHlyaWdodCAoQykgMjAxMiwgMjAxMyBBcHBsZSBJbmMuIEFsbCByaWdodHMgcmVzZXJ2ZWQu CiAjIENvcHlyaWdodCAoQykgMjAxMiBab2x0YW4gSG9ydmF0aCwgQWRvYmUgU3lzdGVtcyBJbmNv cnBvcmF0ZWQuIEFsbCByaWdodHMgcmVzZXJ2ZWQuCiAjCiAjIFJlZGlzdHJpYnV0aW9uIGFuZCB1 c2UgaW4gc291cmNlIGFuZCBiaW5hcnkgZm9ybXMsIHdpdGggb3Igd2l0aG91dApAQCAtMTUxLDkg KzE1MiwyMiBAQCBjbGFzcyBQZXJmVGVzdChvYmplY3QpOgogICAgICAgICBtZWRpYW4gPSBzb3J0 ZWRfdmFsdWVzW21pZGRsZV0gaWYgbGVuKHNvcnRlZF92YWx1ZXMpICUgMiBlbHNlIChzb3J0ZWRf dmFsdWVzW21pZGRsZSAtIDFdICsgc29ydGVkX3ZhbHVlc1ttaWRkbGVdKSAvIDIKICAgICAgICAg c3RkZXYgPSBtYXRoLnNxcnQoc3F1YXJlX3N1bSAvIChsZW4oc29ydGVkX3ZhbHVlcykgLSAxKSkg aWYgbGVuKHNvcnRlZF92YWx1ZXMpID4gMSBlbHNlIDAKIAotICAgICAgICBfbG9nLmluZm8oJ1JF U1VMVCAlcz0gJXMgJXMnICUgKHRlc3RfbmFtZSwgbWVhbiwgdW5pdCkpCi0gICAgICAgIF9sb2cu aW5mbygnbWVkaWFuPSAlcyAlcywgc3RkZXY9ICVzICVzLCBtaW49ICVzICVzLCBtYXg9ICVzICVz JyAlCi0gICAgICAgICAgICAobWVkaWFuLCB1bml0LCBzdGRldiwgdW5pdCwgc29ydGVkX3ZhbHVl c1swXSwgdW5pdCwgc29ydGVkX3ZhbHVlc1stMV0sIHVuaXQpKQorICAgICAgICBkZWYgaGlnaGVz dF9kZWNpbWFsX3BsYWNlKHgpOgorICAgICAgICAgICAgcmV0dXJuIGludChtYXRoLmZsb29yKG1h dGgubG9nKHgsIDEwKSkpCisKKyAgICAgICAgZGVmIHJvdW5kX3RvX3NpZ19maWcoeCwgc2lnX2Zp Zyk6CisgICAgICAgICAgICByZXR1cm4gcm91bmQoeCwgLWhpZ2hlc3RfZGVjaW1hbF9wbGFjZSh4 KSArIHNpZ19maWcgLSAxKQorCisgICAgICAgIHN0ZGV2X2RlY2ltYWxfcGxhY2UgPSAtaGlnaGVz dF9kZWNpbWFsX3BsYWNlKHN0ZGV2KSArIDEKKyAgICAgICAgaWYgc3RkZXZfZGVjaW1hbF9wbGFj ZSA8PSAwOgorICAgICAgICAgICAgcm91bmRfdG9fc3RkZXYgPSBsYW1iZGEgeDogaW50KHJvdW5k KHgpKQorICAgICAgICBlbHNlOgorICAgICAgICAgICAgcm91bmRfdG9fc3RkZXYgPSBsYW1iZGEg eDogcm91bmQoeCwgc3RkZXZfZGVjaW1hbF9wbGFjZSkKKworICAgICAgICBfbG9nLmluZm8oJ1JF U1VMVCAlcz0gJXMgJXMnICUgKHRlc3RfbmFtZSwgcm91bmRfdG9fc3RkZXYobWVhbiksIHVuaXQp KQorICAgICAgICBfbG9nLmluZm8oJ21lZGlhbj0gJXMgJXMsIHN0ZGV2PSAlcyAlJSwgbWluPSAl cyAlcywgbWF4PSAlcyAlcycgJQorICAgICAgICAgICAgKHJvdW5kX3RvX3N0ZGV2KG1lZGlhbiks IHVuaXQsIHJvdW5kX3RvX3NpZ19maWcoc3RkZXYgLyBtZWFuICogMTAwLCAzKSwKKyAgICAgICAg ICAgIHJvdW5kX3RvX3N0ZGV2KHNvcnRlZF92YWx1ZXNbMF0pLCB1bml0LCByb3VuZF90b19zdGRl dihzb3J0ZWRfdmFsdWVzWy0xXSksIHVuaXQpKQogCiAgICAgX2Rlc2NyaXB0aW9uX3JlZ2V4ID0g cmUuY29tcGlsZShyJ15EZXNjcmlwdGlvbjogKD9QPGRlc2NyaXB0aW9uPi4qKSQnLCByZS5JR05P UkVDQVNFKQogICAgIF9tZXRyaWNzX3JlZ2V4ID0gcmUuY29tcGlsZShyJ14oP1A8bWV0cmljPlRp bWV8TWFsbG9jfEpTIEhlYXApOicpCkluZGV4OiBUb29scy9TY3JpcHRzL3dlYmtpdHB5L3BlcmZv cm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0c3J1bm5lcl9pbnRlZ3JhdGlvbnRlc3QucHkKPT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PQotLS0gVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdHNy dW5uZXJfaW50ZWdyYXRpb250ZXN0LnB5CShyZXZpc2lvbiAxNDQ0NzgpCisrKyBUb29scy9TY3Jp cHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0c3J1bm5lcl9pbnRlZ3JhdGlv bnRlc3QucHkJKHdvcmtpbmcgY29weSkKQEAgLTEsNCArMSw1IEBACiAjIENvcHlyaWdodCAoQykg MjAxMiBHb29nbGUgSW5jLiBBbGwgcmlnaHRzIHJlc2VydmVkLgorIyBDb3B5cmlnaHQgKEMpIDIw MTIsIDIwMTMgQXBwbGUgSW5jLiBBbGwgcmlnaHRzIHJlc2VydmVkLgogIwogIyBSZWRpc3RyaWJ1 dGlvbiBhbmQgdXNlIGluIHNvdXJjZSBhbmQgYmluYXJ5IGZvcm1zLCB3aXRoIG9yIHdpdGhvdXQK ICMgbW9kaWZpY2F0aW9uLCBhcmUgcGVybWl0dGVkIHByb3ZpZGVkIHRoYXQgdGhlIGZvbGxvd2lu ZyBjb25kaXRpb25zIGFyZQpAQCAtODYsOCArODcsOCBAQCBtYXggMTUxMCBtcwogIiIiCiAKICAg ICBvdXRwdXQgPSAiIiJSdW5uaW5nIEJpbmRpbmdzL2V2ZW50LXRhcmdldC13cmFwcGVyLmh0bWwg KDEgb2YgMikKLVJFU1VMVCBCaW5kaW5nczogZXZlbnQtdGFyZ2V0LXdyYXBwZXI6IFRpbWU9IDE0 OTAuMCBtcwotbWVkaWFuPSAxNDg4LjAgbXMsIHN0ZGV2PSAxNS4xMzkzNSBtcywgbWluPSAxNDcx LjAgbXMsIG1heD0gMTUxMC4wIG1zCitSRVNVTFQgQmluZGluZ3M6IGV2ZW50LXRhcmdldC13cmFw cGVyOiBUaW1lPSAxNDkwIG1zCittZWRpYW49IDE0ODggbXMsIHN0ZGV2PSAxLjAyICUsIG1pbj0g MTQ3MSBtcywgbWF4PSAxNTEwIG1zCiBGaW5pc2hlZDogMC4xIHMKIAogIiIiCkBAIC0xMDEsMTcg KzEwMiwxNyBAQCBjbGFzcyBTb21lUGFyc2VyVGVzdERhdGE6CiBJZ25vcmluZyB3YXJtLXVwIHJ1 biAoMTExNSkKIAogVGltZToKLXZhbHVlcyAxMDgwLCAxMTIwLCAxMDk1LCAxMTAxLCAxMTA0IG1z Ci1hdmcgMTEwMCBtcwotbWVkaWFuIDExMDEgbXMKLXN0ZGV2IDE0LjUwODYxIG1zCi1taW4gMTA4 MCBtcwotbWF4IDExMjAgbXMKK3ZhbHVlcyAxMC44MywgMTEuMjAsIDEwLjk1LCAxMS4wMSwgMTEu MDQgbXMKK2F2ZyAxMS4wMSBtcworbWVkaWFuIDExLjAxIG1zCitzdGRldiAwLjEzNTAxIG1zCitt aW4gMTAuODMgbXMKK21heCAxMS4yMCBtcwogIiIiCiAKICAgICBvdXRwdXQgPSAiIiJSdW5uaW5n IFBhcnNlci9zb21lLXBhcnNlci5odG1sICgyIG9mIDIpCi1SRVNVTFQgUGFyc2VyOiBzb21lLXBh cnNlcjogVGltZT0gMTEwMC4wIG1zCi1tZWRpYW49IDExMDEuMCBtcywgc3RkZXY9IDE0LjUwODYx IG1zLCBtaW49IDEwODAuMCBtcywgbWF4PSAxMTIwLjAgbXMKK1JFU1VMVCBQYXJzZXI6IHNvbWUt cGFyc2VyOiBUaW1lPSAxMS4wMSBtcworbWVkaWFuPSAxMS4wMSBtcywgc3RkZXY9IDEuMjMgJSwg bWluPSAxMC44MyBtcywgbWF4PSAxMS4yIG1zCiBGaW5pc2hlZDogMC4xIHMKIAogIiIiCkBAIC0x NDgsMTIgKzE0OSwxMiBAQCBtYXggNTQ4MDAwIGJ5dGVzCiAKICAgICBvdXRwdXQgPSAiIiJSdW5u aW5nIDEgdGVzdHMKIFJ1bm5pbmcgUGFyc2VyL21lbW9yeS10ZXN0Lmh0bWwgKDEgb2YgMSkKLVJF U1VMVCBQYXJzZXI6IG1lbW9yeS10ZXN0OiBUaW1lPSAxMTAwLjAgbXMKLW1lZGlhbj0gMTEwMS4w IG1zLCBzdGRldj0gMTQuNTA4NjEgbXMsIG1pbj0gMTA4MC4wIG1zLCBtYXg9IDExMjAuMCBtcwot UkVTVUxUIFBhcnNlcjogbWVtb3J5LXRlc3Q6IEpTSGVhcD0gODMwMDAwLjAgYnl0ZXMKLW1lZGlh bj0gODI5MDAwLjAgYnl0ZXMsIHN0ZGV2PSAxMzc4NC4wNDg3NSBieXRlcywgbWluPSA4MTEwMDAu MCBieXRlcywgbWF4PSA4NDgwMDAuMCBieXRlcwotUkVTVUxUIFBhcnNlcjogbWVtb3J5LXRlc3Q6 IE1hbGxvYz0gNTI5MDAwLjAgYnl0ZXMKLW1lZGlhbj0gNTI5MDAwLjAgYnl0ZXMsIHN0ZGV2PSAx NDEyNC40NDY4OSBieXRlcywgbWluPSA1MTEwMDAuMCBieXRlcywgbWF4PSA1NDgwMDAuMCBieXRl cworUkVTVUxUIFBhcnNlcjogbWVtb3J5LXRlc3Q6IFRpbWU9IDExMDAgbXMKK21lZGlhbj0gMTEw MSBtcywgc3RkZXY9IDEuMzIgJSwgbWluPSAxMDgwIG1zLCBtYXg9IDExMjAgbXMKK1JFU1VMVCBQ YXJzZXI6IG1lbW9yeS10ZXN0OiBKU0hlYXA9IDgzMDAwMCBieXRlcworbWVkaWFuPSA4MjkwMDAg Ynl0ZXMsIHN0ZGV2PSAxLjY2ICUsIG1pbj0gODExMDAwIGJ5dGVzLCBtYXg9IDg0ODAwMCBieXRl cworUkVTVUxUIFBhcnNlcjogbWVtb3J5LXRlc3Q6IE1hbGxvYz0gNTI5MDAwIGJ5dGVzCittZWRp YW49IDUyOTAwMCBieXRlcywgc3RkZXY9IDIuNjcgJSwgbWluPSA1MTEwMDAgYnl0ZXMsIG1heD0g NTQ4MDAwIGJ5dGVzCiBGaW5pc2hlZDogMC4xIHMKICIiIgogCg==