172968 2017-06-06 03:54:36 -0700 Compute the final score using geometric mean in Speedometer 2.0 2017-09-27 12:53:45 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 172339 1 mathias rniwa addyo bmeurer buildbot cdumez ehsan ggaren lforschler mathias mjs pan.deng rniwa saam shiyu.zhang tianyou.li webkit-bug-importer oldest_to_newest 1315953 0 mathias 2017-06-06 03:54:36 -0700 Currently, Speedometer uses the arithmetic mean of individual test results. As a result, Speedometer is essentially dominated by 3 individual benchmarks. It would be nice to get that fixed. I’m happy to submit a patch that uses the geometric mean instead. 1316093 1 rniwa 2017-06-06 10:56:30 -0700 This is by design. We're weighting each test case based on how slow they're. 1322226 2 bmeurer 2017-06-23 01:53:12 -0700 With arithmetic mean, the overall score is a lot less useful. This way Speedometer 2 is basically a Ember/AngularJS/React-Redux benchmark. And this will remain, since Ember does essentially more than other tests that merely test the view layer (to get a serious comparison you'd probably need to replace Ember with Glimmer.js). Using geometric mean instead would make it a bit more useful, since changes for other frameworks would also be reflected in the overall score (in both Safari and Chrome). 1327472 3 tianyou.li 2017-07-11 08:21:55 -0700 As stated in http://browserbench.org/Speedometer/, "Speedometer is not meant to compare the performance of different JavaScript frameworks.". While current Speedometer score is more favorable of particular JS frameworks which take longer execution time for example Ember.js. Would it be possible to use geomean for score calculation which may help for other fast-running JS frameworks to contribute a bit more to the overall Speedometer score compared with original arithmetic mean calculation? 1342710 4 rniwa 2017-08-26 22:12:02 -0700 I think we need to revisit this more carefully. In the latest version of Speedometer 2.0, Preact runs in 554ms whereas Inferno takes 1533ms on Safari. Since the total time is 6533, Preact only accounts for ~0.8% of the total score whilst Inferno accounts for ~23% of the total score. At that point, there's almost no practical value in measuring Preact's runtime. 1342714 5 rniwa 2017-08-26 22:24:11 -0700 (In reply to Ryosuke Niwa from comment #4) > I think we need to revisit this more carefully. In the latest version of > Speedometer 2.0, Preact runs in 554ms whereas Inferno takes 1533ms on > Safari. Since the total time is 6533, Preact only accounts for ~0.8% of the > total score whilst Inferno accounts for ~23% of the total score. At that > point, there's almost no practical value in measuring Preact's runtime. Sorry, I meant to say Preact runs in 54ms, not 554ms. 1342718 6 rniwa 2017-08-26 23:41:35 -0700 Another reason we should probably consider using geomean is that we now have both release & debug builds of Ember.js after https://trac.webkit.org/changeset/221205 and https://trac.webkit.org/changeset/221206. We did this because we noticed that debug build was 4x slower and therefore constitutes a fundamentally different kind of a test. However, if we used arithmetic mean to compute the score, then we’re effectively giving 4x more weight to debug build of Ember.js compared to its release build even though only ~5% of websites that use Ember.js use debug builds in production. 1342727 7 mathias 2017-08-27 08:15:31 -0700 (In reply to Ryosuke Niwa from comment #6) > Another reason we should probably consider using geomean is that we now have > both release & debug builds of Ember.js after > https://trac.webkit.org/changeset/221205 and > https://trac.webkit.org/changeset/221206. > > We did this because we noticed that debug build was 4x slower and therefore > constitutes a fundamentally different kind of a test. > > However, if we used arithmetic mean to compute the score, then we’re > effectively giving 4x more weight to debug build of Ember.js compared to its > release build even though only ~5% of websites that use Ember.js use debug > builds in production. IMHO that just means we should remove the debug build of Ember.js from the benchmark altogether. 1343035 8 addyo 2017-08-28 19:41:20 -0700 There are a few possible options here: 1. Switch to the Geometric mean. Avoids an issue where the lower execution times of frameworks like Vue and Preact don't contribute much to the final score. Also avoids Speedometer appearing to highlight the cost of some frameworks more than others. 2. Adopt a hybrid approach of measuring both Arithmetic and Geometric means, taking an average of the two. 3. Minimize the impact to overall scores by excluding the Ember debug build. This may not be sufficient alone. 4. Consider other weighting factors to each implementation to avoid any specific framework contributing more to the score than others. 1343051 9 rniwa 2017-08-28 20:39:55 -0700 (In reply to Mathias Bynens from comment #7) > (In reply to Ryosuke Niwa from comment #6) > > Another reason we should probably consider using geomean is that we now have > > both release & debug builds of Ember.js after > > https://trac.webkit.org/changeset/221205 and > > https://trac.webkit.org/changeset/221206. > > > > We did this because we noticed that debug build was 4x slower and therefore > > constitutes a fundamentally different kind of a test. > > > > However, if we used arithmetic mean to compute the score, then we’re > > effectively giving 4x more weight to debug build of Ember.js compared to its > > release build even though only ~5% of websites that use Ember.js use debug > > builds in production. > > IMHO that just means we should remove the debug build of Ember.js from the > benchmark altogether. The goal of the Speedometer benchmark is to measure plausible ways DOM APIs will be used, not necessary only the most popular way, or most optimized way. Since 5% of websites that use ember.js use debug build, we should include it in the benchmark given how radically different its performance characteristics is. Additionally, this doesn't solve the problem that Vue.js contributes less than 1% of the total score whereas Inferno contributes more than 23% at least in Safari. (In reply to Addy Osmani from comment #8) > There are a few possible options here: > > 1. Switch to the Geometric mean. Avoids an issue where the lower execution > times of frameworks like Vue and Preact don't contribute much to the final > score. Also avoids Speedometer appearing to highlight the cost of some > frameworks more than others. We should probably do this. > 2. Adopt a hybrid approach of measuring both Arithmetic and Geometric means, > taking an average of the two. This has one problem that we're still going to give ~2x more weight to debug build of ember.js compared to release build of ember.js > 3. Minimize the impact to overall scores by excluding the Ember debug build. > This may not be sufficient alone. Right. Just removing debug build of ember.js doesn't solve the issue of Inferno account for ~23% of the test score while Vue.js accounts for less than 1%. > 4. Consider other weighting factors to each implementation to avoid any > specific framework contributing more to the score than others. Given we don't know have a good understanding of how popular each framework / library is, I don't think we could reasonably do this. And it's subject to a lot of interpretations and opinions. 1345387 10 319888 rniwa 2017-09-05 02:29:24 -0700 Created attachment 319888 Changes the score computation 1345466 11 319888 saam 2017-09-05 09:03:41 -0700 Comment on attachment 319888 Changes the score computation View in context: https://bugs.webkit.org/attachment.cgi?id=319888&action=review > PerformanceTests/Speedometer/resources/benchmark-runner.js:285 > + values.sort(function (a, b) { return a - b }); // Avoid the loss of significance for the sum. Do you want to compute product over the sorted array as well? 1345758 12 rniwa 2017-09-05 19:10:00 -0700 (In reply to Saam Barati from comment #11) > Comment on attachment 319888 [details] > Changes the score computation > > View in context: > https://bugs.webkit.org/attachment.cgi?id=319888&action=review > > > PerformanceTests/Speedometer/resources/benchmark-runner.js:285 > > + values.sort(function (a, b) { return a - b }); // Avoid the loss of significance for the sum. > > Do you want to compute product over the sorted array as well? No, the computation of a product doesn't suffer from the loss of significance. 1345764 13 rniwa 2017-09-05 19:37:43 -0700 Committed r221659: <http://trac.webkit.org/changeset/221659> 1353717 14 webkit-bug-importer 2017-09-27 12:53:45 -0700 <rdar://problem/34694227> 319888 2017-09-05 02:29:24 -0700 2017-09-05 09:03:41 -0700 Changes the score computation bug-172968-20170905022922.patch text/plain 12148 rniwa SW5kZXg6IFBlcmZvcm1hbmNlVGVzdHMvQ2hhbmdlTG9nCj09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFBlcmZvcm1h bmNlVGVzdHMvQ2hhbmdlTG9nCShyZXZpc2lvbiAyMjE2MTIpCisrKyBQZXJmb3JtYW5jZVRlc3Rz L0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwzICsxLDUyIEBACisyMDE3LTA5LTA1ICBS eW9zdWtlIE5pd2EgIDxybml3YUB3ZWJraXQub3JnPgorCisgICAgICAgIENvbXB1dGUgdGhlIGZp bmFsIHNjb3JlIHVzaW5nIGdlb21ldHJpYyBtZWFuIGluIFNwZWVkb21ldGVyIDIuMAorICAgICAg ICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTcyOTY4CisKKyAgICAg ICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgTWFrZSBTcGVlZG9tZXRl ciAyLjAgdXNlIHRoZSBnZW9tZXRyaWMgbWVhbiBvZiB0aGUgc3VidG90YWwgb2YgZWFjaCB0ZXN0 IHN1aXRlIGluc3RlYWQgb2YgdGhlIHRvdGFsLi4KKworICAgICAgICBJbiBTcGVlZG9tZXRlciAx LjAsIHdlIHVzZWQgdGhlIHRvdGFsIHRpbWUgdG8gY29tcHV0ZSB0aGUgZmluYWwgc2NvcmUgYmVj YXVzZSB3ZSB3YW50ZWQgdG8gbWFrZQorICAgICAgICB0aGUgc2xvd2VzdCBmcmFtZXdvcmsgYW5k IGxpYnJhcnkgZmFzdGVyLiBUaGUgZmFzdGVzdCBzdWl0ZSAoRmxpZ2h0SlMpIHN0aWxsIGFjY291 bnRlZCBmb3IgfjYlIGFuZAorICAgICAgICB0aGUgc2xvd2VzdCBjYXNlIChSZWFjdCkgYWNjb3Vu dGVkIGZvciB+MjUlIHNvIHdlIGZlbHQgdGhlIHRvdGFsIHRpbWUsIG9yIHRoZSBhcml0aG1ldGlj IG1lYW4gd2l0aAorICAgICAgICBhIGNvbnN0YW50IGZhY3Rvciwgd2FzIGEgZ29vZCBtZXRyaWMg dG8gdHJhY2suCisKKyAgICAgICAgSW4gdGhlIGxhdGVzdCB2ZXJzaW9uIG9mIFNwZWVkb21ldGVy IDIuMCwgaG93ZXZlciwgdGhlIGZhc3Rlc3Qgc3VpdGUgKFByZWFjdCkgcnVucyBpbiB+NTVtcyB3 aGVyZWFzCisgICAgICAgIHRoZSBzbG93ZXN0IHN1aXRlIChJbmZlcm5vKSB0YWtlcyAxLjVzIG9u IFNhZmFyaS4gU2luY2UgdGhlIHRvdGFsIHRpbWUgaXMgNi41cywgUHJlYWN0J3Mgc3VpdGUgb25s eQorICAgICAgICBhY2NvdW50cyBmb3IgfjAuOCUgb2YgdGhlIHRvdGFsIHNjb3JlIHdoaWxlIElu ZmVybm8ncyBzdWl0ZSBhY2NvdW50cyBmb3IgfjIzJSBvZiB0aGUgdG90YWwgc2NvcmUuCisgICAg ICAgIFNpbmNlIHRoZSBnb2FsIG9mIFNwZWVkb21ldGVyIGlzIHRvIGFwcHJveGltYXRlIGRpZmZl cmVudCBraW5kcyBvZiBET00gQVBJIHVzZSBwYXR0ZXJucyBvbiB0aGUgV2ViLAorICAgICAgICB3 ZSB3YW50IGVhY2ggZnJhbWV3b3JrICYgbGlicmFyeSB0byBoYXZlIHNvbWUgbWVhc3VyZW1lbnQg aW1wYWN0IG9uIHRoZSBvdmVyYWxsIGJlbmNobWFyayBzY29yZS4KKworICAgICAgICBGdXJ0aGVy bW9yZSwgYWZ0ZXIgcjIyMTIwNSwgd2UncmUgdGVzdGluZyBib3RoIGRlYnVnIGJ1aWxkIG9mIEVt YmVyLmpzIGFzIHdlbGwgYXMgcmVsZWFzZSBidWlsZC4KKyAgICAgICAgU2luY2UgZGVidWcgYnVp bGQgaXMgNHggc2xvd2VyLCB1c2luZyB0aGUgdG90YWwgdGltZSBvciB0aGUgYXJpdGhtZXRpYyBt ZWFuIHRoZXJlb2Ygd2lsbCBlZmZlY3RpdmVseQorICAgICAgICBnaXZlIDR4IGFzIG11Y2ggd2Vp Z2h0IHRvIGRlYnVnIGJ1aWxkIG9mIEVtYmVyLmpzIHJlbGF0aXZlIHRvIHJlbGVhc2UgYnVpbGQg b2YgRW1iZXIuanMuIEdpdmVuIG9ubHkKKyAgICAgICAgfjUlIG9mIHdlYnNpdGVzIHRoYXQgZGVw bG95IEVtYmVyLmpzIHVzZSBkZWJ1ZyBidWlsZCwgdGhpcyB3ZWlnaHRpbmcgaXMgY2xlYXJseSBu b3QgcmlnaHQuCisKKyAgICAgICAgVGhpcyBwYXRjaCwgdGhlcmVmb3JlLCByZXBsYWNlcyB0aGUg YXJpdGhtZXRpYyBtZWFuIGJ5IHRoZSBnZW9tZXRyaWMgbWVhbiB0byBjb21wdXRlIHRoZSBmaW5h bCBzY29yZS4KKyAgICAgICAgSXQgYWxzbyBtb3ZlcyB0aGUgY29kZSB0byBjb21wdXRlIHRoZSBm aW5hbCBzY29yZSB0byBCZW5jaG1hcmtSdW5uZXIgdG8gYmUgc2hhcmVkIGJldHdlZW4gbWFpbi5q cworICAgICAgICBhbmQgSW50ZXJhY3RpdmVSdW5uZXIuaHRtbC4KKworICAgICAgICAqIFNwZWVk b21ldGVyL0ludGVyYWN0aXZlUnVubmVyLmh0bWw6CisgICAgICAgICguZGlkUnVuU3VpdGVzKTog U2hvdyBnZW9tZXRyaWMgbWVhbiwgYXJpdGhtZXRpYyBtZWFuLCB0b3RhbCwgYXMgd2VsbCBhcyB0 aGUgc2NvcmUgZm9yIGNvbXBsZXRlbmVzcworICAgICAgICBzaW5jZSB0aGlzIGlzIGEgZGVidWdn aW5nIHBhZ2UgZm9yIGRldmVsb3BlcnMuCisgICAgICAgICogU3BlZWRvbWV0ZXIvcmVzb3VyY2Vz L2JlbmNobWFyay1ydW5uZXIuanM6CisgICAgICAgIChCZW5jaG1hcmtSdW5uZXIucHJvdG90eXBl LnN0ZXApOiBBZGRlZCBtZWFuLCBnZW9tZWFuLCBhbmQgc2NvcmUgYXMgbWVhc3VyZWRWYWx1ZXMn IHByb3BlcnRpZXMuCisgICAgICAgIChCZW5jaG1hcmtSdW5uZXIucHJvdG90eXBlLl9ydW5UZXN0 QW5kUmVjb3JkUmVzdWx0cyk6IFJlbW92ZWQgdGhlIGRlYWQgY29kZS4KKyAgICAgICAgKEJlbmNo bWFya1J1bm5lci5wcm90b3R5cGUuX2ZpbmFsaXplKTogQ29tcHV0ZSBhbmQgYWRkIHRvdGFsLCBh cml0aG1ldGljIG1lYW4gKGp1c3QgbWVhbiBpbiB0aGUgY29kZSksCisgICAgICAgIGFuZCBnZW9t ZXRyaWMgbWVhbiAoZ2VvbWVhbikgdG8gbWVhc3VyZWRWYWx1ZXMuCisgICAgICAgICogU3BlZWRv bWV0ZXIvcmVzb3VyY2VzL21haW4uanM6CisgICAgICAgICh3aW5kb3cuYmVuY2htYXJrQ2xpZW50 KTogUmVwbGFjZWQgdGVzdHNDb3VudCBieSBzdGVwc0NvdW50IGFuZCBfdGltZVZhbHVlcyBieSBf bWVhc3VyZWRWYWx1ZXNMaXN0LgorICAgICAgICAod2luZG93LmJlbmNobWFya0NsaWVudC53aWxs UnVuVGVzdCk6CisgICAgICAgICh3aW5kb3cuYmVuY2htYXJrQ2xpZW50LmRpZFJ1blRlc3QpOgor ICAgICAgICAod2luZG93LmJlbmNobWFya0NsaWVudC5kaWRSdW5TdWl0ZXMpOiBTdG9yZSBtZWFz dXJlZFZhbHVlcyBvYmplY3QgaW5zdGVhZCBvZiBqdXN0IHRoZSB0b3RhbCB0aW1lLgorICAgICAg ICAod2luZG93LmJlbmNobWFya0NsaWVudC5kaWRGaW5pc2hMYXN0SXRlcmF0aW9uKToKKyAgICAg ICAgKHdpbmRvdy5iZW5jaG1hcmtDbGllbnQuX2NvbXB1dGVSZXN1bHRzKToKKyAgICAgICAgKHdp bmRvdy5iZW5jaG1hcmtDbGllbnQuX2NvbXB1dGVSZXN1bHRzLnZhbHVlRm9yVW5pdCk6IFJlbmFt ZWQgZnJvbSB0b3RhbFRpbWVJbkRpc3BsYXlVbml0LiBOb3cgc2ltcGx5CisgICAgICAgIHJldHJp ZXZlcyB0aGUgdmFsdWVzIGNvbXB1dGVkIGJ5IEJlbmNobWFya1J1bm5lcidzX2ZpbmFsaXplLgor ICAgICAgICAoc3RhcnRCZW5jaG1hcmspOgorICAgICAgICAoY29tcHV0ZVNjb3JlKTogRGVsZXRl ZC4KKwogMjAxNy0wOS0wNSAgU2hpeXUgWmhhbmcgIDxzaGl5dS56aGFuZ0BpbnRlbC5jb20+CiAK ICAgICAgICAgU3BlZWRvbWV0ZXIgMi4wOiBBZGQgZHVtbXkgbm9kZSB0byBub3RpZnkgYXBwIGlz IHJlYWR5IGZvciBCYWNrYm9uZSBzdWl0ZQpJbmRleDogUGVyZm9ybWFuY2VUZXN0cy9TcGVlZG9t ZXRlci9JbnRlcmFjdGl2ZVJ1bm5lci5odG1sCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFBlcmZvcm1hbmNlVGVz dHMvU3BlZWRvbWV0ZXIvSW50ZXJhY3RpdmVSdW5uZXIuaHRtbAkocmV2aXNpb24gMjIxNjEwKQor KysgUGVyZm9ybWFuY2VUZXN0cy9TcGVlZG9tZXRlci9JbnRlcmFjdGl2ZVJ1bm5lci5odG1sCSh3 b3JraW5nIGNvcHkpCkBAIC0xNDEsNyArMTQxLDEwIEBAIGZ1bmN0aW9uIHN0YXJ0VGVzdCgpIHsK ICAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgICAgICAgcmVzdWx0cyArPSBzdWl0ZU5hbWUg KyAnIDogJyArIHN1aXRlUmVzdWx0cy50b3RhbCArICcgbXNcbic7CiAgICAgICAgICAgICB9Cisg ICAgICAgICAgICByZXN1bHRzICs9ICdBcml0aGVtdGljIE1lYW4gOiAnICsgbWVhc3VyZWRWYWx1 ZXMubWVhbiAgKyAnIG1zXG4nOworICAgICAgICAgICAgcmVzdWx0cyArPSAnR2VvbWV0cmljIE1l YW4gOiAnICsgbWVhc3VyZWRWYWx1ZXMuZ2VvbWVhbiAgKyAnIG1zXG4nOwogICAgICAgICAgICAg cmVzdWx0cyArPSAnVG90YWwgOiAnICsgbWVhc3VyZWRWYWx1ZXMudG90YWwgKyAnIG1zXG4nOwor ICAgICAgICAgICAgcmVzdWx0cyArPSAnU2NvcmUgOiAnICsgbWVhc3VyZWRWYWx1ZXMuc2NvcmUg KyAnIHJwbVxuJzsKIAogICAgICAgICAgICAgaWYgKCFyZXN1bHRzKQogICAgICAgICAgICAgICAg IHJldHVybjsKSW5kZXg6IFBlcmZvcm1hbmNlVGVzdHMvU3BlZWRvbWV0ZXIvcmVzb3VyY2VzL2Jl bmNobWFyay1ydW5uZXIuanMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gUGVyZm9ybWFuY2VUZXN0cy9TcGVlZG9t ZXRlci9yZXNvdXJjZXMvYmVuY2htYXJrLXJ1bm5lci5qcwkocmV2aXNpb24gMjIxNjEwKQorKysg UGVyZm9ybWFuY2VUZXN0cy9TcGVlZG9tZXRlci9yZXNvdXJjZXMvYmVuY2htYXJrLXJ1bm5lci5q cwkod29ya2luZyBjb3B5KQpAQCAtMTk0LDcgKzE5NCw3IEBAIEJlbmNobWFya1N0YXRlLnByb3Rv dHlwZS5wcmVwYXJlQ3VycmVudFMKIEJlbmNobWFya1J1bm5lci5wcm90b3R5cGUuc3RlcCA9IGZ1 bmN0aW9uIChzdGF0ZSkgewogICAgIGlmICghc3RhdGUpIHsKICAgICAgICAgc3RhdGUgPSBuZXcg QmVuY2htYXJrU3RhdGUodGhpcy5fc3VpdGVzKTsKLSAgICAgICAgdGhpcy5fbWVhc3VyZWRWYWx1 ZXMgPSB7dGVzdHM6IHt9LCB0b3RhbDogMH07CisgICAgICAgIHRoaXMuX21lYXN1cmVkVmFsdWVz ID0ge3Rlc3RzOiB7fSwgdG90YWw6IDAsIG1lYW46IE5hTiwgZ2VvbWVhbjogTmFOLCBzY29yZTog TmFOfTsKICAgICB9CiAKICAgICB2YXIgc3VpdGUgPSBzdGF0ZS5jdXJyZW50U3VpdGUoKTsKQEAg LTIwNyw3ICsyMDcsNiBAQCBCZW5jaG1hcmtSdW5uZXIucHJvdG90eXBlLnN0ZXAgPSBmdW5jdGlv CiAKICAgICBpZiAoc3RhdGUuaXNGaXJzdFRlc3QoKSkgewogICAgICAgICB0aGlzLl9yZW1vdmVG cmFtZSgpOwotICAgICAgICB0aGlzLl9tYXN1cmVkVmFsdWVzRm9yQ3VycmVudFN1aXRlID0ge307 CiAgICAgICAgIHZhciBzZWxmID0gdGhpczsKICAgICAgICAgcmV0dXJuIHN0YXRlLnByZXBhcmVD dXJyZW50U3VpdGUodGhpcywgdGhpcy5fYXBwZW5kRnJhbWUoKSkudGhlbihmdW5jdGlvbiAocHJl cGFyZVJldHVyblZhbHVlKSB7CiAgICAgICAgICAgICBzZWxmLl9wcmVwYXJlUmV0dXJuVmFsdWUg PSBwcmVwYXJlUmV0dXJuVmFsdWU7CkBAIC0yNjAsNyArMjU5LDYgQEAgQmVuY2htYXJrUnVubmVy LnByb3RvdHlwZS5fcnVuVGVzdEFuZFJlYwogICAgICAgICAgICAgc2VsZi5fbWVhc3VyZWRWYWx1 ZXMudGVzdHNbc3VpdGUubmFtZV0gPSBzdWl0ZVJlc3VsdHM7CiAgICAgICAgICAgICBzdWl0ZVJl c3VsdHMudGVzdHNbdGVzdC5uYW1lXSA9IHt0ZXN0czogeydTeW5jJzogc3luY1RpbWUsICdBc3lu Yyc6IGFzeW5jVGltZX0sIHRvdGFsOiB0b3RhbH07CiAgICAgICAgICAgICBzdWl0ZVJlc3VsdHMu dG90YWwgKz0gdG90YWw7Ci0gICAgICAgICAgICBzZWxmLl9tZWFzdXJlZFZhbHVlcy50b3RhbCAr PSB0b3RhbDsKIAogICAgICAgICAgICAgaWYgKHNlbGYuX2NsaWVudCAmJiBzZWxmLl9jbGllbnQu ZGlkUnVuVGVzdCkKICAgICAgICAgICAgICAgICBzZWxmLl9jbGllbnQuZGlkUnVuVGVzdChzdWl0 ZSwgdGVzdCk7CkBAIC0yNzUsOCArMjczLDI2IEBAIEJlbmNobWFya1J1bm5lci5wcm90b3R5cGUu X3J1blRlc3RBbmRSZWMKIEJlbmNobWFya1J1bm5lci5wcm90b3R5cGUuX2ZpbmFsaXplID0gZnVu Y3Rpb24gKCkgewogICAgIHRoaXMuX3JlbW92ZUZyYW1lKCk7CiAKLSAgICBpZiAodGhpcy5fY2xp ZW50ICYmIHRoaXMuX2NsaWVudC5kaWRSdW5TdWl0ZXMpCisgICAgaWYgKHRoaXMuX2NsaWVudCAm JiB0aGlzLl9jbGllbnQuZGlkUnVuU3VpdGVzKSB7CisgICAgICAgIHZhciBwcm9kdWN0ID0gMTsK KyAgICAgICAgdmFyIHZhbHVlcyA9IFtdOworICAgICAgICBmb3IgKHZhciBzdWl0ZU5hbWUgaW4g dGhpcy5fbWVhc3VyZWRWYWx1ZXMudGVzdHMpIHsKKyAgICAgICAgICAgIHZhciBzdWl0ZVRvdGFs ID0gdGhpcy5fbWVhc3VyZWRWYWx1ZXMudGVzdHNbc3VpdGVOYW1lXS50b3RhbDsKKyAgICAgICAg ICAgIHByb2R1Y3QgKj0gc3VpdGVUb3RhbDsKKyAgICAgICAgICAgIHZhbHVlcy5wdXNoKHN1aXRl VG90YWwpOworICAgICAgICB9CisKKyAgICAgICAgdmFsdWVzLnNvcnQoZnVuY3Rpb24gKGEsIGIp IHsgcmV0dXJuIGEgLSBiIH0pOyAvLyBBdm9pZCB0aGUgbG9zcyBvZiBzaWduaWZpY2FuY2UgZm9y IHRoZSBzdW0uCisgICAgICAgIHZhciB0b3RhbCA9IHZhbHVlcy5yZWR1Y2UoZnVuY3Rpb24gKGEs IGIpIHsgcmV0dXJuIGEgKyBiIH0pOworICAgICAgICB2YXIgZ2VvbWVhbiA9IE1hdGgucG93KHBy b2R1Y3QsIDEgLyB2YWx1ZXMubGVuZ3RoKTsKKworICAgICAgICB2YXIgY29ycmVjdGlvbkZhY3Rv ciA9IDM7IC8vIFRoaXMgZmFjdG9yIG1ha2VzIHRoZSB0ZXN0IHNjb3JlIGxvb2sgcmVhc29uYWJs eSBmaXQgd2l0aGluIDAgdG8gMTQwLgorICAgICAgICB0aGlzLl9tZWFzdXJlZFZhbHVlcy50b3Rh bCA9IHRvdGFsOworICAgICAgICB0aGlzLl9tZWFzdXJlZFZhbHVlcy5tZWFuID0gdG90YWwgLyB2 YWx1ZXMubGVuZ3RoOworICAgICAgICB0aGlzLl9tZWFzdXJlZFZhbHVlcy5nZW9tZWFuID0gZ2Vv bWVhbjsKKyAgICAgICAgdGhpcy5fbWVhc3VyZWRWYWx1ZXMuc2NvcmUgPSA2MCAqIDEwMDAgLyBn ZW9tZWFuIC8gY29ycmVjdGlvbkZhY3RvcjsKICAgICAgICAgdGhpcy5fY2xpZW50LmRpZFJ1blN1 aXRlcyh0aGlzLl9tZWFzdXJlZFZhbHVlcyk7CisgICAgfQogCiAgICAgaWYgKHRoaXMuX3J1bk5l eHRJdGVyYXRpb24pCiAgICAgICAgIHRoaXMuX3J1bk5leHRJdGVyYXRpb24oKTsKSW5kZXg6IFBl cmZvcm1hbmNlVGVzdHMvU3BlZWRvbWV0ZXIvcmVzb3VyY2VzL21haW4uanMKPT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQot LS0gUGVyZm9ybWFuY2VUZXN0cy9TcGVlZG9tZXRlci9yZXNvdXJjZXMvbWFpbi5qcwkocmV2aXNp b24gMjIxNjEwKQorKysgUGVyZm9ybWFuY2VUZXN0cy9TcGVlZG9tZXRlci9yZXNvdXJjZXMvbWFp bi5qcwkod29ya2luZyBjb3B5KQpAQCAtMSw5ICsxLDkgQEAKIHdpbmRvdy5iZW5jaG1hcmtDbGll bnQgPSB7CiAgICAgZGlzcGxheVVuaXQ6ICdydW5zL21pbicsCiAgICAgaXRlcmF0aW9uQ291bnQ6 IDEwLAotICAgIHRlc3RzQ291bnQ6IG51bGwsCisgICAgc3RlcENvdW50OiBudWxsLAogICAgIHN1 aXRlc0NvdW50OiBudWxsLAotICAgIF90aW1lVmFsdWVzOiBbXSwKKyAgICBfbWVhc3VyZWRWYWx1 ZXNMaXN0OiBbXSwKICAgICBfZmluaXNoZWRUZXN0Q291bnQ6IDAsCiAgICAgX3Byb2dyZXNzQ29t cGxldGVkOiBudWxsLAogICAgIHdpbGxBZGRUZXN0RnJhbWU6IGZ1bmN0aW9uIChmcmFtZSkgewpA QCAtMTMsMTcgKzEzLDE3IEBAIHdpbmRvdy5iZW5jaG1hcmtDbGllbnQgPSB7CiAgICAgICAgIGZy YW1lLnN0eWxlLnRvcCA9IG1haW4ub2Zmc2V0VG9wICsgcGFyc2VJbnQoc3R5bGUuYm9yZGVyVG9w V2lkdGgpICsgcGFyc2VJbnQoc3R5bGUucGFkZGluZ1RvcCkgKyAncHgnOwogICAgIH0sCiAgICAg d2lsbFJ1blRlc3Q6IGZ1bmN0aW9uIChzdWl0ZSwgdGVzdCkgewotICAgICAgICBkb2N1bWVudC5n ZXRFbGVtZW50QnlJZCgnaW5mbycpLnRleHRDb250ZW50ID0gc3VpdGUubmFtZSArICcgKCAnICsg dGhpcy5fZmluaXNoZWRUZXN0Q291bnQgKyAnIC8gJyArIHRoaXMudGVzdHNDb3VudCArICcgKSc7 CisgICAgICAgIGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKCdpbmZvJykudGV4dENvbnRlbnQgPSBz dWl0ZS5uYW1lICsgJyAoICcgKyB0aGlzLl9maW5pc2hlZFRlc3RDb3VudCArICcgLyAnICsgdGhp cy5zdGVwQ291bnQgKyAnICknOwogICAgIH0sCiAgICAgZGlkUnVuVGVzdDogZnVuY3Rpb24gKCkg ewogICAgICAgICB0aGlzLl9maW5pc2hlZFRlc3RDb3VudCsrOwotICAgICAgICB0aGlzLl9wcm9n cmVzc0NvbXBsZXRlZC5zdHlsZS53aWR0aCA9ICh0aGlzLl9maW5pc2hlZFRlc3RDb3VudCAqIDEw MCAvIHRoaXMudGVzdHNDb3VudCkgKyAnJSc7CisgICAgICAgIHRoaXMuX3Byb2dyZXNzQ29tcGxl dGVkLnN0eWxlLndpZHRoID0gKHRoaXMuX2ZpbmlzaGVkVGVzdENvdW50ICogMTAwIC8gdGhpcy5z dGVwQ291bnQpICsgJyUnOwogICAgIH0sCiAgICAgZGlkUnVuU3VpdGVzOiBmdW5jdGlvbiAobWVh c3VyZWRWYWx1ZXMpIHsKLSAgICAgICAgdGhpcy5fdGltZVZhbHVlcy5wdXNoKG1lYXN1cmVkVmFs dWVzLnRvdGFsKTsKKyAgICAgICAgdGhpcy5fbWVhc3VyZWRWYWx1ZXNMaXN0LnB1c2gobWVhc3Vy ZWRWYWx1ZXMpOwogICAgIH0sCiAgICAgd2lsbFN0YXJ0Rmlyc3RJdGVyYXRpb246IGZ1bmN0aW9u ICgpIHsKLSAgICAgICAgdGhpcy5fdGltZVZhbHVlcyA9IFtdOworICAgICAgICB0aGlzLl9tZWFz dXJlZFZhbHVlc0xpc3QgPSBbXTsKICAgICAgICAgdGhpcy5fZmluaXNoZWRUZXN0Q291bnQgPSAw OwogICAgICAgICB0aGlzLl9wcm9ncmVzc0NvbXBsZXRlZCA9IGRvY3VtZW50LmdldEVsZW1lbnRC eUlkKCdwcm9ncmVzcy1jb21wbGV0ZWQnKTsKICAgICAgICAgZG9jdW1lbnQuZ2V0RWxlbWVudEJ5 SWQoJ2xvZ28tbGluaycpLm9uY2xpY2sgPSBmdW5jdGlvbiAoZXZlbnQpIHsgZXZlbnQucHJldmVu dERlZmF1bHQoKTsgcmV0dXJuIGZhbHNlOyB9CkBAIC0zMSw3ICszMSw3IEBAIHdpbmRvdy5iZW5j aG1hcmtDbGllbnQgPSB7CiAgICAgZGlkRmluaXNoTGFzdEl0ZXJhdGlvbjogZnVuY3Rpb24gKCkg ewogICAgICAgICBkb2N1bWVudC5nZXRFbGVtZW50QnlJZCgnbG9nby1saW5rJykub25jbGljayA9 IG51bGw7CiAKLSAgICAgICAgdmFyIHJlc3VsdHMgPSB0aGlzLl9jb21wdXRlUmVzdWx0cyh0aGlz Ll90aW1lVmFsdWVzLCB0aGlzLmRpc3BsYXlVbml0KTsKKyAgICAgICAgdmFyIHJlc3VsdHMgPSB0 aGlzLl9jb21wdXRlUmVzdWx0cyh0aGlzLl9tZWFzdXJlZFZhbHVlc0xpc3QsIHRoaXMuZGlzcGxh eVVuaXQpOwogCiAgICAgICAgIHRoaXMuX3VwZGF0ZUdhdWdlTmVlZGxlKHJlc3VsdHMubWVhbik7 CiAgICAgICAgIGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKCdyZXN1bHQtbnVtYmVyJykudGV4dENv bnRlbnQgPSByZXN1bHRzLmZvcm1hdHRlZE1lYW47CkBAIC00NywxMiArNDcsMTIgQEAgd2luZG93 LmJlbmNobWFya0NsaWVudCA9IHsKICAgICAgICAgfSBlbHNlCiAgICAgICAgICAgICBzaG93UmVz dWx0c1N1bW1hcnkoKTsKICAgICB9LAotICAgIF9jb21wdXRlUmVzdWx0czogZnVuY3Rpb24gKHRp bWVWYWx1ZXMsIGRpc3BsYXlVbml0KSB7CisgICAgX2NvbXB1dGVSZXN1bHRzOiBmdW5jdGlvbiAo bWVhc3VyZWRWYWx1ZXNMaXN0LCBkaXNwbGF5VW5pdCkgewogICAgICAgICB2YXIgc3VpdGVzQ291 bnQgPSB0aGlzLnN1aXRlc0NvdW50OwotICAgICAgICBmdW5jdGlvbiB0b3RhbFRpbWVJbkRpc3Bs YXlVbml0KHRpbWUpIHsKKyAgICAgICAgZnVuY3Rpb24gdmFsdWVGb3JVbml0KG1lYXN1cmVkVmFs dWVzKSB7CiAgICAgICAgICAgICBpZiAoZGlzcGxheVVuaXQgPT0gJ21zJykKLSAgICAgICAgICAg ICAgICByZXR1cm4gdGltZTsKLSAgICAgICAgICAgIHJldHVybiBjb21wdXRlU2NvcmUodGltZSk7 CisgICAgICAgICAgICAgICAgcmV0dXJuIG1lYXN1cmVkVmFsdWVzLmdlb21lYW47CisgICAgICAg ICAgICByZXR1cm4gbWVhc3VyZWRWYWx1ZXMuc2NvcmU7CiAgICAgICAgIH0KIAogICAgICAgICBm dW5jdGlvbiBzaWdGaWdGcm9tUGVyY2VudERlbHRhKHBlcmNlbnREZWx0YSkgewpAQCAtNjQsNyAr NjQsNyBAQCB3aW5kb3cuYmVuY2htYXJrQ2xpZW50ID0gewogICAgICAgICAgICAgcmV0dXJuIG51 bWJlci50b1ByZWNpc2lvbihNYXRoLm1heChub25EZWNpbWFsRGlnaXRDb3VudCwgTWF0aC5taW4o Niwgc2lnRmlnKSkpOwogICAgICAgICB9CiAKLSAgICAgICAgdmFyIHZhbHVlcyA9IHRpbWVWYWx1 ZXMubWFwKHRvdGFsVGltZUluRGlzcGxheVVuaXQpOworICAgICAgICB2YXIgdmFsdWVzID0gbWVh c3VyZWRWYWx1ZXNMaXN0Lm1hcCh2YWx1ZUZvclVuaXQpOwogICAgICAgICB2YXIgc3VtID0gdmFs dWVzLnJlZHVjZShmdW5jdGlvbiAoYSwgYikgeyByZXR1cm4gYSArIGI7IH0sIDApOwogICAgICAg ICB2YXIgYXJpdGhtZXRpY01lYW4gPSBzdW0gLyB2YWx1ZXMubGVuZ3RoOwogICAgICAgICB2YXIg bWVhblNpZ0ZpZyA9IDQ7CkBAIC04Myw4ICs4Myw4IEBAIHdpbmRvdy5iZW5jaG1hcmtDbGllbnQg PSB7CiAgICAgICAgIHZhciBmb3JtYXR0ZWRNZWFuID0gdG9TaWdGaWdQcmVjaXNpb24oYXJpdGht ZXRpY01lYW4sIE1hdGgubWF4KG1lYW5TaWdGaWcsIDMpKTsKIAogICAgICAgICByZXR1cm4gewot ICAgICAgICAgICAgZm9ybWF0dGVkVmFsdWVzOiB0aW1lVmFsdWVzLm1hcChmdW5jdGlvbiAodGlt ZSkgewotICAgICAgICAgICAgICAgIHJldHVybiB0b1NpZ0ZpZ1ByZWNpc2lvbih0b3RhbFRpbWVJ bkRpc3BsYXlVbml0KHRpbWUpLCA0KSArICcgJyArIGRpc3BsYXlVbml0OworICAgICAgICAgICAg Zm9ybWF0dGVkVmFsdWVzOiB2YWx1ZXMubWFwKGZ1bmN0aW9uICh2YWx1ZSkgeworICAgICAgICAg ICAgICAgIHJldHVybiB0b1NpZ0ZpZ1ByZWNpc2lvbih2YWx1ZSwgNCkgKyAnICcgKyBkaXNwbGF5 VW5pdDsKICAgICAgICAgICAgIH0pLAogICAgICAgICAgICAgbWVhbjogYXJpdGhtZXRpY01lYW4s CiAgICAgICAgICAgICBmb3JtYXR0ZWRNZWFuOiBmb3JtYXR0ZWRNZWFuLApAQCAtMTk2LDggKzE5 Niw4IEBAIGZ1bmN0aW9uIHN0YXJ0QmVuY2htYXJrKCkgewogICAgIH0KIAogICAgIHZhciBlbmFi bGVkU3VpdGVzID0gU3VpdGVzLmZpbHRlcihmdW5jdGlvbiAoc3VpdGUpIHsgcmV0dXJuICFzdWl0 ZS5kaXNhYmxlZDsgfSk7Ci0gICAgdmFyIHRvdGFsU3VidGVzdENvdW50ID0gZW5hYmxlZFN1aXRl cy5yZWR1Y2UoZnVuY3Rpb24gKHRlc3RzQ291bnQsIHN1aXRlKSB7IHJldHVybiB0ZXN0c0NvdW50 ICsgc3VpdGUudGVzdHMubGVuZ3RoOyB9LCAwKTsKLSAgICBiZW5jaG1hcmtDbGllbnQudGVzdHND b3VudCA9IGJlbmNobWFya0NsaWVudC5pdGVyYXRpb25Db3VudCAqIHRvdGFsU3VidGVzdENvdW50 OworICAgIHZhciB0b3RhbFN1YnRlc3RzQ291bnQgPSBlbmFibGVkU3VpdGVzLnJlZHVjZShmdW5j dGlvbiAodGVzdHNDb3VudCwgc3VpdGUpIHsgcmV0dXJuIHRlc3RzQ291bnQgKyBzdWl0ZS50ZXN0 cy5sZW5ndGg7IH0sIDApOworICAgIGJlbmNobWFya0NsaWVudC5zdGVwQ291bnQgPSBiZW5jaG1h cmtDbGllbnQuaXRlcmF0aW9uQ291bnQgKiB0b3RhbFN1YnRlc3RzQ291bnQ7CiAgICAgYmVuY2ht YXJrQ2xpZW50LnN1aXRlc0NvdW50ID0gZW5hYmxlZFN1aXRlcy5sZW5ndGg7CiAgICAgdmFyIHJ1 bm5lciA9IG5ldyBCZW5jaG1hcmtSdW5uZXIoU3VpdGVzLCBiZW5jaG1hcmtDbGllbnQpOwogICAg IHJ1bm5lci5ydW5NdWx0aXBsZUl0ZXJhdGlvbnMoYmVuY2htYXJrQ2xpZW50Lml0ZXJhdGlvbkNv dW50KTsKQEAgLTIwNSwxMCArMjA1LDYgQEAgZnVuY3Rpb24gc3RhcnRCZW5jaG1hcmsoKSB7CiAg ICAgcmV0dXJuIHRydWU7CiB9CiAKLWZ1bmN0aW9uIGNvbXB1dGVTY29yZSh0aW1lKSB7Ci0gICAg cmV0dXJuIDYwICogMTAwMCAqIGJlbmNobWFya0NsaWVudC5zdWl0ZXNDb3VudCAvIHRpbWU7Ci19 Ci0KIGZ1bmN0aW9uIHNob3dTZWN0aW9uKHNlY3Rpb25JZGVudGlmaWVyLCBwdXNoU3RhdGUpIHsK ICAgICB2YXIgY3VycmVudFNlY3Rpb25FbGVtZW50ID0gZG9jdW1lbnQucXVlcnlTZWxlY3Rvcign c2VjdGlvbi5zZWxlY3RlZCcpOwogICAgIGNvbnNvbGUuYXNzZXJ0KGN1cnJlbnRTZWN0aW9uRWxl bWVudCk7Cg==