61888 2011-06-01 15:49:43 -0700 nrwt: should skip chromium test expectation overrides on upstream bots 2022-03-01 02:49:54 -0800 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 dpranke dpranke abarth ojan senorblanco tony oldest_to_newest 413717 0 dpranke 2011-06-01 15:49:43 -0700 If there are test_expectations checked into a downstream Chromium checkout, they are automatically picked up by both the canaries and the DEPS bots. There is therefore no way to test whether the upstream test_expectations.txt themselves are sufficient to keep the tree green (which one might want to do prior to doing a webkit roll). So, we've decided to modify the code so that the overrides are ignored if the builder_name is set and does not include "(deps)" in the name. A downside of this change is that a chromium-only committer will no longer be able to fix a failure on the canaries by just inserting a downstream expectation. Hopefully this will not be much of an issue in practice. Note that there are two ways to implement this change. We could either make the code generically aware of the significance of "(deps)" and skip the overrides generically, or we could push all of this logic inside the chromium code. There does not appear to be a way to write tests for this that will cover both cases :( I will post patches that demonstrate both approaches, but the chromium-only approach seems slightly simpler for now. 413721 1 95680 dpranke 2011-06-01 15:55:18 -0700 Created attachment 95680 chromium-specific version 413735 2 95683 dpranke 2011-06-01 16:08:04 -0700 Created attachment 95683 generic version 413736 3 dpranke 2011-06-01 16:08:50 -0700 actually, now that I've cleaned up the generic version a bit, it's kind of a toss-up. 413862 4 ojan 2011-06-01 20:17:35 -0700 Meh. Either solution seems fine to me. The chromium-only approach appeals to me in that it's just fewer modified lines/files. I'm not sure DUMMY_BUILDER_NAME needs to use overrides though. It would somewhat simplify that if-check. 414066 5 senorblanco 2011-06-02 06:27:10 -0700 If I understand this correctly, there may be a problem: since Skia rolls into chromium first, if a Skia roll introduces test failures (which are added to downstream test_expectations), there will be no way to get the canaries green without upstreaming the test_expectations. Some of the skia gardeners are not WebKit committers, so they won't be able to do this without going through webkit's bugzilla, commit-queue, etc. I'm not sure this is a blocking issue (since it'll only affect the WebKit gardener, and only until the expectations are upstreamed), but I thought I'd mention it. 414146 6 95680 eric 2011-06-02 08:34:56 -0700 Comment on attachment 95680 chromium-specific version Cleared review? from obsolete attachment 95680 so that this bug does not appear in http://webkit.org/pending-review. If you would like this patch reviewed, please attach it to a new bug (or re-open this bug before marking it for review again). 414200 7 abarth 2011-06-02 09:45:12 -0700 You just need to co-ordinated with the webkit gardener when rolling ski a (which you'll need to do anyway without commit bit). 414261 8 tony 2011-06-02 10:54:17 -0700 I feel the pain for you guys who are trying to improve skia. There's pain on so many sides :( I've also mentioned to evan that it would be possible to roll skia in WebKit first (in Source/WebKit/chromium/DEPS) before rolling it in chromium, but then you have to remember to update two DEPS values when rolling webkit. We could also resurrect From() so that chromium gets the version of skia from WebKit, but then rolling skia would be blocked on webkit rolls. Sorry for derailing this bug. It might be worth brainstorming over email how to make this less painful (maybe we need a ToT chromium, ToT webkit, ToT skia builder-- that would at least tell you which tests to add to test_expectations.txt in advance?). 414263 9 dpranke 2011-06-02 10:55:48 -0700 (In reply to comment #6) > (From update of attachment 95680 [details]) > Cleared review? from obsolete attachment 95680 [details] so that this bug does not appear in http://webkit.org/pending-review. If you would like this patch reviewed, please attach it to a new bug (or re-open this bug before marking it for review again). Actually, I did want that version reviewed, since I think there's a slight agreement that the chromium-only version might be better. Re-setting the flag. 414264 10 dpranke 2011-06-02 10:57:54 -0700 (In reply to comment #4) > Meh. Either solution seems fine to me. The chromium-only approach appeals to me in that it's just fewer modified lines/files. > > I'm not sure DUMMY_BUILDER_NAME needs to use overrides though. It would somewhat simplify that if-check. The reason the DUMMY_BUILDER_NAME variant uses the overrides is so --lint-test-files will pick up and lint the overrides as well as the upstream version; there isn't a good alternative AFAICT, and it seemed important to make sure both files were linted. I also couldn't think of a way that the downstream version would mask any upstream-only lint errors, so it seemed safer. 415073 11 dpranke 2011-06-03 12:04:06 -0700 One additional thought is that we could create *two* downstream expectations files, one that only applied to the DEPS bots and one that only applied to the canaries. That way a skia change that broke the canaries could still be fixed by someone with chromium-only committing rights. That said, I'd be inclined to not do this until it was clear that this was a problem. 415243 12 95680 abarth 2011-06-03 15:38:12 -0700 Comment on attachment 95680 chromium-specific version View in context: https://bugs.webkit.org/attachment.cgi?id=95680&action=review I think we should move forward with this approach. Introducing a yet another test_expectations file is too much complexity. We'll find a way to do Skia rolls without this functionality. > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:265 > + # See http://bugs.webkit.org/show_bug.cgi?id=61888 for the rationale for this. I'd remove this comment. Folks can use svn blame to find the bug number. 415252 13 dpranke 2011-06-03 15:49:55 -0700 (In reply to comment #12) > (From update of attachment 95680 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=95680&action=review > > I think we should move forward with this approach. Introducing a yet another test_expectations file is too much complexity. We'll find a way to do Skia rolls without this functionality. > > > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:265 > > + # See http://bugs.webkit.org/show_bug.cgi?id=61888 for the rationale for this. > > I'd remove this comment. Folks can use svn blame to find the bug number. Okay. 415262 14 dpranke 2011-06-03 16:02:09 -0700 Committed r88073: <http://trac.webkit.org/changeset/88073> 443606 15 senorblanco 2011-07-28 12:21:41 -0700 Update: I think we need to reconsider this change. This is making skia rolls very painful. When we roll skia, in order to prevent the WebKit canaries going red, we have to land an upstream WebKit change to pre-emptively suppress the failures. Then we land the skia roll. However, if this is done before the WebKit roll brings the suppressions into Chrome, we have to *also* suppress the failures in downstream test_expectations. Then rebaseline upstream, and remove the expectations in both places. So up to five patches. The amount of pain this is causing the skia gardener is greater than the amount of good it is causing the WebKit gardener, IMHO. Also, it seems that this change also inadvertently affected the layout test trybots: since they don't have "(deps)" in their name, they also ignore the downstream test_expectations, leading to unexpected failures during try runs. 443620 16 abarth 2011-07-28 12:54:56 -0700 Perhaps skia should roll into WebKit before it rolls into Chromium? 443622 17 senorblanco 2011-07-28 13:05:41 -0700 (In reply to comment #16) > Perhaps skia should roll into WebKit before it rolls into Chromium? Although we considered it, I don't think that's practical at this time. There's a lot of work going on in Skia, and to add the lag of a WebKit roll to that work would be prohibitive. As one datapoint, there are a number of merges going from skia to the release branch right now. To add a WebKit merge to that process seems like it would be a nightmare. Also, since we don't have full coverage of all tests on the WebKit bots, a skia change which introduced a ChromeOS or pyauto regression (or any one of the many bots we don't have WebKit canary coverage for) would require a rollback in skia, and a merge forward in WebKit to fix. 443989 18 abarth 2011-07-29 10:35:22 -0700 > Although we considered it, I don't think that's practical at this time. There's a lot of work going on in Skia, and to add the lag of a WebKit roll to that work would be prohibitive. As one datapoint, there are a number of merges going from skia to the release branch right now. To add a WebKit merge to that process seems like it would be a nightmare. Whether we fix the skia revision in http://trac.webkit.org/browser/trunk/Source/WebKit/chromium/DEPS or http://src.chromium.org/viewvc/chrome/trunk/src/DEPS doesn't have any bearing on how you merge changes to release branches. > Also, since we don't have full coverage of all tests on the WebKit bots, a skia change which introduced a ChromeOS or pyauto regression (or any one of the many bots we don't have WebKit canary coverage for) would require a rollback in skia, and a merge forward in WebKit to fix. We have that problem for every WebKit change. This bug probably isn't the best place to have this discussion. Maybe we should move the discussion to chromium-dev? 444027 19 senorblanco 2011-07-29 11:42:20 -0700 (In reply to comment #18) > > Although we considered it, I don't think that's practical at this time. There's a lot of work going on in Skia, and to add the lag of a WebKit roll to that work would be prohibitive. As one datapoint, there are a number of merges going from skia to the release branch right now. To add a WebKit merge to that process seems like it would be a nightmare. > > Whether we fix the skia revision in http://trac.webkit.org/browser/trunk/Source/WebKit/chromium/DEPS or http://src.chromium.org/viewvc/chrome/trunk/src/DEPS doesn't have any bearing on how you merge changes to release branches. Yeah, I think I was wrong about that -- any merges to the skia Release branch won't require a WebKit merge, since the release branch WebKit DEPS will already be pointing at the Skia release branch (I think?) > > Also, since we don't have full coverage of all tests on the WebKit bots, a skia change which introduced a ChromeOS or pyauto regression (or any one of the many bots we don't have WebKit canary coverage for) would require a rollback in skia, and a merge forward in WebKit to fix. > > We have that problem for every WebKit change. Right, this would just extend the problem to every skia change as well. > This bug probably isn't the best place to have this discussion. Maybe we should move the discussion to chromium-dev? I started to write out the pros and cons of each option, and after considering the above I think merging skia->WebKit actually comes out ahead. I'm going to run it by the skia folks again first to see what they think. I do think that the issue of the trybots not respecting the downstream test_expectations should be fixed, though. 444030 20 abarth 2011-07-29 11:47:46 -0700 > I do think that the issue of the trybots not respecting the downstream test_expectations should be fixed, though. Definitely. We should coordinate with maruel to make that happen. Thanks. 444063 21 dpranke 2011-07-29 13:20:00 -0700 (In reply to comment #20) > > I do think that the issue of the trybots not respecting the downstream test_expectations should be fixed, though. > > Definitely. We should coordinate with maruel to make that happen. > > Thanks. You can simply modify chromium.py to recognize the tryserver builder names and include the file. I don't think maruel needs to do anything. 444067 22 abarth 2011-07-29 13:28:27 -0700 > You can simply modify chromium.py to recognize the tryserver builder names and include the file. I don't think maruel needs to do anything. As these the names: linux_layoutm mac_layout, win_layout ? (from http://build.chromium.org/p/tryserver.chromium/waterfall) 444075 23 senorblanco 2011-07-29 13:40:08 -0700 (In reply to comment #22) > > You can simply modify chromium.py to recognize the tryserver builder names and include the file. I don't think maruel needs to do anything. > > As these the names: > > linux_layoutm mac_layout, win_layout ? > > (from http://build.chromium.org/p/tryserver.chromium/waterfall) There are Release flavours of each as well: linux_layout_rel, mac_layout_rel, win_layout_rel. 444079 24 abarth 2011-07-29 13:42:19 -0700 /me will write a patch. 444087 25 abarth 2011-07-29 14:05:51 -0700 https://bugs.webkit.org/show_bug.cgi?id=65390 95680 2011-06-01 15:55:18 -0700 2022-03-01 02:49:54 -0800 chromium-specific version bug-61888-20110601155517.patch text/plain 3340 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogODc3MDMKZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBi L1Rvb2xzL0NoYW5nZUxvZwppbmRleCAzNmU5ZDE0ZTEyN2M4MzZkMzcwNGQxMDAzMDY1ZjFiNmQ1 M2VlZDc0Li45Mzc1NjJmNWUyM2ZhZDdjZjBhNzUwN2Y4MmIzYjkzNTBmNzIyNzdiIDEwMDY0NAot LS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTMg QEAKKzIwMTEtMDYtMDEgIERpcmsgUHJhbmtlICA8ZHByYW5rZUBjaHJvbWl1bS5vcmc+CisKKyAg ICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgbnJ3dDogc2hvdWxk IHNraXAgY2hyb21pdW0gdGVzdCBleHBlY3RhdGlvbiBvdmVycmlkZXMgb24gdXBzdHJlYW0gYm90 cworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9NjE4ODgK KworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21pdW0u cHk6CisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1 bV91bml0dGVzdC5weToKKwogMjAxMS0wNS0zMCAgQ2FybG9zIEdhcmNpYSBDYW1wb3MgIDxjZ2Fy Y2lhQGlnYWxpYS5jb20+CiAKICAgICAgICAgUmV2aWV3ZWQgYnkgTWFydGluIFJvYmluc29uLgpk aWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJv bWl1bS5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21p dW0ucHkKaW5kZXggOGVjMGUxY2ZkMmVhYmQxMDA1MjJhMDI1NTBhZGJhZDY3OTJkOTM1Ny4uNGVj YjM2OTczYzgzNWVmNWI5MzAwODVhYWI4Zjk1OWM1ZWU4NWQ0MyAxMDA2NDQKLS0tIGEvVG9vbHMv U2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1bS5weQorKysgYi9Ub29s cy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Nocm9taXVtLnB5CkBAIC0yNjIs NiArMjYyLDEzIEBAIGNsYXNzIENocm9taXVtUG9ydChiYXNlLlBvcnQpOgogICAgICAgICByZXR1 cm4gc2VsZi5fZmlsZXN5c3RlbS5yZWFkX3RleHRfZmlsZShleHBlY3RhdGlvbnNfcGF0aCkKIAog ICAgIGRlZiB0ZXN0X2V4cGVjdGF0aW9uc19vdmVycmlkZXMoc2VsZik6CisgICAgICAgICMgU2Vl IGh0dHA6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTYxODg4IGZvciB0aGUgcmF0 aW9uYWxlIGZvciB0aGlzLgorICAgICAgICAjIEZJWE1FOiBJdCBzZWVtcyBiYWQgdGhhdCBydW5f d2Via2l0X3Rlc3RzLnB5IHVzZXMgYSBoYXJkY29kZWQgZHVtbXkKKyAgICAgICAgIyBidWlsZGVy IHN0cmluZyBpbnN0ZWFkIG9mIGp1c3QgdXNpbmcgTm9uZS4KKyAgICAgICAgYnVpbGRlcl9uYW1l ID0gc2VsZi5nZXRfb3B0aW9uKCdidWlsZGVyX25hbWUnLCAnRFVNTVlfQlVJTERFUl9OQU1FJykK KyAgICAgICAgaWYgYnVpbGRlcl9uYW1lICE9ICdEVU1NWV9CVUlMREVSX05BTUUnIGFuZCBub3Qg JyhkZXBzKScgaW4gYnVpbGRlcl9uYW1lOgorICAgICAgICAgICAgcmV0dXJuIE5vbmUKKwogICAg ICAgICB0cnk6CiAgICAgICAgICAgICBvdmVycmlkZXNfcGF0aCA9IHNlbGYucGF0aF9mcm9tX2No cm9taXVtX2Jhc2UoJ3dlYmtpdCcsICd0b29scycsCiAgICAgICAgICAgICAgICAgJ2xheW91dF90 ZXN0cycsICd0ZXN0X2V4cGVjdGF0aW9ucy50eHQnKQpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0 cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1bV91bml0dGVzdC5weSBiL1Rvb2xz L1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvY2hyb21pdW1fdW5pdHRlc3QucHkK aW5kZXggYjg1YWYwNTIzMzI3YTE2NDRhNGJhZmIyMjUzNTVhYzNhMWUyN2EwMC4uNzBiN2UxZDhj MmZhMjcyYWJlYjI4NjIzZWFiNDU4M2MyYmVjOTkwOSAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0 cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9jaHJvbWl1bV91bml0dGVzdC5weQorKysgYi9U b29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Nocm9taXVtX3VuaXR0ZXN0 LnB5CkBAIC0yMjgsNiArMjI4LDI5IEBAIExJTlVYIFdJTiA6IGZhc3QvanMvdmVyeS1nb29kLmpz ID0gVElNRU9VVCBQQVNTIiIiCiAgICAgICAgICAgICBleGNlcHRpb25fcmFpc2VkID0gVHJ1ZQog ICAgICAgICBzZWxmLmFzc2VydEZhbHNlKGV4Y2VwdGlvbl9yYWlzZWQpCiAKKyAgICBkZWYgdGVz dF9vdmVycmlkZXNfYW5kX2J1aWxkZXJfbmFtZXMoc2VsZik6CisgICAgICAgICMgU2VlIGJ1ZyBo dHRwOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD02MTg4OCBmb3IgdGhlIHJhdGlv bmFsZSBmb3IgdGhpcy4KKyAgICAgICAgcG9ydCA9IHNlbGYubWFrZV9wb3J0KCkKKyAgICAgICAg aWYgbm90IHBvcnQ6CisgICAgICAgICAgICByZXR1cm4KKworICAgICAgICBmaWxlc3lzdGVtID0g ZmlsZXN5c3RlbV9tb2NrLk1vY2tGaWxlU3lzdGVtKCkKKyAgICAgICAgcG9ydC5fZmlsZXN5c3Rl bSA9IGZpbGVzeXN0ZW0KKyAgICAgICAgcG9ydC5wYXRoX2Zyb21fY2hyb21pdW1fYmFzZSA9IGxh bWJkYSAqY29tcHM6ICcvJyArICcvJy5qb2luKGNvbXBzKQorCisgICAgICAgIG92ZXJyaWRlc19w YXRoID0gcG9ydC5wYXRoX2Zyb21fY2hyb21pdW1fYmFzZSgnd2Via2l0JywgJ3Rvb2xzJywgJ2xh eW91dF90ZXN0cycsICd0ZXN0X2V4cGVjdGF0aW9ucy50eHQnKQorICAgICAgICBPVkVSUklERVMg PSAnZm9vJworICAgICAgICBmaWxlc3lzdGVtLmZpbGVzW292ZXJyaWRlc19wYXRoXSA9IE9WRVJS SURFUworCisgICAgICAgIHBvcnQuX29wdGlvbnMuYnVpbGRlcl9uYW1lID0gJ0RVTU1ZX0JVSUxE RVJfTkFNRScKKyAgICAgICAgc2VsZi5hc3NlcnRFcXVhbHMocG9ydC50ZXN0X2V4cGVjdGF0aW9u c19vdmVycmlkZXMoKSwgT1ZFUlJJREVTKQorCisgICAgICAgIHBvcnQuX29wdGlvbnMuYnVpbGRl cl9uYW1lID0gJ2J1aWxkZXIgKGRlcHMpJworICAgICAgICBzZWxmLmFzc2VydEVxdWFscyhwb3J0 LnRlc3RfZXhwZWN0YXRpb25zX292ZXJyaWRlcygpLCBPVkVSUklERVMpCisKKyAgICAgICAgcG9y dC5fb3B0aW9ucy5idWlsZGVyX25hbWUgPSAnYnVpbGRlcicKKyAgICAgICAgc2VsZi5hc3NlcnRF cXVhbHMocG9ydC50ZXN0X2V4cGVjdGF0aW9uc19vdmVycmlkZXMoKSwgTm9uZSkKKwogCiBjbGFz cyBDaHJvbWl1bVBvcnRMb2dnaW5nVGVzdChsb2d0ZXN0aW5nLkxvZ2dpbmdUZXN0Q2FzZSk6CiAg ICAgZGVmIHRlc3RfY2hlY2tfc3lzX2RlcHMoc2VsZik6Cg== 95683 2011-06-01 16:08:04 -0700 2011-06-01 16:08:04 -0700 generic version bug-61888-20110601160803.patch text/plain 5717 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogODc3MDMKZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBi L1Rvb2xzL0NoYW5nZUxvZwppbmRleCAzNmU5ZDE0ZTEyN2M4MzZkMzcwNGQxMDAzMDY1ZjFiNmQ1 M2VlZDc0Li41MzFhZWNlMWEwOTM4ZDUzYmJjYTUxZDYwZGQ4MWVkNzkwMDg1YTM0IDEwMDY0NAot LS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTMg QEAKKzIwMTEtMDYtMDEgIERpcmsgUHJhbmtlICA8ZHByYW5rZUBjaHJvbWl1bS5vcmc+CisKKyAg ICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgbnJ3dDogc2hvdWxk IHNraXAgY2hyb21pdW0gdGVzdCBleHBlY3RhdGlvbiBvdmVycmlkZXMgb24gdXBzdHJlYW0gYm90 cworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9NjE4ODgK KworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVz dHNfaW50ZWdyYXRpb250ZXN0LnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL2xheW91dF9wYWNrYWdlL21hbmFnZXIucHk6CisKIDIwMTEtMDUtMzAgIENhcmxvcyBH YXJjaWEgQ2FtcG9zICA8Y2dhcmNpYUBpZ2FsaWEuY29tPgogCiAgICAgICAgIFJldmlld2VkIGJ5 IE1hcnRpbiBSb2JpbnNvbi4KZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5 b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL21hbmFnZXIucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9tYW5hZ2VyLnB5CmluZGV4IDc0YjhmODBm MDNlMzM4MmQ5MGNmZDUwZjliNTYyZDhhZmJiNDIxYzcuLjYyODQxZDFkZjMzMzVlN2ZiYzUyNGFm ZDhhNWEwZjlkOWE5ZGM5YWEgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5 b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL21hbmFnZXIucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93 ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2thZ2UvbWFuYWdlci5weQpAQCAtMjc2LDYg KzI3Niw4IEBAIGNsYXNzIE1hbmFnZXI6CiAgICAgICAgICMgVGhpcyBtYXBzIHdvcmtlciBuYW1l cyB0byB0aGUgc3RhdGUgd2UgYXJlIHRyYWNraW5nIGZvciBlYWNoIG9mIHRoZW0uCiAgICAgICAg IHNlbGYuX3dvcmtlcl9zdGF0ZXMgPSB7fQogCisgICAgICAgICMgU2VlIGJ1ZyAjNjE4ODggZm9y IHRoZSByYXRpb25hbGUgZm9yIHRoaXMuCisgICAgICAgIHNlbGYuX3VzZV9vdmVycmlkZXMgPSBv cHRpb25zLmJ1aWxkZXJfbmFtZSA9PSAnRFVNTVlfQlVJTERFUl9OQU1FJyBvciAnKGRlcHMpJyBp biBvcHRpb25zLmJ1aWxkZXJfbmFtZQogCiAgICAgZGVmIGNvbGxlY3RfdGVzdHMoc2VsZiwgYXJn cywgbGFzdF91bmV4cGVjdGVkX3Jlc3VsdHMpOgogICAgICAgICAiIiJGaW5kIGFsbCB0aGUgZmls ZXMgdG8gdGVzdC4KQEAgLTMxNywyNiArMzE5LDMyIEBAIGNsYXNzIE1hbmFnZXI6CiAKICAgICBk ZWYgbGludF9leHBlY3RhdGlvbnMoc2VsZiwgY29uZmlnKToKICAgICAgICAgcG9ydCA9IHNlbGYu X3BvcnQKKyAgICAgICAgb3ZlcnJpZGVzID0gTm9uZQorICAgICAgICBpZiBzZWxmLl91c2Vfb3Zl cnJpZGVzOgorICAgICAgICAgICAgb3ZlcnJpZGVzID0gcG9ydC50ZXN0X2V4cGVjdGF0aW9uc19v dmVycmlkZXMoKQogICAgICAgICB0ZXN0X2V4cGVjdGF0aW9ucy5UZXN0RXhwZWN0YXRpb25zKAog ICAgICAgICAgICAgcG9ydCwKICAgICAgICAgICAgIE5vbmUsCiAgICAgICAgICAgICBwb3J0LnRl c3RfZXhwZWN0YXRpb25zKCksCiAgICAgICAgICAgICBjb25maWcsCiAgICAgICAgICAgICBzZWxm Ll9vcHRpb25zLmxpbnRfdGVzdF9maWxlcywKLSAgICAgICAgICAgIHBvcnQudGVzdF9leHBlY3Rh dGlvbnNfb3ZlcnJpZGVzKCkpCisgICAgICAgICAgICBvdmVycmlkZXMpCiAKICAgICBkZWYgcGFy c2VfZXhwZWN0YXRpb25zKHNlbGYpOgogICAgICAgICAiIiJQYXJzZSB0aGUgZXhwZWN0YXRpb25z IGZyb20gdGhlIHRlc3RfbGlzdCBmaWxlcyBhbmQgcmV0dXJuIGEgZGF0YQogICAgICAgICBzdHJ1 Y3R1cmUgaG9sZGluZyB0aGVtLiBUaHJvd3MgYW4gZXJyb3IgaWYgdGhlIHRlc3RfbGlzdCBmaWxl cyBoYXZlCiAgICAgICAgIGludmFsaWQgc3ludGF4LiIiIgogICAgICAgICBwb3J0ID0gc2VsZi5f cG9ydAorICAgICAgICBvdmVycmlkZXMgPSBOb25lCisgICAgICAgIGlmIHNlbGYuX3VzZV9vdmVy cmlkZXM6CisgICAgICAgICAgICBvdmVycmlkZXMgPSBwb3J0LnRlc3RfZXhwZWN0YXRpb25zX292 ZXJyaWRlcygpCiAgICAgICAgIHNlbGYuX2V4cGVjdGF0aW9ucyA9IHRlc3RfZXhwZWN0YXRpb25z LlRlc3RFeHBlY3RhdGlvbnMoCiAgICAgICAgICAgICBwb3J0LAogICAgICAgICAgICAgc2VsZi5f dGVzdF9maWxlcywKICAgICAgICAgICAgIHBvcnQudGVzdF9leHBlY3RhdGlvbnMoKSwKICAgICAg ICAgICAgIHBvcnQudGVzdF9jb25maWd1cmF0aW9uKCksCiAgICAgICAgICAgICBzZWxmLl9vcHRp b25zLmxpbnRfdGVzdF9maWxlcywKLSAgICAgICAgICAgIHBvcnQudGVzdF9leHBlY3RhdGlvbnNf b3ZlcnJpZGVzKCkpCisgICAgICAgICAgICBvdmVycmlkZXMpCiAKICAgICAjIEZJWE1FOiBUaGlz IG1ldGhvZCBpcyB3YXkgdG9vIGxvbmcgYW5kIG5lZWRzIHRvIGJlIGJyb2tlbiBpbnRvIHBpZWNl cy4KICAgICBkZWYgcHJlcGFyZV9saXN0c19hbmRfcHJpbnRfb3V0cHV0KHNlbGYpOgpkaWZmIC0t Z2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2thZ2Uv bWFuYWdlcl91bml0dGVzdC5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3Rz L2xheW91dF9wYWNrYWdlL21hbmFnZXJfdW5pdHRlc3QucHkKaW5kZXggNDNjZWQ1YTQzNjNjYjJk Njk4MmZjNTBjMWNiYmQwM2ZlMjRmMDkyZC4uZmJmOWFmM2Y2OGViNDFmNGU0ZGQ5MjQ2MGUxNmJk MzFhMWJlNmYwNyAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvbGF5b3V0X3BhY2thZ2UvbWFuYWdlcl91bml0dGVzdC5weQorKysgYi9Ub29scy9TY3JpcHRz L3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9tYW5hZ2VyX3VuaXR0ZXN0LnB5 CkBAIC00OSw3ICs0OSw5IEBAIGNsYXNzIE1hbmFnZXJUZXN0KHVuaXR0ZXN0LlRlc3RDYXNlKToK ICAgICAgICAgIyBwdXQgdGhlIGh0dHAgdGVzdHMgZmlyc3QgaW4gdGhlIHF1ZXVlLgogICAgICAg ICBwb3J0ID0gTW9jaygpCiAgICAgICAgIHBvcnQuX2ZpbGVzeXN0ZW0gPSBmaWxlc3lzdGVtX21v Y2suTW9ja0ZpbGVTeXN0ZW0oKQotICAgICAgICBtYW5hZ2VyID0gTWFuYWdlcldyYXBwZXIocG9y dD1wb3J0LCBvcHRpb25zPU1vY2soKSwgcHJpbnRlcj1Nb2NrKCkpCisgICAgICAgIG9wdGlvbnMg PSBNb2NrKCkKKyAgICAgICAgb3B0aW9ucy5idWlsZGVyX25hbWUgPSAnRFVNTVlfQlVJTERFUl9O QU1FJworICAgICAgICBtYW5hZ2VyID0gTWFuYWdlcldyYXBwZXIocG9ydD1wb3J0LCBvcHRpb25z PW9wdGlvbnMsIHByaW50ZXI9TW9jaygpKQogCiAgICAgICAgIHRlc3RfbGlzdCA9IFsKICAgICAg ICAgICAiTGF5b3V0VGVzdHMvd2Vic29ja2V0L3Rlc3RzL3VuaWNvZGUuaHRtIiwKZGlmZiAtLWdp dCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHNf aW50ZWdyYXRpb250ZXN0LnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMv cnVuX3dlYmtpdF90ZXN0c19pbnRlZ3JhdGlvbnRlc3QucHkKaW5kZXggZjIzMmNhMjkxZDFhYTZi YWZkNjgyZThmZjEyZWExM2E1ZTM2ZTNkNC4uNjJmM2VkYmY1Nzc1YzVlNWRkMzY5NmE1ZDY2ZTg5 YTg0MTEyNWE3ZCAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvcnVuX3dlYmtpdF90ZXN0c19pbnRlZ3JhdGlvbnRlc3QucHkKKysrIGIvVG9vbHMvU2NyaXB0 cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0c19pbnRlZ3JhdGlvbnRlc3Qu cHkKQEAgLTYwNyw2ICs2MDcsMjkgQEAgY2xhc3MgTWFpblRlc3QodW5pdHRlc3QuVGVzdENhc2Up OgogICAgICAgICAgICAgICAgICAgICAgICAgaW4gcmVndWxhcl9vdXRwdXQuZ2V0KCkpCiAKIAor Y2xhc3MgQ2hyb21pdW1PdmVycmlkZXNUZXN0KHVuaXR0ZXN0LlRlc3RDYXNlKToKKyAgICBkZWYg cnVuX3Rlc3Qoc2VsZiwgYnVpbGRlcl9hcmdzLCBvdmVycmlkZV9zdHIsIGV4cGVjdGVkX3Jlc3Vs dCk6CisgICAgICAgIG9wdGlvbnMsIHBhcnNlZF9hcmdzID0gcGFyc2VfYXJncyhidWlsZGVyX2Fy Z3MgKyBbJ2ZhaWx1cmVzL2V4cGVjdGVkL2ltYWdlLmh0bWwnXSwKKyAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHRlc3RzX2luY2x1ZGVkPVRydWUpCisgICAgICAgIHBv cnRfb2JqID0gcG9ydC5nZXQocG9ydF9uYW1lPW9wdGlvbnMucGxhdGZvcm0sIG9wdGlvbnM9b3B0 aW9ucywKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICB1c2VyPW1vY2t0b29sLk1vY2tVc2Vy KCksIGZpbGVzeXN0ZW09Tm9uZSkKKyAgICAgICAgcG9ydF9vYmoudGVzdF9leHBlY3RhdGlvbnNf b3ZlcnJpZGVzID0gbGFtYmRhOiBvdmVycmlkZV9zdHIKKyAgICAgICAgcmVzLCBfLCBfID0gcnVu X2FuZF9jYXB0dXJlKHBvcnRfb2JqLCBvcHRpb25zLCBwYXJzZWRfYXJncykKKyAgICAgICAgc2Vs Zi5hc3NlcnRFcXVhbHMocmVzLCBleHBlY3RlZF9yZXN1bHQpCisKKyAgICBkZWYgdGVzdF9vdmVy cmlkZXNfYXJlX25vdF9pbmNsdWRlZF9pZl9ub3RfZGVwc19ib3Qoc2VsZik6CisgICAgICAgIHNl bGYucnVuX3Rlc3QoWyctLWJ1aWxkZXItbmFtZScsICd0ZXN0J10sICdCVUdYIDogZmFpbHVyZXMv ZXhwZWN0ZWQvaW1hZ2UuaHRtbCA9IFBBU1NcbicsIDApCisgICAgICAgIHNlbGYucnVuX3Rlc3Qo WyctLWJ1aWxkZXItbmFtZScsICd0ZXN0JywgJy0tbGludC10ZXN0LWZpbGVzJ10sICdzeW50YXgg ZXJyb3InLCAwKQorCisgICAgZGVmIHRlc3Rfb3ZlcnJpZGVzX2FyZV9pbmNsdWRlZF9mb3JfZGVw c19ib3RzKHNlbGYpOgorICAgICAgICBzZWxmLnJ1bl90ZXN0KFsnLS1idWlsZGVyLW5hbWUnLCAn dGVzdCAoZGVwcyknXSwgJ0JVR1ggOiBmYWlsdXJlcy9leHBlY3RlZC9pbWFnZS5odG1sID0gUEFT U1xuJywgMSkKKyAgICAgICAgc2VsZi5ydW5fdGVzdChbJy0tYnVpbGRlci1uYW1lJywgJ3Rlc3Qg KGRlcHMpJywgJy0tbGludC10ZXN0LWZpbGVzJ10sICdzeW50YXggZXJyb3InLCAtMSkKKworICAg IGRlZiB0ZXN0X292ZXJyaWRlc19hcmVfaW5jbHVkZWRfaWZfbm9fYnVpbGRlcihzZWxmKToKKyAg ICAgICAgc2VsZi5ydW5fdGVzdChbXSwgJ0JVR1ggOiBmYWlsdXJlcy9leHBlY3RlZC9pbWFnZS5o dG1sID0gUEFTU1xuJywgMSkKKyAgICAgICAgc2VsZi5ydW5fdGVzdChbJy0tbGludC10ZXN0LWZp bGVzJ10sICdzeW50YXggZXJyb3InLCAtMSkKKworCiBNYWluVGVzdCA9IHNraXBfaWYoTWFpblRl c3QsIHN5cy5wbGF0Zm9ybSA9PSAnY3lnd2luJyBhbmQgY29tcGFyZV92ZXJzaW9uKHN5cywgJzIu NicpWzBdIDwgMCwgJ25ldy1ydW4td2Via2l0LXRlc3RzIHRlc3RzIGhhbmcgb24gQ3lnd2luIFB5 dGhvbiAyLjUuMicpCiAKIAo=