97112 2012-09-19 07:57:50 -0700 REGRESSION (r126189): No more mismatch-did-not-occur failures in reftests 2012-09-26 10:46:52 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P1 Normal --- 1 zan zan abarth dglazkov dpranke ojan ossy peter+ews simon.fraser webkit.review.bot oldest_to_newest 723571 0 zan 2012-09-19 07:57:50 -0700 On GTK (and possibly other platforms as well, though most skip the test) when running fast/harness/sample-fail-mismatch-reftest.html the following output is shown, originating from SingleTestRunner[1]: fast/harness/sample-fail-mismatch-reftest.html -> ref test hashes matched but diff failed The test is expected to (and should) fail but actually passes, that is the reftest matches with its expected mismatch and that's considered an error. The output indicates ImageDiff does not produce the diff image event though there are no differences. I believe this is the current behavior of pixel dumping code throughout platforms when the actual hash matches the expected one[3][4][5]. [1] http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py#L328 [2] http://trac.webkit.org/changeset/126189 [3] http://trac.webkit.org/browser/trunk/Tools/DumpRenderTree/PixelDumpSupport.cpp#L71 [4] http://trac.webkit.org/browser/trunk/Tools/DumpRenderTree/qt/DumpRenderTreeQt.cpp#L962 [5] http://trac.webkit.org/browser/trunk/Tools/DumpRenderTree/chromium/TestShell.cpp#L708 723734 1 dpranke 2012-09-19 13:00:51 -0700 I'm a bit confused ... that log message says that although the hashes were the same, the images were different, and so there was a mismatch (hence the test ran "as expected"). That seems to be the opposite of what you're saying? FWIW, I just ran this on chromium mac (on lion) and the hashes were actually different. I am rebuilding the apple-mac port to check there. 723755 2 zan 2012-09-19 13:19:28 -0700 (In reply to comment #1) > I'm a bit confused ... that log message says that although the hashes were the same, the images were different, and so there was a mismatch (hence the test ran "as expected"). > > That seems to be the opposite of what you're saying? Indeed the hashes are the same (as they should be) but the image of the reference driver output is None because DumpRenderTrees don't seem to output diff images when the hashes match. The Port.diff_image method returns a (True, 0, None) tuple when either image content is missing[1], and this is considered as a difference between reference and actual output in the current SingleTestRunner._compare_output_with_reference code, meaning the test in question passes (mismatch occurred) while it should fail. > > FWIW, I just ran this on chromium mac (on lion) and the hashes were actually different. I am rebuilding the apple-mac port to check there. The hashes shouldn't differ, though. Don't know what's happening there. [1] http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L346 723798 3 dpranke 2012-09-19 13:53:01 -0700 (In reply to comment #2) > (In reply to comment #1) > > I'm a bit confused ... that log message says that although the hashes were the same, the images were different, and so there was a mismatch (hence the test ran "as expected"). > > > > That seems to be the opposite of what you're saying? > > Indeed the hashes are the same (as they should be) but the image of the reference driver output is None because DumpRenderTrees don't seem to output diff images when the hashes match. The Port.diff_image method returns a (True, 0, None) tuple when either image content is missing[1], and this is considered as a difference between reference and actual output in the current SingleTestRunner._compare_output_with_reference code, meaning the test in question passes (mismatch occurred) while it should fail. > Hm. Okay, well, we seem to have two choices. We can either (1) not pass the hash to the reference test as input (so that we always generate both PNGs) or (2) assume that if the hashes match we don't need to actually diff the two images. I'm inclined to do the former, since we've seen problems with hashes before, and it's not clear if there's any real perf impact to getting the second PNG. FWIW, though, both chromium linux and apple mac also produce different hashes for the two inputs. > > > > FWIW, I just ran this on chromium mac (on lion) and the hashes were actually different. I am rebuilding the apple-mac port to check there. > > The hashes shouldn't differ, though. Don't know what's happening there. > > > [1] http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L346 724612 4 zan 2012-09-20 10:24:55 -0700 (In reply to comment #3) > > Hm. Okay, well, we seem to have two choices. We can either (1) not pass the hash to the reference test as input (so that we always generate both PNGs) or (2) assume that if the hashes match we don't need to actually diff the two images. > > I'm inclined to do the former, since we've seen problems with hashes before, and it's not clear if there's any real perf impact to getting the second PNG. > I'll upload a patch that doesn't pass the hash to driver input. On a related note, this would be easier to spot if the test was reported as unexpectedly passing. But the test is reported as such only if pixel testing is enabled, despite the test itself being a reftest. Also discovered is the following problem - the expectation for the failing test is not taken into account (the failure is not expected) if in the following format: fast/harness/sample-fail-mismatch-reftest.html [ ImageOnlyFailure ] The following two expectations seem to work, though: (1) fast/harness/sample-fail-mismatch-reftest.html [ ImageOnlyFailure WontFix ] (2) Bug(GTK) fast/harness/sample-fail-mismatch-reftest.html [ ImageOnlyFailure ] I'll open two separate bugs for these issues. 724617 5 164941 zan 2012-09-20 10:33:40 -0700 Created attachment 164941 Patch 724633 6 164941 peter+ews 2012-09-20 10:48:51 -0700 Comment on attachment 164941 Patch Attachment 164941 did not pass cr-android-ews (chromium-android): Output: http://queues.webkit.org/results/13954252 724745 7 zan 2012-09-20 12:16:31 -0700 (In reply to comment #4) > On a related note, this would be easier to spot if the test was reported as unexpectedly passing. But the test is reported as such only if pixel testing is enabled, despite the test itself being a reftest. Also discovered is the following problem - the expectation for the failing test is not taken into account (the failure is not expected) if in the following format: > fast/harness/sample-fail-mismatch-reftest.html [ ImageOnlyFailure ] > > The following two expectations seem to work, though: > (1) fast/harness/sample-fail-mismatch-reftest.html [ ImageOnlyFailure WontFix ] > (2) Bug(GTK) fast/harness/sample-fail-mismatch-reftest.html [ ImageOnlyFailure ] The first issue is being handled in bug #97424, the other issue is not really an issue, the expectation should either have the WontFix modifier or the Bug(*) modifier. So no problem there after all. 724845 8 164941 dpranke 2012-09-20 14:09:44 -0700 Comment on attachment 164941 Patch can we add a test for this? 725318 9 zan 2012-09-21 02:41:09 -0700 (In reply to comment #8) > (From update of attachment 164941 [details]) > can we add a test for this? (In reply to comment #8) > (From update of attachment 164941 [details]) > can we add a test for this? Ideally yes, though MockDRT still doesn't seem to be up to date after not-that-recent per-test pixel dumping changes. 727856 10 zan 2012-09-25 12:09:17 -0700 > (In reply to comment #8) > > (From update of attachment 164941 [details] [details]) > > can we add a test for this? > > Ideally yes, though MockDRT still doesn't seem to be up to date after not-that-recent per-test pixel dumping changes. After closer reinspection I'm not finding any practical way of how to test this. Testing MockDRT behavior is not helping to test the code changes in SingleTestRunner and the latter class doesn't have its own unit tests. I'd be glad to receive some pointers on how to approach this or at very least just commit the patch (to be honest most of the latest changes to this code haven't included any tests). 727948 11 165666 dpranke 2012-09-25 14:12:10 -0700 Created attachment 165666 Patch 727954 12 dpranke 2012-09-25 14:15:51 -0700 (In reply to comment #10) > After closer reinspection I'm not finding any practical way of how to test this. Testing MockDRT behavior is not helping to test the code changes in SingleTestRunner and the latter class doesn't have its own unit tests. I'd be glad to receive some pointers on how to approach this I was looking for integration-level tests that make sure that we're testing how reftest mismatches are handled; it turns out that the way the Test port was implemented wasn't matching the normal ports' behavior, so I've updated it to return no image if the hashes match (and updated diff_image() to match the base implementation if it gets None). MockDRT could probably use a similar patch but we don't need it for this. > to be honest most of the latest changes to this code haven't included any tests). Pointers? If that was true, that would either be because the change wasn't for something I considered worth testing (e.g. the text in a log message), or I thought it was covered by existing tests. If there were substantive changes that went in w/o tests I'd be sad (probably means I missed something in the review). 728264 13 zan 2012-09-26 00:16:24 -0700 (In reply to comment #12) > (In reply to comment #10) > > After closer reinspection I'm not finding any practical way of how to test this. Testing MockDRT behavior is not helping to test the code changes in SingleTestRunner and the latter class doesn't have its own unit tests. I'd be glad to receive some pointers on how to approach this > > I was looking for integration-level tests that make sure that we're testing how reftest mismatches are handled; it turns out that the way the Test port was implemented wasn't matching the normal ports' behavior, so I've updated it to return no image if the hashes match (and updated diff_image() to match the base implementation if it gets None). MockDRT could probably use a similar patch but we don't need it for this. > Aha, thanks, makes sense. > > to be honest most of the latest changes to this code haven't included any tests). > > Pointers? If that was true, that would either be because the change wasn't for something I considered worth testing (e.g. the text in a log message), or I thought it was covered by existing tests. If there were substantive changes that went in w/o tests I'd be sad (probably means I missed something in the review). To be fair nothing substantial went in without being tested, though admittedly I was considering this patch unsubstantial as well. I apologize if I came around as offensive or doubtful, I didn't mean to. Thanks again, I'll land the patch shortly. 728265 14 zan 2012-09-26 00:17:52 -0700 Committed r129600: <http://trac.webkit.org/changeset/129600> 728727 15 dpranke 2012-09-26 10:46:52 -0700 (In reply to comment #13) > To be fair nothing substantial went in without being tested, though admittedly I was considering this patch unsubstantial as well. I apologize if I came around as offensive or doubtful, I didn't mean to. No offense taken, I was just wondering if I missed something. 164941 2012-09-20 10:33:40 -0700 2012-09-25 14:12:08 -0700 Patch bug-97112-20120920103305.patch text/plain 2376 zan U3VidmVyc2lvbiBSZXZpc2lvbjogMTI5MTM5CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZjMzZTZlZDBjNWYxZmNhZDRmNjljMDQ5MWNlYWJhYTEy ZDU2YzdlYi4uZWM4NzFjNzNkYWU3MWQzNzJmNzIyZjQ0MDA4MDA3OTU0MDYzY2E4MyAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE5 IEBACisyMDEyLTA5LTIwICBaYW4gRG9iZXJzZWsgIDx6YW5kb2JlcnNla0BnbWFpbC5jb20+CisK KyAgICAgICAgUkVHUkVTU0lPTiAocjEyNjE4OSk6IE5vIG1vcmUgbWlzbWF0Y2gtZGlkLW5vdC1v Y2N1ciBmYWlsdXJlcyBpbiByZWZ0ZXN0cworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9y Zy9zaG93X2J1Zy5jZ2k/aWQ9OTcxMTIKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9P UFMhKS4KKworICAgICAgICBEb24ndCB1c2UgdGhlIHJlZnRlc3QncyBpbWFnZSBoYXNoIHdoZW4g Z2F0aGVyaW5nIG91dHB1dCBvZiBpdHMgcmVmZXJlbmNlLgorICAgICAgICBUaGlzIGluZGlyZWN0 bHkgZm9yY2VzIHRoZSBkcml2ZXIgdG8gcmV0dXJuIHRoZSByZWZlcmVuY2UncyBpbWFnZSBvdXRw dXQKKyAgICAgICAgYXMgd2VsbC4gVGhpcyBzcGVjaWZpY2FsbHkgc29sdmVzIHRoZSBwcm9ibGVt IHdoZW4gdGhlIHJlZnRlc3QgYW5kIGl0cworICAgICAgICBtaXNtYXRjaCByZWZlcmVuY2UgYXJl IGVxdWFsIGJ1dCB0aGUgdW5leHBlY3RlZCBtYXRjaCBmYWlsdXJlIGlzIG5vdAorICAgICAgICBy ZXBvcnRlZCBkdWUgdG8gbWlzc2luZyBpbWFnZSBvdXRwdXQgb2YgdGhlIHJlZmVyZW5jZSBmaWxl LgorCisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMv c2luZ2xlX3Rlc3RfcnVubmVyLnB5OgorICAgICAgICAoU2luZ2xlVGVzdFJ1bm5lci5fcnVuX3Jl ZnRlc3QpOgorCiAyMDEyLTA5LTIwICBTdGVwaGVuIENoZW5uZXkgIDxzY2hlbm5leUBjaHJvbWl1 bS5vcmc+CiAKICAgICAgICAgW0Nocm9taXVtXSBEUlQgZG9lcyBub3Qgc3VwcG9ydCAtLWR1bXAt YWxsLXBpeGVscyBmbGFnCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91 dF90ZXN0cy9jb250cm9sbGVycy9zaW5nbGVfdGVzdF9ydW5uZXIucHkgYi9Ub29scy9TY3JpcHRz L3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9zaW5nbGVfdGVzdF9ydW5uZXIucHkK aW5kZXggZWRhYzhlYmIzMjgyNjRhOTMxNDE1YTJiZjlmZjdjNDZkZWJhNDI4ZS4uNDZkMjE4Y2Ey Yjk5Yzc0ODRiNGQyZThjNThjZDcxM2I5MDg5ZmYwNiAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0 cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvc2luZ2xlX3Rlc3RfcnVubmVyLnB5 CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL3Np bmdsZV90ZXN0X3J1bm5lci5weQpAQCAtMjk0LDcgKzI5NCw3IEBAIGNsYXNzIFNpbmdsZVRlc3RS dW5uZXIob2JqZWN0KToKICAgICAgICAgcHV0QWxsTWlzbWF0Y2hCZWZvcmVNYXRjaCA9IHNvcnRl ZAogICAgICAgICBmb3IgZXhwZWN0YXRpb24sIHJlZmVyZW5jZV9maWxlbmFtZSBpbiBwdXRBbGxN aXNtYXRjaEJlZm9yZU1hdGNoKHNlbGYuX3JlZmVyZW5jZV9maWxlcyk6CiAgICAgICAgICAgICBy ZWZlcmVuY2VfdGVzdF9uYW1lID0gc2VsZi5fcG9ydC5yZWxhdGl2ZV90ZXN0X2ZpbGVuYW1lKHJl ZmVyZW5jZV9maWxlbmFtZSkKLSAgICAgICAgICAgIHJlZmVyZW5jZV9vdXRwdXQgPSBzZWxmLl9k cml2ZXIucnVuX3Rlc3QoRHJpdmVySW5wdXQocmVmZXJlbmNlX3Rlc3RfbmFtZSwgc2VsZi5fdGlt ZW91dCwgdGVzdF9vdXRwdXQuaW1hZ2VfaGFzaCwgc2hvdWxkX3J1bl9waXhlbF90ZXN0PVRydWUp LCBzZWxmLl9zdG9wX3doZW5fZG9uZSkKKyAgICAgICAgICAgIHJlZmVyZW5jZV9vdXRwdXQgPSBz ZWxmLl9kcml2ZXIucnVuX3Rlc3QoRHJpdmVySW5wdXQocmVmZXJlbmNlX3Rlc3RfbmFtZSwgc2Vs Zi5fdGltZW91dCwgTm9uZSwgc2hvdWxkX3J1bl9waXhlbF90ZXN0PVRydWUpLCBzZWxmLl9zdG9w X3doZW5fZG9uZSkKICAgICAgICAgICAgIHRlc3RfcmVzdWx0ID0gc2VsZi5fY29tcGFyZV9vdXRw dXRfd2l0aF9yZWZlcmVuY2UocmVmZXJlbmNlX291dHB1dCwgdGVzdF9vdXRwdXQsIHJlZmVyZW5j ZV9maWxlbmFtZSwgZXhwZWN0YXRpb24gPT0gJyE9JykKIAogICAgICAgICAgICAgaWYgKGV4cGVj dGF0aW9uID09ICchPScgYW5kIHRlc3RfcmVzdWx0LmZhaWx1cmVzKSBvciAoZXhwZWN0YXRpb24g PT0gJz09JyBhbmQgbm90IHRlc3RfcmVzdWx0LmZhaWx1cmVzKToK 165666 2012-09-25 14:12:10 -0700 2012-09-25 14:15:59 -0700 Patch bug-97112-20120925141130.patch text/plain 3911 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogMTI5NDE3CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNzMyODY4ZTJiMzA5MDdiN2QxYmE4YzcwNTljNDk5OWNk MmNlOTljNi4uMGMwNTJhOWJlYmI4ZDIwMzY3MDJlM2I1YzM3M2VkOTJkZjNiOWRmNiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDIy IEBACisyMDEyLTA5LTI1ICBaYW4gRG9iZXJzZWsgIDx6YW5kb2JlcnNla0BnbWFpbC5jb20+CisK KyAgICAgICAgUkVHUkVTU0lPTiAocjEyNjE4OSk6IE5vIG1vcmUgbWlzbWF0Y2gtZGlkLW5vdC1v Y2N1ciBmYWlsdXJlcyBpbiByZWZ0ZXN0cworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9y Zy9zaG93X2J1Zy5jZ2k/aWQ9OTcxMTIKKworICAgICAgICBSZXZpZXdlZCBieSBEaXJrIFByYW5r ZS4KKworICAgICAgICBEb24ndCB1c2UgdGhlIHJlZnRlc3QncyBpbWFnZSBoYXNoIHdoZW4gZ2F0 aGVyaW5nIG91dHB1dCBvZiBpdHMgcmVmZXJlbmNlLgorICAgICAgICBUaGlzIGluZGlyZWN0bHkg Zm9yY2VzIHRoZSBkcml2ZXIgdG8gcmV0dXJuIHRoZSByZWZlcmVuY2UncyBpbWFnZSBvdXRwdXQK KyAgICAgICAgYXMgd2VsbC4gVGhpcyBzcGVjaWZpY2FsbHkgc29sdmVzIHRoZSBwcm9ibGVtIHdo ZW4gdGhlIHJlZnRlc3QgYW5kIGl0cworICAgICAgICBtaXNtYXRjaCByZWZlcmVuY2UgYXJlIGVx dWFsIGJ1dCB0aGUgdW5leHBlY3RlZCBtYXRjaCBmYWlsdXJlIGlzIG5vdAorICAgICAgICByZXBv cnRlZCBkdWUgdG8gbWlzc2luZyBpbWFnZSBvdXRwdXQgb2YgdGhlIHJlZmVyZW5jZSBmaWxlLgor CisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvc2lu Z2xlX3Rlc3RfcnVubmVyLnB5OgorICAgICAgICAoU2luZ2xlVGVzdFJ1bm5lci5fcnVuX3JlZnRl c3QpOgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvdGVzdC5w eToKKyAgICAgICAgKFRlc3RQb3J0LmRpZmZfaW1hZ2UpOgorICAgICAgICAoVGVzdERyaXZlci5y dW5fdGVzdCk6CisKIDIwMTItMDktMjQgIFNoZXJpZmYgQm90ICA8d2Via2l0LnJldmlldy5ib3RA Z21haWwuY29tPgogCiAgICAgICAgIFVucmV2aWV3ZWQsIHJvbGxpbmcgb3V0IHIxMjkzNDMuCmRp ZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVy cy9zaW5nbGVfdGVzdF9ydW5uZXIucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90 ZXN0cy9jb250cm9sbGVycy9zaW5nbGVfdGVzdF9ydW5uZXIucHkKaW5kZXggMDliNGUxZDlmYmZi Njc5MThlNTUzNTcxNzI5YWIyYzc4MjUwNjNiOS4uY2IxZjAyMjE4MmNhNmRiZjhlYTQ3ZDc0N2Yx YzM2MGM0OTY1Mjg1MiAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRf dGVzdHMvY29udHJvbGxlcnMvc2luZ2xlX3Rlc3RfcnVubmVyLnB5CisrKyBiL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL3NpbmdsZV90ZXN0X3J1bm5lci5w eQpAQCAtMjk0LDcgKzI5NCw3IEBAIGNsYXNzIFNpbmdsZVRlc3RSdW5uZXIob2JqZWN0KToKICAg ICAgICAgcHV0QWxsTWlzbWF0Y2hCZWZvcmVNYXRjaCA9IHNvcnRlZAogICAgICAgICBmb3IgZXhw ZWN0YXRpb24sIHJlZmVyZW5jZV9maWxlbmFtZSBpbiBwdXRBbGxNaXNtYXRjaEJlZm9yZU1hdGNo KHNlbGYuX3JlZmVyZW5jZV9maWxlcyk6CiAgICAgICAgICAgICByZWZlcmVuY2VfdGVzdF9uYW1l ID0gc2VsZi5fcG9ydC5yZWxhdGl2ZV90ZXN0X2ZpbGVuYW1lKHJlZmVyZW5jZV9maWxlbmFtZSkK LSAgICAgICAgICAgIHJlZmVyZW5jZV9vdXRwdXQgPSBzZWxmLl9kcml2ZXIucnVuX3Rlc3QoRHJp dmVySW5wdXQocmVmZXJlbmNlX3Rlc3RfbmFtZSwgc2VsZi5fdGltZW91dCwgdGVzdF9vdXRwdXQu aW1hZ2VfaGFzaCwgc2hvdWxkX3J1bl9waXhlbF90ZXN0PVRydWUpLCBzZWxmLl9zdG9wX3doZW5f ZG9uZSkKKyAgICAgICAgICAgIHJlZmVyZW5jZV9vdXRwdXQgPSBzZWxmLl9kcml2ZXIucnVuX3Rl c3QoRHJpdmVySW5wdXQocmVmZXJlbmNlX3Rlc3RfbmFtZSwgc2VsZi5fdGltZW91dCwgTm9uZSwg c2hvdWxkX3J1bl9waXhlbF90ZXN0PVRydWUpLCBzZWxmLl9zdG9wX3doZW5fZG9uZSkKICAgICAg ICAgICAgIHRlc3RfcmVzdWx0ID0gc2VsZi5fY29tcGFyZV9vdXRwdXRfd2l0aF9yZWZlcmVuY2Uo cmVmZXJlbmNlX291dHB1dCwgdGVzdF9vdXRwdXQsIHJlZmVyZW5jZV9maWxlbmFtZSwgZXhwZWN0 YXRpb24gPT0gJyE9JykKIAogICAgICAgICAgICAgaWYgKGV4cGVjdGF0aW9uID09ICchPScgYW5k IHRlc3RfcmVzdWx0LmZhaWx1cmVzKSBvciAoZXhwZWN0YXRpb24gPT0gJz09JyBhbmQgbm90IHRl c3RfcmVzdWx0LmZhaWx1cmVzKToKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkv bGF5b3V0X3Rlc3RzL3BvcnQvdGVzdC5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL3BvcnQvdGVzdC5weQppbmRleCBjYmMwZGJlZDFiZWRhNzI4MjU3ZjJkYWQ5MGRiNzYw ZWEwZWVkYTJlLi5kMDA4ZDk5NWQ1YTQ5ZDA0ZTAyOGQ2NzFhYzZiMmUyNDY3NTBlMzliIDEwMDY0 NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L3Rlc3QucHkK KysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC90ZXN0LnB5CkBA IC00MDUsNiArNDA1LDEwIEBAIGNsYXNzIFRlc3RQb3J0KFBvcnQpOgogCiAgICAgZGVmIGRpZmZf aW1hZ2Uoc2VsZiwgZXhwZWN0ZWRfY29udGVudHMsIGFjdHVhbF9jb250ZW50cywgdG9sZXJhbmNl PU5vbmUpOgogICAgICAgICBkaWZmZWQgPSBhY3R1YWxfY29udGVudHMgIT0gZXhwZWN0ZWRfY29u dGVudHMKKyAgICAgICAgaWYgbm90IGFjdHVhbF9jb250ZW50cyBhbmQgbm90IGV4cGVjdGVkX2Nv bnRlbnRzOgorICAgICAgICAgICAgcmV0dXJuIChOb25lLCAwLCBOb25lKQorICAgICAgICBpZiBu b3QgYWN0dWFsX2NvbnRlbnRzIG9yIG5vdCBleHBlY3RlZF9jb250ZW50czoKKyAgICAgICAgICAg IHJldHVybiAoVHJ1ZSwgMCwgTm9uZSkKICAgICAgICAgaWYgJ3JlZicgaW4gZXhwZWN0ZWRfY29u dGVudHM6CiAgICAgICAgICAgICBhc3NlcnQgdG9sZXJhbmNlID09IDAKICAgICAgICAgaWYgZGlm ZmVkOgpAQCAtNTY4LDcgKzU3MiwxMSBAQCBjbGFzcyBUZXN0RHJpdmVyKERyaXZlcik6CiAgICAg ICAgIGlmIHN0b3Bfd2hlbl9kb25lOgogICAgICAgICAgICAgc2VsZi5zdG9wKCkKIAotICAgICAg ICByZXR1cm4gRHJpdmVyT3V0cHV0KGFjdHVhbF90ZXh0LCB0ZXN0LmFjdHVhbF9pbWFnZSwgdGVz dC5hY3R1YWxfY2hlY2tzdW0sIGF1ZGlvLAorICAgICAgICBpZiB0ZXN0LmFjdHVhbF9jaGVja3N1 bSA9PSB0ZXN0X2lucHV0LmltYWdlX2hhc2g6CisgICAgICAgICAgICBpbWFnZSA9IE5vbmUKKyAg ICAgICAgZWxzZToKKyAgICAgICAgICAgIGltYWdlID0gdGVzdC5hY3R1YWxfaW1hZ2UKKyAgICAg ICAgcmV0dXJuIERyaXZlck91dHB1dChhY3R1YWxfdGV4dCwgaW1hZ2UsIHRlc3QuYWN0dWFsX2No ZWNrc3VtLCBhdWRpbywKICAgICAgICAgICAgIGNyYXNoPXRlc3QuY3Jhc2ggb3IgdGVzdC53ZWJf cHJvY2Vzc19jcmFzaCwgY3Jhc2hlZF9wcm9jZXNzX25hbWU9Y3Jhc2hlZF9wcm9jZXNzX25hbWUs CiAgICAgICAgICAgICBjcmFzaGVkX3BpZD1jcmFzaGVkX3BpZCwgY3Jhc2hfbG9nPWNyYXNoX2xv ZywKICAgICAgICAgICAgIHRlc3RfdGltZT10aW1lLnRpbWUoKSAtIHN0YXJ0X3RpbWUsIHRpbWVv dXQ9dGVzdC50aW1lb3V0LCBlcnJvcj10ZXN0LmVycm9yKQo=