216536 2020-09-15 06:06:11 -0700 Add further temporary debug logging for wptserve 2020-09-16 16:40:21 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED WONTFIX https://bugs.webkit.org/show_bug.cgi?id=215829 P2 Normal --- 1 gsnedders gsnedders aakash_jain ap clopez ews-watchlist youennf oldest_to_newest 1688655 0 gsnedders 2020-09-15 06:06:11 -0700 This just checks what processes and open files we have, to try and see why we're running into "too many open files" 1688656 1 408814 gsnedders 2020-09-15 06:11:24 -0700 Created attachment 408814 Patch 1688657 2 ews-watchlist 2020-09-15 06:12:29 -0700 This patch modifies the imported WPT tests. Please ensure that any changes on the tests (not coming from a WPT import) are exported to WPT. Please see https://trac.webkit.org/wiki/WPTExportProcess 1688691 3 408814 aakash_jain 2020-09-15 08:15:18 -0700 Comment on attachment 408814 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=408814&action=review > LayoutTests/imported/w3c/web-platform-tests/tools/wptserve/wptserve/response.py:254 > + self.logger.debug(subprocess.check_output(["ps", "aux"]) missing parenthesis 1688709 4 408824 gsnedders 2020-09-15 09:15:52 -0700 Created attachment 408824 Patch 1688725 5 aakash_jain 2020-09-15 09:43:27 -0700 Where are we running into "too many open files"? Any bug/radar with the details? 1688728 6 gsnedders 2020-09-15 09:50:24 -0700 See the see also bug: 215829 :) 1688755 7 408834 gsnedders 2020-09-15 11:06:43 -0700 Created attachment 408834 Patch 1688773 8 408838 gsnedders 2020-09-15 11:30:59 -0700 Created attachment 408838 Patch 1688850 9 gsnedders 2020-09-15 14:25:54 -0700 This entire approach is doomed to failure, because lsof can block and make everything worse. 1688851 10 408838 aakash_jain 2020-09-15 14:27:07 -0700 Comment on attachment 408838 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=408838&action=review > LayoutTests/imported/w3c/web-platform-tests/tools/wptserve/wptserve/response.py:257 > + self.logger.debug(subprocess.check_output(["ps", "aux"])) This ('ps aux' and 'lsof -P +c0') will produce huge amount of output (tens of thousands of lines). Are you sure you want to add that much output in the logs? Maybe you just need a count of number of lines of these. Also, when the machine is already running out of file handles, these commands might just slow down/hang the machine further. This might explain why https://ews-build.webkit.org/#/builders/30/builds/17561/steps/11/logs/stdio hanged with this patch. 1689257 11 ap 2020-09-16 16:40:18 -0700 We can manually run a job on one of the machines dumping lsof every few seconds, this way we could catch it right before the condition occurs. As long as it's not a super quick explosion, that could be good enough. 408814 2020-09-15 06:11:24 -0700 2020-09-15 09:15:48 -0700 Patch bug-216536-20200915151123.patch text/plain 1734 gsnedders U3VidmVyc2lvbiBSZXZpc2lvbjogMjY3MDgzCmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9pbXBv cnRlZC93M2MvQ2hhbmdlTG9nIGIvTGF5b3V0VGVzdHMvaW1wb3J0ZWQvdzNjL0NoYW5nZUxvZwpp bmRleCA5OTdhNmY2YmM4MDE5YTQ1NGYxMDdhMzIyYTgxZDM3ZDQ5NGUxYTg5Li44NDY5MGQ2Y2Qx ODczYzkxNjljZTY1N2JmMWNlZmU2MGUxODA1ZGI5IDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9p bXBvcnRlZC93M2MvQ2hhbmdlTG9nCisrKyBiL0xheW91dFRlc3RzL2ltcG9ydGVkL3czYy9DaGFu Z2VMb2cKQEAgLTEsMyArMSwxMyBAQAorMjAyMC0wOS0xNSAgU2FtIFNuZWRkb24gIDxnc25lZGRl cnNAYXBwbGUuY29tPgorCisgICAgICAgIEFkZCB0ZW1wb3JhcnkgZGVidWdnaW5nIHRvIHdwdHNl cnZlCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMTY1 MzYKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIHdl Yi1wbGF0Zm9ybS10ZXN0cy90b29scy93cHRzZXJ2ZS93cHRzZXJ2ZS9yZXNwb25zZS5weToKKyAg ICAgICAgKFJlc3BvbnNlLnNldF9lcnJvcik6CisKIDIwMjAtMDktMTUgIFJvYiBCdWlzICA8cmJ1 aXNAaWdhbGlhLmNvbT4KIAogICAgICAgICBDb252ZXJ0IHJ1bnRpbWUgZmxhZyB0byBzZXR0aW5n IGZvciBsYXp5IGltYWdlIGxvYWRpbmcKZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL2ltcG9ydGVk L3czYy93ZWItcGxhdGZvcm0tdGVzdHMvdG9vbHMvd3B0c2VydmUvd3B0c2VydmUvcmVzcG9uc2Uu cHkgYi9MYXlvdXRUZXN0cy9pbXBvcnRlZC93M2Mvd2ViLXBsYXRmb3JtLXRlc3RzL3Rvb2xzL3dw dHNlcnZlL3dwdHNlcnZlL3Jlc3BvbnNlLnB5CmluZGV4IDY4NmU3YzZlN2Y2MWVlZGQ3OTUzOWQ0 Yzk3MjZhMGFiMGNhZDkzOTcuLjhkNjdlMjg0YjIyYTdhNjRhNmMxODc5NmRjMzczNDZiYjBiMDVl YjMgMTAwNjQ0Ci0tLSBhL0xheW91dFRlc3RzL2ltcG9ydGVkL3czYy93ZWItcGxhdGZvcm0tdGVz dHMvdG9vbHMvd3B0c2VydmUvd3B0c2VydmUvcmVzcG9uc2UucHkKKysrIGIvTGF5b3V0VGVzdHMv aW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy90b29scy93cHRzZXJ2ZS93cHRzZXJ2ZS9y ZXNwb25zZS5weQpAQCAtMyw2ICszLDcgQEAgZnJvbSBkYXRldGltZSBpbXBvcnQgZGF0ZXRpbWUs IHRpbWVkZWx0YQogZnJvbSBpbyBpbXBvcnQgQnl0ZXNJTwogaW1wb3J0IGpzb24KIGltcG9ydCBz b2NrZXQKK2ltcG9ydCBzdWJwcm9jZXNzCiBpbXBvcnQgdXVpZAogCiBmcm9tIGhwYWNrLnN0cnVj dCBpbXBvcnQgSGVhZGVyVHVwbGUKQEAgLTI0OSw2ICsyNTAsMTAgQEAgY2xhc3MgUmVzcG9uc2Uo b2JqZWN0KToKICAgICAgICAgc2VsZi5jb250ZW50ID0gZGF0YQogICAgICAgICBpZiBtZXNzYWdl OgogICAgICAgICAgICAgc2VsZi5sb2dnZXIuZXJyb3IobWVzc2FnZSkKKyAgICAgICAgICAgIHNl bGYubG9nZ2VyLmRlYnVnKCJwcyBhdXg6IikKKyAgICAgICAgICAgIHNlbGYubG9nZ2VyLmRlYnVn KHN1YnByb2Nlc3MuY2hlY2tfb3V0cHV0KFsicHMiLCAiYXV4Il0pCisgICAgICAgICAgICBzZWxm LmxvZ2dlci5kZWJ1ZygibHNvZjoiKQorICAgICAgICAgICAgc2VsZi5sb2dnZXIuZGVidWcoc3Vi cHJvY2Vzcy5jaGVja19vdXRwdXQoWyJsc29mIiwgIi1QIiwgIitjMCJdKQogCiAKIGNsYXNzIE11 bHRpcGFydENvbnRlbnQob2JqZWN0KToK 408824 2020-09-15 09:15:52 -0700 2020-09-15 11:06:40 -0700 Patch bug-216536-20200915181550.patch text/plain 1736 gsnedders U3VidmVyc2lvbiBSZXZpc2lvbjogMjY3MDgzCmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9pbXBv cnRlZC93M2MvQ2hhbmdlTG9nIGIvTGF5b3V0VGVzdHMvaW1wb3J0ZWQvdzNjL0NoYW5nZUxvZwpp bmRleCA5OTdhNmY2YmM4MDE5YTQ1NGYxMDdhMzIyYTgxZDM3ZDQ5NGUxYTg5Li44NDY5MGQ2Y2Qx ODczYzkxNjljZTY1N2JmMWNlZmU2MGUxODA1ZGI5IDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9p bXBvcnRlZC93M2MvQ2hhbmdlTG9nCisrKyBiL0xheW91dFRlc3RzL2ltcG9ydGVkL3czYy9DaGFu Z2VMb2cKQEAgLTEsMyArMSwxMyBAQAorMjAyMC0wOS0xNSAgU2FtIFNuZWRkb24gIDxnc25lZGRl cnNAYXBwbGUuY29tPgorCisgICAgICAgIEFkZCB0ZW1wb3JhcnkgZGVidWdnaW5nIHRvIHdwdHNl cnZlCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMTY1 MzYKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIHdl Yi1wbGF0Zm9ybS10ZXN0cy90b29scy93cHRzZXJ2ZS93cHRzZXJ2ZS9yZXNwb25zZS5weToKKyAg ICAgICAgKFJlc3BvbnNlLnNldF9lcnJvcik6CisKIDIwMjAtMDktMTUgIFJvYiBCdWlzICA8cmJ1 aXNAaWdhbGlhLmNvbT4KIAogICAgICAgICBDb252ZXJ0IHJ1bnRpbWUgZmxhZyB0byBzZXR0aW5n IGZvciBsYXp5IGltYWdlIGxvYWRpbmcKZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL2ltcG9ydGVk L3czYy93ZWItcGxhdGZvcm0tdGVzdHMvdG9vbHMvd3B0c2VydmUvd3B0c2VydmUvcmVzcG9uc2Uu cHkgYi9MYXlvdXRUZXN0cy9pbXBvcnRlZC93M2Mvd2ViLXBsYXRmb3JtLXRlc3RzL3Rvb2xzL3dw dHNlcnZlL3dwdHNlcnZlL3Jlc3BvbnNlLnB5CmluZGV4IDY4NmU3YzZlN2Y2MWVlZGQ3OTUzOWQ0 Yzk3MjZhMGFiMGNhZDkzOTcuLjk3ZjNkZjI5NjUzYWRmZDU1NzVkNWVhYjgwMWY0NzY5M2FiYmEx ZTYgMTAwNjQ0Ci0tLSBhL0xheW91dFRlc3RzL2ltcG9ydGVkL3czYy93ZWItcGxhdGZvcm0tdGVz dHMvdG9vbHMvd3B0c2VydmUvd3B0c2VydmUvcmVzcG9uc2UucHkKKysrIGIvTGF5b3V0VGVzdHMv aW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy90b29scy93cHRzZXJ2ZS93cHRzZXJ2ZS9y ZXNwb25zZS5weQpAQCAtMyw2ICszLDcgQEAgZnJvbSBkYXRldGltZSBpbXBvcnQgZGF0ZXRpbWUs IHRpbWVkZWx0YQogZnJvbSBpbyBpbXBvcnQgQnl0ZXNJTwogaW1wb3J0IGpzb24KIGltcG9ydCBz b2NrZXQKK2ltcG9ydCBzdWJwcm9jZXNzCiBpbXBvcnQgdXVpZAogCiBmcm9tIGhwYWNrLnN0cnVj dCBpbXBvcnQgSGVhZGVyVHVwbGUKQEAgLTI0OSw2ICsyNTAsMTAgQEAgY2xhc3MgUmVzcG9uc2Uo b2JqZWN0KToKICAgICAgICAgc2VsZi5jb250ZW50ID0gZGF0YQogICAgICAgICBpZiBtZXNzYWdl OgogICAgICAgICAgICAgc2VsZi5sb2dnZXIuZXJyb3IobWVzc2FnZSkKKyAgICAgICAgICAgIHNl bGYubG9nZ2VyLmRlYnVnKCJwcyBhdXg6IikKKyAgICAgICAgICAgIHNlbGYubG9nZ2VyLmRlYnVn KHN1YnByb2Nlc3MuY2hlY2tfb3V0cHV0KFsicHMiLCAiYXV4Il0pKQorICAgICAgICAgICAgc2Vs Zi5sb2dnZXIuZGVidWcoImxzb2Y6IikKKyAgICAgICAgICAgIHNlbGYubG9nZ2VyLmRlYnVnKHN1 YnByb2Nlc3MuY2hlY2tfb3V0cHV0KFsibHNvZiIsICItUCIsICIrYzAiXSkpCiAKIAogY2xhc3Mg TXVsdGlwYXJ0Q29udGVudChvYmplY3QpOgo= 408834 2020-09-15 11:06:43 -0700 2020-09-15 11:30:55 -0700 Patch bug-216536-20200915200642.patch text/plain 1805 gsnedders U3VidmVyc2lvbiBSZXZpc2lvbjogMjY3MDgzCmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9pbXBv cnRlZC93M2MvQ2hhbmdlTG9nIGIvTGF5b3V0VGVzdHMvaW1wb3J0ZWQvdzNjL0NoYW5nZUxvZwpp bmRleCA5OTdhNmY2YmM4MDE5YTQ1NGYxMDdhMzIyYTgxZDM3ZDQ5NGUxYTg5Li44NDY5MGQ2Y2Qx ODczYzkxNjljZTY1N2JmMWNlZmU2MGUxODA1ZGI5IDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9p bXBvcnRlZC93M2MvQ2hhbmdlTG9nCisrKyBiL0xheW91dFRlc3RzL2ltcG9ydGVkL3czYy9DaGFu Z2VMb2cKQEAgLTEsMyArMSwxMyBAQAorMjAyMC0wOS0xNSAgU2FtIFNuZWRkb24gIDxnc25lZGRl cnNAYXBwbGUuY29tPgorCisgICAgICAgIEFkZCB0ZW1wb3JhcnkgZGVidWdnaW5nIHRvIHdwdHNl cnZlCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMTY1 MzYKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIHdl Yi1wbGF0Zm9ybS10ZXN0cy90b29scy93cHRzZXJ2ZS93cHRzZXJ2ZS9yZXNwb25zZS5weToKKyAg ICAgICAgKFJlc3BvbnNlLnNldF9lcnJvcik6CisKIDIwMjAtMDktMTUgIFJvYiBCdWlzICA8cmJ1 aXNAaWdhbGlhLmNvbT4KIAogICAgICAgICBDb252ZXJ0IHJ1bnRpbWUgZmxhZyB0byBzZXR0aW5n IGZvciBsYXp5IGltYWdlIGxvYWRpbmcKZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL2ltcG9ydGVk L3czYy93ZWItcGxhdGZvcm0tdGVzdHMvdG9vbHMvd3B0c2VydmUvd3B0c2VydmUvcmVzcG9uc2Uu cHkgYi9MYXlvdXRUZXN0cy9pbXBvcnRlZC93M2Mvd2ViLXBsYXRmb3JtLXRlc3RzL3Rvb2xzL3dw dHNlcnZlL3dwdHNlcnZlL3Jlc3BvbnNlLnB5CmluZGV4IDY4NmU3YzZlN2Y2MWVlZGQ3OTUzOWQ0 Yzk3MjZhMGFiMGNhZDkzOTcuLjkwZWM3MzQzYjdlMDUwZDY1YjRiODcxZTZkMWRiYmIwZjFmNWY0 MGYgMTAwNjQ0Ci0tLSBhL0xheW91dFRlc3RzL2ltcG9ydGVkL3czYy93ZWItcGxhdGZvcm0tdGVz dHMvdG9vbHMvd3B0c2VydmUvd3B0c2VydmUvcmVzcG9uc2UucHkKKysrIGIvTGF5b3V0VGVzdHMv aW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy90b29scy93cHRzZXJ2ZS93cHRzZXJ2ZS9y ZXNwb25zZS5weQpAQCAtMyw2ICszLDggQEAgZnJvbSBkYXRldGltZSBpbXBvcnQgZGF0ZXRpbWUs IHRpbWVkZWx0YQogZnJvbSBpbyBpbXBvcnQgQnl0ZXNJTwogaW1wb3J0IGpzb24KIGltcG9ydCBz b2NrZXQKK2ltcG9ydCBzdWJwcm9jZXNzCitpbXBvcnQgc3lzCiBpbXBvcnQgdXVpZAogCiBmcm9t IGhwYWNrLnN0cnVjdCBpbXBvcnQgSGVhZGVyVHVwbGUKQEAgLTI0OSw2ICsyNTEsMTEgQEAgY2xh c3MgUmVzcG9uc2Uob2JqZWN0KToKICAgICAgICAgc2VsZi5jb250ZW50ID0gZGF0YQogICAgICAg ICBpZiBtZXNzYWdlOgogICAgICAgICAgICAgc2VsZi5sb2dnZXIuZXJyb3IobWVzc2FnZSkKKyAg ICAgICAgICAgIGlmIHN5cy5wbGF0Zm9ybSAhPSAid2luMzIiOgorICAgICAgICAgICAgICAgIHNl bGYubG9nZ2VyLmRlYnVnKCJwcyBhdXg6IikKKyAgICAgICAgICAgICAgICBzZWxmLmxvZ2dlci5k ZWJ1ZyhzdWJwcm9jZXNzLmNoZWNrX291dHB1dChbInBzIiwgImF1eCJdKSkKKyAgICAgICAgICAg ICAgICBzZWxmLmxvZ2dlci5kZWJ1ZygibHNvZjoiKQorICAgICAgICAgICAgICAgIHNlbGYubG9n Z2VyLmRlYnVnKHN1YnByb2Nlc3MuY2hlY2tfb3V0cHV0KFsibHNvZiIsICItUCIsICIrYzAiXSkp CiAKIAogY2xhc3MgTXVsdGlwYXJ0Q29udGVudChvYmplY3QpOgo= 408838 2020-09-15 11:30:59 -0700 2020-09-15 14:27:07 -0700 Patch bug-216536-20200915203058.patch text/plain 1865 gsnedders U3VidmVyc2lvbiBSZXZpc2lvbjogMjY3MDgzCmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9pbXBv cnRlZC93M2MvQ2hhbmdlTG9nIGIvTGF5b3V0VGVzdHMvaW1wb3J0ZWQvdzNjL0NoYW5nZUxvZwpp bmRleCA5OTdhNmY2YmM4MDE5YTQ1NGYxMDdhMzIyYTgxZDM3ZDQ5NGUxYTg5Li44NDY5MGQ2Y2Qx ODczYzkxNjljZTY1N2JmMWNlZmU2MGUxODA1ZGI5IDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9p bXBvcnRlZC93M2MvQ2hhbmdlTG9nCisrKyBiL0xheW91dFRlc3RzL2ltcG9ydGVkL3czYy9DaGFu Z2VMb2cKQEAgLTEsMyArMSwxMyBAQAorMjAyMC0wOS0xNSAgU2FtIFNuZWRkb24gIDxnc25lZGRl cnNAYXBwbGUuY29tPgorCisgICAgICAgIEFkZCB0ZW1wb3JhcnkgZGVidWdnaW5nIHRvIHdwdHNl cnZlCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMTY1 MzYKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIHdl Yi1wbGF0Zm9ybS10ZXN0cy90b29scy93cHRzZXJ2ZS93cHRzZXJ2ZS9yZXNwb25zZS5weToKKyAg ICAgICAgKFJlc3BvbnNlLnNldF9lcnJvcik6CisKIDIwMjAtMDktMTUgIFJvYiBCdWlzICA8cmJ1 aXNAaWdhbGlhLmNvbT4KIAogICAgICAgICBDb252ZXJ0IHJ1bnRpbWUgZmxhZyB0byBzZXR0aW5n IGZvciBsYXp5IGltYWdlIGxvYWRpbmcKZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL2ltcG9ydGVk L3czYy93ZWItcGxhdGZvcm0tdGVzdHMvdG9vbHMvd3B0c2VydmUvd3B0c2VydmUvcmVzcG9uc2Uu cHkgYi9MYXlvdXRUZXN0cy9pbXBvcnRlZC93M2Mvd2ViLXBsYXRmb3JtLXRlc3RzL3Rvb2xzL3dw dHNlcnZlL3dwdHNlcnZlL3Jlc3BvbnNlLnB5CmluZGV4IDY4NmU3YzZlN2Y2MWVlZGQ3OTUzOWQ0 Yzk3MjZhMGFiMGNhZDkzOTcuLjUyZTMwYjEzMTc4YTcwY2I0OWNlZDI1NTNkMWUwZjE0OTYzZGU3 ZmEgMTAwNjQ0Ci0tLSBhL0xheW91dFRlc3RzL2ltcG9ydGVkL3czYy93ZWItcGxhdGZvcm0tdGVz dHMvdG9vbHMvd3B0c2VydmUvd3B0c2VydmUvcmVzcG9uc2UucHkKKysrIGIvTGF5b3V0VGVzdHMv aW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy90b29scy93cHRzZXJ2ZS93cHRzZXJ2ZS9y ZXNwb25zZS5weQpAQCAtMyw2ICszLDggQEAgZnJvbSBkYXRldGltZSBpbXBvcnQgZGF0ZXRpbWUs IHRpbWVkZWx0YQogZnJvbSBpbyBpbXBvcnQgQnl0ZXNJTwogaW1wb3J0IGpzb24KIGltcG9ydCBz b2NrZXQKK2ltcG9ydCBzdWJwcm9jZXNzCitpbXBvcnQgc3lzCiBpbXBvcnQgdXVpZAogCiBmcm9t IGhwYWNrLnN0cnVjdCBpbXBvcnQgSGVhZGVyVHVwbGUKQEAgLTI0OSw2ICsyNTEsMTIgQEAgY2xh c3MgUmVzcG9uc2Uob2JqZWN0KToKICAgICAgICAgc2VsZi5jb250ZW50ID0gZGF0YQogICAgICAg ICBpZiBtZXNzYWdlOgogICAgICAgICAgICAgc2VsZi5sb2dnZXIuZXJyb3IobWVzc2FnZSkKKyAg ICAgICAgICAgIGlmIHN5cy5wbGF0Zm9ybSBub3QgaW4gKCJ3aW4zMiIsICJjeWd3aW4iKToKKyAg ICAgICAgICAgICAgICAjIGN5Z3dpbiBkb2Vzbid0IGhhdmUgbHNvZgorICAgICAgICAgICAgICAg IHNlbGYubG9nZ2VyLmRlYnVnKCJwcyBhdXg6IikKKyAgICAgICAgICAgICAgICBzZWxmLmxvZ2dl ci5kZWJ1ZyhzdWJwcm9jZXNzLmNoZWNrX291dHB1dChbInBzIiwgImF1eCJdKSkKKyAgICAgICAg ICAgICAgICBzZWxmLmxvZ2dlci5kZWJ1ZygibHNvZjoiKQorICAgICAgICAgICAgICAgIHNlbGYu bG9nZ2VyLmRlYnVnKHN1YnByb2Nlc3MuY2hlY2tfb3V0cHV0KFsibHNvZiIsICItUCIsICIrYzAi XSkpCiAKIAogY2xhc3MgTXVsdGlwYXJ0Q29udGVudChvYmplY3QpOgo=