219189 2020-11-19 20:20:41 -0800 Switch EWS workers for JSC-ARMv7-32bits build and test queues to a new machine. 2020-12-02 11:15:28 -0800 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=219444 InRadar P2 Normal --- 1 clopez clopez aakash_jain angelos dpino jbedard pmatos ticaiolima webkit-bug-importer oldest_to_newest 1709365 0 clopez 2020-11-19 20:20:41 -0800 We currently run the EWS for ARMv7 JSCOnly on a x86_64 machine that cross-compiles for ARMv7 and then the tests run on a Raspberry Pi via the --remote-config-file option for run-javascriptcore-tests script This lately was not running very smoothly, as it was taking sometimes too much time to complete the testing. But now we have an ARM server that can do things natively and much faster than we have now. I bench-marked it and it can build JSCOnly in 5 minutes and runs the whole run-javascriptcore-tests also in 5 minutes. 1709367 1 414646 clopez 2020-11-19 20:31:44 -0800 Created attachment 414646 Patch 1709470 2 aakash_jain 2020-11-20 07:23:51 -0800 (In reply to Carlos Alberto Lopez Perez from comment #0) > I bench-marked it and it can build JSCOnly in 5 minutes and runs the whole run-javascriptcore-tests also in 5 minutes. For a clear comparison, can you also state similar timing for current setup. Also, is the plan to do this for other JSC queues as well? 1709479 3 414646 aakash_jain 2020-11-20 07:45:02 -0800 Comment on attachment 414646 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=414646&action=review > Tools/CISupport/ews-build/config.json:589 > + "workernames": ["igalia-jsc32-armv7-ews-02"] This is switching worker only for tester queue, not for builder queue. 1709484 4 clopez 2020-11-20 08:09:19 -0800 (In reply to Aakash Jain from comment #3) > Comment on attachment 414646 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=414646&action=review > > > Tools/CISupport/ews-build/config.json:589 > > + "workernames": ["igalia-jsc32-armv7-ews-02"] > > This is switching worker only for tester queue, not for builder queue. right.. the switch of the builder one is not reflected on the patch, but internally we are switching the machine assinged to igalia-jsc32-armv7-ews 1710205 5 webkit-bug-importer 2020-11-26 20:21:13 -0800 <rdar://problem/71757055> 1711230 6 clopez 2020-12-01 17:27:32 -0800 (In reply to Aakash Jain from comment #2) > (In reply to Carlos Alberto Lopez Perez from comment #0) > > I bench-marked it and it can build JSCOnly in 5 minutes and runs the whole run-javascriptcore-tests also in 5 minutes. > For a clear comparison, can you also state similar timing for current setup. > Current queue builds as fast as this (since it cross-builds on x86_64) but the test step takes 1 hour to run. > Also, is the plan to do this for other JSC queues as well? We are trying to get better hardware to improve also the timings on the MIPS queue, but I don't think it will be fast enough to build WebKit. So very likely we will continue to cross-build for that architecture and continue running tests on the MIPS queue with the --remote-config-file option 1711311 7 414646 aakash_jain 2020-12-02 06:02:42 -0800 Comment on attachment 414646 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=414646&action=review > Tools/CISupport/ews-build/steps.py:1617 > + self.command.extend(['--no-testmasm', '--no-testair', '--no-testb3', '--no-testdfg', '--no-testapi']) what was the reason that igalia jsc-armv7 queue needed these parameters (e.g.: --no-testmasm) earlier, and what's the reason it doesn't need it now? > Tools/CISupport/ews-build/steps.py:1622 > + self.command.extend(['--memory-limited', '--verbose']) why verbose? 1711315 8 clopez 2020-12-02 06:15:24 -0800 (In reply to Aakash Jain from comment #7) > Comment on attachment 414646 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=414646&action=review > > > Tools/CISupport/ews-build/steps.py:1617 > > + self.command.extend(['--no-testmasm', '--no-testair', '--no-testb3', '--no-testdfg', '--no-testapi']) > > what was the reason that igalia jsc-armv7 queue needed these parameters > (e.g.: --no-testmasm) earlier, and what's the reason it doesn't need it now? > Previously the tests were running on a remote board (a Raspberry Pi) and the built-product was copied to the board at the moment of starting the tests. The function responsible of copying this built-product (prepareBundle() at Tools/Scripts/run-jsc-stress-tests) was only copying the jsc binary but not the testmasm, testair, testb3, testdfb, testapi binaries. Now the tests run natively on the same host that hosts the buildbot worker, so no copying of the built-product to a board is involved. > > Tools/CISupport/ews-build/steps.py:1622 > > + self.command.extend(['--memory-limited', '--verbose']) > > why verbose? It gives information about which exact test is running at the time. IMHO the stdout log gives more information that way. It is also how we run the JSC tests on the JSCOnly post-commit bot at https://build.webkit.org/waterfall?category=misc 1711316 9 aakash_jain 2020-12-02 06:28:59 -0800 (In reply to Carlos Alberto Lopez Perez from comment #8) > Now the tests run natively on the same host that hosts the buildbot worker, so no copying of the built-product to a board is involved. The build and test are still on different bots. bot on builder queue builds, upload the archive, and bot on tester queue downloads it. 1711321 10 clopez 2020-12-02 07:07:30 -0800 (In reply to Aakash Jain from comment #9) > (In reply to Carlos Alberto Lopez Perez from comment #8) > > Now the tests run natively on the same host that hosts the buildbot worker, so no copying of the built-product to a board is involved. > The build and test are still on different bots. bot on builder queue builds, > upload the archive, and bot on tester queue downloads it. Right. I think maybe I was not clear enough. I will rephrase what I was trying to mean. - Now both machines are armv7 machines: The build-queue builds natively and the test-queue simply executes the binaries it downloads from the build-queue. - Previously the build-queue was a x84_64 machine cross-building for armv7 and the test-queue was another x86_64 machine that ran the test step remotely on a RPi board via the "--remote-config-file" option for the script run-javascriptcore-tests. 1711323 11 aakash_jain 2020-12-02 07:10:00 -0800 sounds good 1711347 12 ews-feeder 2020-12-02 08:39:27 -0800 Committed r270348: <https://trac.webkit.org/changeset/270348> All reviewed patches have been landed. Closing bug and clearing flags on attachment 414646. 414646 2020-11-19 20:31:44 -0800 2020-12-02 08:39:27 -0800 Patch bug-219189-20201120053143.patch text/plain 6585 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjcwMDY2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNDRjMjRmNDU4OThjMzUyNWI3Zjk4ZDNmN2M1MGE5ZmVh ZjY3NzA4Ni4uMzE5NjBjNmUzMDNmMTMyNmU2NmY5YzBiYTY1ZGRkNzk2NTMyMTE1MiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDMw IEBACisyMDIwLTExLTE5ICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIFN3aXRjaCBFV1Mgd29ya2VycyBmb3IgSlNDLUFSTXY3LTMyYml0 cyBidWlsZCBhbmQgdGVzdCBxdWV1ZXMgdG8gYSBuZXcgbWFjaGluZS4KKyAgICAgICAgaHR0cHM6 Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTIxOTE4OQorCisgICAgICAgIFJldmll d2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFRoaXMgc3dpdGNoZXMgdGhlIHdvcmtl cnMgZm9yIHRoZSBKU0MtQVJNdjctMzJiaXRzLUJ1aWxkLUVXUyBhbmQKKyAgICAgICAgSlNDLUFS TXY3LTMyYml0cy1UZXN0LUVXUyBxdWV1ZXMgdG8gYSBuZXcgQVJNIHNlcnZlciB0aGF0IHdpbGwg cnVuCisgICAgICAgIHRoaW5ncyBmYXN0ZXIgdGhhbiB0aGUgcHJldmlvdXMgY29tYmluYXRpb24g b2YgY3Jvc3MtYnVpbGRlciArIFJQaXMKKworICAgICAgICBTaW5jZSBub3cgdGhlIHRldHMgcnVu IG5hdGl2ZWx5IHRoZXJlIGlzIG5vIG5lZWQgZm9yIHBhc3NpbmcgdGhlCisgICAgICAgICItLXJl bW90ZS1jb25maWctZmlsZSIgc3dpdGNoIHRvIHRoZSBydW4tamF2YXNjcmlwdGNvcmUtdGVzdHMg c2NyaXB0LgorCisgICAgICAgIEhvd2V2ZXIgIi0tbWVtb3J5LWxpbWl0ZWQiIGlzIHN0aWxsIG5l ZWRlZCBiZWNhdXNlIHRoZSBpc3N1ZXMgd2l0aAorICAgICAgICBmYWlsdXJlcyBvbiBtZW1vcnkg aW50ZW5zaXZlIHRlc3RzIGhhcHBlbiBvbiBhbGwgTGludXggbWFjaGluZXMuCisgICAgICAgIFNv IHRoZSBjaGVjayB0byBwYXNzIHRoaXMgZmxhZyBpcyBnZW5lcmFsaXplZCBsaWtlIHdlIGRvIGZv ciBhbGwKKyAgICAgICAgTGludXggcG9ydHMgb24gdGhlIGJ1aWxkLndlYmtpdC5vcmcgYnVpbGRi b3QgY29uZmlnLgorCisgICAgICAgICogQ0lTdXBwb3J0L2V3cy1idWlsZC9jb25maWcuanNvbjoK KyAgICAgICAgKiBDSVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzLnB5OgorICAgICAgICAoUnVuSmF2 YVNjcmlwdENvcmVUZXN0cy5zdGFydCk6CisgICAgICAgICogQ0lTdXBwb3J0L2V3cy1idWlsZC9z dGVwc191bml0dGVzdC5weToKKyAgICAgICAgKFRlc3RSdW5KYXZhU2NyaXB0Q29yZVRlc3RzLnRl c3RfcmVtb3RlX3N1Y2Nlc3MpOgorICAgICAgICAoVGVzdFJ1bkphdmFTY3JpcHRDb3JlVGVzdHMu dGVzdF9kZmdfYWlyX2FuZF9zdHJlc3NfdGVzdF9mYWlsdXJlKToKKyAgICAgICAgKFRlc3RSZVJ1 bkphdmFTY3JpcHRDb3JlVGVzdHMudGVzdF9yZW1vdGVfc3VjY2Vzcyk6CisKIDIwMjAtMTEtMTkg IEphbWVzIERhcnBpbmlhbiAgPGpkYXJwaW5pYW5AY2hyb21pdW0ub3JnPgogCiAgICAgICAgIElt cHJvdmUgdXBkYXRlLWFuZ2xlIHNjcmlwdCBhbmQgbW92ZSB0byBUb29scy9TY3JpcHRzCmRpZmYg LS1naXQgYS9Ub29scy9DSVN1cHBvcnQvZXdzLWJ1aWxkL2NvbmZpZy5qc29uIGIvVG9vbHMvQ0lT dXBwb3J0L2V3cy1idWlsZC9jb25maWcuanNvbgppbmRleCA4YjE3NWNhODg3M2IzYmQ0MzhhNTY5 Y2IyZTliNGFkMWVlYjlhZGI4Li4zMDJhMzRkMmYwMDdmNGM4NGM1YTQ3YThlM2JhNjM3OTMwMWY2 OGY1IDEwMDY0NAotLS0gYS9Ub29scy9DSVN1cHBvcnQvZXdzLWJ1aWxkL2NvbmZpZy5qc29uCisr KyBiL1Rvb2xzL0NJU3VwcG9ydC9ld3MtYnVpbGQvY29uZmlnLmpzb24KQEAgLTU4Niw4ICs1ODYs NyBAQAogICAgICAgInBsYXRmb3JtIjogImpzYy1vbmx5IiwKICAgICAgICJjb25maWd1cmF0aW9u IjogInJlbGVhc2UiLAogICAgICAgImFyY2hpdGVjdHVyZXMiOiBbImFybXY3Il0sCi0gICAgICAi d29ya2VybmFtZXMiOiBbImlnYWxpYS1qc2MzMi1hcm12Ny1ld3MiXSwKLSAgICAgICJyZW1vdGVz IjogIi4uLy4uL0VXUy10ZXN0LWRldmljZXMuanNvbiIKKyAgICAgICJ3b3JrZXJuYW1lcyI6IFsi aWdhbGlhLWpzYzMyLWFybXY3LWV3cy0wMiJdCiAgICAgfSwKICAgICB7CiAgICAgICAibmFtZSI6 ICJKU0MtaTM4Ni0zMmJpdHMtRVdTIiwKZGlmZiAtLWdpdCBhL1Rvb2xzL0NJU3VwcG9ydC9ld3Mt YnVpbGQvc3RlcHMucHkgYi9Ub29scy9DSVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzLnB5CmluZGV4 IDdiYTIxMmQ1NWNmOTMzYjM4NmE2NmU4MTNjNzM0OTFlMWI3Y2VlZWMuLmM4ZmRlMTM2NWZjZWJh MTdjYTliOTNjMTE5MWI1NDJmYWJhZjYwY2YgMTAwNjQ0Ci0tLSBhL1Rvb2xzL0NJU3VwcG9ydC9l d3MtYnVpbGQvc3RlcHMucHkKKysrIGIvVG9vbHMvQ0lTdXBwb3J0L2V3cy1idWlsZC9zdGVwcy5w eQpAQCAtMTYxMyw3ICsxNjEzLDE0IEBAIGNsYXNzIFJ1bkphdmFTY3JpcHRDb3JlVGVzdHMoc2hl bGwuVGVzdCk6CiAKICAgICAgICAgcGxhdGZvcm0gPSBzZWxmLmdldFByb3BlcnR5KCdwbGF0Zm9y bScpCiAgICAgICAgIGlmIHBsYXRmb3JtID09ICdqc2Mtb25seScgYW5kIHJlbW90ZXNmaWxlOgot ICAgICAgICAgICAgc2VsZi5jb21tYW5kLmV4dGVuZChbJy0tbm8tdGVzdG1hc20nLCAnLS1uby10 ZXN0YWlyJywgJy0tbm8tdGVzdGIzJywgJy0tbm8tdGVzdGRmZycsICctLW5vLXRlc3RhcGknLCAn LS1tZW1vcnktbGltaXRlZCddKQorICAgICAgICAgICAgIyBGSVhNRTogdGhlIGJ1bmRsZSBjb3Bp ZWQgdG8gdGhlIHJlbW90ZSBzaG91bGQgaW5jbHVkZSB0aGUgdGVzdGFpciwgdGVzdGIzLCB0ZXN0 YXBpLCBldGMuLiBiaW5hcmllcworICAgICAgICAgICAgc2VsZi5jb21tYW5kLmV4dGVuZChbJy0t bm8tdGVzdG1hc20nLCAnLS1uby10ZXN0YWlyJywgJy0tbm8tdGVzdGIzJywgJy0tbm8tdGVzdGRm ZycsICctLW5vLXRlc3RhcGknXSkKKworICAgICAgICAjIExpbnV4IGJvdHMgaGF2ZSBjdXJyZW50 bHkgcHJvYmxlbXMgd2l0aCBKU0MgdGVzdHMgdGhhdCB0cnkgdG8gdXNlIGxhcmdlIGFtb3VudHMg b2YgbWVtb3J5LgorICAgICAgICAjIENoZWNrOiBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93 X2J1Zy5jZ2k/aWQ9MTc1MTQwCisgICAgICAgIGlmIHBsYXRmb3JtIGluICgnZ3RrJywgJ3dwZScs ICdqc2Mtb25seScpOgorICAgICAgICAgICAgc2VsZi5jb21tYW5kLmV4dGVuZChbJy0tbWVtb3J5 LWxpbWl0ZWQnLCAnLS12ZXJib3NlJ10pCisKICAgICAgICAgYXBwZW5kQ3VzdG9tQnVpbGRGbGFn cyhzZWxmLCBzZWxmLmdldFByb3BlcnR5KCdwbGF0Zm9ybScpLCBzZWxmLmdldFByb3BlcnR5KCdm dWxsUGxhdGZvcm0nKSkKICAgICAgICAgcmV0dXJuIHNoZWxsLlRlc3Quc3RhcnQoc2VsZikKIApk aWZmIC0tZ2l0IGEvVG9vbHMvQ0lTdXBwb3J0L2V3cy1idWlsZC9zdGVwc191bml0dGVzdC5weSBi L1Rvb2xzL0NJU3VwcG9ydC9ld3MtYnVpbGQvc3RlcHNfdW5pdHRlc3QucHkKaW5kZXggMzM1YTI4 ZWNhOTgzYmFlMzgzYmVhNDJiOGM1YTIyNzMzMWEyNDhiOS4uZDRjYjk1ODA3MTk1OTk5Mjk4MDlm MmY2MTNmMGY3NDc0NWNlYzBhNSAxMDA2NDQKLS0tIGEvVG9vbHMvQ0lTdXBwb3J0L2V3cy1idWls ZC9zdGVwc191bml0dGVzdC5weQorKysgYi9Ub29scy9DSVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBz X3VuaXR0ZXN0LnB5CkBAIC0xMjc5LDcgKzEyNzksNyBAQCBjbGFzcyBUZXN0UnVuSmF2YVNjcmlw dENvcmVUZXN0cyhCdWlsZFN0ZXBNaXhpbkFkZGl0aW9ucywgdW5pdHRlc3QuVGVzdENhc2UpOgog ICAgICAgICBzZWxmLmV4cGVjdFJlbW90ZUNvbW1hbmRzKAogICAgICAgICAgICAgRXhwZWN0U2hl bGwod29ya2Rpcj0nd2tkaXInLAogICAgICAgICAgICAgICAgICAgICAgICAgbG9nRW52aXJvbj1G YWxzZSwKLSAgICAgICAgICAgICAgICAgICAgICAgIGNvbW1hbmQ9WydwZXJsJywgJ1Rvb2xzL1Nj cmlwdHMvcnVuLWphdmFzY3JpcHRjb3JlLXRlc3RzJywgJy0tbm8tYnVpbGQnLCAnLS1uby1mYWls LWZhc3QnLCAnLS1qc29uLW91dHB1dD17MH0nLmZvcm1hdChzZWxmLmpzb25GaWxlTmFtZSksICct LXJlbGVhc2UnLCAnLS1yZW1vdGUtY29uZmlnLWZpbGU9cmVtb3RlLW1hY2hpbmVzLmpzb24nLCAn LS1uby10ZXN0bWFzbScsICctLW5vLXRlc3RhaXInLCAnLS1uby10ZXN0YjMnLCAnLS1uby10ZXN0 ZGZnJywgJy0tbm8tdGVzdGFwaScsICctLW1lbW9yeS1saW1pdGVkJywgJy0tanNjLW9ubHknXSwK KyAgICAgICAgICAgICAgICAgICAgICAgIGNvbW1hbmQ9WydwZXJsJywgJ1Rvb2xzL1NjcmlwdHMv cnVuLWphdmFzY3JpcHRjb3JlLXRlc3RzJywgJy0tbm8tYnVpbGQnLCAnLS1uby1mYWlsLWZhc3Qn LCAnLS1qc29uLW91dHB1dD17MH0nLmZvcm1hdChzZWxmLmpzb25GaWxlTmFtZSksICctLXJlbGVh c2UnLCAnLS1yZW1vdGUtY29uZmlnLWZpbGU9cmVtb3RlLW1hY2hpbmVzLmpzb24nLCAnLS1uby10 ZXN0bWFzbScsICctLW5vLXRlc3RhaXInLCAnLS1uby10ZXN0YjMnLCAnLS1uby10ZXN0ZGZnJywg Jy0tbm8tdGVzdGFwaScsICctLW1lbW9yeS1saW1pdGVkJywgJy0tdmVyYm9zZScsICctLWpzYy1v bmx5J10sCiAgICAgICAgICAgICAgICAgICAgICAgICBsb2dmaWxlcz17J2pzb24nOiBzZWxmLmpz b25GaWxlTmFtZX0sCiAgICAgICAgICAgICAgICAgICAgICAgICApCiAgICAgICAgICAgICArIDAs CkBAIC0xMzc1LDcgKzEzNzUsNyBAQCBjbGFzcyBUZXN0UnVuSmF2YVNjcmlwdENvcmVUZXN0cyhC dWlsZFN0ZXBNaXhpbkFkZGl0aW9ucywgdW5pdHRlc3QuVGVzdENhc2UpOgogICAgICAgICAgICAg RXhwZWN0U2hlbGwod29ya2Rpcj0nd2tkaXInLAogICAgICAgICAgICAgICAgICAgICAgICAgbG9n RW52aXJvbj1GYWxzZSwKICAgICAgICAgICAgICAgICAgICAgICAgIGxvZ2ZpbGVzPXsnanNvbic6 IHNlbGYuanNvbkZpbGVOYW1lfSwKLSAgICAgICAgICAgICAgICAgICAgICAgIGNvbW1hbmQ9Wydw ZXJsJywgJ1Rvb2xzL1NjcmlwdHMvcnVuLWphdmFzY3JpcHRjb3JlLXRlc3RzJywgJy0tbm8tYnVp bGQnLCAnLS1uby1mYWlsLWZhc3QnLCAnLS1qc29uLW91dHB1dD17MH0nLmZvcm1hdChzZWxmLmpz b25GaWxlTmFtZSksICctLXJlbGVhc2UnLCAnLS1qc2Mtb25seSddLAorICAgICAgICAgICAgICAg ICAgICAgICAgY29tbWFuZD1bJ3BlcmwnLCAnVG9vbHMvU2NyaXB0cy9ydW4tamF2YXNjcmlwdGNv cmUtdGVzdHMnLCAnLS1uby1idWlsZCcsICctLW5vLWZhaWwtZmFzdCcsICctLWpzb24tb3V0cHV0 PXswfScuZm9ybWF0KHNlbGYuanNvbkZpbGVOYW1lKSwgJy0tcmVsZWFzZScsICctLW1lbW9yeS1s aW1pdGVkJywgJy0tdmVyYm9zZScsICctLWpzYy1vbmx5J10sCiAgICAgICAgICAgICAgICAgICAg ICAgICApCiAgICAgICAgICAgICArIDIKICAgICAgICAgICAgICsgRXhwZWN0U2hlbGwubG9nKCdq c29uJywgc3Rkb3V0PXNlbGYuanNjX2RmZ19haXJfYW5kX3N0cmVzc190ZXN0X2ZhaWx1cmUpLApA QCAtMTQxOSw3ICsxNDE5LDcgQEAgY2xhc3MgVGVzdFJlUnVuSmF2YVNjcmlwdENvcmVUZXN0cyhU ZXN0UnVuSmF2YVNjcmlwdENvcmVUZXN0cyk6CiAgICAgICAgIHNlbGYuZXhwZWN0UmVtb3RlQ29t bWFuZHMoCiAgICAgICAgICAgICBFeHBlY3RTaGVsbCh3b3JrZGlyPSd3a2RpcicsCiAgICAgICAg ICAgICAgICAgICAgICAgICBsb2dFbnZpcm9uPUZhbHNlLAotICAgICAgICAgICAgICAgICAgICAg ICAgY29tbWFuZD1bJ3BlcmwnLCAnVG9vbHMvU2NyaXB0cy9ydW4tamF2YXNjcmlwdGNvcmUtdGVz dHMnLCAnLS1uby1idWlsZCcsICctLW5vLWZhaWwtZmFzdCcsICctLWpzb24tb3V0cHV0PXswfScu Zm9ybWF0KHNlbGYuanNvbkZpbGVOYW1lKSwgJy0tcmVsZWFzZScsICctLXJlbW90ZS1jb25maWct ZmlsZT1yZW1vdGUtbWFjaGluZXMuanNvbicsICctLW5vLXRlc3RtYXNtJywgJy0tbm8tdGVzdGFp cicsICctLW5vLXRlc3RiMycsICctLW5vLXRlc3RkZmcnLCAnLS1uby10ZXN0YXBpJywgJy0tbWVt b3J5LWxpbWl0ZWQnLCAnLS1qc2Mtb25seSddLAorICAgICAgICAgICAgICAgICAgICAgICAgY29t bWFuZD1bJ3BlcmwnLCAnVG9vbHMvU2NyaXB0cy9ydW4tamF2YXNjcmlwdGNvcmUtdGVzdHMnLCAn LS1uby1idWlsZCcsICctLW5vLWZhaWwtZmFzdCcsICctLWpzb24tb3V0cHV0PXswfScuZm9ybWF0 KHNlbGYuanNvbkZpbGVOYW1lKSwgJy0tcmVsZWFzZScsICctLXJlbW90ZS1jb25maWctZmlsZT1y ZW1vdGUtbWFjaGluZXMuanNvbicsICctLW5vLXRlc3RtYXNtJywgJy0tbm8tdGVzdGFpcicsICct LW5vLXRlc3RiMycsICctLW5vLXRlc3RkZmcnLCAnLS1uby10ZXN0YXBpJywgJy0tbWVtb3J5LWxp bWl0ZWQnLCAnLS12ZXJib3NlJywgJy0tanNjLW9ubHknXSwKICAgICAgICAgICAgICAgICAgICAg ICAgIGxvZ2ZpbGVzPXsnanNvbic6IHNlbGYuanNvbkZpbGVOYW1lfSwKICAgICAgICAgICAgICAg ICAgICAgICAgICkgKwogICAgICAgICAgICAgMCwK