186115 2018-05-30 16:18:57 -0700 test262-runner should run in verbose mode in automation 2018-05-31 03:31:44 -0700 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 ryanhaddad webkit-unassigned aakash_jain commit-queue leo lforschler realdawei webkit-bug-importer ysuzuki oldest_to_newest 1428646 0 ryanhaddad 2018-05-30 16:18:57 -0700 test262-runner should run in verbose mode in automation so that: 1. It provides output so that we can get a better idea what is going wrong when something like this happens: https://build.webkit.org/builders/Apple%20Sierra%20Debug%20Test262%20%28Tests%29/builds/2594/steps/test262-test/logs/stdio 2. It doesn't hit the '1200 seconds without output' timeout if it is legitimately running tests, but taking longer than 1200 seconds. 1428669 1 leo 2018-05-30 16:40:27 -0700 Thanks for this feedback, Ryan. I've been wondering if we should have verbose mode in the build bot for a while, now you confirmed my thoughts. I have one extra question: should we report all the test files in the verbose mode? Right now it's limited to report all the failures (including the known failures). I believe if we print a line - or even a dot - for each passing test will support the '1200 seconds without output' better 1428683 2 ryanhaddad 2018-05-30 16:58:24 -0700 (In reply to Leo Balter from comment #1) > Thanks for this feedback, Ryan. I've been wondering if we should have > verbose mode in the build bot for a while, now you confirmed my thoughts. > > I have one extra question: should we report all the test files in the > verbose mode? Right now it's limited to report all the failures (including > the known failures). I think it is fine to print everything in verbose mode, including known failures. We do this with 'run-webkit-tests' in automation (i.e. https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20(Tests)/builds/9702/steps/layout-test/logs/stdio). > I believe if we print a line - or even a dot - for each passing test will > support the '1200 seconds without output' better Yes, any output should prevent the step from hitting the buildbot timeout. 1428711 3 341633 leo 2018-05-30 18:39:12 -0700 Created attachment 341633 Patch 1428712 4 leo 2018-05-30 18:42:19 -0700 Ryan, I uploaded a patch with the verbose mode toggled for the build bot + dots for known passing tests. It still prints a full "NEW PASS $filename" for tests the were previously failing. I decided to try the dots as I believe this still makes some readable output. I'm also not printing anything the skipped tests, I don't think this will be a problem, but it's easy to also print anything for these cases. 1428717 5 341636 leo 2018-05-30 18:50:22 -0700 Created attachment 341636 Patch 1428718 6 leo 2018-05-30 18:52:20 -0700 I gave a better look to the wpt report and I believe I was being unnecessarily conservative on printing the passing and skipped files. The new update just prints everything, it's nice to tell what is being surfaced by the runner. 1428794 7 341636 ysuzuki 2018-05-31 02:54:28 -0700 Comment on attachment 341636 Patch Let's land it and watch on the bot results! 1428804 8 341636 commit-queue 2018-05-31 03:21:00 -0700 Comment on attachment 341636 Patch Clearing flags on attachment: 341636 Committed r232343: <https://trac.webkit.org/changeset/232343> 1428805 9 commit-queue 2018-05-31 03:21:02 -0700 All reviewed patches have been landed. Closing bug. 1428807 10 webkit-bug-importer 2018-05-31 03:31:44 -0700 <rdar://problem/40680506> 341633 2018-05-30 18:39:12 -0700 2018-05-30 18:50:19 -0700 Patch bug-186115-20180530213911.patch text/plain 3036 leo U3VidmVyc2lvbiBSZXZpc2lvbjogMjMyMzIwCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNGQwYjhkOTRmZjQ2ZTk3OGJlYzE0NjBiYmQ4ZTMyYjBj OWYzNGZhYy4uNjAwZjY5ZjMwZWMwZTQwZmQ2MDk0MTlmMTc3ZDE5YWUxNWFlMTIwMyAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE3 IEBACisyMDE4LTA1LTMwICBMZW8gQmFsdGVyICA8bGVvbmFyZG8uYmFsdGVyQGdtYWlsLmNvbT4K KworICAgICAgICB0ZXN0MjYyLXJ1bm5lciBzaG91bGQgcnVuIGluIHZlcmJvc2UgbW9kZSBpbiBh dXRvbWF0aW9uCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9p ZD0xODYxMTUKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAg ICBUaGlzIHBhdGNoIHRvZ2dsZXMgdGhlIHZlcmJvc2UgbW9kZSBpbiB0aGUgYnVpbGQgYm90IGFu ZCByZXBvcnRzIGEgc2ltcGxlIGRvdCBmb3IgZWFjaCBwYXNzaW5nIHRlc3QKKyAgICAgICAgaW4g b3JkZXIgdG8gYXZvaWQgdGhlICcxMjAwIHNlY29uZHMgd2l0aG91dCBvdXRwdXQnIGlzc3VlLgor ICAgICAgICAqIEJ1aWxkU2xhdmVTdXBwb3J0L2J1aWxkLndlYmtpdC5vcmctY29uZmlnL3N0ZXBz LnB5OgorICAgICAgICAoUnVuVGVzdDI2MlRlc3RzKToKKyAgICAgICAgKiBTY3JpcHRzL3Rlc3Qy NjIvUnVubmVyLnBtOgorICAgICAgICAocHJvY2Vzc1Jlc3VsdCk6CisKIDIwMTgtMDUtMzAgIFlv dWVubiBGYWJsZXQgIDx5b3Vlbm5AYXBwbGUuY29tPgogCiAgICAgICAgIFJlbmFtZSBDcm9zc09y aWdpblJlc291cmNlUG9saWN5IHNhbWUgdG8gc2FtZS1vcmlnaW4KZGlmZiAtLWdpdCBhL1Rvb2xz L0J1aWxkU2xhdmVTdXBwb3J0L2J1aWxkLndlYmtpdC5vcmctY29uZmlnL3N0ZXBzLnB5IGIvVG9v bHMvQnVpbGRTbGF2ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvc3RlcHMucHkKaW5k ZXggYTc4NDczMzcwZGU2M2I2YjNkYWU1YmEwZjg1OWM1M2VlYjExNDRkOC4uMDYzMWU4NmUzYzU1 ZDg0YzA1MDVhZjY5ODU3NTZlOTBhNTNkNWZmMSAxMDA2NDQKLS0tIGEvVG9vbHMvQnVpbGRTbGF2 ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvc3RlcHMucHkKKysrIGIvVG9vbHMvQnVp bGRTbGF2ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvc3RlcHMucHkKQEAgLTM2Nyw3 ICszNjcsNyBAQCBjbGFzcyBSdW5UZXN0MjYyVGVzdHMoVGVzdFdpdGhGYWlsdXJlQ291bnQpOgog ICAgIGRlc2NyaXB0aW9uID0gWyJ0ZXN0MjYyLXRlc3RzIHJ1bm5pbmciXQogICAgIGRlc2NyaXB0 aW9uRG9uZSA9IFsidGVzdDI2Mi10ZXN0cyJdCiAgICAgZmFpbGVkVGVzdHNGb3JtYXRTdHJpbmcg PSAiJWQgVGVzdDI2MiB0ZXN0JXMgZmFpbGVkIgotICAgIGNvbW1hbmQgPSBbInBlcmwiLCAiLi9U b29scy9TY3JpcHRzL3Rlc3QyNjItcnVubmVyIiwgV2l0aFByb3BlcnRpZXMoIi0tJShjb25maWd1 cmF0aW9uKXMiKV0KKyAgICBjb21tYW5kID0gWyJwZXJsIiwgIi4vVG9vbHMvU2NyaXB0cy90ZXN0 MjYyLXJ1bm5lciIsICItLXZlcmJvc2UiLCBXaXRoUHJvcGVydGllcygiLS0lKGNvbmZpZ3VyYXRp b24pcyIpXQogCiAgICAgZGVmIHN0YXJ0KHNlbGYpOgogICAgICAgICBhcHBlbmRDdXN0b21CdWls ZEZsYWdzKHNlbGYsIHNlbGYuZ2V0UHJvcGVydHkoJ3BsYXRmb3JtJyksIHNlbGYuZ2V0UHJvcGVy dHkoJ2Z1bGxQbGF0Zm9ybScpKQpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy90ZXN0MjYyL1J1 bm5lci5wbSBiL1Rvb2xzL1NjcmlwdHMvdGVzdDI2Mi9SdW5uZXIucG0KaW5kZXggZWMzMGQwMWZk NGNkNTgzNjVkMmJhMGNhYzJlMzE1NmQyOTU5YjYxOS4uYWFlYmU2NzkwYjNlOTBmZWFmODc5NTcy NjVlNTRkNTYxZGNmYzcxNyAxMDA3NTUKLS0tIGEvVG9vbHMvU2NyaXB0cy90ZXN0MjYyL1J1bm5l ci5wbQorKysgYi9Ub29scy9TY3JpcHRzL3Rlc3QyNjIvUnVubmVyLnBtCkBAIC02NzgsMjIgKzY3 OCwyNSBAQCBzdWIgcHJvY2Vzc1Jlc3VsdCB7CiAgICAgICAgIG15ICRmYWlsTXNnID0gJyc7CiAg ICAgICAgICRmYWlsTXNnID0gIkZBSUwgJGZpbGUgKCRzY2VuYXJpbylcbiIgaWYgKCRwcmludEZh aWx1cmUgb3IgJHZlcmJvc2UpOwogCisgICAgICAgIG15ICRwcmVmaXhNc2cgPSAnJzsKICAgICAg ICAgbXkgJHN1ZmZpeE1zZyA9ICcnOwogCiAgICAgICAgIGlmICgkdmVyYm9zZSkgeworICAgICAg ICAgICAgJHByZWZpeE1zZyA9ICJcbiI7CiAgICAgICAgICAgICBteSAkZmVhdHVyZXNMaXN0ID0g Jyc7CiAgICAgICAgICAgICAkZmVhdHVyZXNMaXN0ID0gIlxuRmVhdHVyZXM6ICIgLiBqb2luKCcs ICcsIEB7ICRkYXRhLT57ZmVhdHVyZXN9IH0pIGlmICRkYXRhLT57ZmVhdHVyZXN9OwogICAgICAg ICAgICAgJHN1ZmZpeE1zZyA9ICIkcmVzdWx0JGZlYXR1cmVzTGlzdFxuXG4iOwogICAgICAgICB9 CiAKLSAgICAgICAgcHJpbnQgIiRuZXdGYWlsJGZhaWxNc2ckc3VmZml4TXNnIjsKKyAgICAgICAg cHJpbnQgIiRwcmVmaXhNc2ckbmV3RmFpbCRmYWlsTXNnJHN1ZmZpeE1zZyI7CiAKICAgICAgICAg JHJlc3VsdGRhdGF7cmVzdWx0fSA9ICdGQUlMJzsKICAgICAgICAgJHJlc3VsdGRhdGF7ZXJyb3J9 ID0gJGN1cnJlbnRmYWlsdXJlOwogICAgIH0gZWxzaWYgKCRzY2VuYXJpbyBuZSAnc2tpcCcgJiYg ISRjdXJyZW50ZmFpbHVyZSkgewogICAgICAgICBpZiAoJGV4cGVjdGVkZmFpbHVyZSkgewotICAg ICAgICAgICAgcHJpbnQgIk5FVyBQQVNTICRmaWxlICgkc2NlbmFyaW8pXG4iOwotICAgICAgICAg ICAgcHJpbnQgIlxuIiBpZiAkdmVyYm9zZTsKKyAgICAgICAgICAgIHByaW50ICJcbk5FVyBQQVNT ICRmaWxlICgkc2NlbmFyaW8pXG4iOworICAgICAgICB9IGVsc2lmICgkdmVyYm9zZSkgeworICAg ICAgICAgICAgcHJpbnQgJy4nOwogICAgICAgICB9CiAKICAgICAgICAgJHJlc3VsdGRhdGF7cmVz dWx0fSA9ICdQQVNTJzsK 341636 2018-05-30 18:50:22 -0700 2018-05-31 03:21:00 -0700 Patch bug-186115-20180530215021.patch text/plain 2869 leo U3VidmVyc2lvbiBSZXZpc2lvbjogMjMyMzIwCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNGQwYjhkOTRmZjQ2ZTk3OGJlYzE0NjBiYmQ4ZTMyYjBj OWYzNGZhYy4uZTc5YzlmNzNhZGNlMDZmY2I4N2E1NTU2ZGU3OTRjZjRkOGE1ZjJmOCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE3 IEBACisyMDE4LTA1LTMwICBMZW8gQmFsdGVyICA8bGVvbmFyZG8uYmFsdGVyQGdtYWlsLmNvbT4K KworICAgICAgICB0ZXN0MjYyLXJ1bm5lciBzaG91bGQgcnVuIGluIHZlcmJvc2UgbW9kZSBpbiBh dXRvbWF0aW9uCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9p ZD0xODYxMTUKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAg ICBUaGlzIHBhdGNoIHRvZ2dsZXMgdGhlIHZlcmJvc2UgbW9kZSBpbiB0aGUgYnVpbGQgYm90IGFu ZCByZXBvcnRzIGV2ZXJ5IHRlc3QgZmlsZQorICAgICAgICBpbiBvcmRlciB0byBhdm9pZCB0aGUg JzEyMDAgc2Vjb25kcyB3aXRob3V0IG91dHB1dCcgaXNzdWUuCisgICAgICAgICogQnVpbGRTbGF2 ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvc3RlcHMucHk6CisgICAgICAgIChSdW5U ZXN0MjYyVGVzdHMpOgorICAgICAgICAqIFNjcmlwdHMvdGVzdDI2Mi9SdW5uZXIucG06CisgICAg ICAgIChwcm9jZXNzUmVzdWx0KToKKwogMjAxOC0wNS0zMCAgWW91ZW5uIEZhYmxldCAgPHlvdWVu bkBhcHBsZS5jb20+CiAKICAgICAgICAgUmVuYW1lIENyb3NzT3JpZ2luUmVzb3VyY2VQb2xpY3kg c2FtZSB0byBzYW1lLW9yaWdpbgpkaWZmIC0tZ2l0IGEvVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQv YnVpbGQud2Via2l0Lm9yZy1jb25maWcvc3RlcHMucHkgYi9Ub29scy9CdWlsZFNsYXZlU3VwcG9y dC9idWlsZC53ZWJraXQub3JnLWNvbmZpZy9zdGVwcy5weQppbmRleCBhNzg0NzMzNzBkZTYzYjZi M2RhZTViYTBmODU5YzUzZWViMTE0NGQ4Li4wNjMxZTg2ZTNjNTVkODRjMDUwNWFmNjk4NTc1NmU5 MGE1M2Q1ZmYxIDEwMDY0NAotLS0gYS9Ub29scy9CdWlsZFNsYXZlU3VwcG9ydC9idWlsZC53ZWJr aXQub3JnLWNvbmZpZy9zdGVwcy5weQorKysgYi9Ub29scy9CdWlsZFNsYXZlU3VwcG9ydC9idWls ZC53ZWJraXQub3JnLWNvbmZpZy9zdGVwcy5weQpAQCAtMzY3LDcgKzM2Nyw3IEBAIGNsYXNzIFJ1 blRlc3QyNjJUZXN0cyhUZXN0V2l0aEZhaWx1cmVDb3VudCk6CiAgICAgZGVzY3JpcHRpb24gPSBb InRlc3QyNjItdGVzdHMgcnVubmluZyJdCiAgICAgZGVzY3JpcHRpb25Eb25lID0gWyJ0ZXN0MjYy LXRlc3RzIl0KICAgICBmYWlsZWRUZXN0c0Zvcm1hdFN0cmluZyA9ICIlZCBUZXN0MjYyIHRlc3Ql cyBmYWlsZWQiCi0gICAgY29tbWFuZCA9IFsicGVybCIsICIuL1Rvb2xzL1NjcmlwdHMvdGVzdDI2 Mi1ydW5uZXIiLCBXaXRoUHJvcGVydGllcygiLS0lKGNvbmZpZ3VyYXRpb24pcyIpXQorICAgIGNv bW1hbmQgPSBbInBlcmwiLCAiLi9Ub29scy9TY3JpcHRzL3Rlc3QyNjItcnVubmVyIiwgIi0tdmVy Ym9zZSIsIFdpdGhQcm9wZXJ0aWVzKCItLSUoY29uZmlndXJhdGlvbilzIildCiAKICAgICBkZWYg c3RhcnQoc2VsZik6CiAgICAgICAgIGFwcGVuZEN1c3RvbUJ1aWxkRmxhZ3Moc2VsZiwgc2VsZi5n ZXRQcm9wZXJ0eSgncGxhdGZvcm0nKSwgc2VsZi5nZXRQcm9wZXJ0eSgnZnVsbFBsYXRmb3JtJykp CmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3Rlc3QyNjIvUnVubmVyLnBtIGIvVG9vbHMvU2Ny aXB0cy90ZXN0MjYyL1J1bm5lci5wbQppbmRleCBlYzMwZDAxZmQ0Y2Q1ODM2NWQyYmEwY2FjMmUz MTU2ZDI5NTliNjE5Li5kYWI0YjFmZGI5MjIzZjYzMTFmYzE1MDk0ZWNkNTI1ODE2NjA5NmIzIDEw MDc1NQotLS0gYS9Ub29scy9TY3JpcHRzL3Rlc3QyNjIvUnVubmVyLnBtCisrKyBiL1Rvb2xzL1Nj cmlwdHMvdGVzdDI2Mi9SdW5uZXIucG0KQEAgLTY4Myw3ICs2ODMsNyBAQCBzdWIgcHJvY2Vzc1Jl c3VsdCB7CiAgICAgICAgIGlmICgkdmVyYm9zZSkgewogICAgICAgICAgICAgbXkgJGZlYXR1cmVz TGlzdCA9ICcnOwogICAgICAgICAgICAgJGZlYXR1cmVzTGlzdCA9ICJcbkZlYXR1cmVzOiAiIC4g am9pbignLCAnLCBAeyAkZGF0YS0+e2ZlYXR1cmVzfSB9KSBpZiAkZGF0YS0+e2ZlYXR1cmVzfTsK LSAgICAgICAgICAgICRzdWZmaXhNc2cgPSAiJHJlc3VsdCRmZWF0dXJlc0xpc3RcblxuIjsKKyAg ICAgICAgICAgICRzdWZmaXhNc2cgPSAiJHJlc3VsdCRmZWF0dXJlc0xpc3RcbiI7CiAgICAgICAg IH0KIAogICAgICAgICBwcmludCAiJG5ld0ZhaWwkZmFpbE1zZyRzdWZmaXhNc2ciOwpAQCAtNjkz LDExICs2OTMsMTUgQEAgc3ViIHByb2Nlc3NSZXN1bHQgewogICAgIH0gZWxzaWYgKCRzY2VuYXJp byBuZSAnc2tpcCcgJiYgISRjdXJyZW50ZmFpbHVyZSkgewogICAgICAgICBpZiAoJGV4cGVjdGVk ZmFpbHVyZSkgewogICAgICAgICAgICAgcHJpbnQgIk5FVyBQQVNTICRmaWxlICgkc2NlbmFyaW8p XG4iOwotICAgICAgICAgICAgcHJpbnQgIlxuIiBpZiAkdmVyYm9zZTsKKyAgICAgICAgfSBlbHNp ZiAoJHZlcmJvc2UpIHsKKyAgICAgICAgICAgIHByaW50ICJQQVNTICRmaWxlICgkc2NlbmFyaW8p XG4iOwogICAgICAgICB9CiAKICAgICAgICAgJHJlc3VsdGRhdGF7cmVzdWx0fSA9ICdQQVNTJzsK ICAgICB9IGVsc2UgeworICAgICAgICBpZiAoJHZlcmJvc2UpIHsKKyAgICAgICAgICAgIHByaW50 ICJTS0lQICRmaWxlXG4iOworICAgICAgICB9CiAgICAgICAgICRyZXN1bHRkYXRhe3Jlc3VsdH0g PSAnU0tJUCc7CiAgICAgfQogCg==