207118 2020-02-03 08:28:51 -0800 run-javascriptcore-tests: Don't split binary results 2020-06-03 14:57:43 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 jbedard jbedard aakash_jain ap ryanhaddad webkit-bug-importer zhifei_fang oldest_to_newest 1613956 0 jbedard 2020-02-03 08:28:51 -0800 Experience over the last 2 months has shown that attempting to split up results from a single JSC test binary doesn't really work. We should just report aggregated results for a single binary. 1613959 1 389515 jbedard 2020-02-03 08:30:41 -0800 Created attachment 389515 Patch 1613960 2 jbedard 2020-02-03 08:32:15 -0800 https://results.webkit.org/suites?suite=javascriptcore-tests demonstrates how my original approach basically failed. 1614018 3 389515 zhifei_fang 2020-02-03 10:13:36 -0800 Comment on attachment 389515 Patch unofficially r+, in long term we should unify all the test output format, and let those binaries generate that standard output 1614025 4 aakash_jain 2020-02-03 10:23:34 -0800 (In reply to Jonathan Bedard from comment #0) > Experience over the last 2 months has shown that attempting to split up results from a single JSC test binary doesn't really work. What's the issue? 1614039 5 jbedard 2020-02-03 10:46:04 -0800 (In reply to Aakash Jain from comment #4) > (In reply to Jonathan Bedard from comment #0) > > Experience over the last 2 months has shown that attempting to split up results from a single JSC test binary doesn't really work. > What's the issue? We can't really do it consistently, a sub-test which fails will sometimes have a different name than when it passes and often, exception messages will be tagged as actual test names. 1614044 6 aakash_jain 2020-02-03 10:59:54 -0800 rs=me 1614062 7 jbedard 2020-02-03 11:37:43 -0800 Committed r255578: <https://trac.webkit.org/changeset/255578> 1614064 8 webkit-bug-importer 2020-02-03 11:41:44 -0800 <rdar://problem/59118504> 389515 2020-02-03 08:30:41 -0800 2020-06-03 14:57:43 -0700 Patch bug-207118-20200203083040.patch text/plain 4254 jbedard SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1NTU2MykKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDEzIEBACisyMDIwLTAyLTAzICBKb25hdGhhbiBCZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNv bT4KKworICAgICAgICBydW4tamF2YXNjcmlwdGNvcmUtdGVzdHM6IERvbid0IHNwbGl0IGJpbmFy eSByZXN1bHRzCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9p ZD0yMDcxMTgKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAg ICAqIFNjcmlwdHMvcnVuLWphdmFzY3JpcHRjb3JlLXRlc3RzOgorICAgICAgICAocnVuVGVzdCk6 IE9ubHkgcmVwb3J0IGFnZ3JlZ2F0ZWQgYmluYXJ5IHJlc3VsdHMuCisKIDIwMjAtMDItMDMgIFph biBEb2JlcnNlayAgPHpkb2JlcnNla0BpZ2FsaWEuY29tPgogCiAgICAgICAgIFVzZSBzdGQ6Omh5 cG90KCkgd2hlcmUgcG9zc2libGUKSW5kZXg6IFRvb2xzL1NjcmlwdHMvcnVuLWphdmFzY3JpcHRj b3JlLXRlc3RzCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL1NjcmlwdHMvcnVuLWphdmFzY3JpcHRjb3Jl LXRlc3RzCShyZXZpc2lvbiAyNTU1NTkpCisrKyBUb29scy9TY3JpcHRzL3J1bi1qYXZhc2NyaXB0 Y29yZS10ZXN0cwkod29ya2luZyBjb3B5KQpAQCAtNTQ5LDgxICs1NDksMTQgQEAgc3ViIHJ1blRl c3QgewogICAgIH0KIAogICAgIG15ICR0ZXN0UmVzdWx0ID0gMDsKLSAgICBteSAkZmFpbHVyZUZs YWcgPSAwOwogICAgIG15ICRsYXN0T3B0aW1pemVMZXZlbDsKIAogICAgIG9wZW4oVEVTVCwgIi18 IiwgIkBjb21tYW5kIDI+JjEiKSBvciBkaWUgIkZhaWxlZCB0byBydW4gQGNvbW1hbmQiOwogICAg IHdoaWxlICggbXkgJGxpbmUgPSA8VEVTVD4gKSB7Ci0gICAgICAgIG15ICRzdWJUZXN0TmFtZTsK ICAgICAgICAgcHJpbnQgJGxpbmUgaWYgKCR2ZXJib3NlKTsKLQotICAgICAgICAjIGUuZy46ICdP MjogdGVzdFJvdFJXaXRoSW1tU2hpZnQodWludDY0LW1heCwgdWludDMyLW1pbik6IE9LIScnCi0g ICAgICAgIGlmICgkbGluZSA9fiAvKE9cZCk6IChcUyspKFwoLipcKSk6IChbQS1aYS16XSspIVxu LykgewotICAgICAgICAgICAgJGxhc3RPcHRpbWl6ZUxldmVsID0gJDE7Ci0gICAgICAgICAgICAk c3ViVGVzdE5hbWUgPSAiJHRlc3ROYW1lLiQxLiQyJDMiOwotICAgICAgICAgICAgJGZhaWx1cmVG bGFnICs9IDEgaWYgKCQ0IG5lICJPSyIpOwotICAgICAgICB9Ci0gICAgICAgICMgQXNzZXJ0aW9u IGluIGJpbmFyeQotICAgICAgICAjIGUuZy46IEFTU0VSVElPTiBGQUlMRUQ6IHRoaW5nc1swXSA9 PSAyCi0gICAgICAgIGVsc2lmICgkbGluZSA9fiAvQVNTRVJUSU9OIEZBSUxFRDovKSB7Ci0gICAg ICAgICAgICAkZmFpbHVyZUZsYWcgKz0gMTsKLSAgICAgICAgICAgIG5leHQ7Ci0gICAgICAgIH0K LSAgICAgICAgIyBlLmcuOiBmaWxlL3BhdGgoMzk3KSA6IHZvaWQgKGFub255bW91cyBuYW1lc3Bh Y2UpOjp0ZXN0U2h1ZmZsZVNpbXBsZUJyb2FkY2FzdCgpCi0gICAgICAgIGVsc2lmICgkZmFpbHVy ZUZsYWcgJiYgJGxpbmUgPX4gLy4qXChcZCtcKSA6IFxTKyAuKjo6KFxTKykoXCguKlwpKS8pIHsK LSAgICAgICAgICAgIGlmIChkZWZpbmVkICRsYXN0T3B0aW1pemVMZXZlbCkgewotICAgICAgICAg ICAgICAgICRzdWJUZXN0TmFtZSA9ICIkdGVzdE5hbWUuJGxhc3RPcHRpbWl6ZUxldmVsLiQxJDIi OwotICAgICAgICAgICAgfSBlbHNlIHsKLSAgICAgICAgICAgICAgICAkc3ViVGVzdE5hbWUgPSAi JHRlc3ROYW1lLiQxJDIiOwotICAgICAgICAgICAgfQotICAgICAgICB9Ci0gICAgICAgICMgc3Vi LXRlc3QgZnJvbSB0aGUgYy1hcGksIG9mIHRoZSBmb3JtOgotICAgICAgICAjIGUuZy46ICd0ZXN0 YXBpWzE1NDIyOjE2OTA0NzkwXSBwYXJhbGxlbFByb21pc2VSZXNvbHZlVGVzdCgpOiBkb25lLicn Ci0gICAgICAgIGVsc2lmICgkbGluZSA9fiAvXFtcZCs6XGQrXF0gKFxTKykoKTogZG9uZVwuXG4v KSB7Ci0gICAgICAgICAgICAkc3ViVGVzdE5hbWUgPSAiJHRlc3ROYW1lLm9iamVjdGl2ZS1jLiQx IjsKLSAgICAgICAgfQotICAgICAgICAjIEVuZCBvZiBDLUFQSSB0ZXN0cyAobWVhbmluZyB3ZSBq dXN0IGZpbmlzaGVkIHRoZSB1bmdyb3VwZWQgbWFpbiB0ZXN0cykKLSAgICAgICAgZWxzaWYgKCRs aW5lID1+IC9TdGFydGluZyBDLUFQSSB0ZXN0cyBpbiBDKysvKSB7Ci0gICAgICAgICAgICAkc3Vi VGVzdE5hbWUgPSAiJHRlc3ROYW1lLm9iamVjdGl2ZS1jLm1haW4iOwotICAgICAgICB9Ci0gICAg ICAgICMgRW5kIG9mIHRlc3RhcGkKLSAgICAgICAgIyBlLmcuOiBGQUlMOiBTb21lIHRlc3RzIGZh aWxlZC4KLSAgICAgICAgZWxzaWYgKCRsaW5lID1+IC9GQUlMOiBTb21lIHRlc3RzIGZhaWxlZFwu LykgewotICAgICAgICAgICAgJGZhaWx1cmVGbGFnID0gMDsKLSAgICAgICAgICAgIG5leHQ7Ci0g ICAgICAgIH0KLSAgICAgICAgIyBUZXN0IGNhc2VzIGluc2lkZSBBUEkgdGVzdHMKLSAgICAgICAg IyBlLmcuOiAndGVzdGFwaVsxNTQyMjoxNjkwNDc5MF0gVEVTVDogIjIgKyAyIjogUEFTU0VEJwot ICAgICAgICBlbHNpZiAoJGxpbmUgPX4gL1xbXGQrOlxkK1xdIFRFU1Q6ICIuKiI6IChbQS1aXSsp XG4vKSB7Ci0gICAgICAgICAgICAkZmFpbHVyZUZsYWcgKz0gMSBpZiAoJDEgbmUgIlBBU1NFRCIg JiYgJDEgbmUgIlBBU1MiKTsKLSAgICAgICAgICAgIG5leHQ7Ci0gICAgICAgIH0KLSAgICAgICAg IyBlLmcuOiAndGVzdEJpdFhvckFuZEFuZEFyZ3MoMCwgMCx1aW50MzItbWF4KTogT0shJycKLSAg ICAgICAgZWxzaWYgKCRsaW5lID1+IC8oXFMrKShcKC4qXCkpOiAoW0EtWmEtel0rKSFcbi8pIHsK LSAgICAgICAgICAgICRzdWJUZXN0TmFtZSA9ICIkdGVzdE5hbWUuJDEkMiI7Ci0gICAgICAgICAg ICAkZmFpbHVyZUZsYWcgKz0gMSBpZiAoJDMgbmUgIk9LIik7Ci0gICAgICAgIH0KLSAgICAgICAg IyBHZW5lcmFsIHRlc3RjYXNlIHN0YXRlbWVudCwgZmlsdGVyZWQgdG8gZXhjbHVkZSBzb21lIGp1 bmsgbG9nZ2luZwotICAgICAgICAjIGUuZy46ICdQQVNTOiBERkcgc2NyaXB0IHRpbWVkIG91dCBh cyBleHBlY3RlZCB3aGVuIG5vIGNhbGxiYWNrIGlzIHNwZWNpZmllZC4nJwotICAgICAgICBlbHNp ZiAoJGxpbmUgPX4gLyhbQS1aXSspOiAoLiopXC4qXG4vICYmICEoJGxpbmUgPX4gL1tBLVpdKzog XGQrXG4vIHx8ICRsaW5lID1+IC9cW1xkKzpcZCtcXSBbQS1aXSs6LykpIHsKLSAgICAgICAgICAg ICRzdWJUZXN0TmFtZSA9ICIkdGVzdE5hbWUuJDIiOwotICAgICAgICAgICAgJGZhaWx1cmVGbGFn ICs9IDEgaWYgKCQxIG5lICJQQVNTIik7Ci0gICAgICAgIH0KLSAgICAgICAgIyBlLmcuOiAndGVz dCBkZXNjcmlwdGlvbjogUEFTU0VEJwotICAgICAgICBlbHNpZiAoJGxpbmUgPX4gLy4qOiAoW0Et Wl0rKVxuLykgewotICAgICAgICAgICAgJGZhaWx1cmVGbGFnICs9IDEgaWYgKCQxIG5lICJQQVNT RUQiICYmICQxIG5lICJQQVNTIik7Ci0gICAgICAgICAgICBuZXh0OwotICAgICAgICB9IGVsc2Ug ewotICAgICAgICAgICAgbmV4dDsKLSAgICAgICAgfQotCi0gICAgICAgIGlmICghZGVmaW5lZCAk cmVwb3J0RGF0YXskc3ViVGVzdE5hbWV9KSB7Ci0gICAgICAgICAgICBwcmludCAiJHN1YlRlc3RO YW1lIGZhaWxlZFxuIiBpZiAoJGZhaWx1cmVGbGFnICYmICEkdmVyYm9zZSk7Ci0gICAgICAgICAg ICAkcmVwb3J0RGF0YXskc3ViVGVzdE5hbWV9ID0gJGZhaWx1cmVGbGFnID8ge2FjdHVhbCA9PiAi RkFJTCJ9IDoge2FjdHVhbCA9PiAiUEFTUyJ9OwotICAgICAgICB9Ci0gICAgICAgICRmYWlsdXJl RmxhZyA9IDA7CiAgICAgfQogICAgICR0ZXN0UmVzdWx0ID0gY2xvc2UoVEVTVCkgPyAwIDogJD87 CisgICAgJHJlcG9ydERhdGF7JHRlc3ROYW1lfSA9ICR0ZXN0UmVzdWx0ID8ge2FjdHVhbCA9PiAi RkFJTCJ9IDoge2FjdHVhbCA9PiAiUEFTUyJ9OwogCiAgICAgbXkgJGV4aXRTdGF0dXMgPSBleGl0 U3RhdHVzKCR0ZXN0UmVzdWx0KTsKICAgICBwcmludCAiJHRlc3ROYW1lIGNvbXBsZXRlZCB3aXRo IHJjPSR0ZXN0UmVzdWx0ICgkZXhpdFN0YXR1cylcblxuIjsK