210880 2020-04-22 15:18:24 -0700 run-webkit-tests should not fail if all tests found to run are skipped. 2020-05-13 11:37:53 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 jlewis3 jlewis3 ap ews-watchlist glenn jbedard webkit-bug-importer oldest_to_newest 1644529 0 jlewis3 2020-04-22 15:18:24 -0700 run-webkit-tests should not fail if all tests found to run are skipped. Currently if tests are found to run but all of those tests are skipped, we return a failing value of -1. We should still fail if there are no test to run, but we shouldn't if all tests are skipped. 1644531 1 webkit-bug-importer 2020-04-22 15:18:48 -0700 <rdar://problem/62213993> 1644537 2 jbedard 2020-04-22 15:28:37 -0700 I don't know about this. Only selecting skipped tests seems like something we want to surface. The exit code really only matter in automation, and if we're skipping all tests, shouldn't we just avoid running that particular set of tests on that particular configuration? 1644554 3 jlewis3 2020-04-22 15:42:34 -0700 So I see your point here but after talking with Alexey, this was something we discussed as being a situation that we should't fail on. Perhaps he can provide a bit more behind his thoughts. 1644560 4 ap 2020-04-22 15:52:13 -0700 Skipping and moving tests is a separate decision from what we run in automation. It is possible that we have some tests run one day, and they get moved to a different suite another day. I don't think that we'd want to change buildbot configuration in lockstep. 1652032 5 399204 jlewis3 2020-05-12 17:07:27 -0700 Created attachment 399204 Patch 1652234 6 399204 jbedard 2020-05-13 08:06:26 -0700 Comment on attachment 399204 Patch r+ing because this behavior is changed correctly, although I disagree that we should be changing this behavior. Since Alexey, Matt and our other bot watchers are for this change, so I'm not going to block it. 1652309 7 jlewis3 2020-05-13 11:37:53 -0700 Committed r261629: <https://trac.webkit.org/changeset/261629> 399204 2020-05-12 17:07:27 -0700 2020-05-13 08:06:26 -0700 Patch bug-210880-20200512170727.patch text/plain 6294 jlewis3 U3VidmVyc2lvbiBSZXZpc2lvbjogMjYxNTg3CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggYTZjNjk5YjllOTk0NTcwYmEwYmI2NDk5YmI5NzhjYmQ4 ZTc4YmJjMS4uM2Y1N2RjZTVkYjExOTQwMzIwMzYwMTc5ZmNkYWM3ZGIyZTBlNDM0MSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDI0 IEBACisyMDIwLTA1LTEyICBNYXR0IExld2lzICA8amxld2lzM0BhcHBsZS5jb20+CisKKyAgICAg ICAgUnVuLXdlYmtpdC10ZXN0cyBzaG91bGQgbm90IGZhaWwgaWYgYWxsIHRlc3RzIGZvdW5kIHRv IHJ1biBhcmUgc2tpcHBlZC4KKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19i dWcuY2dpP2lkPTIxMDg4MAorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgor CisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbWFu YWdlci5weToKKyAgICAgICAgKE1hbmFnZXIucnVuKTogQWRkZWQgYSBjaGVjayB0byBzZWUgaWYg d2Ugc2tpcGVkIGFsbCB0ZXN0cy4gSWYgc28gV2UgcmV0dXJuIGEgc3VjY2Vzc2Z1bCBleGl0Cisg ICAgICAgIGNvZGUgYXMgd2UgcGVyZm9ybWVkIGFzIGV4cGVjdGVkLgorICAgICAgICAqIFNjcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL21vZGVscy90ZXN0X3J1bl9yZXN1bHRzLnB5OgorICAg ICAgICAoUnVuRGV0YWlscy5fX2luaXRfXyk6IEFkZGVkIGluIGFuIGFkZGl0aW9uYWwgY2xhc3Mg dmFyaWFibGUgdG8gYmUgdXNlZCBhcyBhIHdheSB0byByZWNvcmQgYW4KKyAgICAgICAgYWxsIHNr aXBwZWQgcmVzdWx0IGNoZWNrLgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rl c3RzL3J1bl93ZWJraXRfdGVzdHMucHk6CisgICAgICAgIChtYWluKTogQWRkZWQgYSBhbGwgdGVz dCBza2lwcGVkIGNoZWNrIGFuZCB3ZSByZXR1cm4gZWFybHkgd2l0aCB0aGUgZXhpdCBjb2RlLgor ICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHNf aW50ZWdyYXRpb250ZXN0LnB5OiBBZGRlZCBhIG5ldyB0ZXN0IGFuZCByZW1vdmVkIHRoZSBvbGQK KyAgICAgICAgb25lcyB0aGF0IG5vIGxvbmdlciB0ZXN0IHRoZSBjb3JyZWN0IGZ1bmN0aW9uYWxp dHkuCisgICAgICAgIChSdW5UZXN0LnRlc3RfYWxsX3Rlc3RzX3NraXBwZWQpOgorICAgICAgICAo UnVuVGVzdC50ZXN0X25vX3Rlc3RzX2ZvdW5kKTogRGVsZXRlZC4KKyAgICAgICAgKFJ1blRlc3Qu dGVzdF9ub190ZXN0c19mb3VuZF8yKTogRGVsZXRlZC4KKwogMjAyMC0wNS0xMiAgQWxleCBDaHJp c3RlbnNlbiAgPGFjaHJpc3RlbnNlbkB3ZWJraXQub3JnPgogCiAgICAgICAgIFtpT1MgU2ltdWxh dG9yXSBUZXN0V2ViS2l0QVBJLl9XS0Rvd25sb2FkLlJlc3VtZSBpcyBhIGZsYWt5IGNyYXNoCmRp ZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVy cy9tYW5hZ2VyLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJv bGxlcnMvbWFuYWdlci5weQppbmRleCA5ZTljOWJmOTEyMjZlMjhiOWJmNTUzMzgwM2U1NzE0N2E0 MWRlZTVkLi5mNThiMWM4NjJhMDcwZjMwM2M4YjU3NDZjNWJlNDQxNGQzNDA2NDM1IDEwMDY0NAot LS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9tYW5h Z2VyLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xs ZXJzL21hbmFnZXIucHkKQEAgLTIxOCw3ICsyMTgsMTIgQEAgY2xhc3MgTWFuYWdlcihvYmplY3Qp OgogICAgICAgICBzZWxmLl9wcmludGVyLnByaW50X2ZvdW5kKGxlbihhZ2dyZWdhdGVfdGVzdF9u YW1lcyksIGxlbihhZ2dyZWdhdGVfdGVzdHMpLCBzZWxmLl9vcHRpb25zLnJlcGVhdF9lYWNoLCBz ZWxmLl9vcHRpb25zLml0ZXJhdGlvbnMpCiAgICAgICAgIHN0YXJ0X3RpbWUgPSB0aW1lLnRpbWUo KQogCi0gICAgICAgICMgQ2hlY2sgdG8gbWFrZSBzdXJlIHdlJ3JlIG5vdCBza2lwcGluZyBldmVy eSB0ZXN0LgorICAgICAgICAjIENoZWNrIHRvIHNlZSBpZiBhbGwgdGVzdHMgd2UgYXJlIHJ1bm5p bmcgYXJlIHNraXBwZWQuCisgICAgICAgIGlmIHRlc3RzX3RvX3NraXAgPT0gdG90YWxfdGVzdHM6 CisgICAgICAgICAgICBfbG9nLmVycm9yKCJBbGwgdGVzdHMgc2tpcHBlZC4iKQorICAgICAgICAg ICAgcmV0dXJuIHRlc3RfcnVuX3Jlc3VsdHMuUnVuRGV0YWlscyhleGl0X2NvZGU9MCwgc2tpcHBl ZF9hbGxfdGVzdHM9VHJ1ZSkKKworICAgICAgICAjIENoZWNrIHRvIG1ha2Ugc3VyZSB3ZSBoYXZl IG5vIHRlc3RzIHRvIHJ1biB0aGF0IGFyZSBub3Qgc2tpcHBlZC4KICAgICAgICAgaWYgbm90IHN1 bShbbGVuKHRlc3RzKSBmb3IgdGVzdHMgaW4gaXRlcnZhbHVlcyh0ZXN0c190b19ydW5fYnlfZGV2 aWNlKV0pOgogICAgICAgICAgICAgX2xvZy5jcml0aWNhbCgnTm8gdGVzdHMgdG8gcnVuLicpCiAg ICAgICAgICAgICByZXR1cm4gdGVzdF9ydW5fcmVzdWx0cy5SdW5EZXRhaWxzKGV4aXRfY29kZT0t MSkKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL21vZGVs cy90ZXN0X3J1bl9yZXN1bHRzLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvbW9kZWxzL3Rlc3RfcnVuX3Jlc3VsdHMucHkKaW5kZXggZWFjZGE4NTgwZWU4NzJjZjZlNGQ2 MWFhOWI5ZWIyOTExMGQ0ZWMyYy4uMmMyM2MwNGVkZjBlYmMyYjNjYWExYjM2M2NiOTM1MjMyY2U0 ZGZiZCAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbW9k ZWxzL3Rlc3RfcnVuX3Jlc3VsdHMucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlv dXRfdGVzdHMvbW9kZWxzL3Rlc3RfcnVuX3Jlc3VsdHMucHkKQEAgLTE1NywxMiArMTU3LDEzIEBA IGNsYXNzIFRlc3RSdW5SZXN1bHRzKG9iamVjdCk6CiAKIAogY2xhc3MgUnVuRGV0YWlscyhvYmpl Y3QpOgotICAgIGRlZiBfX2luaXRfXyhzZWxmLCBleGl0X2NvZGUsIHN1bW1hcml6ZWRfcmVzdWx0 cz1Ob25lLCBpbml0aWFsX3Jlc3VsdHM9Tm9uZSwgcmV0cnlfcmVzdWx0cz1Ob25lLCBlbmFibGVk X3BpeGVsX3Rlc3RzX2luX3JldHJ5PUZhbHNlKToKKyAgICBkZWYgX19pbml0X18oc2VsZiwgZXhp dF9jb2RlLCBzdW1tYXJpemVkX3Jlc3VsdHM9Tm9uZSwgaW5pdGlhbF9yZXN1bHRzPU5vbmUsIHJl dHJ5X3Jlc3VsdHM9Tm9uZSwgZW5hYmxlZF9waXhlbF90ZXN0c19pbl9yZXRyeT1GYWxzZSwgc2tp cHBlZF9hbGxfdGVzdHM9RmFsc2UpOgogICAgICAgICBzZWxmLmV4aXRfY29kZSA9IGV4aXRfY29k ZQogICAgICAgICBzZWxmLnN1bW1hcml6ZWRfcmVzdWx0cyA9IHN1bW1hcml6ZWRfcmVzdWx0cwog ICAgICAgICBzZWxmLmluaXRpYWxfcmVzdWx0cyA9IGluaXRpYWxfcmVzdWx0cwogICAgICAgICBz ZWxmLnJldHJ5X3Jlc3VsdHMgPSByZXRyeV9yZXN1bHRzCiAgICAgICAgIHNlbGYuZW5hYmxlZF9w aXhlbF90ZXN0c19pbl9yZXRyeSA9IGVuYWJsZWRfcGl4ZWxfdGVzdHNfaW5fcmV0cnkKKyAgICAg ICAgc2VsZi5za2lwcGVkX2FsbF90ZXN0cyA9IHNraXBwZWRfYWxsX3Rlc3RzCiAKIAogZGVmIF9p bnRlcnByZXRfdGVzdF9mYWlsdXJlcyhmYWlsdXJlcyk6CmRpZmYgLS1naXQgYS9Ub29scy9TY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzLnB5IGIvVG9vbHMvU2Ny aXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0cy5weQppbmRleCBlMGUx OTI3MGJiYTVkZDU1NDIxN2M1MjFhNzAyYjNiOGU3OGVhNWNmLi4wZTFhNDU1YzU2ODY3M2VhNDA2 ODRmOWI4ZDEwMDg4OWIwMjEyYmVjIDEwMDc1NQotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5 L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Vi a2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHkKQEAgLTg5LDYgKzg5LDggQEAg ZGVmIG1haW4oYXJndiwgc3Rkb3V0LCBzdGRlcnIpOgogICAgICAgICBvcHRpb25zLmFkZGl0aW9u YWxfZW52X3Zhci5hcHBlbmQoJ0pTQ19tYXhQZXJUaHJlYWRTdGFja1VzYWdlPScgKyBzdHIoc3Rh Y2tTaXplSW5CeXRlcykpCiAgICAgICAgIG9wdGlvbnMuYWRkaXRpb25hbF9lbnZfdmFyLmFwcGVu ZCgnX19YUENfSlNDX21heFBlclRocmVhZFN0YWNrVXNhZ2U9JyArIHN0cihzdGFja1NpemVJbkJ5 dGVzKSkKICAgICAgICAgcnVuX2RldGFpbHMgPSBydW4ocG9ydCwgb3B0aW9ucywgYXJncywgc3Rk ZXJyKQorICAgICAgICBpZiBydW5fZGV0YWlscy5leGl0X2NvZGUgIT0gLTEgYW5kIHJ1bl9kZXRh aWxzLnNraXBwZWRfYWxsX3Rlc3RzOgorICAgICAgICAgICAgcmV0dXJuIHJ1bl9kZXRhaWxzLmV4 aXRfY29kZQogICAgICAgICBpZiBydW5fZGV0YWlscy5leGl0X2NvZGUgIT0gLTEgYW5kIG5vdCBy dW5fZGV0YWlscy5pbml0aWFsX3Jlc3VsdHMua2V5Ym9hcmRfaW50ZXJydXB0ZWQ6CiAgICAgICAg ICAgICBib3RfcHJpbnRlciA9IGJ1aWxkYm90X3Jlc3VsdHMuQnVpbGRCb3RQcmludGVyKHN0ZG91 dCwgb3B0aW9ucy5kZWJ1Z19yd3RfbG9nZ2luZykKICAgICAgICAgICAgIGJvdF9wcmludGVyLnBy aW50X3Jlc3VsdHMocnVuX2RldGFpbHMpCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX2ludGVncmF0aW9udGVzdC5weSBiL1Rv b2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHNfaW50ZWdy YXRpb250ZXN0LnB5CmluZGV4IDMwODAyYmRiZjgzZGEwMTJjYTY2ZGM1YzFkNjI1YTZhOTc1NDUw ZmIuLmFiYmQxOGUwMDU1MDZmNTMwNjQ4NDdkZDg1NjczODNjMDNiMjRmZTggMTAwNjQ0Ci0tLSBh L1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHNfaW50 ZWdyYXRpb250ZXN0LnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3Rz L3J1bl93ZWJraXRfdGVzdHNfaW50ZWdyYXRpb250ZXN0LnB5CkBAIC0yODAsMTUgKzI4MCwxMCBA QCBjbGFzcyBSdW5UZXN0KHVuaXR0ZXN0LlRlc3RDYXNlLCBTdHJlYW1UZXN0aW5nTWl4aW4pOgog ICAgICAgICAgICAgXywgcmVndWxhcl9vdXRwdXQsIF8gPSBsb2dnaW5nX3J1bihbJ2ZhaWx1cmVz L2V4cGVjdGVkL2tleWJvYXJkLmh0bWwnLCAncGFzc2VzL3RleHQuaHRtbCcsICctLWNoaWxkLXBy b2Nlc3NlcycsICcyJywgJy0tZm9yY2UnXSwgdGVzdHNfaW5jbHVkZWQ9VHJ1ZSwgc2hhcmVkX3Bv cnQ9RmFsc2UpCiAgICAgICAgICAgICBzZWxmLmFzc2VydFRydWUoYW55KFsnSW50ZXJydXB0ZWQs IGV4aXRpbmcnIGluIGxpbmUgZm9yIGxpbmUgaW4gcmVndWxhcl9vdXRwdXQuZ2V0dmFsdWUoKS5z cGxpdGxpbmVzKCldKSkKIAotICAgIGRlZiB0ZXN0X25vX3Rlc3RzX2ZvdW5kKHNlbGYpOgotICAg ICAgICBkZXRhaWxzLCBlcnIsIF8gPSBsb2dnaW5nX3J1bihbJ3Jlc291cmNlcyddLCB0ZXN0c19p bmNsdWRlZD1UcnVlKQotICAgICAgICBzZWxmLmFzc2VydEVxdWFsKGRldGFpbHMuZXhpdF9jb2Rl LCAtMSkKLSAgICAgICAgc2VsZi5hc3NlcnRDb250YWlucyhlcnIsICdObyB0ZXN0cyB0byBydW4u XG4nKQotCi0gICAgZGVmIHRlc3Rfbm9fdGVzdHNfZm91bmRfMihzZWxmKToKKyAgICBkZWYgdGVz dF9hbGxfdGVzdHNfc2tpcHBlZChzZWxmKToKICAgICAgICAgZGV0YWlscywgZXJyLCBfID0gbG9n Z2luZ19ydW4oWydmb28nXSwgdGVzdHNfaW5jbHVkZWQ9VHJ1ZSkKLSAgICAgICAgc2VsZi5hc3Nl cnRFcXVhbChkZXRhaWxzLmV4aXRfY29kZSwgLTEpCi0gICAgICAgIHNlbGYuYXNzZXJ0Q29udGFp bnMoZXJyLCAnTm8gdGVzdHMgdG8gcnVuLlxuJykKKyAgICAgICAgc2VsZi5hc3NlcnRFcXVhbChk ZXRhaWxzLmV4aXRfY29kZSwgMCkKKyAgICAgICAgc2VsZi5hc3NlcnRDb250YWlucyhlcnIsICdB bGwgdGVzdHMgc2tpcHBlZC5cbicpCiAKICAgICBkZWYgdGVzdF9uYXR1cmFsX29yZGVyKHNlbGYp OgogICAgICAgICB0ZXN0c190b19ydW4gPSBbJ3Bhc3Nlcy9hdWRpby5odG1sJywgJ2ZhaWx1cmVz L2V4cGVjdGVkL3RleHQuaHRtbCcsICdmYWlsdXJlcy9leHBlY3RlZC9taXNzaW5nX3RleHQuaHRt bCcsICdwYXNzZXMvYXJncy5odG1sJ10K