82233 2012-03-26 12:18:48 -0700 Retry crashing tests serially at the end of NRWT on Apple Mac 2015-02-05 09:51:22 -0800 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 mhahnenberg mhahnenberg abarth ap dpranke ggaren mitz ojan ossy webkit.review.bot oldest_to_newest 587982 0 mhahnenberg 2012-03-26 12:18:48 -0700 A few of the daemons/system services/frameworks on the Mac bots can't handle being pummeled by many threads simultaneously while we're running all the layout tests concurrently. This causes some of the tests to randomly fail occasionally when it isn't actually an issue with the test. If we were to run all of these potentially crashing tests serially at the end of NRWT (which we already do for non-crashing tests), this might alleviate the problem. 588038 1 133873 mhahnenberg 2012-03-26 13:05:12 -0700 Created attachment 133873 Patch 588053 2 133873 dpranke 2012-03-26 13:19:42 -0700 Comment on attachment 133873 Patch Note that the tests that are retried are still actually run concurrently, so this patch may not do what you think it should do. I'm sure I had a patch at one point that did run things serially, but I guess I never committed it; it would be easy to make that change and I can't really think of a good reason not to do it, so you could either add that to this or add a separate patch to do so, if you like, or you could file a bug and I could do it at some point. All that aside, I'm a bit torn by this patch. I can see the argument in retrying the crashes to see if they're consistently crashing or just flaky under load; I don't really have a problem with that. However, I'm not sure that i think a flaky crash should be ignored (which is what would happen by default). Is it hard (or impossible) to fix things so that DRT and WTR don't actually crash? As an actual comment on the crash, I think I would prefer this to be a method on the port itself (like port.should_retry_crashes()), rather than a test against the port name. 588072 3 ggaren 2012-03-26 13:32:27 -0700 > Is it hard (or impossible) to fix things so that DRT and WTR don't actually crash? Like the bug description says, the crash isn't in DRT or WTR (or WebKit, for that matter): It's a bug in certain "daemons/system services/frameworks on the Mac bots". Fixing those crashes it out of scope for the WebKit project. > Note that the tests that are retried are still actually run concurrently, so this patch may not do what you think it should do. Mark, it sounds like you should write a follow-up patch to have retried tests run serially. (Possibly just change the concurrency setting before retrying?) Based on my testing a year ago, this would also fix flakiness in filesystem/database and animation tests, so it's probably appropriate for all ports. > However, I'm not sure that i think a flaky crash should be ignored (which is what would happen by default). What would you suggest instead? We'd like a solution that can give an unambiguous "PASS" (green) or "FAIL" (red) result for a given test run. 588077 4 133879 mhahnenberg 2012-03-26 13:35:35 -0700 Created attachment 133879 Patch 588109 5 dpranke 2012-03-26 14:00:09 -0700 (In reply to comment #3) > > Is it hard (or impossible) to fix things so that DRT and WTR don't actually crash? > > Like the bug description says, the crash isn't in DRT or WTR (or WebKit, for that matter): It's a bug in certain "daemons/system services/frameworks on the Mac bots". Fixing those crashes it out of scope for the WebKit project. > Well, for daemons and system services, presumably you could make DRT/WTR/WebKit resilient to crashes in other processes. For frameworks obviously that's harder; however, in either case I would be concerned that crashes represent potential security issues and shouldn't be ignored. Obviously, having tests crash all the time is a good way to make sure they aren't ignored, but I agree that randomly flaky tests don't help anything. > > However, I'm not sure that i think a flaky crash should be ignored (which is what would happen by default). > > What would you suggest instead? > The alternative I was considering was that we could retry the test to see if it crashed twice, but still consider it a FAIL even if it only crashed once. > We'd like a solution that can give an unambiguous "PASS" (green) or "FAIL" (red) result for a given test run. Sure, wouldn't we all? :) 588111 6 133879 dpranke 2012-03-26 14:01:04 -0700 Comment on attachment 133879 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=133879&action=review > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:901 > + # On Apple Mac, we retry crashes due to https://bugs.webkit.org/show_bug.cgi?id=82233 this comment should move to mac.py now. Please fix that before landing ... 590662 7 mhahnenberg 2012-03-28 17:34:20 -0700 This was fixed in http://trac.webkit.org/changeset/112189. 1067084 8 ossy 2015-02-05 05:31:58 -0800 Is it still needed? Apple buildbots run with --no-retry-failures for a while without a regular flakiness. Should we still retry crashes for local users only? 1067111 9 ap 2015-02-05 09:51:22 -0800 > Apple buildbots run with --no-retry-failures for a while We still have retrying enabled on debug bots and on EWS. But as far as retrying crashed tests goes, I do not understand why we do that. AFAIK we still report it as a hard failure when first try crashes, and second one doesn't. 133873 2012-03-26 13:05:12 -0700 2012-03-26 13:35:32 -0700 Patch bug-82233-20120326130511.patch text/plain 1749 mhahnenberg U3VidmVyc2lvbiBSZXZpc2lvbjogMTEyMTE2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggODkwNzQ4MDk4MjIyMDJjZWMxYTJkMjVjODRiZWZhZGVj ODdhNDIzMS4uMTQzNmUzMDdlMTZlODZiN2ZkYzNmNjU5NDgzYTc0NmJhYWE1ZWVjYiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDEz IEBACisyMDEyLTAzLTI2ICBNYXJrIEhhaG5lbmJlcmcgIDxtaGFobmVuYmVyZ0BhcHBsZS5jb20+ CisKKyAgICAgICAgUmV0cnkgY3Jhc2hpbmcgdGVzdHMgc2VyaWFsbHkgYXQgdGhlIGVuZCBvZiBO UldUIG9uIEFwcGxlIE1hYworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1 Zy5jZ2k/aWQ9ODIyMzMKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKwor ICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL21hbmFn ZXIucHk6CisgICAgICAgIChNYW5hZ2VyLnJ1bik6IAorCiAyMDEyLTAzLTI2ICBMZW8gWWFuZyAg PGxlby55YW5nQHRvcmNobW9iaWxlLmNvbS5jbj4KIAogICAgICAgICBbQmxhY2tCZXJyeV0gU2tl bGV0b24gY29kZSBvZiBBc3luY0ZpbGVTeXN0ZW1CbGFja0JlcnJ5LmNwcApkaWZmIC0tZ2l0IGEv VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbWFuYWdlci5w eSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL21hbmFn ZXIucHkKaW5kZXggZWQ3ODZlNTNkMTg2ZjM4MWQ1Yjc3MDQ0NTJhMThiZjJlNGM2NTQ2ZC4uMGJi NzMxMTZmY2YzZjZmZTM4ODI4ZDk3N2VkNGNiYzcwOGY1ZjJlMiAxMDA2NDQKLS0tIGEvVG9vbHMv U2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbWFuYWdlci5weQorKysg Yi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9tYW5hZ2Vy LnB5CkBAIC04OTgsNyArODk4LDggQEAgY2xhc3MgTWFuYWdlcihvYmplY3QpOgogCiAgICAgICAg ICMgV2UgZXhjbHVkZSB0aGUgY3Jhc2hlcyBmcm9tIHRoZSBsaXN0IG9mIHJlc3VsdHMgdG8gcmV0 cnksIGJlY2F1c2UKICAgICAgICAgIyB3ZSB3YW50IHRvIHRyZWF0IGV2ZW4gYSBwb3RlbnRpYWxs eSBmbGFreSBjcmFzaCBhcyBhbiBlcnJvci4KLSAgICAgICAgZmFpbHVyZXMgPSBzZWxmLl9nZXRf ZmFpbHVyZXMocmVzdWx0X3N1bW1hcnksIGluY2x1ZGVfY3Jhc2hlcz1GYWxzZSwgaW5jbHVkZV9t aXNzaW5nPUZhbHNlKQorICAgICAgICAjIE9uIEFwcGxlIE1hYywgd2UgcmV0cnkgY3Jhc2hlcyBk dWUgdG8gaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTgyMjMzCisgICAg ICAgIGZhaWx1cmVzID0gc2VsZi5fZ2V0X2ZhaWx1cmVzKHJlc3VsdF9zdW1tYXJ5LCBpbmNsdWRl X2NyYXNoZXM9c2VsZi5fcG9ydC5wb3J0X25hbWUgPT0gIm1hYyIsIGluY2x1ZGVfbWlzc2luZz1G YWxzZSkKICAgICAgICAgcmV0cnlfc3VtbWFyeSA9IHJlc3VsdF9zdW1tYXJ5CiAgICAgICAgIHdo aWxlIChsZW4oZmFpbHVyZXMpIGFuZCBzZWxmLl9vcHRpb25zLnJldHJ5X2ZhaWx1cmVzIGFuZCBu b3Qgc2VsZi5fcmV0cnlpbmcgYW5kIG5vdCBpbnRlcnJ1cHRlZCBhbmQgbm90IGtleWJvYXJkX2lu dGVycnVwdGVkKToKICAgICAgICAgICAgIF9sb2cuaW5mbygnJykK 133879 2012-03-26 13:35:35 -0700 2012-03-26 14:01:04 -0700 Patch bug-82233-20120326133534.patch text/plain 3217 mhahnenberg U3VidmVyc2lvbiBSZXZpc2lvbjogMTEyMTE2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggODkwNzQ4MDk4MjIyMDJjZWMxYTJkMjVjODRiZWZhZGVj ODdhNDIzMS4uMTQzNmUzMDdlMTZlODZiN2ZkYzNmNjU5NDgzYTc0NmJhYWE1ZWVjYiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDEz IEBACisyMDEyLTAzLTI2ICBNYXJrIEhhaG5lbmJlcmcgIDxtaGFobmVuYmVyZ0BhcHBsZS5jb20+ CisKKyAgICAgICAgUmV0cnkgY3Jhc2hpbmcgdGVzdHMgc2VyaWFsbHkgYXQgdGhlIGVuZCBvZiBO UldUIG9uIEFwcGxlIE1hYworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1 Zy5jZ2k/aWQ9ODIyMzMKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKwor ICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL21hbmFn ZXIucHk6CisgICAgICAgIChNYW5hZ2VyLnJ1bik6IAorCiAyMDEyLTAzLTI2ICBMZW8gWWFuZyAg PGxlby55YW5nQHRvcmNobW9iaWxlLmNvbS5jbj4KIAogICAgICAgICBbQmxhY2tCZXJyeV0gU2tl bGV0b24gY29kZSBvZiBBc3luY0ZpbGVTeXN0ZW1CbGFja0JlcnJ5LmNwcApkaWZmIC0tZ2l0IGEv VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbWFuYWdlci5w eSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL21hbmFn ZXIucHkKaW5kZXggZWQ3ODZlNTNkMTg2ZjM4MWQ1Yjc3MDQ0NTJhMThiZjJlNGM2NTQ2ZC4uMWNl ZDJlZjViOWU5NjQ1OWFmZjMzOTI1MjFlZmMzYjA2OWU2NDVkMSAxMDA2NDQKLS0tIGEvVG9vbHMv U2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbWFuYWdlci5weQorKysg Yi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9tYW5hZ2Vy LnB5CkBAIC04OTgsNyArODk4LDggQEAgY2xhc3MgTWFuYWdlcihvYmplY3QpOgogCiAgICAgICAg ICMgV2UgZXhjbHVkZSB0aGUgY3Jhc2hlcyBmcm9tIHRoZSBsaXN0IG9mIHJlc3VsdHMgdG8gcmV0 cnksIGJlY2F1c2UKICAgICAgICAgIyB3ZSB3YW50IHRvIHRyZWF0IGV2ZW4gYSBwb3RlbnRpYWxs eSBmbGFreSBjcmFzaCBhcyBhbiBlcnJvci4KLSAgICAgICAgZmFpbHVyZXMgPSBzZWxmLl9nZXRf ZmFpbHVyZXMocmVzdWx0X3N1bW1hcnksIGluY2x1ZGVfY3Jhc2hlcz1GYWxzZSwgaW5jbHVkZV9t aXNzaW5nPUZhbHNlKQorICAgICAgICAjIE9uIEFwcGxlIE1hYywgd2UgcmV0cnkgY3Jhc2hlcyBk dWUgdG8gaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTgyMjMzCisgICAg ICAgIGZhaWx1cmVzID0gc2VsZi5fZ2V0X2ZhaWx1cmVzKHJlc3VsdF9zdW1tYXJ5LCBpbmNsdWRl X2NyYXNoZXM9c2VsZi5fcG9ydC5zaG91bGRfcmV0cnlfY3Jhc2hlcygpLCBpbmNsdWRlX21pc3Np bmc9RmFsc2UpCiAgICAgICAgIHJldHJ5X3N1bW1hcnkgPSByZXN1bHRfc3VtbWFyeQogICAgICAg ICB3aGlsZSAobGVuKGZhaWx1cmVzKSBhbmQgc2VsZi5fb3B0aW9ucy5yZXRyeV9mYWlsdXJlcyBh bmQgbm90IHNlbGYuX3JldHJ5aW5nIGFuZCBub3QgaW50ZXJydXB0ZWQgYW5kIG5vdCBrZXlib2Fy ZF9pbnRlcnJ1cHRlZCk6CiAgICAgICAgICAgICBfbG9nLmluZm8oJycpCmRpZmYgLS1naXQgYS9U b29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jhc2UucHkgYi9Ub29scy9T Y3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jhc2UucHkKaW5kZXggMDQxMTI3NWRh Njk2M2VhY2I2MTQ0N2I3NDdiMDM5ZjIyNWYxOWY0MC4uY2YyOTg4NDQ1ZWMzYjJkMDU4ZmNiNzlm MDYzZWVlNGEwYWJlMWViMCAxMDA3NTUKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlv dXRfdGVzdHMvcG9ydC9iYXNlLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL3BvcnQvYmFzZS5weQpAQCAtMTU0LDYgKzE1NCw5IEBAIGNsYXNzIFBvcnQob2JqZWN0 KToKICAgICAgICAgICAgIHNlbGYuX3ByZXR0eV9wYXRjaF9hdmFpbGFibGUgPSBzZWxmLmNoZWNr X3ByZXR0eV9wYXRjaChsb2dnaW5nPUZhbHNlKQogICAgICAgICByZXR1cm4gc2VsZi5fcHJldHR5 X3BhdGNoX2F2YWlsYWJsZQogCisgICAgZGVmIHNob3VsZF9yZXRyeV9jcmFzaGVzKHNlbGYpOgor ICAgICAgICByZXR1cm4gRmFsc2UKKwogICAgIGRlZiBkZWZhdWx0X2NoaWxkX3Byb2Nlc3Nlcyhz ZWxmKToKICAgICAgICAgIiIiUmV0dXJuIHRoZSBudW1iZXIgb2YgRHVtcFJlbmRlclRyZWUgaW5z dGFuY2VzIHRvIHVzZSBmb3IgdGhpcyBwb3J0LiIiIgogICAgICAgICBjcHVfY291bnQgPSBzZWxm Ll9leGVjdXRpdmUuY3B1X2NvdW50KCkKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0 cHkvbGF5b3V0X3Rlc3RzL3BvcnQvbWFjLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlv dXRfdGVzdHMvcG9ydC9tYWMucHkKaW5kZXggZTRjMDIzZmU3ZjM0ZWZkMmMwMTE2YzA5YTkyNWM4 MDZiOTE0ZmM4ZS4uNWY4YTExZjA4M2YyOWZmNDMzZWRmMDk3YmQ3YTA0MDEwZWI2YWI2OSAxMDA2 NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9tYWMucHkK KysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9tYWMucHkKQEAg LTU3LDYgKzU3LDkgQEAgY2xhc3MgTWFjUG9ydChBcHBsZVBvcnQpOgogICAgICAgICAjIFRoaXMg cmVwcmVzZW50cyB0aGUgbW9zdCByZWNlbnRseS1zaGlwcGluZyB2ZXJzaW9uIG9mIHRoZSBvcGVy YXRpbmcgc3lzdGVtLgogICAgICAgICByZXR1cm4gc2VsZi5WRVJTSU9OX0ZBTExCQUNLX09SREVS Wy0yXQogCisgICAgZGVmIHNob3VsZF9yZXRyeV9jcmFzaGVzKHNlbGYpOgorICAgICAgICByZXR1 cm4gVHJ1ZQorCiAgICAgZGVmIGJhc2VsaW5lX3BhdGgoc2VsZik6CiAgICAgICAgIGlmIHNlbGYu bmFtZSgpID09IHNlbGYuX21vc3RfcmVjZW50X3ZlcnNpb24oKToKICAgICAgICAgICAgICMgQmFz ZWxpbmVzIGZvciB0aGUgbW9zdCByZWNlbnRseSBzaGlwaW5nIHZlcnNpb24gc2hvdWxkIGdvIGlu dG8gJ21hYycsIG5vdCAnbWFjLWZvbycuCg==