96041 2012-09-06 16:56:25 -0700 Chromium Linux EWS bots and CQ bots are flaky 2012-09-17 13:50:24 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED WORKSFORME P2 Normal --- 1 tony tony abarth dpranke jamesr japhet wjmaclean oldest_to_newest 713863 0 tony 2012-09-06 16:56:25 -0700 The bots keep failing the layout tests and retrying. This is causing the queue to get really slow. Filing this bug for tracking and discussion. Looking at the logs, it looks like the platform/chromium-linux/compositing/gestures are often failing image diffs. I ssh'ed to the machine and looked at the results. The actual results for some of those tests are a solid black 800x600 png. We don't see this failure on the build.webkit.org or build.chromium.org waterfalls. These tests should use the software path, right? I see a few other failures, but it's not clear to me if the bots would process faster if we marked platform/chromium-linux/compositing/gestures as flaky. 713864 1 tony 2012-09-06 16:57:16 -0700 The platform/chromium-linux/compositing/gestures tests were added on Aug 22. It's not clear to me if the flakiness started around then or after that. 713869 2 jamesr 2012-09-06 17:06:00 -0700 Because there's "compositing" in the path these will use the h/w path (which is backed by osmesa). These are new tests and I'm not shocked that they are kind of messed up. Let's skip them or mark them flaky and let wjmaclean@ work on fixing them. They aren't worth holding everything else up. 713878 3 162623 tony 2012-09-06 17:14:40 -0700 Created attachment 162623 Patch 713880 4 162623 abarth 2012-09-06 17:18:24 -0700 Comment on attachment 162623 Patch ok 713881 5 abarth 2012-09-06 17:18:31 -0700 Thanks for investigating. 713887 6 tony 2012-09-06 17:20:51 -0700 Committed r127803: <http://trac.webkit.org/changeset/127803> 713889 7 162623 tony 2012-09-06 17:21:38 -0700 Comment on attachment 162623 Patch This is just speculative, so I'm keeping the bug open. Hopefully the cr-linux queue will clear overnight. 713959 8 tony 2012-09-06 18:25:02 -0700 Looking at the CQ now, there are 5 runs that failed. Fails a bunch of compositing tests: http://webkit-commit-queue.appspot.com/results/13778383 2 http cache tests with missing results: http://webkit-commit-queue.appspot.com/results/13775546 http://webkit-commit-queue.appspot.com/results/13785213 http://webkit-commit-queue.appspot.com/results/13785209 http://webkit-commit-queue.appspot.com/results/13765808 I wonder if the http cache tests is related to https://bugs.webkit.org/show_bug.cgi?id=93195 . Not sure why they suddenly became flaky. 714645 9 tony 2012-09-07 10:05:27 -0700 Looking at the ews bot, 2 http tests seem super flaky: http/tests/cache/stopped-revalidation.html = MISSING http/tests/cache/subresource-expiration-1.html = MISSING Here are the diffs: http://pastebin.com/hLfRDTmp http://pastebin.com/vk2uz3dh Looks like neither test is registering dumpAsText() and the second test is getting the output from the first test. I think we have a bug for tests getting out of sync. I'm going to mark these 2 tests as flaky while we investigate. It looks like notifyDone is getting out of sync with the tests. Maybe we're not properly clearing the work queue between tests? 714654 10 tony 2012-09-07 10:08:46 -0700 http://trac.webkit.org/changeset/127883 714712 11 jamesr 2012-09-07 10:39:46 -0700 One of the platform/chromium-linux/compositing/gestures tests involves a navigation - perhaps it's mucking things up? 714777 12 tony 2012-09-07 11:19:09 -0700 Now I'm seeing http/tests/cache/subresource-expiration-2.html = MISSING http/tests/cache/subresource-failover-to-network.html = MISSING But I am able to repro with: new-run-webkit-tests --no-new-test-results --skip-failing-tests --verbose http I'll do some digging . . . 714792 13 tony 2012-09-07 11:28:51 -0700 http://trac.webkit.org/changeset/127897 Turns out that http/tests/cache/cancel-during-revalidation-succeeded.html is causing the 2 following tests to fail. Skipping cancel-during-revalidation-succeeded.html seems to fix the problem on my machine. Nate, do you think you can take a look? 714944 14 tony 2012-09-07 13:59:29 -0700 http://trac.webkit.org/changeset/127916 is a revert of http://trac.webkit.org/changeset/127803, which skipped the compositing/gestures tests. Other compositing tests were failing the same way, so I put that back. The cr-linux ews bot seems to be running smoother since skipping the http test, even with the compositing test failures. 715210 15 jamesr 2012-09-07 18:36:19 -0700 Skipped the directory in http://trac.webkit.org/changeset/127954. Let's see if that helps. James - can you please take a look at this when you get a chance? If it does turn out to be these tests then I'm pretty sure that indicates a real problem in the code they test that we need to address. 715989 16 wjmaclean 2012-09-10 06:00:23 -0700 (In reply to comment #15) > Skipped the directory in http://trac.webkit.org/changeset/127954. Let's see if that helps. James - can you please take a look at this when you get a chance? If it does turn out to be these tests then I'm pretty sure that indicates a real problem in the code they test that we need to address. Sure, I'll look and see what's going on. 716187 17 abarth 2012-09-10 09:56:11 -0700 We're still getting failures in platform/chromium/compositing 722142 18 tony 2012-09-17 12:05:31 -0700 The bots have been running OK for the past week. Maybe we should file separate bugs for the flaky HTTP test and the compositing tests and close this bug out? 722145 19 abarth 2012-09-17 12:08:14 -0700 SGTM 722238 20 tony 2012-09-17 13:50:24 -0700 https://bugs.webkit.org/show_bug.cgi?id=96950 https://bugs.webkit.org/show_bug.cgi?id=96951 162623 2012-09-06 17:14:40 -0700 2012-09-06 17:21:38 -0700 Patch bug-96041-20120906171420.patch text/plain 1514 tony U3VidmVyc2lvbiBSZXZpc2lvbjogMTI3ODAyCmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9DaGFu Z2VMb2cgYi9MYXlvdXRUZXN0cy9DaGFuZ2VMb2cKaW5kZXggM2Y4ODhiY2ZkMjY2MTY1MjkwMjRk NjFiMWMwZGQ1MTVmZTM1MDk1MS4uOTYyNGViOGU2MTM3ZTJmNmQ3Yzg3NTBkOWRiODBmZGEwODc5 ZDE0OSAxMDA2NDQKLS0tIGEvTGF5b3V0VGVzdHMvQ2hhbmdlTG9nCisrKyBiL0xheW91dFRlc3Rz L0NoYW5nZUxvZwpAQCAtMSwzICsxLDE3IEBACisyMDEyLTA5LTA2ICBUb255IENoYW5nICA8dG9u eUBjaHJvbWl1bS5vcmc+CisKKyAgICAgICAgQ2hyb21pdW0gTGludXggRVdTIGJvdHMgYW5kIENR IGJvdHMgYXJlIGZsYWt5CisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVn LmNnaT9pZD05NjA0MQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisg ICAgICAgIFNraXAgcGxhdGZvcm0vY2hyb21pdW0tbGludXgvY29tcG9zaXRpbmcvZ2VzdHVyZXMg dGVzdHMgb24gdGhlCisgICAgICAgIENocm9taXVtIExpbnV4IEVXUyBhbmQgQ1EgYm90cy4gQnkg bWFya2luZyB0aGVtIGFzIGZsYWt5LCB0aGV5IHdvbid0CisgICAgICAgIGJlIHJ1biAodGhlIGJv dHMgdXNlIC0tc2tpcC1mYWlsaW5nLXRlc3RzIHdoaWNoIGNhdXNlcyB0aGVtIHRvIHNraXAKKyAg ICAgICAgdGhlc2UgdGVzdHMpLiAgVGhlc2Ugd2lsbCBzdGlsbCBydW4gb24gdGhlIHdhdGVyZmFs bCBib3RzLgorCisgICAgICAgICogcGxhdGZvcm0vY2hyb21pdW0vVGVzdEV4cGVjdGF0aW9uczoK KwogMjAxMi0wOS0wNiAgQWRhbSBCYXJ0aCAgPGFiYXJ0aEB3ZWJraXQub3JnPgogCiAgICAgICAg IE5ldyB0ZXN0cyBpbnRyb2R1Y2VkIGluIHIxMjc3MDQgZmFpbApkaWZmIC0tZ2l0IGEvTGF5b3V0 VGVzdHMvcGxhdGZvcm0vY2hyb21pdW0vVGVzdEV4cGVjdGF0aW9ucyBiL0xheW91dFRlc3RzL3Bs YXRmb3JtL2Nocm9taXVtL1Rlc3RFeHBlY3RhdGlvbnMKaW5kZXggN2YzNjM1MjA0NjczN2U1ODg0 Mzk0ZWI5MjQ5ZTJlNWJhOWUxZTg5MC4uZDg0MTZiMzA0M2VjYzg0ZGFhOTY3YmM1NWNlMDNiMGJl NDM1ZmY5NyAxMDA2NDQKLS0tIGEvTGF5b3V0VGVzdHMvcGxhdGZvcm0vY2hyb21pdW0vVGVzdEV4 cGVjdGF0aW9ucworKysgYi9MYXlvdXRUZXN0cy9wbGF0Zm9ybS9jaHJvbWl1bS9UZXN0RXhwZWN0 YXRpb25zCkBAIC0zNTk0LDMgKzM1OTQsNSBAQCBCVUdDUjE0NjQwMSBBTkRST0lEIDogZmFzdC9j c3Mvc3RpY2t5L3N0aWNreS13cml0aW5nLW1vZGUtdmVydGljYWwtbHIuaHRtbCA9IElNQQogQlVH V0s5NTc5OSBNQUMgOiB0b3VjaGFkanVzdG1lbnQvaWZyYW1lLWJvdW5kYXJ5Lmh0bWwgPSBURVhU CiAKIEJVR1dLOTU4MTMgOiBmYXN0L2lubmVySFRNTC9pbm5lckhUTUwtaWZyYW1lLmh0bWwgPSBU RVhUIFBBU1MKKworQlVHV0s5NjA0MSBMSU5VWCA6IHBsYXRmb3JtL2Nocm9taXVtLWxpbnV4L2Nv bXBvc2l0aW5nL2dlc3R1cmVzID0gUEFTUyBJTUFHRQo=