46452 2010-09-23 23:53:35 -0700 [chromium] Updated test expectations to match the bots using new auto-update script. 2011-01-18 22:02:37 -0800 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) Other OS X 10.5 RESOLVED FIXED P2 Normal --- 0 koz koz commit-queue noel.gordon ojan tony oldest_to_newest 284261 0 koz 2010-09-23 23:53:35 -0700 [chromium] Updated test expectations to match the bots using new auto-update script. 284263 1 68656 koz 2010-09-23 23:56:49 -0700 Created attachment 68656 Patch 284299 2 68656 commit-queue 2010-09-24 02:13:33 -0700 Comment on attachment 68656 Patch Clearing flags on attachment: 68656 Committed r68244: <http://trac.webkit.org/changeset/68244> 284300 3 commit-queue 2010-09-24 02:13:38 -0700 All reviewed patches have been landed. Closing bug. 284544 4 tony 2010-09-24 11:00:20 -0700 Where can I learn more about this script? What will the work flow be for converting BUG_AUTO into bugs? As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. 284650 5 koz 2010-09-24 13:24:26 -0700 Ojan and I have been working on improving the webkit/tools/layout_tests/webkitpy/layout_tests/update_expectations_from_dashboard.py script. Its usage is undocumented, but in short it takes JSON data as input generated from http://test-results.appspot.com/dashboards/flakiness_dashboard.html#expectationsUpdate=true and uses it to modify test_expectations.txt. The modifications to the script are unreviewed, but once committed I'll add a section to the wiki explaining how to use it at http://dev.chromium.org/developers/testing/flakiness-dashboard. > What will the work flow be for converting BUG_AUTO into bugs? I'm not sure - Ojan, can you chime in? > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. Yep, that sounds like a good idea. Could you provide a more specific heuristic? 284688 6 tony 2010-09-24 14:16:27 -0700 (In reply to comment #5) > > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. > > Yep, that sounds like a good idea. Could you provide a more specific heuristic? I'm not sure. I just bring it up because it looks like the last 5 entries to test_expectations.txt are just slow tests. But maybe not. One of the tests is for Release builds. 285049 7 ojan 2010-09-25 14:02:00 -0700 (In reply to comment #4) > Where can I learn more about this script? The working version is still not checked in. We'll document it once it's usable. This was our first pass at actually using it. > What will the work flow be for converting BUG_AUTO into bugs? That's a good question. I don't have a good answer. I was planning on bringing this up on chromium-dev soon. I'm open to suggestions. > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. I've been meaning to get rid of SLOW. I think it's too complicated. Instead, we should just have a long timeout but give a short timeout to tests that we expect to timeout. How does that sound? 285497 8 tony 2010-09-27 09:29:46 -0700 (In reply to comment #7) > (In reply to comment #4) > > What will the work flow be for converting BUG_AUTO into bugs? > > That's a good question. I don't have a good answer. I was planning on bringing this up on chromium-dev soon. I'm open to suggestions. I think whoever runs the tool should fill in bug numbers, right? We shouldn't check in expectations without bugs filed. > > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. > > I've been meaning to get rid of SLOW. I think it's too complicated. Instead, we should just have a long timeout but give a short timeout to tests that we expect to timeout. How does that sound? I suspect that over time, more tests will time out (the long timeout) and the full test run will gradually get slower. I prefer fast by default with exceptions to make things slower. Maybe you're hoping that the auto-update script will detect and mark tests as slow? I'm not sure how easy that will be to do. 68656 2010-09-23 23:56:49 -0700 2010-09-24 02:13:33 -0700 Patch bug-46452-20100924165646.patch text/plain 4168 koz ZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL0NoYW5nZUxvZyBiL0xheW91dFRlc3RzL0NoYW5nZUxv ZwppbmRleCA5YzdhZWEzOTE4OTVkYjc0NzljNzhlMDM4MTM3YjIyNGM1YWJmNDJjLi45ZmVkMzg2 ZDk0OGM1YzZjOGZkN2Y4MWE4YzM3MDI2ZWEyMDNjYWE5IDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0 cy9DaGFuZ2VMb2cKKysrIGIvTGF5b3V0VGVzdHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTQgQEAK KzIwMTAtMDktMjMgIEphbWVzIEtvemlhbnNraSAgPGtvekBjaHJvbWl1bS5vcmc+CisKKyAgICAg ICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgW2Nocm9taXVtXSBVcGRh dGVkIHRlc3QgZXhwZWN0YXRpb25zIHRvIG1hdGNoIHRoZSBib3RzIHVzaW5nIG5ldyBhdXRvLXVw ZGF0ZSBzY3JpcHQuCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNn aT9pZD00NjQ1MgorCisgICAgICAgIFVwZGF0ZWQgZXhwZWN0YXRpb25zIHRvIG1hdGNoIHRoZSBi b3RzIHVzaW5nIHRoZSBuZXcgYXV0by11cGRhdGUgc2NyaXB0LgorCisgICAgICAgICogcGxhdGZv cm0vY2hyb21pdW0vdGVzdF9leHBlY3RhdGlvbnMudHh0OgorCiAyMDEwLTA5LTIzICBTaW1vbiBG cmFzZXIgIDxzaW1vbi5mcmFzZXJAYXBwbGUuY29tPgogCiAgICAgICAgIFJldmlld2VkIGJ5IERh biBCZXJuc3RlaW4uCmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9wbGF0Zm9ybS9jaHJvbWl1bS90 ZXN0X2V4cGVjdGF0aW9ucy50eHQgYi9MYXlvdXRUZXN0cy9wbGF0Zm9ybS9jaHJvbWl1bS90ZXN0 X2V4cGVjdGF0aW9ucy50eHQKaW5kZXggOTMxY2MxNmMxYjY5MTRhNTI3YzUxZTJiNTgxYjQ3MzA5 YWVlNGIxNi4uZWYyYjM1MGFiY2RhODllNzliNDFkZjdlNDhlZTY5NjJmN2E4ODIyMyAxMDA2NDQK LS0tIGEvTGF5b3V0VGVzdHMvcGxhdGZvcm0vY2hyb21pdW0vdGVzdF9leHBlY3RhdGlvbnMudHh0 CisrKyBiL0xheW91dFRlc3RzL3BsYXRmb3JtL2Nocm9taXVtL3Rlc3RfZXhwZWN0YXRpb25zLnR4 dApAQCAtMjQ5Myw3ICsyNDkzLDkgQEAgQlVHNDU5OTEgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJk LmZpbGxTdHlsZS5wYXJzZS5jdXJyZW50LmJhc2ljLmh0bWwgPSBURVhUCiBCVUc0NTk5MSA6IGNh bnZhcy9waGlsaXAvdGVzdHMvMmQuZmlsbFN0eWxlLnBhcnNlLmN1cnJlbnQuY2hhbmdlZC5odG1s ID0gVEVYVAogQlVHNDU5OTEgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLmZpbGxTdHlsZS5wYXJz ZS5jdXJyZW50LnJlbW92ZWQuaHRtbCA9IFRFWFQKIEJVRzQ1OTkxIDogY2FudmFzL3BoaWxpcC90 ZXN0cy8yZC5maWxsU3R5bGUucGFyc2Uuc3lzdGVtLmh0bWwgPSBURVhUCi1CVUc0NTk5MSA6IGNh bnZhcy9waGlsaXAvdGVzdHMvMmQuZ3JhZGllbnQucmFkaWFsLmNvbmUudG9wLmh0bWwgPSBURVhU IFRJTUVPVVQKK0JVRzQ1OTkxIE1BQyBXSU4gREVCVUcgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJk LmdyYWRpZW50LnJhZGlhbC5jb25lLnRvcC5odG1sID0gVEVYVAorQlVHNDU5OTEgUkVMRUFTRSA6 IGNhbnZhcy9waGlsaXAvdGVzdHMvMmQuZ3JhZGllbnQucmFkaWFsLmNvbmUudG9wLmh0bWwgPSBU RVhUCitCVUc0NTk5MSBMSU5VWCBERUJVRyA6IGNhbnZhcy9waGlsaXAvdGVzdHMvMmQuZ3JhZGll bnQucmFkaWFsLmNvbmUudG9wLmh0bWwgPSBURVhUIFRJTUVPVVQKIEJVRzQ1OTkxIDogY2FudmFz L3BoaWxpcC90ZXN0cy8yZC5ncmFkaWVudC5yYWRpYWwubmVnYXRpdmUuaHRtbCA9IFRFWFQKIEJV RzQ1OTkxIDogY2FudmFzL3BoaWxpcC90ZXN0cy8yZC5pbWFnZURhdGEuY3JlYXRlMS50eXBlLmh0 bWwgPSBURVhUCiBCVUc0NTk5MSA6IGNhbnZhcy9waGlsaXAvdGVzdHMvMmQuaW1hZ2VEYXRhLmNy ZWF0ZTIudHlwZS5odG1sID0gVEVYVApAQCAtMjUwMiw5ICsyNTA0LDkgQEAgQlVHNDU5OTEgOiBj YW52YXMvcGhpbGlwL3Rlc3RzLzJkLmltYWdlRGF0YS5nZXQudHlwZS5odG1sID0gVEVYVAogQlVH NDU5OTEgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLmltYWdlRGF0YS5vYmplY3Qucm91bmQuaHRt bCA9IFRFWFQKIEJVRzQ1OTkxIDogY2FudmFzL3BoaWxpcC90ZXN0cy8yZC5pbWFnZURhdGEub2Jq ZWN0LndyYXAuaHRtbCA9IFRFWFQKIEJVRzQ1OTkxIDogY2FudmFzL3BoaWxpcC90ZXN0cy8yZC5t aXNzaW5nYXJncy5odG1sID0gVEVYVAotQlVHNDU5OTEgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJk LnBhdGguYXJjVG8uY29pbmNpZGUuMS5odG1sID0gVEVYVAorQlVHNDU5OTEgTElOVVggV0lOIERF QlVHIDogY2FudmFzL3BoaWxpcC90ZXN0cy8yZC5wYXRoLmFyY1RvLmNvaW5jaWRlLjEuaHRtbCA9 IFRFWFQKIEJVRzQ1OTkxIDogY2FudmFzL3BoaWxpcC90ZXN0cy8yZC5wYXRoLnJlY3Qud2luZGlu Zy5odG1sID0gVEVYVAotQlVHNDU5OTEgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLnBhdGguc3Ry b2tlLnBydW5lLmFyYy5odG1sID0gVEVYVAorQlVHNDU5OTEgV0lOIERFQlVHIDogY2FudmFzL3Bo aWxpcC90ZXN0cy8yZC5wYXRoLnN0cm9rZS5wcnVuZS5hcmMuaHRtbCA9IFRFWFQKIEJVRzQ1OTkx IDogY2FudmFzL3BoaWxpcC90ZXN0cy8yZC5wYXR0ZXJuLmltYWdlLmJyb2tlbi5odG1sID0gVEVY VAogQlVHNDU5OTEgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLnBhdHRlcm4uaW1hZ2UuaW5jb21w bGV0ZS5odG1sID0gVEVYVAogQlVHNDU5OTEgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLnBhdHRl cm4uaW1hZ2UudW5kZWZpbmVkLmh0bWwgPSBURVhUCkBAIC0yNTc2LDcgKzI1NzgsOCBAQCBCVUc0 NTk5MSBNQUMgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLmdyYWRpZW50LnJhZGlhbC5jb25lLmZy b250Lmh0bWwgPSBURVhUCiBCVUc0NTk5MSBNQUMgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLmdy YWRpZW50LnJhZGlhbC5pbnNpZGUyLmh0bWwgPSBURVhUCiBCVUc0NTk5MSBNQUMgOiBjYW52YXMv cGhpbGlwL3Rlc3RzLzJkLmdyYWRpZW50LnJhZGlhbC5pbnNpZGUzLmh0bWwgPSBURVhUCiBCVUc0 NTk5MSBNQUMgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLmdyYWRpZW50LnJhZGlhbC5vdXRzaWRl MS5odG1sID0gVEVYVAotQlVHNDU5OTEgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLmltYWdlRGF0 YS5nZXQuc291cmNlLm91dHNpZGUuaHRtbCA9IFRFWFQKK0JVRzQ1OTkxIERFQlVHIDogY2FudmFz L3BoaWxpcC90ZXN0cy8yZC5pbWFnZURhdGEuZ2V0LnNvdXJjZS5vdXRzaWRlLmh0bWwgPSBURVhU CitCVUc0NTk5MSBNQUMgUkVMRUFTRSA6IGNhbnZhcy9waGlsaXAvdGVzdHMvMmQuaW1hZ2VEYXRh LmdldC5zb3VyY2Uub3V0c2lkZS5odG1sID0gVEVYVAogQlVHNDU5OTEgTUFDIDogY2FudmFzL3Bo aWxpcC90ZXN0cy8yZC5saW5lLmNhcC5jbG9zZWQuaHRtbCA9IFRFWFQKIEJVRzQ1OTkxIE1BQyA6 IGNhbnZhcy9waGlsaXAvdGVzdHMvMmQubGluZS5qb2luLnBhcmFsbGVsLmh0bWwgPSBURVhUCiBC VUc0NTk5MSBNQUMgOiBjYW52YXMvcGhpbGlwL3Rlc3RzLzJkLmxpbmUubWl0ZXIubGluZWVkZ2Uu aHRtbCA9IFRFWFQKQEAgLTMxMDAsMyArMzEwMyw5IEBAIEJVR0pBUEhFVCA6IGh0dHAvdGVzdHMv bmF2aWdhdGlvbi9waW5nLWNyb3NzLW9yaWdpbi5odG1sID0gVEVYVCBUSU1FT1VUCiAKIC8vIFN0 YXJ0ZWQgY3Jhc2hpbmcgYXMgcGFydCBvZiB0aGUgV0sgcm9sbCA2ODEwMDo2ODE3OC4KIEJVRzU2 NzMyIDogc3RvcmFnZS9pbmRleGVkZGIvdHJhbnNhY3Rpb24tYmFzaWNzLmh0bWwgPSBDUkFTSAor CitCVUdfQVVUTyBMSU5VWCBXSU4gREVCVUcgOiBhbmltYXRpb25zL3N1c3BlbmQtcmVzdW1lLWFu aW1hdGlvbi5odG1sID0gUEFTUyBURVhUCitCVUdfQVVUTyBMSU5VWCBSRUxFQVNFIDogYW5pbWF0 aW9ucy8zZC90cmFuc2Zvcm0tb3JpZ2luLXZzLWZ1bmN0aW9ucy5odG1sID0gUEFTUyBURVhUCitC VUdfQVVUTyBXSU4gREVCVUcgOiBhbmltYXRpb25zLzNkL3RyYW5zZm9ybS1vcmlnaW4tdnMtZnVu Y3Rpb25zLmh0bWwgPSBQQVNTIFRFWFQKK0JVR19BVVRPIFdJTiBERUJVRyA6IGFuaW1hdGlvbnMv bXVsdGlwbGUta2V5ZnJhbWVzLmh0bWwgPSBQQVNTIFRFWFQKK0JVR19BVVRPIFdJTiBERUJVRyA6 IGNhbnZhcy9waGlsaXAvdGVzdHMvMmQudGV4dC5kcmF3LmFsaWduLmNlbnRlci5odG1sID0gUEFT UyBURVhUCg==