[chromium] Updated test expectations to match the bots using new auto-update script.
Created attachment 68656 [details] Patch
Comment on attachment 68656 [details] Patch Clearing flags on attachment: 68656 Committed r68244: <http://trac.webkit.org/changeset/68244>
All reviewed patches have been landed. Closing bug.
Where can I learn more about this script? What will the work flow be for converting BUG_AUTO into bugs? As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug.
Ojan and I have been working on improving the webkit/tools/layout_tests/webkitpy/layout_tests/update_expectations_from_dashboard.py script. Its usage is undocumented, but in short it takes JSON data as input generated from http://test-results.appspot.com/dashboards/flakiness_dashboard.html#expectationsUpdate=true and uses it to modify test_expectations.txt. The modifications to the script are unreviewed, but once committed I'll add a section to the wiki explaining how to use it at http://dev.chromium.org/developers/testing/flakiness-dashboard. > What will the work flow be for converting BUG_AUTO into bugs? I'm not sure - Ojan, can you chime in? > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. Yep, that sounds like a good idea. Could you provide a more specific heuristic?
(In reply to comment #5) > > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. > > Yep, that sounds like a good idea. Could you provide a more specific heuristic? I'm not sure. I just bring it up because it looks like the last 5 entries to test_expectations.txt are just slow tests. But maybe not. One of the tests is for Release builds.
(In reply to comment #4) > Where can I learn more about this script? The working version is still not checked in. We'll document it once it's usable. This was our first pass at actually using it. > What will the work flow be for converting BUG_AUTO into bugs? That's a good question. I don't have a good answer. I was planning on bringing this up on chromium-dev soon. I'm open to suggestions. > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. I've been meaning to get rid of SLOW. I think it's too complicated. Instead, we should just have a long timeout but give a short timeout to tests that we expect to timeout. How does that sound?
(In reply to comment #7) > (In reply to comment #4) > > What will the work flow be for converting BUG_AUTO into bugs? > > That's a good question. I don't have a good answer. I was planning on bringing this up on chromium-dev soon. I'm open to suggestions. I think whoever runs the tool should fill in bug numbers, right? We shouldn't check in expectations without bugs filed. > > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug. > > I've been meaning to get rid of SLOW. I think it's too complicated. Instead, we should just have a long timeout but give a short timeout to tests that we expect to timeout. How does that sound? I suspect that over time, more tests will time out (the long timeout) and the full test run will gradually get slower. I prefer fast by default with exceptions to make things slower. Maybe you're hoping that the auto-update script will detect and mark tests as slow? I'm not sure how easy that will be to do.