Bug 46452 – [chromium] Updated test expectations to match the bots using new auto-update script.

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[James Kozianski]

[chromium] Updated test expectations to match the bots using new auto-update script.

[James Kozianski]

Created attachment 68656 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 68656 [details]
Patch

Clearing flags on attachment: 68656

Committed r68244: <http://trac.webkit.org/changeset/68244>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Tony Chang]

Where can I learn more about this script?  What will the work flow be for converting BUG_AUTO into bugs?

As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug.

[James Kozianski]

Ojan and I have been working on improving the webkit/tools/layout_tests/webkitpy/layout_tests/update_expectations_from_dashboard.py script. Its usage is undocumented, but in short it takes JSON data as input generated from http://test-results.appspot.com/dashboards/flakiness_dashboard.html#expectationsUpdate=true and uses it to modify test_expectations.txt.

The modifications to the script are unreviewed, but once committed I'll add a section to the wiki explaining how to use it at http://dev.chromium.org/developers/testing/flakiness-dashboard.

> What will the work flow be for converting BUG_AUTO into bugs?

I'm not sure - Ojan, can you chime in?


> As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug.

Yep, that sounds like a good idea. Could you provide a more specific heuristic?

[Tony Chang]

(In reply to comment #5)
> > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug.
> 
> Yep, that sounds like a good idea. Could you provide a more specific heuristic?

I'm not sure.  I just bring it up because it looks like the last 5 entries to test_expectations.txt are just slow tests.  But maybe not.  One of the tests is for Release builds.

[Ojan Vafai]

(In reply to comment #4)
> Where can I learn more about this script?  

The working version is still not checked in. We'll document it once it's usable. This was our first pass at actually using it.

> What will the work flow be for converting BUG_AUTO into bugs?

That's a good question. I don't have a good answer. I was planning on bringing this up on chromium-dev soon. I'm open to suggestions.

> As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug.

I've been meaning to get rid of SLOW. I think it's too complicated. Instead, we should just have a long timeout but give a short timeout to tests that we expect to timeout. How does that sound?

[Tony Chang]

(In reply to comment #7)
> (In reply to comment #4)
> > What will the work flow be for converting BUG_AUTO into bugs?
> 
> That's a good question. I don't have a good answer. I was planning on bringing this up on chromium-dev soon. I'm open to suggestions.

I think whoever runs the tool should fill in bug numbers, right?  We shouldn't check in expectations without bugs filed.

> > As a feature request, maybe it can guess when to use SLOW if the test is only failing in debug.
> 
> I've been meaning to get rid of SLOW. I think it's too complicated. Instead, we should just have a long timeout but give a short timeout to tests that we expect to timeout. How does that sound?

I suspect that over time, more tests will time out (the long timeout) and the full test run will gradually get slower.  I prefer fast by default with exceptions to make things slower.

Maybe you're hoping that the auto-update script will detect and mark tests as slow?  I'm not sure how easy that will be to do.