NEW 73978
Ease specifying narrow test_expectations lines for multiple failures on same test 
https://bugs.webkit.org/show_bug.cgi?id=73978
Summary Ease specifying narrow test_expectations lines for multiple failures on same ...
Vincent Scheib
Reported 2011-12-06 20:36:50 PST
It is best to define a test expectation as narrowly as possible such that configurations that are succeeding continue to provide regression detection. This can be challenging when a test is failing in multiple ways. E.g. if debug versions are SLOW but all LEOPARD CPU-CG builders are crashing. Either new-run-webkit-tests could tolerate overlapping expectations for tests, e.g.: BUG LEOPARD : test.html = CRASH PASS BUG MAC RELEASE : test.html = TIMEOUT PASS Or tools such as the chrome flakiness dashboard could generate suggestions of how to narrowly define failures in non-overlapping ways.
Attachments
Add attachment proposed patch, testcase, etc.
Dirk Pranke
Comment 1 2012-09-21 16:11:04 PDT
I think the general consensus these days is that the TestExpectations-managing infrastructure is too complicated and we need to simplify things more, so I'm not inclined to add more new syntax to support this or to tolerate overlapping expectations. I think it is interesting to consider improving the tools to make sure that expectations are as narrowly scoped as possible, though. Does that seem reasonable to you?
Note You need to log in before you can comment on or make changes to this bug.