Bug 73978
| Summary: | Ease specifying narrow test_expectations lines for multiple failures on same test | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Vincent Scheib <scheib> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | NEW | ||
| Severity: | Normal | CC: | dpranke |
| Priority: | P2 | Keywords: | NRWT |
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
Vincent Scheib
It is best to define a test expectation as narrowly as possible such that configurations that are succeeding continue to provide regression detection. This can be challenging when a test is failing in multiple ways. E.g. if debug versions are SLOW but all LEOPARD CPU-CG builders are crashing.
Either new-run-webkit-tests could tolerate overlapping expectations for tests, e.g.:
BUG LEOPARD : test.html = CRASH PASS
BUG MAC RELEASE : test.html = TIMEOUT PASS
Or tools such as the chrome flakiness dashboard could generate suggestions of how to narrowly define failures in non-overlapping ways.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Dirk Pranke
I think the general consensus these days is that the TestExpectations-managing infrastructure is too complicated and we need to simplify things more, so I'm not inclined to add more new syntax to support this or to tolerate overlapping expectations.
I think it is interesting to consider improving the tools to make sure that expectations are as narrowly scoped as possible, though.
Does that seem reasonable to you?