Bug 47293
| Summary: | run-webkit-tests should identify if new failing/passing platform specific results match general expected results | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Joseph Pecoraro <joepeck> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | NEW | ||
| Severity: | Normal | CC: | ap, ddkilzer, simon.fraser |
| Priority: | P2 | ||
| Version: | 528+ (Nightly build) | ||
| Hardware: | PC | ||
| OS: | OS X 10.5 | ||
Joseph Pecoraro
If a platform has expected results for a failing test, and eventually that test starts passing,
it would be great if run-webkit-tests could point out that the new results match the
general, non-platform-specific results.
Example:
LayoutTests/platform/fake-platform/fast/blah-expected.txt => "FAIL - not yet supported"
LayoutTests/fast/blah-expected.txt => "PASS"
After implementing "blah" and running tests you could see something like:
Tests where results did not match expected results:
fast/blah.html expected | actual | diff | pretty diff | *
Tests now matching non-platform-specific results:
fast/blah.html
It would also be helpful if run-webkit-tests pointed out that the results are platform specific
for a dumpAsText() test.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
David Kilzer (:ddkilzer)
Other possibilities:
- Have run-webkit-tests (with a special flag) not only mark the test as passing for this mode, but also delete the platform-specific result on disk.
- Have run-webkit-tests add a --lint-results switch (or similar) that would (for a given platform) check that its most specific result matches the immediate upstream result and (optionally delete it when that happens).
d
David Kilzer (:ddkilzer)
Maybe this is just another way of running --new-baseline, though?
See Bug 141913: run-webkit-tests --new-baseline should not write new result (or delete existing result) if upstream results match