Bug 47293 – run-webkit-tests should identify if new failing/passing platform specific results match general expected results

• Status: NEW
• Component: Tools / Tests

[Joseph Pecoraro]

If a platform has expected results for a failing test, and eventually that test starts passing,
it would be great if run-webkit-tests could point out that the new results match the
general, non-platform-specific results.

Example:

  LayoutTests/platform/fake-platform/fast/blah-expected.txt => "FAIL - not yet supported"
  LayoutTests/fast/blah-expected.txt => "PASS"

After implementing "blah" and running tests you could see something like:

  Tests where results did not match expected results:
  fast/blah.html	expected | actual | diff | pretty diff | *

  Tests now matching non-platform-specific results:
  fast/blah.html

It would also be helpful if run-webkit-tests pointed out that the results are platform specific
for a dumpAsText() test.

[David Kilzer (:ddkilzer)]

Other possibilities:

- Have run-webkit-tests (with a special flag) not only mark the test as passing for this mode, but also delete the platform-specific result on disk.

- Have run-webkit-tests add a --lint-results switch (or similar) that would (for a given platform) check that its most specific result matches the immediate upstream result and (optionally delete it when that happens).
d

[David Kilzer (:ddkilzer)]

Maybe this is just another way of running --new-baseline, though?

See Bug 141913: run-webkit-tests --new-baseline should not write new result (or delete existing result) if upstream results match