|Summary:||Cleanup: Expected Results Where No Test Exists|
|Product:||WebKit||Reporter:||Joseph Pecoraro <joepeck>|
|Component:||Tools / Tests||Assignee:||Nobody <webkit-unassigned>|
|Severity:||Normal||CC:||ddkilzer, dev+webkit, joepeck|
|Version:||528+ (Nightly build)|
|OS:||OS X 10.5|
Description Joseph Pecoraro 2011-03-01 11:05:50 PST
Orphaned files? This detects "-expected.*" files which doesn't seem to have an associated test file.
Comment 1 Joseph Pecoraro 2011-03-01 11:06:52 PST
Created attachment 84252 [details] [SCRIPT] Find Expected Results Without Test Script I used to find the issues.
Comment 2 Joseph Pecoraro 2011-03-01 11:10:05 PST
Created attachment 84253 [details] [OUTPUT] Missing Tests Summary: 426 expected results missing tests. 500 unpaired pngs/checksums. Should all "-expected.png" files have an "-expected.checksum"? I print those out as warnings. Also, if there is a test in "platform/x/.../foo.html" I assume that test can never have expected results outside of platform x. If that isn't the case, like a chain of platform dependencies, then this could be slightly wrong.
Comment 3 David Kilzer (:ddkilzer) 2011-03-02 07:38:17 PST
(In reply to comment #2) > Should all "-expected.png" files have an "-expected.checksum"? I print those > out as warnings. Yes, that's the fast path for determining if -actual.png and -expected.png results match. > Also, if there is a test in "platform/x/.../foo.html" I assume that test can never > have expected results outside of platform x. If that isn't the case, like a > chain of platform dependencies, then this could be slightly wrong. That is correct. Although I suppose platform/mac could have different results in platform/mac-leopard and platform/mac-snowleopard, although I'm not sure whether [old-]run-webkit-tests actually handles this or not.
Comment 4 David Kilzer (:ddkilzer) 2011-03-02 07:39:40 PST
(In reply to comment #1) > Created an attachment (id=84252) [details] > [SCRIPT] Find Expected Results Without Test > > Script I used to find the issues. You should also put this up for review!
Comment 5 Joseph Pecoraro 2011-03-03 14:23:16 PST
I noticed at least one issue with my script. There should be a list of fallback directories to search in. # For expected result: LayoutTests/platform/chromium-gpu-win/foo/bar/x-expected.txt we check: # # LayoutTests/platform/chromium-gpu-win/foo/bar/x.* # LayoutTests/platform/chromium-gpu/foo/bar/x.* # LayoutTests/platform/chromium/foo/bar/x.* # LayoutTests/foo/bar/x.* # # If no test file is found its an orphan! Updated this and I now only see: 393 expected results missing tests. 500 unpaired pngs/checksums. I'll do a little more verification before writing a patch to remove the excess files.
Comment 6 Joseph Pecoraro 2011-03-03 14:53:12 PST
Heh, doing a quick verification I found 1 weird expected result. Turns out my script was working correctly: LayoutTests/platform/mac-leopard/platform/mac/svg/batik/text/smallFonts-expected.txt Should be nixed because no test exists at: LayoutTests/platform/mac/svg/batik/text/smallFonts.* So I think I'll write up a patch to remove these.
Comment 7 Joseph Pecoraro 2011-03-04 00:37:39 PST
Turns out the fallback behaviors, at least in run-webkit-tests, are quite weird. They are defined in Tools/Scripts/webkitpy/layout_tests/ports/*. So, a perfect script would need to mimic deduplicate-tests, and follow the idiosyncrasies in the fallback behavior of some ports.