Bug 55485

Summary: Cleanup: Expected Results Where No Test Exists
Product: WebKit Reporter: Joseph Pecoraro <joepeck>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: ddkilzer, dev+webkit, joepeck
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: PC   
OS: OS X 10.5   
Attachments:
Description Flags
[SCRIPT] Find Expected Results Without Test
none
[OUTPUT] Missing Tests none

Description Joseph Pecoraro 2011-03-01 11:05:50 PST
Orphaned files? This detects "-expected.*" files which doesn't seem to have an associated test file.
Comment 1 Joseph Pecoraro 2011-03-01 11:06:52 PST
Created attachment 84252 [details]
[SCRIPT] Find Expected Results Without Test

Script I used to find the issues.
Comment 2 Joseph Pecoraro 2011-03-01 11:10:05 PST
Created attachment 84253 [details]
[OUTPUT] Missing Tests

Summary:

    426 expected results missing tests.
    500 unpaired pngs/checksums.

Should all "-expected.png" files have an "-expected.checksum"? I print those
out as warnings.

Also, if there is a test in "platform/x/.../foo.html" I assume that test can never
have expected results outside of platform x. If that isn't the case, like a
chain of platform dependencies, then this could be slightly wrong.
Comment 3 David Kilzer (:ddkilzer) 2011-03-02 07:38:17 PST
(In reply to comment #2)
> Should all "-expected.png" files have an "-expected.checksum"? I print those
> out as warnings.

Yes, that's the fast path for determining if -actual.png and -expected.png results match.

> Also, if there is a test in "platform/x/.../foo.html" I assume that test can never
> have expected results outside of platform x. If that isn't the case, like a
> chain of platform dependencies, then this could be slightly wrong.

That is correct.  Although I suppose platform/mac could have different results in platform/mac-leopard and platform/mac-snowleopard, although I'm not sure whether [old-]run-webkit-tests actually handles this or not.
Comment 4 David Kilzer (:ddkilzer) 2011-03-02 07:39:40 PST
(In reply to comment #1)
> Created an attachment (id=84252) [details]
> [SCRIPT] Find Expected Results Without Test
> 
> Script I used to find the issues.

You should also put this up for review!
Comment 5 Joseph Pecoraro 2011-03-03 14:23:16 PST
I noticed at least one issue with my script. There should be a list of fallback directories to search in.

# For expected result: LayoutTests/platform/chromium-gpu-win/foo/bar/x-expected.txt we check:
#
#    LayoutTests/platform/chromium-gpu-win/foo/bar/x.*
#    LayoutTests/platform/chromium-gpu/foo/bar/x.*
#    LayoutTests/platform/chromium/foo/bar/x.*
#    LayoutTests/foo/bar/x.*
#
# If no test file is found its an orphan!

Updated this and I now only see:

    393 expected results missing tests.
    500 unpaired pngs/checksums.

I'll do a little more verification before writing a patch to remove the excess files.
Comment 6 Joseph Pecoraro 2011-03-03 14:53:12 PST
Heh, doing a quick verification I found 1 weird expected result.
Turns out my script was working correctly:

  LayoutTests/platform/mac-leopard/platform/mac/svg/batik/text/smallFonts-expected.txt

Should be nixed because no test exists at:

  LayoutTests/platform/mac/svg/batik/text/smallFonts.*

So I think I'll write up a patch to remove these.
Comment 7 Joseph Pecoraro 2011-03-04 00:37:39 PST
Turns out the fallback behaviors, at least in run-webkit-tests, are quite weird.
They are defined in Tools/Scripts/webkitpy/layout_tests/ports/*. So, a perfect
script would need to mimic deduplicate-tests, and follow the idiosyncrasies 
in the fallback behavior of some ports.