Bug 66040

Summary: REGRESSION: nrwt does not use the right fallback sequence for platform-specific tests
Product: WebKit Reporter: Ryosuke Niwa <rniwa@webkit.org>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned@lists.webkit.org>
Status: RESOLVED INVALID    
Severity: Critical CC: abarth@webkit.org, andersca@apple.com, dpranke@chromium.org, eric@webkit.org, tony@chromium.org
Priority: P1    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 66093    
Bug Blocks: 64491    

Description From 2011-08-11 01:15:09 PST
platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html has been failing on Snow Leopard since andersca has landed his Lion results on http://trac.webkit.org/changeset/92496.

Looking at the result.html, it appears that it's grabbing the expected result from platform/mac/ instead of platform/mac-snowleopard:
http://build.webkit.org/results/SnowLeopard%20Intel%20Debug%20(Tests)/r92823%20(1624)/results.html
------- Comment #1 From 2011-08-11 01:16:28 PST -------
This is a serious bug as there's currently no way of putting different expected results for Lion, Snow Leopard, and Leopard.
------- Comment #2 From 2011-08-11 12:40:19 PST -------
I suspect NRWT just doesn't know anything about lion, and thus assumes that mac/ is Snow Leopard.
------- Comment #3 From 2011-08-11 12:46:27 PST -------
NRWT knows about the platform/mac-snow-leopard directory. There isn't an expected result for platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html there, so it uses the one from platform/mac. So, NRWT seems to be working correctly.

Is the complaint that a rebaselining tool isn't doing the right thing, perhaps? Or is there something else stopping someone from putting a baseline in the SL dir?
------- Comment #4 From 2011-08-11 13:04:58 PST -------
(In reply to comment #3)
> NRWT knows about the platform/mac-snow-leopard directory. There isn't an expected result for platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html there, so it uses the one from platform/mac. So, NRWT seems to be working correctly.
> 
> Is the complaint that a rebaselining tool isn't doing the right thing, perhaps? Or is there something else stopping someone from putting a baseline in the SL dir?

snowleopard is snow supposed to fall back to lion before mac.

I'll change NRWT to match ORWT.
------- Comment #5 From 2011-08-11 13:07:04 PST -------
(In reply to comment #3)
> NRWT knows about the platform/mac-snow-leopard directory. There isn't an expected result for platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html there

There is!
http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt

> so it uses the one from platform/mac. So, NRWT seems to be working correctly.

NRWT should be grabbing the result from mac-snowleopard instead of mac.
------- Comment #6 From 2011-08-11 13:10:24 PST -------
(In reply to comment #4)
> (In reply to comment #3)
> > Is the complaint that a rebaselining tool isn't doing the right thing, perhaps? Or is there something else stopping someone from putting a baseline in the SL dir?
> 
> snowleopard is snow supposed to fall back to lion before mac.

Really?  That doesn't sound right but if that's what ORWT does then I guess NRWT can do that as well.  But I don't think that'll solve problem either because we don't have http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac-lion/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt, and even if we did, it'll be Lion specific result.

The problem is that NRWT isn't looking for the expected result in mac-snowleopard before mac when the test itself is in mac.
------- Comment #7 From 2011-08-11 13:12:36 PST -------
The SL expected result for LayoutTests/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html would be in LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt (to differentiate from a test at LayoutTests/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html).

Maybe ORWT doesn't work that way?
------- Comment #8 From 2011-08-11 13:16:14 PST -------
(In reply to comment #7)
> The SL expected result for LayoutTests/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html would be in LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt (to differentiate from a test at LayoutTests/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html).
> 
> Maybe ORWT doesn't work that way?

ORWT assumes the expected result be in  LayoutTests/platform/mac-snow-leopard/fast/text/international/ instead of LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/
------- Comment #9 From 2011-08-11 13:23:51 PST -------
Why are there results checked into platform/*/platform then?  Some of them date back as far as 2009 (pre-NRWT):

http://trac.webkit.org/changeset/45852
------- Comment #10 From 2011-08-11 13:24:23 PST -------
(In reply to comment #8)
> (In reply to comment #7)
> > The SL expected result for LayoutTests/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html would be in LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt (to differentiate from a test at LayoutTests/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html).
> > 
> > Maybe ORWT doesn't work that way?
> 
> ORWT assumes the expected result be in  LayoutTests/platform/mac-snow-leopard/fast/text/international/ instead of LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/

That sounds broken to me. I agree with Tony.

To further complicate things, I'll note that there is also a LayoutTests/platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html file itself.

I.e., there are platform-specific tests for both mac and mac-sl with the same "generic" name :(. In theory, the platform/mac-snowleopard/f/t/i/G-P-v-m-a-expected.txt file could be interpreted as the "generic" result for that test, or be the platform-specific result for the generic f/t/i test (if it existed).
------- Comment #11 From 2011-08-11 13:25:46 PST -------
(In reply to comment #9)
> Why are there results checked into platform/*/platform then?  Some of them date back as far as 2009 (pre-NRWT):
> 
> http://trac.webkit.org/changeset/45852

Huh, I might be wrong then.
------- Comment #12 From 2011-08-11 13:30:21 PST -------
So this bug may be invalid after all.
------- Comment #13 From 2011-08-11 13:54:31 PST -------
I just implemented support for Lion fallback orders in bug 66093.
------- Comment #14 From 2011-10-12 14:26:05 PST -------
Unclear if this should still be open?