66040 2011-08-11 01:15:09 -0700 REGRESSION: nrwt does not use the right fallback sequence for platform-specific tests 2011-10-12 14:27:23 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED INVALID P1 Critical --- 66093 64491 1 rniwa webkit-unassigned abarth andersca dpranke eric tony oldest_to_newest 449653 0 rniwa 2011-08-11 01:15:09 -0700 platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html has been failing on Snow Leopard since andersca has landed his Lion results on http://trac.webkit.org/changeset/92496. Looking at the result.html, it appears that it's grabbing the expected result from platform/mac/ instead of platform/mac-snowleopard: http://build.webkit.org/results/SnowLeopard%20Intel%20Debug%20(Tests)/r92823%20(1624)/results.html 449654 1 rniwa 2011-08-11 01:16:28 -0700 This is a serious bug as there's currently no way of putting different expected results for Lion, Snow Leopard, and Leopard. 449938 2 eric 2011-08-11 12:40:19 -0700 I suspect NRWT just doesn't know anything about lion, and thus assumes that mac/ is Snow Leopard. 449944 3 dpranke 2011-08-11 12:46:27 -0700 NRWT knows about the platform/mac-snow-leopard directory. There isn't an expected result for platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html there, so it uses the one from platform/mac. So, NRWT seems to be working correctly. Is the complaint that a rebaselining tool isn't doing the right thing, perhaps? Or is there something else stopping someone from putting a baseline in the SL dir? 449962 4 eric 2011-08-11 13:04:58 -0700 (In reply to comment #3) > NRWT knows about the platform/mac-snow-leopard directory. There isn't an expected result for platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html there, so it uses the one from platform/mac. So, NRWT seems to be working correctly. > > Is the complaint that a rebaselining tool isn't doing the right thing, perhaps? Or is there something else stopping someone from putting a baseline in the SL dir? snowleopard is snow supposed to fall back to lion before mac. I'll change NRWT to match ORWT. 449964 5 rniwa 2011-08-11 13:07:04 -0700 (In reply to comment #3) > NRWT knows about the platform/mac-snow-leopard directory. There isn't an expected result for platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html there There is! http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt > so it uses the one from platform/mac. So, NRWT seems to be working correctly. NRWT should be grabbing the result from mac-snowleopard instead of mac. 449968 6 rniwa 2011-08-11 13:10:24 -0700 (In reply to comment #4) > (In reply to comment #3) > > Is the complaint that a rebaselining tool isn't doing the right thing, perhaps? Or is there something else stopping someone from putting a baseline in the SL dir? > > snowleopard is snow supposed to fall back to lion before mac. Really? That doesn't sound right but if that's what ORWT does then I guess NRWT can do that as well. But I don't think that'll solve problem either because we don't have http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac-lion/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt, and even if we did, it'll be Lion specific result. The problem is that NRWT isn't looking for the expected result in mac-snowleopard before mac when the test itself is in mac. 449972 7 tony 2011-08-11 13:12:36 -0700 The SL expected result for LayoutTests/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html would be in LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt (to differentiate from a test at LayoutTests/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html). Maybe ORWT doesn't work that way? 449979 8 rniwa 2011-08-11 13:16:14 -0700 (In reply to comment #7) > The SL expected result for LayoutTests/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html would be in LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt (to differentiate from a test at LayoutTests/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html). > > Maybe ORWT doesn't work that way? ORWT assumes the expected result be in LayoutTests/platform/mac-snow-leopard/fast/text/international/ instead of LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/ 449984 9 tony 2011-08-11 13:23:51 -0700 Why are there results checked into platform/*/platform then? Some of them date back as far as 2009 (pre-NRWT): http://trac.webkit.org/changeset/45852 449985 10 dpranke 2011-08-11 13:24:23 -0700 (In reply to comment #8) > (In reply to comment #7) > > The SL expected result for LayoutTests/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html would be in LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment-expected.txt (to differentiate from a test at LayoutTests/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html). > > > > Maybe ORWT doesn't work that way? > > ORWT assumes the expected result be in LayoutTests/platform/mac-snow-leopard/fast/text/international/ instead of LayoutTests/platform/mac-snow-leopard/platform/mac/fast/text/international/ That sounds broken to me. I agree with Tony. To further complicate things, I'll note that there is also a LayoutTests/platform/mac-snowleopard/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html file itself. I.e., there are platform-specific tests for both mac and mac-sl with the same "generic" name :(. In theory, the platform/mac-snowleopard/f/t/i/G-P-v-m-a-expected.txt file could be interpreted as the "generic" result for that test, or be the platform-specific result for the generic f/t/i test (if it existed). 449988 11 rniwa 2011-08-11 13:25:46 -0700 (In reply to comment #9) > Why are there results checked into platform/*/platform then? Some of them date back as far as 2009 (pre-NRWT): > > http://trac.webkit.org/changeset/45852 Huh, I might be wrong then. 449999 12 rniwa 2011-08-11 13:30:21 -0700 So this bug may be invalid after all. 450013 13 eric 2011-08-11 13:54:31 -0700 I just implemented support for Lion fallback orders in bug 66093. 482710 14 eric 2011-10-12 14:26:05 -0700 Unclear if this should still be open?