77054 2012-01-25 15:53:26 -0800 chromium should not fallback to Mac for results 2012-10-24 12:44:14 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 eric webkit-unassigned abarth dpranke epoger jamesr ojan rniwa tony oldest_to_newest 542717 0 eric 2012-01-25 15:53:26 -0800 That makes no sense, whatsoever. I can't imagine that it's saving us any space these days. https://docs.google.com/drawings/d/1z65SKkWrD4Slm6jobIphHwwRADyUtjOAxwGBVKBY8Kc/edit?hl=en_US It just causes immense confusion, like with https://bugs.webkit.org/show_bug.cgi?id=76947#c8 542735 1 dpranke 2012-01-25 16:03:54 -0800 Last time this came up ( https://lists.webkit.org/pipermail/webkit-dev/2011-July/017598.html ) I wrote a script to check, and the number came back as chromium-mac-snowleopard picked up 11,000+ results from platform/mac :). I posted the script I used to https://bugs.webkit.org/show_bug.cgi?id=64426 ; I have no idea if it still runs but if not, it probably wouldn't be hard to make it work again. Feel free to check and see if the numbers have changed enough to make a difference. 542737 2 eric 2012-01-25 16:06:24 -0800 I'm OK with chromium-mac still taking its circuitous route through mac-*, that's fine. But it makes no sense for chromium-linux to fallback through mac IMO. :) 542738 3 dpranke 2012-01-25 16:07:15 -0800 (In reply to comment #0) > That makes no sense, whatsoever. > > I can't imagine that it's saving us any space these days. > > https://docs.google.com/drawings/d/1z65SKkWrD4Slm6jobIphHwwRADyUtjOAxwGBVKBY8Kc/edit?hl=en_US > > It just causes immense confusion, like with https://bugs.webkit.org/show_bug.cgi?id=76947#c8 Also, for the record, it was originally a launch goal for Chromium to exactly match Safari's results, pixel-for-pixel. So it made perfect sense. Given that we've switched to Skia, I'm not sure that it still does. Adding Eliot and JamesR, who might have more informed opinions of the current state of affairs. 542739 4 eric 2012-01-25 16:10:10 -0800 Turns out that *another* one of my fixes from last week hit this same problem: https://bugs.webkit.org/show_bug.cgi?id=76760#c11 I'm making changes where Mac has different/broken behavior and I expect to be able to add a mac-only expected failure, instead I'm having to add both a mac-only expected failure and a chromium-only "yes we really do pass this" expectation. :( 542740 5 jamesr 2012-01-25 16:11:13 -0800 (In reply to comment #0) > That makes no sense, whatsoever. > > I can't imagine that it's saving us any space these days. > > https://docs.google.com/drawings/d/1z65SKkWrD4Slm6jobIphHwwRADyUtjOAxwGBVKBY8Kc/edit?hl=en_US > > It just causes immense confusion, like with https://bugs.webkit.org/show_bug.cgi?id=76947#c8 Who is it confusing other than you? We've always fallen back on platform/mac. I think other platforms do too. Another consideration is that for many tests the only expectations of any sort are in platform/mac and we consider missing expectations a failure. It's also useful to reference the platform/mac baselines when looking at new tests since Apple folks will often check in only platform/mac results with new tests (if they check any expectations in at all). 542745 6 dpranke 2012-01-25 16:13:57 -0800 okay, bugzilla's collisions on comments are really annoying (again!) .. (In reply to comment #2) > I'm OK with chromium-mac still taking its circuitous route through mac-*, that's fine. But it makes no sense for chromium-linux to fallback through mac IMO. :) In the data I posted on that thread, chromium-linux pulled > 1000 .txt and > 1000 .png's from the mac directory :) Again, maybe the numbers have changed and we should probably look again. The fact that it is "best practice" to check in expected failures on the non-Chromium ports certainly can aggrevate this, and perhaps it makes sense to remove platform-mac from the chromium tree given this behavior (although being able to distinguish platform-specific pass from platform-specific failure would definitely help here). As JamesR notes, there is also the fact that chromium runs the tests in platform/mac :). I have argued that we should be skipping those, though. 542753 7 tony 2012-01-25 16:35:00 -0800 (In reply to comment #6) > As JamesR notes, there is also the fact that chromium runs the tests in platform/mac :). I have argued that we should be skipping those, though. James isn't talking about the tests in platform/mac, he's saying that if you work on mac and you add a new pixel test, you'll probably only check in a pixel result for mac. The Chromium Win/Linux bots which run pixel tests will flag the new test as MISSING unless we look in the mac directory. It would probably be easy to write a script to see how many tests have pixel results in only platform/mac. 749925 8 eric 2012-10-24 12:40:32 -0700 This is solved now, right? 749932 9 dpranke 2012-10-24 12:44:14 -0700 Yes.