Bug 77054 - chromium should not fallback to Mac for results
Summary: chromium should not fallback to Mac for results
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-01-25 15:53 PST by Eric Seidel (no email)
Modified: 2012-10-24 12:44 PDT (History)
7 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Eric Seidel (no email) 2012-01-25 15:53:26 PST
That makes no sense, whatsoever.

I can't imagine that it's saving us any space these days.

https://docs.google.com/drawings/d/1z65SKkWrD4Slm6jobIphHwwRADyUtjOAxwGBVKBY8Kc/edit?hl=en_US

It just causes immense confusion, like with https://bugs.webkit.org/show_bug.cgi?id=76947#c8
Comment 1 Dirk Pranke 2012-01-25 16:03:54 PST
Last time this came up ( https://lists.webkit.org/pipermail/webkit-dev/2011-July/017598.html ) I wrote a script to check, and the number came back as chromium-mac-snowleopard picked up 11,000+ results from platform/mac :). 

I posted the script I used to https://bugs.webkit.org/show_bug.cgi?id=64426 ; I have no idea if it still runs but if not, it probably wouldn't be hard to make it work again. 

Feel free to check and see if the numbers have changed enough to make a difference.
Comment 2 Eric Seidel (no email) 2012-01-25 16:06:24 PST
I'm OK with chromium-mac still taking its circuitous route through mac-*, that's fine.  But it makes no sense for chromium-linux to fallback through mac IMO. :)
Comment 3 Dirk Pranke 2012-01-25 16:07:15 PST
(In reply to comment #0)
> That makes no sense, whatsoever.
> 
> I can't imagine that it's saving us any space these days.
> 
> https://docs.google.com/drawings/d/1z65SKkWrD4Slm6jobIphHwwRADyUtjOAxwGBVKBY8Kc/edit?hl=en_US
> 
> It just causes immense confusion, like with https://bugs.webkit.org/show_bug.cgi?id=76947#c8

Also, for the record, it was originally a launch goal for Chromium to exactly match Safari's results, pixel-for-pixel. So it made perfect sense. 

Given that we've switched to Skia, I'm not sure that it still does. Adding Eliot and JamesR, who might have more informed opinions of the current state of affairs.
Comment 4 Eric Seidel (no email) 2012-01-25 16:10:10 PST
Turns out that *another* one of my fixes from last week hit this same problem:
https://bugs.webkit.org/show_bug.cgi?id=76760#c11

I'm making changes where Mac has different/broken behavior and I expect to be able to add a mac-only expected failure, instead I'm having to add both a mac-only expected failure and a chromium-only "yes we really do pass this" expectation. :(
Comment 5 James Robinson 2012-01-25 16:11:13 PST
(In reply to comment #0)
> That makes no sense, whatsoever.
> 
> I can't imagine that it's saving us any space these days.
> 
> https://docs.google.com/drawings/d/1z65SKkWrD4Slm6jobIphHwwRADyUtjOAxwGBVKBY8Kc/edit?hl=en_US
> 
> It just causes immense confusion, like with https://bugs.webkit.org/show_bug.cgi?id=76947#c8

Who is it confusing other than you? We've always fallen back on platform/mac. I think other platforms do too.

Another consideration is that for many tests the only expectations of any sort are in platform/mac and we consider missing expectations a failure.  It's also useful to reference the platform/mac baselines when looking at new tests since Apple folks will often check in only platform/mac results with new tests (if they check any expectations in at all).
Comment 6 Dirk Pranke 2012-01-25 16:13:57 PST
okay, bugzilla's collisions on comments are really annoying (again!) ..

(In reply to comment #2)
> I'm OK with chromium-mac still taking its circuitous route through mac-*, that's fine.  But it makes no sense for chromium-linux to fallback through mac IMO. :)

In the data I posted on that thread, chromium-linux pulled > 1000 .txt and > 1000 .png's from the mac directory :) Again, maybe the numbers have changed and we should probably look again.

The fact that it is "best practice" to check in expected failures on the non-Chromium ports certainly can aggrevate this, and perhaps it makes sense to remove platform-mac from the chromium tree given this behavior (although being able to distinguish platform-specific pass from platform-specific failure would definitely help here).

As JamesR notes, there is also the fact that chromium runs the tests in platform/mac :). I have argued that we should be skipping those, though.
Comment 7 Tony Chang 2012-01-25 16:35:00 PST
(In reply to comment #6)

> As JamesR notes, there is also the fact that chromium runs the tests in platform/mac :). I have argued that we should be skipping those, though.

James isn't talking about the tests in platform/mac, he's saying that if you work on mac and you add a new pixel test, you'll probably only check in a pixel result for mac.  The Chromium Win/Linux bots which run pixel tests will flag the new test as MISSING unless we look in the mac directory.

It would probably be easy to write a script to see how many tests have pixel results in only platform/mac.
Comment 8 Eric Seidel (no email) 2012-10-24 12:40:32 PDT
This is solved now, right?
Comment 9 Dirk Pranke 2012-10-24 12:44:14 PDT
Yes.