Bug 90191 – rebaselining requires a bot for each platform directory under LayoutTests

• Status: NEW
• Component: Tools / Tests

[Ojan Vafai]

Once 90186 is resolved, rebaseline-test, rebaseline-expectations and the rebaseline button in garden-o-matic will all work like the following:
1. Grab the results from each bot and put it in the most specific platform directory.
2. Merge results that are equal into the parent's (in fallback order) directory.

This means that if a port does not have a bot that runs the test for each platform directory, then the merging is extremely sub-optimal (leaves copies of the file in platform directories where they could be merged and put next to the file or in the platform/mac directory).

At a quick glance, Chromium, EFL and GTK have bots for all their platform directories. Apple and Qt do not.

What's the best way to address this? We could have the Apple/Qt bots put the results in a less specific directory, but that would mean that rebaseline would do the wrong thing in some cases (e.g. where Apple's SnowLeopard and Lion ports give different results).

[Adam Barth]

What does it mean to have a results directory if there's not bot that exercises it?  I guess there might be some corner cases involving future releases of Mac OS X that are exercised internally at Apple but not publicly...

[Ojan Vafai]

(In reply to comment #1)
> What does it mean to have a results directory if there's not bot that exercises it?  I guess there might be some corner cases involving future releases of Mac OS X that are exercised internally at Apple but not publicly...

They could be tested not on build.webkit.org. For example, Chromium only has full-bot coverage at build.chromium.org. In general, I agree though, it doesn't make sense to have a platform directory without a buildbot.

Qt has 9 directories, but only one bot that runs the layout tests. We could  special-case Qt to always put the results in the platform/qt directory.

Lucas, is there a reason Apple doesn't run SnowLeopard, Mt. Lion and Window-WebKit2 bots? We could special-case Apple bots to always put results in the platform/mac or platform/win directory.

Also, why is there a win-7sp0 directory that's separate from the win directory? Do you really get different results for different Win7 service packs? Does Apple Win 7 Release (Tests) run SP1 or SP0?

For completeness sake: there's also the blackberry directory and the google-chrome-* directories. But I think we can ignore all of those for the purposes of baseline merging for now, as in, we can make merging pretend those ports don't exist.

[Lucas Forschler]

>Lucas, is there a reason Apple doesn't run SnowLeopard, Mt. Lion and Window-WebKit2 bots? We could >special-case Apple bots to always put results in the platform/mac or platform/win directory.

We don't have Mtn Lion bots on webkit.org because Mtn Lion is not released yet.  When Mtn Lion is released we will be bringing up bots

For Snow Leopard bots, it's mostly a hardware resources issue, so I don't expect that we'll be expanding our hardware resources for Snowleopard anytime soon.  I'm not sure on Windows bots, maybe Jessie can give insight there?

>Also, why is there a win-7sp0 directory that's separate from the win directory? Do you really get different >results for different Win7 service packs? Does Apple Win 7 Release (Tests) run SP1 or SP0?

I don't know specifics here, Jessie, do you know, or can you point us to someone who might?

[Ojan Vafai]

To be clear, I don't have any preference of how people should set up their bots. I just want to come up with a solution whereby we only end up with a single expected file for tests where all the ports get the same result that doesn't involve humans manually moving things around.

Maybe the best short-term solution is to have Apple bots always put results in the mac/win directories and Qt bots always put results in the qt directory.

[Ojan Vafai]

Bug 90204 does the fudging to make things work for now. Not sure if we want to keep this open for cleaning up the state of the platform directories.