RESOLVED WONTFIX Bug 96842
[chromium] lots of layout test failures on 10.7.4 
https://bugs.webkit.org/show_bug.cgi?id=96842
Summary [chromium] lots of layout test failures on 10.7.4
Dirk Pranke
Reported 2012-09-14 16:59:42 PDT
it appears that things changed in 10.7.4 such that tests that run fine on 10.7.3 produce different baselines on 10.7.4. More details to follow.
Attachments
Add attachment proposed patch, testcase, etc.
Dirk Pranke
Comment 1 2012-09-17 14:48:53 PDT
fwiw, I have a local platform/chromium-mac-10_7_4 directory with ~160ish layout test results that I think account for most if not all of the failures. I'm not posting this to the bug for now since I haven't really audited all of them, and it's unclear quite yet what to do with this. Per zmo, it seems that we don't necessarily want to just upgrade the bots to 10.7.4 because that'll cause a lot of gpu tests to fail. On the other hand, 10.7.4 is what our customers are running, so arguably we should just suck it up. We can: 1) create a 10.7.4 platform directory and keep it up-to-date by hand 2) spin up a 10.7.4 bot 3) wait for 10.7.5 and hope that fixes the gpu issues 4) suppress the failures as IMAGE (and IMAGE+TEXT etc) so that we don't need to track the baselines 5) ignore the problem and hope it goes away :) Let me know if anyone has any strong leanings here.
Ojan Vafai
Comment 2 2012-09-17 15:42:41 PDT
My vote would be for 1+2. But I'd be fine with any combination of 1, 3, 4 or 5. Seems like we should at least do one of 1 or 4 though. 4 seems like the least amount of effort in the short-term.
Dirk Pranke
Comment 3 2012-09-17 15:45:26 PDT
I'll run locally w/ my 10.7.4 baselines for a while and see how much churn there is. Maybe updating by hand won't be too bad ... I'd prefer to avoid 2) if we can, and I'm not a bug fan of 4) since I'm trying to move away from suppressions (and 160+ tests is a lot to suppress if we have no reason to expect this to get fixed soon).
Tony Chang
Comment 4 2012-09-18 12:04:48 PDT
How will the gpu tests fail? Are these real failures that our users are seeing or is it a bug in our test infrastructure?
Dirk Pranke
Comment 5 2012-09-18 12:22:36 PDT
zmo can confirm but I believe they reflect real bugs in 10.7.4 that our users see.
Tony Chang
Comment 6 2012-09-18 12:27:32 PDT
(In reply to comment #5) > zmo can confirm but I believe they reflect real bugs in 10.7.4 that our users see. In that case, I would just update the bots to 10.7.4 and either rebaseline or mark them in TestExpectations. The failures are real, so it seems fine for us to mark them as such in TestExpectations. I'm not sure what benefit we get by running the tests on 10.7.3 if that's not what our users are running.
Dirk Pranke
Comment 7 2012-09-18 15:11:23 PDT
So, after running with a set of 10.7.4-specific baselines for a while, I'm not seeing any real gpu-related weirdness during the layout tests; they seem fairly stable. Given that, and a general consensus that we should probably be testing what our customers are likely running, it seems like we should update the lion layout test bots to 10.7.4 and update the baselines. I am not proposing updating the other chromium bots; I'll let zhenyao worry about those. Anyone disagree?
Dirk Pranke
Comment 8 2012-10-02 18:37:06 PDT
We've upgraded the canaries to 10.7.5 (and rebaselined), so I'm closing this as WONTFIX ...
Note You need to log in before you can comment on or make changes to this bug.