Bug 124312 – Occasionally, hundreds of tests fail with small text diffs on Mavericks

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Tim Horton]

Like http://build.webkit.org/results/Apple%20Mavericks%20Debug%20WK2%20(Tests)/r159226%20(433)/results.html.

This has affected multiple bots today and yesterday, Mavericks and Mountain Lion, WK1 and WK2. Don’t know what’s going on. Maybe Antti can take a look, since he’s been working in this area?

[Antti Koivisto]

It is not likely to be simple line layout related as there are failing tests on both paths. I don't have great ideas. Uninitialized variable somewhere? Do we have know around which revisions this started?

[Antti Koivisto]

The only instances I could find were on Mavericks debug bots. First one was Nov 5th http://build.webkit.org/builders/Apple%20Mavericks%20Debug%20WK2%20%28Tests%29/builds/254
(the logs don't go much further than that).

[Tim Horton]

Looking at the WKTR debug logs, it’s not like one worker gets stuck in a bad state. *Two* workers are producing the bad results, and have passes sprinkled in between fails (I suppose it is feasible that the passes either don’t have text or are DumpAsText or something).

[Tim Horton]

I’m specifically looking at http://build.webkit.org/builders/Apple%20Mavericks%20Debug%20WK2%20%28Tests%29/builds/448/steps/layout-test/logs/stdio

worker 2 and 3 produce most of the failures, though 1 joins in at some point.

So I think it’s unlikely that it’s state left behind by a bad test.

[Tim Horton]

(In reply to comment #2)
> The only instances I could find were on Mavericks debug bots. First one was Nov 5th http://build.webkit.org/builders/Apple%20Mavericks%20Debug%20WK2%20%28Tests%29/builds/254
> (the logs don't go much further than that).

You’re right, I don’t see why I thought it was happening on Mountain Lion.

[Tim Horton]

Huh, the ref-test-image-diffs that coincide with the 500+ failures are interesting. The most interesting one I found is http://build.webkit.org/results/Apple%20Mavericks%20Debug%20WK1%20(Tests)/r159327%20(573)/fast/regions/auto-size/autoheight-maxheight-region-diffs.html

[Tim Horton]

Created attachment 217007 [details]
expected

[Tim Horton]

Created attachment 217008 [details]
actual

[Tim Horton]

Uploaded that one here in case it can shed some light and so that it doesn’t get disappeared.

[Antti Koivisto]

Weird. Looks like some sort of antialiasing or subpixel positioning kicking in.

[Tim Horton]

Being Mavericks only calls into question the code that disables screen-font-substitution (but enables it in the tests).

[Tim Horton]

(In reply to comment #11)
> Being Mavericks only calls into question the code that disables screen-font-substitution (but enables it in the tests).

I can reproduce the same 3px change in http://build.webkit.org/results/Apple%20Mavericks%20Debug%20WK2%20(Tests)/r158682%20(254)/compositing/geometry/abs-position-inside-opacity-diff.txt by toggling screen-font-substitution in Safari.

[Tim Horton]

Created attachment 217091 [details]
experiment

[Alexey Proskuryakov]

Comment on attachment 217091 [details]
experiment

I think it might be better to simply delete the code. As one of the options we have is to move the behavior to the script, having a FIXME in C++ code may be misleading.

[Tim Horton]

Experiment landed in http://trac.webkit.org/changeset/159364. Antti, let's keep an eye out and see if we see it happen any more.

[Tim Horton]

Created attachment 217225 [details]
restore defaults deletion

ap and mitz may have more to discuss about the general strategy of r158652, but in the meantime we can restore this functionality in a place where it will not cause a race between processes.

[Tim Horton]

Created attachment 217226 [details]
got some extra change log in there

[WebKit Commit Bot]

Comment on attachment 217226 [details]
got some extra change log in there

Clearing flags on attachment: 217226

Committed r159453: <http://trac.webkit.org/changeset/159453>