Bug 120755 – Lots of pixel tests are failing on first run, so tests cannot finish

• Status: REOPENED
• Component: Tools / Tests

[WebKit Commit Bot]

This is an automatically generated bug from the commit-queue.
animations/cross-fade-background-image.html has been flaky on the commit-queue.

animations/cross-fade-background-image.html was authored by [e-mail] and [e-mail].
http://trac.webkit.org/browser/trunk/LayoutTests/animations/cross-fade-background-image.html

The commit-queue just saw animations/cross-fade-background-image.html flake (image diff) while processing attachment 210550 [details] on bug 119955.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4

The bots will update this with information from each new failure.

If you believe this bug to be fixed or invalid, feel free to close.  The bots will re-open if the flake re-occurs.

If you would like to track this test fix with another bug, please close this bug as a duplicate.  The bots will follow the duplicate chain when making future comments.

[WebKit Commit Bot]

Created attachment 210581 [details]
Archive of layout-test-results from webkit-cq-02

[Dirk Schulze]

It might be that there are platform differences when compositing two images. I have the same Mac version and it passes for me. This is a bit strange, since the PNG that I had before passed on the server as well. Not sure why the ref test doesn't. The ref tests worked on the EWS bots too.

Needs investigation.

[Alexey Proskuryakov]

I can reproduce this locally. The issue is that this test is color profile sensitive, and run-webkit-tests only forces a color profile for pixel tests, not for reftests.

Pixel tests are also used when retrying tests, so these kinds of failures are essentially hidden - the tests are "flaky", but that's not reported loudly enough. Also, regular color profile on some machines likely just happens to match what tests want.

Looking at attached test results, we are now getting too many of these failures, so run-webkit-tests doesn't retry, so it finally started reporting those failures.

It may be that our tools are stuck until someone who understands color WebCore management can have a look.

[WebKit Commit Bot]

The commit-queue just saw animations/cross-fade-background-image.html flake (image diff) while processing attachment 210723 [details] on bug 120847.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4

[WebKit Commit Bot]

Created attachment 210755 [details]
Archive of layout-test-results from webkit-cq-02

[Dirk Schulze]

For now we could either turn it back to a pixel test or just check DRT. IMO there would not be much of a difference since we sadly don't run pixel tests on the server.

[Alexey Proskuryakov]

This does kill EWS and other tools.

I can't figure out why this became more serious only now, but the only fix I can think of anyway it to roll out <http://trac.webkit.org/changeset/115601>. We need to set the profile not just when running pixel tests, but for reftests too, and that means that we can just as well set it unconditionally.

[Alexey Proskuryakov]

Created attachment 210762 [details]
proposed fix

[Alexey Proskuryakov]

*** Bug 120727 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120729 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120730 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120728 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120731 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120732 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120734 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120733 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120735 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120736 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120737 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120738 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120739 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120740 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120741 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120742 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120743 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120744 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120745 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120746 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120747 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120748 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120749 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120750 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120751 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120752 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120753 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 120754 has been marked as a duplicate of this bug. ***

[Ryosuke Niwa]

Comment on attachment 210762 [details]
proposed fix

View in context: https://bugs.webkit.org/attachment.cgi?id=210762&action=review

> Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:153
> -        if self._options.pixel_tests:
> -            self._printer.write_update("Starting pixel test helper ...")
> -            self._port.start_helper()
> +        self._printer.write_update("Starting pixel test helper ...")
> +        self._port.start_helper()

We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now.
Perhaps we should do this only for Mac port?

[Alexey Proskuryakov]

Committed <http://trac.webkit.org/r155196>.

Looks like this may not help EWS though! I know that this fixes it for me locally, but why is EWS still yellow? I'll have to keep looking.

> We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now.

Yeah, I guess that would be helpful for people who only ever run fast/js tests. Otherwise, we have reftests everywhere, even in http directory.

> Perhaps we should do this only for Mac port?

Currently, only the Mac port has a helper, it's a no-op for others.

[Dirk Pranke]

(In reply to comment #37)
> We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now.

Given that we don't have a global list of reftests (we check each test on demand), you'd have to check that up front, and that could be fairly expensive.

[Dirk Schulze]

(In reply to comment #38)
> Committed <http://trac.webkit.org/r155196>.
> 
> Looks like this may not help EWS though! I know that this fixes it for me locally, but why is EWS still yellow? I'll have to keep looking.

Currently the expected file on animations/cross-fade-background-image.html has a result that worked for me on my machine. If this test is failing now, I can submit another result.

[Alexey Proskuryakov]

Looks like animations/cross-fade-background-image.html now passes on WK1, but it's still failing (with color management badness) on WK2.

We might need a separate new bug for this test specifically.

[WebKit Commit Bot]

Re-opened since this is blocked by bug 120919

[Ryosuke Niwa]

The EWS bots were simply encountering the display initialization problem. The right fix is to log into each one of them via VNC/screen share.

[WebKit Commit Bot]

The commit-queue just saw fast/inline/layout-after-inserting-nested-br.html flake (image diff) while processing attachment 210780 [details] on bug 120884.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4

[WebKit Commit Bot]

Created attachment 210870 [details]
Archive of layout-test-results from webkit-cq-01

[WebKit Commit Bot]

The commit-queue just saw fast/shapes/shape-inside/shape-inside-overflow-fixed-dimensions-block-content.html flake (image diff) while processing attachment 210780 [details] on bug 120884.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4

[WebKit Commit Bot]

Created attachment 210874 [details]
Archive of layout-test-results from webkit-cq-01

[WebKit Commit Bot]

The commit-queue just saw fast/shapes/shape-outside-floats/shape-outside-floats-image-002.html flake (image diff) while processing attachment 210820 [details] on bug 120911.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4

[WebKit Commit Bot]

Created attachment 210896 [details]
Archive of layout-test-results from webkit-cq-03

[WebKit Commit Bot]

The commit-queue just saw fast/shapes/shape-outside-floats/shape-outside-floats-image-001.html flake (image diff) while processing attachment 210820 [details] on bug 120911.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4

[WebKit Commit Bot]

Created attachment 210897 [details]
Archive of layout-test-results from webkit-cq-03

[Ryosuke Niwa]

I've rebooted all EWS bots, manually logged into them via screen sharing, and manually set sRGB as the color profile. We'll see if test failures will go away now.  If not, then something must have regressed in DRT/WTR or WebKit itself.

[Ryosuke Niwa]

Some bots are still seeing image only failures. However, it appears that tests do pass on those bots immediately after I've manually logged into them via screen sharing.

I've realized that screen saver was enabled on some of them so I've disabled them manually. Is it possible for screen saver somehow affect the color profile or accelerated compositing?

[Ryosuke Niwa]

Not fixed.

Somehow a different color profile is used when accelerated compositing is enabled.

[Mark Hahnenberg]

Is this bug still a thing? css2.1/20110323/border-conflict-element-001d.htm	 is flaking a lot on the Mavericks Debug WK2 bots, and the bug I found for that test was duped to this one.

[Alexey Proskuryakov]

Color management troubles are still a thing indeed, although I'm not sure if this bug is the best way to track fixing them.

For now, I've switched the bot that had trouble with css2.1/20110323/border-conflict-element-001d.htm to sRGB manually, which should resolve this (and I believe someone else - maybe Beth - has switched one or more bots to sRGB this week already).

(In reply to Ryosuke Niwa from comment #52)
> I've rebooted all EWS bots, manually logged into them via screen sharing,
> and manually set sRGB as the color profile. We'll see if test failures will
> go away now.  If not, then something must have regressed in DRT/WTR or
> WebKit itself.

[Boris Rangelov]

(In reply to izma143 from comment #68)
> (В отговор на Ryosuke Niwa от коментар #52 )
> > Рестартирах всички EWS ботове, ръчно влязох в тях чрез споделяне на екрана > и ръчно зададох sRGB като цветен профил. Ще видим дали неуспешните тестове > ще изчезнат сега. Ако не, тогава нещо трябва да е регресирало в DRT/WTR или > самия WebKit.