Bug 120755

Summary: Lots of pixel tests are failing on first run, so tests cannot finish
Product: WebKit Reporter: WebKit Commit Bot <commit-queue>
Component: Tools / TestsAssignee: Alexey Proskuryakov <ap>
Status: REOPENED ---    
Severity: Blocker CC: ap, cabatimi, cyberflixtvappco, dino, dpranke, glenn, krit, lodi, mhahnenberg, mjs, moonarya123, ngockhanhlam87, rexlsf99, rniwa, simon.fraser, sj12152251, thorton, todobancosvenezuela, vodrickcarter987
Priority: P1 Keywords: MakingBotsRed
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 120919    
Bug Blocks: 50856    
Attachments:
Description Flags
Archive of layout-test-results from webkit-cq-02
none
Archive of layout-test-results from webkit-cq-02
none
proposed fix
rniwa: review+
Archive of layout-test-results from webkit-cq-01
none
Archive of layout-test-results from webkit-cq-01
none
Archive of layout-test-results from webkit-cq-03
none
Archive of layout-test-results from webkit-cq-03 none

Description WebKit Commit Bot 2013-09-04 23:48:00 PDT
This is an automatically generated bug from the commit-queue.
animations/cross-fade-background-image.html has been flaky on the commit-queue.

animations/cross-fade-background-image.html was authored by krit@webkit.org and thorton@apple.com.
http://trac.webkit.org/browser/trunk/LayoutTests/animations/cross-fade-background-image.html

The commit-queue just saw animations/cross-fade-background-image.html flake (image diff) while processing attachment 210550 [details] on bug 119955.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4

The bots will update this with information from each new failure.

If you believe this bug to be fixed or invalid, feel free to close.  The bots will re-open if the flake re-occurs.

If you would like to track this test fix with another bug, please close this bug as a duplicate.  The bots will follow the duplicate chain when making future comments.
Comment 1 WebKit Commit Bot 2013-09-04 23:48:03 PDT
Created attachment 210581 [details]
Archive of layout-test-results from webkit-cq-02
Comment 2 Dirk Schulze 2013-09-04 23:59:04 PDT
It might be that there are platform differences when compositing two images. I have the same Mac version and it passes for me. This is a bit strange, since the PNG that I had before passed on the server as well. Not sure why the ref test doesn't. The ref tests worked on the EWS bots too.

Needs investigation.
Comment 3 Alexey Proskuryakov 2013-09-05 14:44:40 PDT
I can reproduce this locally. The issue is that this test is color profile sensitive, and run-webkit-tests only forces a color profile for pixel tests, not for reftests.

Pixel tests are also used when retrying tests, so these kinds of failures are essentially hidden - the tests are "flaky", but that's not reported loudly enough. Also, regular color profile on some machines likely just happens to match what tests want.

Looking at attached test results, we are now getting too many of these failures, so run-webkit-tests doesn't retry, so it finally started reporting those failures.

It may be that our tools are stuck until someone who understands color WebCore management can have a look.
Comment 4 WebKit Commit Bot 2013-09-06 06:34:46 PDT
The commit-queue just saw animations/cross-fade-background-image.html flake (image diff) while processing attachment 210723 [details] on bug 120847.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4
Comment 5 WebKit Commit Bot 2013-09-06 06:34:50 PDT
Created attachment 210755 [details]
Archive of layout-test-results from webkit-cq-02
Comment 6 Dirk Schulze 2013-09-06 06:49:32 PDT
For now we could either turn it back to a pixel test or just check DRT. IMO there would not be much of a difference since we sadly don't run pixel tests on the server.
Comment 7 Alexey Proskuryakov 2013-09-06 09:32:45 PDT
This does kill EWS and other tools.

I can't figure out why this became more serious only now, but the only fix I can think of anyway it to roll out <http://trac.webkit.org/changeset/115601>. We need to set the profile not just when running pixel tests, but for reftests too, and that means that we can just as well set it unconditionally.
Comment 8 Alexey Proskuryakov 2013-09-06 09:39:41 PDT
Created attachment 210762 [details]
proposed fix
Comment 9 Alexey Proskuryakov 2013-09-06 09:42:42 PDT
*** Bug 120727 has been marked as a duplicate of this bug. ***
Comment 10 Alexey Proskuryakov 2013-09-06 09:42:51 PDT
*** Bug 120729 has been marked as a duplicate of this bug. ***
Comment 11 Alexey Proskuryakov 2013-09-06 09:42:55 PDT
*** Bug 120730 has been marked as a duplicate of this bug. ***
Comment 12 Alexey Proskuryakov 2013-09-06 09:43:03 PDT
*** Bug 120728 has been marked as a duplicate of this bug. ***
Comment 13 Alexey Proskuryakov 2013-09-06 09:43:12 PDT
*** Bug 120731 has been marked as a duplicate of this bug. ***
Comment 14 Alexey Proskuryakov 2013-09-06 09:43:21 PDT
*** Bug 120732 has been marked as a duplicate of this bug. ***
Comment 15 Alexey Proskuryakov 2013-09-06 09:43:30 PDT
*** Bug 120734 has been marked as a duplicate of this bug. ***
Comment 16 Alexey Proskuryakov 2013-09-06 09:43:38 PDT
*** Bug 120733 has been marked as a duplicate of this bug. ***
Comment 17 Alexey Proskuryakov 2013-09-06 09:43:45 PDT
*** Bug 120735 has been marked as a duplicate of this bug. ***
Comment 18 Alexey Proskuryakov 2013-09-06 09:43:49 PDT
*** Bug 120736 has been marked as a duplicate of this bug. ***
Comment 19 Alexey Proskuryakov 2013-09-06 09:43:53 PDT
*** Bug 120737 has been marked as a duplicate of this bug. ***
Comment 20 Alexey Proskuryakov 2013-09-06 09:43:57 PDT
*** Bug 120738 has been marked as a duplicate of this bug. ***
Comment 21 Alexey Proskuryakov 2013-09-06 09:44:01 PDT
*** Bug 120739 has been marked as a duplicate of this bug. ***
Comment 22 Alexey Proskuryakov 2013-09-06 09:44:05 PDT
*** Bug 120740 has been marked as a duplicate of this bug. ***
Comment 23 Alexey Proskuryakov 2013-09-06 09:44:09 PDT
*** Bug 120741 has been marked as a duplicate of this bug. ***
Comment 24 Alexey Proskuryakov 2013-09-06 09:44:13 PDT
*** Bug 120742 has been marked as a duplicate of this bug. ***
Comment 25 Alexey Proskuryakov 2013-09-06 09:44:23 PDT
*** Bug 120743 has been marked as a duplicate of this bug. ***
Comment 26 Alexey Proskuryakov 2013-09-06 09:44:26 PDT
*** Bug 120744 has been marked as a duplicate of this bug. ***
Comment 27 Alexey Proskuryakov 2013-09-06 09:44:30 PDT
*** Bug 120745 has been marked as a duplicate of this bug. ***
Comment 28 Alexey Proskuryakov 2013-09-06 09:44:34 PDT
*** Bug 120746 has been marked as a duplicate of this bug. ***
Comment 29 Alexey Proskuryakov 2013-09-06 09:44:38 PDT
*** Bug 120747 has been marked as a duplicate of this bug. ***
Comment 30 Alexey Proskuryakov 2013-09-06 09:44:44 PDT
*** Bug 120748 has been marked as a duplicate of this bug. ***
Comment 31 Alexey Proskuryakov 2013-09-06 09:44:49 PDT
*** Bug 120749 has been marked as a duplicate of this bug. ***
Comment 32 Alexey Proskuryakov 2013-09-06 09:44:53 PDT
*** Bug 120750 has been marked as a duplicate of this bug. ***
Comment 33 Alexey Proskuryakov 2013-09-06 09:44:57 PDT
*** Bug 120751 has been marked as a duplicate of this bug. ***
Comment 34 Alexey Proskuryakov 2013-09-06 09:45:07 PDT
*** Bug 120752 has been marked as a duplicate of this bug. ***
Comment 35 Alexey Proskuryakov 2013-09-06 09:45:11 PDT
*** Bug 120753 has been marked as a duplicate of this bug. ***
Comment 36 Alexey Proskuryakov 2013-09-06 09:45:14 PDT
*** Bug 120754 has been marked as a duplicate of this bug. ***
Comment 37 Ryosuke Niwa 2013-09-06 10:30:06 PDT
Comment on attachment 210762 [details]
proposed fix

View in context: https://bugs.webkit.org/attachment.cgi?id=210762&action=review

> Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:153
> -        if self._options.pixel_tests:
> -            self._printer.write_update("Starting pixel test helper ...")
> -            self._port.start_helper()
> +        self._printer.write_update("Starting pixel test helper ...")
> +        self._port.start_helper()

We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now.
Perhaps we should do this only for Mac port?
Comment 38 Alexey Proskuryakov 2013-09-06 11:14:30 PDT
Committed <http://trac.webkit.org/r155196>.

Looks like this may not help EWS though! I know that this fixes it for me locally, but why is EWS still yellow? I'll have to keep looking.

> We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now.

Yeah, I guess that would be helpful for people who only ever run fast/js tests. Otherwise, we have reftests everywhere, even in http directory.

> Perhaps we should do this only for Mac port?

Currently, only the Mac port has a helper, it's a no-op for others.
Comment 39 Dirk Pranke 2013-09-06 11:41:10 PDT
(In reply to comment #37)
> We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now.

Given that we don't have a global list of reftests (we check each test on demand), you'd have to check that up front, and that could be fairly expensive.
Comment 40 Dirk Schulze 2013-09-06 11:44:09 PDT
(In reply to comment #38)
> Committed <http://trac.webkit.org/r155196>.
> 
> Looks like this may not help EWS though! I know that this fixes it for me locally, but why is EWS still yellow? I'll have to keep looking.

Currently the expected file on animations/cross-fade-background-image.html has a result that worked for me on my machine. If this test is failing now, I can submit another result.
Comment 41 Alexey Proskuryakov 2013-09-06 12:28:12 PDT
Looks like animations/cross-fade-background-image.html now passes on WK1, but it's still failing (with color management badness) on WK2.

We might need a separate new bug for this test specifically.
Comment 42 WebKit Commit Bot 2013-09-06 21:28:22 PDT
Re-opened since this is blocked by bug 120919
Comment 43 Ryosuke Niwa 2013-09-06 21:36:57 PDT
The EWS bots were simply encountering the display initialization problem. The right fix is to log into each one of them via VNC/screen share.
Comment 44 WebKit Commit Bot 2013-09-06 22:32:10 PDT
The commit-queue just saw fast/inline/layout-after-inserting-nested-br.html flake (image diff) while processing attachment 210780 [details] on bug 120884.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4
Comment 45 WebKit Commit Bot 2013-09-06 22:32:16 PDT
Created attachment 210870 [details]
Archive of layout-test-results from webkit-cq-01
Comment 46 WebKit Commit Bot 2013-09-06 22:32:27 PDT
The commit-queue just saw fast/shapes/shape-inside/shape-inside-overflow-fixed-dimensions-block-content.html flake (image diff) while processing attachment 210780 [details] on bug 120884.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4
Comment 47 WebKit Commit Bot 2013-09-06 22:32:32 PDT
Created attachment 210874 [details]
Archive of layout-test-results from webkit-cq-01
Comment 48 WebKit Commit Bot 2013-09-06 22:34:11 PDT
The commit-queue just saw fast/shapes/shape-outside-floats/shape-outside-floats-image-002.html flake (image diff) while processing attachment 210820 [details] on bug 120911.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4
Comment 49 WebKit Commit Bot 2013-09-06 22:34:17 PDT
Created attachment 210896 [details]
Archive of layout-test-results from webkit-cq-03
Comment 50 WebKit Commit Bot 2013-09-06 22:34:26 PDT
The commit-queue just saw fast/shapes/shape-outside-floats/shape-outside-floats-image-001.html flake (image diff) while processing attachment 210820 [details] on bug 120911.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.8.4
Comment 51 WebKit Commit Bot 2013-09-06 22:34:32 PDT
Created attachment 210897 [details]
Archive of layout-test-results from webkit-cq-03
Comment 52 Ryosuke Niwa 2013-09-06 23:25:40 PDT
I've rebooted all EWS bots, manually logged into them via screen sharing, and manually set sRGB as the color profile. We'll see if test failures will go away now.  If not, then something must have regressed in DRT/WTR or WebKit itself.
Comment 53 Ryosuke Niwa 2013-09-06 23:51:13 PDT
Some bots are still seeing image only failures. However, it appears that tests do pass on those bots immediately after I've manually logged into them via screen sharing.

I've realized that screen saver was enabled on some of them so I've disabled them manually. Is it possible for screen saver somehow affect the color profile or accelerated compositing?
Comment 54 Ryosuke Niwa 2013-09-06 23:58:57 PDT
Not fixed.

Somehow a different color profile is used when accelerated compositing is enabled.
Comment 55 Mark Hahnenberg 2014-05-23 19:26:25 PDT
Is this bug still a thing? css2.1/20110323/border-conflict-element-001d.htm	 is flaking a lot on the Mavericks Debug WK2 bots, and the bug I found for that test was duped to this one.
Comment 56 Alexey Proskuryakov 2014-05-23 23:44:36 PDT
Color management troubles are still a thing indeed, although I'm not sure if this bug is the best way to track fixing them.

For now, I've switched the bot that had trouble with css2.1/20110323/border-conflict-element-001d.htm to sRGB manually, which should resolve this (and I believe someone else - maybe Beth - has switched one or more bots to sRGB this week already).