I'm continually confused by pixel test failures that seem color-profile related (e.g. a test failure where the output looks the same, but the shade of green differs). I notice that some of our expected PNG files have a "Generic RGB" profile, and some do not. This might explain things.
Example: 12$ $ sips --getProperty profile *.png /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/composited-iframe-scroll-expected.png profile: Generic RGB Profile /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/fixed-position-iframe-expected.png profile: <nil> /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/iframe-content-flipping-expected.png profile: Generic RGB Profile /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/invisible-iframe-expected.png profile: <nil> /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/invisible-nested-iframe-expected.png profile: <nil> /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/nested-composited-iframe-expected.png profile: <nil> /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/nested-iframe-scrolling-expected.png profile: Generic RGB Profile /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/repaint-after-losing-scrollbars-expected-orig.png profile: Generic RGB Profile /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/repaint-after-losing-scrollbars-expected.png profile: <nil>
composited-iframe-scroll-expected.png was updated by Tony in r84563, and has a color profile. fixed-position-iframe-expected.png was updated by enne in r91597 and does not. Their green colors do not match.
Many, but not all of the PNGs with color profiles were added by Tony. Some were added by me or James in the compositing directory. It may be that PNGs generated on Lion have color profiles.
In r84563, I was just adding checksums. It looks like the original results already had color profiles; for example, r70509 added composited-iframe-scroll-expected.png. I think results generated by Chromium's DRT never have color profiles, but those generated by Apple Mac DRT do have color profiles. Adrienne, did you generate the results in r91597 with Chromium DRT or Apple Mac DRT? Simon, are these failures when you run locally or also on the bots? I thought DRT (both chromium and Apple mac) tried to make sure the monitor color profiles were consistent across runs.
(In reply to comment #4) > Adrienne, did you generate the results in r91597 with Chromium DRT or Apple Mac DRT? I can't remember what I did in this case, but I usually generate all expected results with Chromium DRT on Linux (other than Mac-specific ones).
We should both remove the color profiles from the results files, and fix the diffing tool to ignore them.
Whichever way we go we should teach check-webkit-style how to verify that new PNGs have the correct setting. It sounds like we're thinking that PNGs for test expectations should _not_ have color profiles.
It seems that Lion includes a color profile in the PNG file because we create the bitmap context with CGColorSpaceCreateDeviceRGB(). There doesn't seem to be an easy way to remove this profile.
(In reply to comment #8) > It seems that Lion includes a color profile in the PNG file because we create the bitmap context with CGColorSpaceCreateDeviceRGB(). There doesn't seem to be an easy way to remove this profile. The PNG file format isn't that complicated. If we can remove the whole iCCP section of the file, this shouldn't be too hard to do right before we write the file to disk. Alternately, tools like pngcrush can remove the iCCP section of the file. We could possible hook this into new-run-webkit-tests.
It looks to me like the checksums in some of the images that Tony generated are wrong. For example, in WebKitTestRunner on Mac, compositing/images/content-image-change.html generates a green (rgb(0, 255, 0)) square, and checksums using 00FF00FF pixel values. This gives a hash of bc04d27321e03146cf94bbc61b154f61. However, the PNG file contains a hash of 109e6fe93cf270312fe30f132e38e950.
(In reply to comment #10) > It looks to me like the checksums in some of the images that Tony generated are wrong. > > For example, in WebKitTestRunner on Mac, compositing/images/content-image-change.html generates a green (rgb(0, 255, 0)) square, and checksums using 00FF00FF pixel values. This gives a hash of bc04d27321e03146cf94bbc61b154f61. > > However, the PNG file contains a hash of 109e6fe93cf270312fe30f132e38e950. It's possible that the mac checksums in http://trac.webkit.org/changeset/84274 are incorrect. I think I made those on my local machine (may have been Leopard at the time, I can't remember).
Using the tool changes in bug 68039, it appears that almost all the mac results have bad checksums.
Created attachment 107284 [details] List of files which appear to have bad checksums (on Lion) Using the changes in bug 68039, I generated this list of expected PNG files whose checksum does not match the actual pixels in the image.
So about 15,500 of the 43,700 expected images have bad checksums, on Lion. Looking at a few examples, the checksums in the images match the old .checksum files on disk. Also, some newly added png files have checksums that don't match. That suggests that SL/Lion rendering differences may be at play here. I need to run my tool on SL and see what transpires.
I'm getting different checksums from the same image on SL and Lion. Investigating.
My code that re-generates the checksum from the image gives different results to the original image creation on Lion. So the attachment listing is suspect.
aaaand it depends on the color profile of my main display. If that is set to Generic RGB, most (but not all) of the expected PNGs have the correct checksum.
Created attachment 107436 [details] List of results with bad checksums, generated on SL List of ~8000 files which I think really do have bad checksums.