NEW 67880
Some expected PNG files have color profiles, some do not, possibly causing pixel test failures 
https://bugs.webkit.org/show_bug.cgi?id=67880
Summary Some expected PNG files have color profiles, some do not, possibly causing pi...
Simon Fraser (smfr)
Reported 2011-09-09 19:06:09 PDT
I'm continually confused by pixel test failures that seem color-profile related (e.g. a test failure where the output looks the same, but the shade of green differs). I notice that some of our expected PNG files have a "Generic RGB" profile, and some do not. This might explain things.
Attachments
List of files which appear to have bad checksums (on Lion) (2.58 MB, text/plain)
2011-09-13 22:03 PDT, Simon Fraser (smfr) 		no flags
Details
List of results with bad checksums, generated on SL (1.34 MB, text/plain)
2011-09-14 18:27 PDT, Simon Fraser (smfr) 		no flags
Details
View All Add attachment proposed patch, testcase, etc.
Simon Fraser (smfr)
Comment 1 2011-09-09 19:07:04 PDT
Example: 12$ $ sips --getProperty profile *.png /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/composited-iframe-scroll-expected.png profile: Generic RGB Profile /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/fixed-position-iframe-expected.png profile: <nil> /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/iframe-content-flipping-expected.png profile: Generic RGB Profile /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/invisible-iframe-expected.png profile: <nil> /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/invisible-nested-iframe-expected.png profile: <nil> /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/nested-composited-iframe-expected.png profile: <nil> /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/nested-iframe-scrolling-expected.png profile: Generic RGB Profile /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/repaint-after-losing-scrollbars-expected-orig.png profile: Generic RGB Profile /Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/repaint-after-losing-scrollbars-expected.png profile: <nil>
Simon Fraser (smfr)
Comment 2 2011-09-09 19:09:29 PDT
composited-iframe-scroll-expected.png was updated by Tony in r84563, and has a color profile. fixed-position-iframe-expected.png was updated by enne in r91597 and does not. Their green colors do not match.
Simon Fraser (smfr)
Comment 3 2011-09-09 19:16:29 PDT
Many, but not all of the PNGs with color profiles were added by Tony. Some were added by me or James in the compositing directory. It may be that PNGs generated on Lion have color profiles.
Tony Chang
Comment 4 2011-09-12 09:50:35 PDT
In r84563, I was just adding checksums. It looks like the original results already had color profiles; for example, r70509 added composited-iframe-scroll-expected.png. I think results generated by Chromium's DRT never have color profiles, but those generated by Apple Mac DRT do have color profiles. Adrienne, did you generate the results in r91597 with Chromium DRT or Apple Mac DRT? Simon, are these failures when you run locally or also on the bots? I thought DRT (both chromium and Apple mac) tried to make sure the monitor color profiles were consistent across runs.
Adrienne Walker
Comment 5 2011-09-12 10:02:19 PDT
(In reply to comment #4) > Adrienne, did you generate the results in r91597 with Chromium DRT or Apple Mac DRT? I can't remember what I did in this case, but I usually generate all expected results with Chromium DRT on Linux (other than Mac-specific ones).
Simon Fraser (smfr)
Comment 6 2011-09-12 11:42:29 PDT
We should both remove the color profiles from the results files, and fix the diffing tool to ignore them.
James Robinson
Comment 7 2011-09-12 15:45:34 PDT
Whichever way we go we should teach check-webkit-style how to verify that new PNGs have the correct setting. It sounds like we're thinking that PNGs for test expectations should _not_ have color profiles.
Simon Fraser (smfr)
Comment 8 2011-09-12 16:16:25 PDT
It seems that Lion includes a color profile in the PNG file because we create the bitmap context with CGColorSpaceCreateDeviceRGB(). There doesn't seem to be an easy way to remove this profile.
Tony Chang
Comment 9 2011-09-12 16:34:35 PDT
(In reply to comment #8) > It seems that Lion includes a color profile in the PNG file because we create the bitmap context with CGColorSpaceCreateDeviceRGB(). There doesn't seem to be an easy way to remove this profile. The PNG file format isn't that complicated. If we can remove the whole iCCP section of the file, this shouldn't be too hard to do right before we write the file to disk. Alternately, tools like pngcrush can remove the iCCP section of the file. We could possible hook this into new-run-webkit-tests.
Simon Fraser (smfr)
Comment 10 2011-09-12 16:51:29 PDT
It looks to me like the checksums in some of the images that Tony generated are wrong. For example, in WebKitTestRunner on Mac, compositing/images/content-image-change.html generates a green (rgb(0, 255, 0)) square, and checksums using 00FF00FF pixel values. This gives a hash of bc04d27321e03146cf94bbc61b154f61. However, the PNG file contains a hash of 109e6fe93cf270312fe30f132e38e950.
Tony Chang
Comment 11 2011-09-12 17:00:50 PDT
(In reply to comment #10) > It looks to me like the checksums in some of the images that Tony generated are wrong. > > For example, in WebKitTestRunner on Mac, compositing/images/content-image-change.html generates a green (rgb(0, 255, 0)) square, and checksums using 00FF00FF pixel values. This gives a hash of bc04d27321e03146cf94bbc61b154f61. > > However, the PNG file contains a hash of 109e6fe93cf270312fe30f132e38e950. It's possible that the mac checksums in http://trac.webkit.org/changeset/84274 are incorrect. I think I made those on my local machine (may have been Leopard at the time, I can't remember).
Simon Fraser (smfr)
Comment 12 2011-09-13 18:40:16 PDT
Using the tool changes in bug 68039, it appears that almost all the mac results have bad checksums.
Simon Fraser (smfr)
Comment 13 2011-09-13 22:03:38 PDT
Created attachment 107284 [details] List of files which appear to have bad checksums (on Lion) Using the changes in bug 68039, I generated this list of expected PNG files whose checksum does not match the actual pixels in the image.
Simon Fraser (smfr)
Comment 14 2011-09-13 22:13:48 PDT
So about 15,500 of the 43,700 expected images have bad checksums, on Lion. Looking at a few examples, the checksums in the images match the old .checksum files on disk. Also, some newly added png files have checksums that don't match. That suggests that SL/Lion rendering differences may be at play here. I need to run my tool on SL and see what transpires.
Simon Fraser (smfr)
Comment 15 2011-09-14 13:48:05 PDT
I'm getting different checksums from the same image on SL and Lion. Investigating.
Simon Fraser (smfr)
Comment 16 2011-09-14 15:25:45 PDT
My code that re-generates the checksum from the image gives different results to the original image creation on Lion. So the attachment listing is suspect.
Simon Fraser (smfr)
Comment 17 2011-09-14 15:49:52 PDT
aaaand it depends on the color profile of my main display. If that is set to Generic RGB, most (but not all) of the expected PNGs have the correct checksum.
Simon Fraser (smfr)
Comment 18 2011-09-14 18:27:17 PDT
Created attachment 107436 [details] List of results with bad checksums, generated on SL List of ~8000 files which I think really do have bad checksums.
Note You need to log in before you can comment on or make changes to this bug.