Bug 67880 – Some expected PNG files have color profiles, some do not, possibly causing pixel test failures

• Component: Tools / Tests
• Status: NEW

[Simon Fraser (smfr)]

I'm continually confused by pixel test failures that seem color-profile related (e.g. a test failure where the output looks the same, but the shade of green differs).

I notice that some of our expected PNG files have a "Generic RGB" profile, and some do not. This might explain things.

[Simon Fraser (smfr)]

Example:


12$ $ sips --getProperty profile *.png
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/composited-iframe-scroll-expected.png
  profile: Generic RGB Profile
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/fixed-position-iframe-expected.png
  profile: <nil>
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/iframe-content-flipping-expected.png
  profile: Generic RGB Profile
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/invisible-iframe-expected.png
  profile: <nil>
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/invisible-nested-iframe-expected.png
  profile: <nil>
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/nested-composited-iframe-expected.png
  profile: <nil>
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/nested-iframe-scrolling-expected.png
  profile: Generic RGB Profile
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/repaint-after-losing-scrollbars-expected-orig.png
  profile: Generic RGB Profile
/Volumes/Monster/Development/apple/webkit/WebKit.git/LayoutTests/compositing/iframes/repaint-after-losing-scrollbars-expected.png
  profile: <nil>

[Simon Fraser (smfr)]

composited-iframe-scroll-expected.png was updated by Tony in r84563, and has a color profile.
fixed-position-iframe-expected.png was updated by enne in r91597 and does not.
Their green colors do not match.

[Simon Fraser (smfr)]

Many, but not all of the PNGs with color profiles were added by Tony. Some were added by me or James in the compositing directory. It may be that PNGs generated on Lion have color profiles.

[Tony Chang]

In r84563, I was just adding checksums.  It looks like the original results already had color profiles; for example, r70509 added composited-iframe-scroll-expected.png.

I think results generated by Chromium's DRT never have color profiles, but those generated by Apple Mac DRT do have color profiles.

Adrienne, did you generate the results in r91597 with Chromium DRT or Apple Mac DRT?

Simon, are these failures when you run locally or also on the bots?  I thought DRT (both chromium and Apple mac) tried to make sure the monitor color profiles were consistent across runs.

[Adrienne Walker]

(In reply to comment #4)

> Adrienne, did you generate the results in r91597 with Chromium DRT or Apple Mac DRT?

I can't remember what I did in this case, but I usually generate all expected results with Chromium DRT on Linux (other than Mac-specific ones).

[Simon Fraser (smfr)]

We should both remove the color profiles from the results files, and fix the diffing tool to ignore them.

[James Robinson]

Whichever way we go we should teach check-webkit-style how to verify that new PNGs have the correct setting.  It sounds like we're thinking that PNGs for test expectations should _not_ have color profiles.

[Simon Fraser (smfr)]

It seems that Lion includes a color profile in the PNG file because we create the bitmap context with CGColorSpaceCreateDeviceRGB(). There doesn't seem to be an easy way to remove this profile.

[Tony Chang]

(In reply to comment #8)
> It seems that Lion includes a color profile in the PNG file because we create the bitmap context with CGColorSpaceCreateDeviceRGB(). There doesn't seem to be an easy way to remove this profile.

The PNG file format isn't that complicated.  If we can remove the whole iCCP section of the file, this shouldn't be too hard to do right before we write the file to disk.

Alternately, tools like pngcrush can remove the iCCP section of the file.  We could possible hook this into new-run-webkit-tests.

[Simon Fraser (smfr)]

It looks to me like the checksums in some of the images that Tony generated are wrong.

For example, in WebKitTestRunner on Mac, compositing/images/content-image-change.html generates a green (rgb(0, 255, 0)) square, and checksums using 00FF00FF pixel values. This gives a hash of bc04d27321e03146cf94bbc61b154f61.

However, the PNG file contains a hash of 109e6fe93cf270312fe30f132e38e950.

[Tony Chang]

(In reply to comment #10)
> It looks to me like the checksums in some of the images that Tony generated are wrong.
> 
> For example, in WebKitTestRunner on Mac, compositing/images/content-image-change.html generates a green (rgb(0, 255, 0)) square, and checksums using 00FF00FF pixel values. This gives a hash of bc04d27321e03146cf94bbc61b154f61.
> 
> However, the PNG file contains a hash of 109e6fe93cf270312fe30f132e38e950.

It's possible that the mac checksums in http://trac.webkit.org/changeset/84274 are incorrect. I think I made those on my local machine (may have been Leopard at the time, I can't remember).

[Simon Fraser (smfr)]

Using the tool changes in bug 68039, it appears that almost all the mac results have bad checksums.

[Simon Fraser (smfr)]

Created attachment 107284 [details]
List of files which appear to have bad checksums (on Lion)

Using the changes in bug 	 68039, I generated this list of expected PNG files whose checksum does not match the actual pixels in the image.

[Simon Fraser (smfr)]

So about 15,500 of the 43,700 expected images have bad checksums, on Lion. Looking at a few examples, the checksums in the images match the old .checksum files on disk. Also, some newly added png files have checksums that don't match.

That suggests that SL/Lion rendering differences may be at play here. I need to run my tool on SL and see what transpires.

[Simon Fraser (smfr)]

I'm getting different checksums from the same image on SL and Lion. Investigating.

[Simon Fraser (smfr)]

My code that re-generates the checksum from the image gives different results to the original image creation on Lion. So the attachment listing is suspect.

[Simon Fraser (smfr)]

aaaand it depends on the color profile of my main display. If that is set to Generic RGB, most (but not all) of the expected PNGs have the correct checksum.

[Simon Fraser (smfr)]

Created attachment 107436 [details]
List of results with bad checksums, generated on SL

List of ~8000 files which I think really do have bad checksums.