Bug 270981 – JPEG with HDR renders incorrectly

• Status: RESOLVED
• Resolution: MOVED
• Component: Images

[Alex Gunnarson]

Created attachment 470364 [details]
Reference image

There's been a regression in displaying the attached HDR JPEG (also available here — https://ibb.co/WgqNmZf). I assume this is the case for at least some other HDR JPEGs. I first noticed it on my own machine (MacBook Pro 16-inch 2021, Sonoma 14.4, Safari 17.4), but via BrowserStack, I've bisected the regression to Safari 17, across a variety of devices, the relevant sample of which is featured below:

Incorrect rendering (extremely dark):
- iPhone 12, iOS 17, Safari — https://ibb.co/DWZC758
- Mac Sonoma, Safari 17.3 — https://ibb.co/0sQJQcB

Correct rendering:
- iPhone 13, iOS 16, Safari — https://ibb.co/NZggPKF
- Mac Ventura, Safari 16.5 — https://ibb.co/9V3XjmV
- iPhone 7, iOS 10, Safari — https://ibb.co/JrdwRYD
- Mac El Capitan, Safari 9.1 — https://ibb.co/HCMfjMx

For what it's worth, this same class of bug also occurs starting in Edge 108 and Chrome 108 on Windows (tested on 8, 10, and 11):
- Correct: Windows 11, Edge 107 — https://ibb.co/3sn371X
- Incorrect: Windows 11, Edge 108 — https://ibb.co/4KFCp6T

Chrome on Mac, and Firefox on any platform, is unaffected by this bug:
- Mac Sonoma, Chrome 122 — https://ibb.co/prKKpwk
- Mac Sonoma, Firefox 123 — https://ibb.co/P5zrXGz

Note that the HDR JPEG was obtained by encoding the JPEG-XL available at https://people.csail.mit.edu/ericchan/hdr/jxl_images/20140606_102418_IMGP0297.jxl via `cjpegli forest.jxl forest-jpegli.jpeg --distance=0 --progressive_level=2`, where `cjpegli` refers to the `jpegli` encoding tool available at https://github.com/libjxl/libjxl/tree/v0.10.1.

[Alex Gunnarson]

Related bug report for Windows Chrome >= 108 here: https://issues.chromium.org/issues/329584683

[Greg]

That is a simple 8-bit JPG encoded with the PQ tone curve. That format is insufficient to avoid banding with HDR and I would not necessarily expect support (JXL or AVIF would be suitable for simple PQ encoding due to their higher bit-depth capabilities).

A JPG with a "gain map" would also be suitable for HDR (and ideal as gain maps better support SDR display than a tone mapped HDR), but will render as just the base SDR image in Safari / WebKit (they have never supported any HDR photo formats I've tested).

Adobe Lightroom / ACR offer great tools for creating your own JPG gain maps: https://gregbenzphotography.com/hdr-photos/jpg-hdr-gain-maps-in-adobe-camera-raw/

[Alex Gunnarson]

Thanks for your comment, Greg! Your site has been instrumental in helping our team make sense of the HDR landscape as we build out a new feature for our web app (shopdeft.com).

Interesting that you note some potential pitfalls around the format of the uploaded image (jpegli). The (co-)author of the JPEG-XL format, Jyrki Alakuijala, said that "Overall I believe jpegli is in its own class of jpeg codecs ... [it] allow[s] 10+ bits of dynamics for '8 bit' jpegs. I believe Jpegli and jpeg xl lead in the highest quality category (5 BPP)" (https://news.ycombinator.com/item?id=36429397). He also notes that "... we can ... jam more bits into the 8-bit formalism than the original authors thought would be possible and extend[] it into 10+ bits while being backward compatible. This can unblock traditional JPEG for high quality HDR uses, too" (https://www.linkedin.com/posts/jyrkialakuijala_jpegli-news-since-its-introduction-in-1992-activity-7158133901616390144-uHkZ).

I'm not yet well-versed in the world of HDR, so help me understand. I read Jyrki's statements as saying that jpegli (the format of the image featured in this bug report) is a high-quality HDR-capable format designed for backwards compatibility. When you say that "That format is insufficient to avoid banding with HDR", and "I would not necessarily expect support", what am I missing?

For what it's worth, viewing the jpegli in Safari 16 on my Mac M1 looks identical to: 
- The jpegli version as viewed in Chrome 122 on my Mac M1
- The JPEG XL version (from which the jpegli derives) as viewed in Safari 17 on my iPhone
- The AVIF version as viewed in Chrome 122 on my Mac M1
- Etc.

and it is distinct from the lossless WebP version as viewed in Chrome 122 on my Mac M1 (in the WebP as I've transcoded it, the HDR data appears to have been truncated to SDR, with the pure white "maxing out"; though it's not washed out, turned to near-black, etc.).

From this I can observe that the jpegli is indeed displaying correctly in HDR in a number of non-Safari-17 browsers, with no discernible (to me) difference in quality.

It appears there are a number of caveats to displaying HDR content, even in modern browsers. Correct me if I'm wrong, Greg, but seems like the best way to display HDR content at this time (assuming an HDR display, so no tone mapping required) is, in decreasing order of preference:
- JPEG XL (Safari 17; prior to this release, JPEG XL didn't appear to have HDR support on Safari)
- AVIF (non-Safari browsers; IIRC Safari 16 had spotty support)
- Single-frame HEVC (Safari < 17)
- JPEG gain maps (I believe jpegli falls into this category?)

[Greg]

Hi Alex- Glad to hear it's been so helpful!

I am unfamiliar with jpegli, though that sounds like a strong endorsement. Ultimately, 8 bits encodes less data than 10 and creates a higher risk of visible banding (quantization error) for a given encoding scheme/parameters. I assume what they likely are trying to do is to use dithering in an intelligent way such that it makes it hard to see banding at 8-bits (and probably selectively to avoid increasing file size by encoding noise where it isn't helpful). But that's just my uninformed guess.

Even if that is successful for an SDR JPG, I would have several concerns extrapolating that success to HDR:
(1) HDR encodes a much large range of luminance - which means each bit is stretched further and the risk of banding increases. You're well beyond the "Barten ramp" at this point and very prone to banding (12-bits required to avoid banding for some images).
(2) HDR is much more likely to use P3 and Rec 20202 gamuts, which adds to that risk.
(3) AVIF is now set to become a major file standard (now that all modern browsers support it) and offers much better file size efficiency. I believe it has the potential to quickly displace JPG once adoption starts to take off.

At this time, I see the HDR file format landscape as:
- ISO JPG gain map is the ideal way to share HDR on the web. It is 100% safe (falls back to SDR on both non-supporting displays and non-supporting browsers). This is the format encoded by Adobe software and Android devices, with Adobe using the spec more fully at this time with a 3-channel vs luminosity only gain map. [Note that Apple has a proprietary JPG gain map standard which is different.]
- Second best is HDR AVIF (no gain map), which is well supported by Chrome, Edge, Brave, and Opera. Safari will always tone map it due to a lack of HDR support, and Firefox fails to render properly (makes a very dark image). With a standard AVIF, tone mapping will be used if the display is SDR or a less-capable HDR display (Chrome-based browsers offer much better tone mapping that Safari at this time). Gain maps produce superior results as the SDR rendition has input from the creator. So I'm not generally a fan of simple HDR AVIF, though I use it on my test site to eliminate SDR fallback for testing/demonstration purposes.
- In the future, the best solution will likely be AVIF with a gain map. We don't have encoders yet, but Chrome already supports it under a dev flag and looks great with samples published by Adobe. With an AVIF gain map, we would get smaller files with the superior flexibility of a gain map to support both SDR and HDR displays. The quality will very likely be higher as well. And it would support transparency (though I haven't see any gain map support for that yet).

JPEG XL (JXL) is a great format, but probably going nowhere fast since Chrome pulled dev support. https://caniuse.com/jpegxl
HEIC is another great format, but with limited support: https://caniuse.com/heif

[Radar WebKit Bug Importer]

<rdar://problem/125174885>

[Said Abou-Hallawa]

This is an issue below WebKit.

[Alex Gunnarson]

The image renders correctly on my Mac (MacBook Pro 16-inch 2021, Sonoma 14.4) in Preview and in Chrome (122), but not in Safari (17.4). Curious how this could be an issue below WebKit. Perhaps some library WebKit depends on that neither Preview nor Chrome do? I'll leave it to you! Thanks.

[Said Abou-Hallawa]

Using the same WebKit revision, Webkit renders the reference image correctly on macOS Monetary but it renders it as a black rectangle on macOS Sonoma.

Preview might be using a system API to render the image different from what WebKit is using. Or Preview and WebKit might be using the same API but with different arguments.

[Alex Gunnarson]

Makes sense! Thank you for isolating!

[Alex Gunnarson]

And Greg — thanks so much for your additional insight! Always extremely helpful. After some testing, my current understanding is that the most reliable HDR support in browsers is to be found in single-frame VP9s at this time. HDR image support is nonexistent in Safari (Firefox fares worse):

- HDR AVIF is tone-mapped to SDR in Safari
- HDR JPEG-XL is also tone-mapped to SDR in Safari (I just realized this the other day)
- HDR-gain-mapped JPEGs (Ultra HDR) are tone-mapped in Safari 
- HDR-via-PQ-ICC-profile JPEGs are tone-mapped in Safari <= 16, but black in Safari 17 (thus this bug report)
- HDR-via-PQ-ICC-profile WebPs are black in Safari 17 (possibly same bug as the JPEG?)
- HDR-via-PQ-ICC-profile PNGs are tone-mapped in Safari
- Surprisingly, even HDR HEICs display unexpectedly in Safari per my tests (very dark, though not black), but that could be an encoding issue on my end

HDR-capable video formats fare better:
- AV1 isn't supported on Apple devices without hardware decoders, but HDR AV1 renders correctly in Chrome
- HDR H.265/HEVC renders great in Safari, but not Chrome or Firefox
- HDR VP9 renders great in Safari, Chrome, and Firefox — the only consistent one so far!
- HDR-via-PQ-ICC-profile VP8 renders pretty close to VP9, but has banding due to its 8-bitness

Also, while single-frame VP9 appears to be the most reliable option, rendering more than 16 single-frame HDR `video` elements at a time (of whatever format and size) in Safari is unreliable. Sometimes I can get ~54 to display, but this can cause the browser to get into a strange state, and sometimes crash. Usually >16 will sporadically fail to render at all. In Firefox I at least get a warning about some sort of initialization error, presumably due to some resource limits.

And HDR gets clipped to SDR in WebGL 2 — I made the mistake of trying :)