Bug 145380 - Add Content-DPR header support
Summary: Add Content-DPR header support
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-05-26 08:51 PDT by Yoav Weiss
Modified: 2016-11-23 03:18 PST (History)
13 users (show)

See Also:


Attachments
Patch (9.95 KB, patch)
2015-05-26 09:08 PDT, Yoav Weiss
yoav: review?
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Yoav Weiss 2015-05-26 08:51:50 PDT
Add Content-DPR header support
Comment 1 Yoav Weiss 2015-05-26 09:08:10 PDT
Created attachment 253706 [details]
Patch
Comment 2 Yoav Weiss 2015-05-26 09:14:48 PDT
This is the first patch in the implementation of Client-Hints, but is also also useful in the context of srcset alone.

It implements "Content-DPR" response header support, so that the server can impact the intrinsic dimensions of the displayed image.

That enables automatic image resizing without taking into account whether the image has CSS dimensions or not. (and therefore enables automatic image resizing without the risk of breaking layout)
Comment 3 Alexey Proskuryakov 2015-05-26 22:20:38 PDT
Comment on attachment 253706 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=253706&action=review

> Source/WebCore/ChangeLog:9
> +        Spec: http://igrigorik.github.io/http-client-hints/#confirming-selected-dpr

Which value wins when Content-DPR doesn't agree with image's internal metadata?

I thought that most formats had that already, making duplication of the same functionality in HTTP harmful.
Comment 4 Yoav Weiss 2015-05-27 00:10:15 PDT
The image's internal meta data determines the initial image dimensions, and then Content-DPR (Just like srcset's x or w descriptors) determine the image density which then scales the original intrinsic dimensions up or down.

So Content-DPR doesn't override the image's meta data but complements it. It does OTOH override srcset's descriptors, in case they were used.
Comment 5 Alexey Proskuryakov 2015-05-27 09:45:39 PDT
What is image density then? I assumed that you were talking about dpi/ppi, but that's internal image metadata. It is already known without http headers.
Comment 6 Yoav Weiss 2015-05-28 00:09:22 PDT
If you are referring to the density info that images contain (e.g. http://www.w3.org/TR/PNG/#11pHYs and the Xdensity, Ydensity in JFIF's APP0 marker), these seem to be largely ignored by both browsers and desktop image viewers, which means we cannot start supporting them without breaking content. 
Also, since the same image can be served in different width & density combinations, I'd say baking this info into the image is not ideal. (e.g. a 600px wide image that is displayed over the entire viewport can be served to a 1x, 600px wide screen or to a 2x, 300px wide screen)

FWIW, the header we're discussing here does the same thing that srcset's descriptors are already doing in markup across all modern browsers.
Comment 7 Alexey Proskuryakov 2015-05-28 01:18:50 PDT
(In reply to comment #6)
> If you are referring to the density info that images contain (e.g.
> http://www.w3.org/TR/PNG/#11pHYs and the Xdensity, Ydensity in JFIF's APP0
> marker), these seem to be largely ignored by both browsers and desktop image
> viewers, which means we cannot start supporting them without breaking
> content. 

What is the specific case that concerns you? Browsers ignore the Content-DPR header field too.

> Also, since the same image can be served in different width & density
> combinations, I'd say baking this info into the image is not ideal. (e.g. a
> 600px wide image that is displayed over the entire viewport can be served to
> a 1x, 600px wide screen or to a 2x, 300px wide screen)

What is the use case for that? One can just use CSS to stretch to 100%.

> FWIW, the header we're discussing here does the same thing that srcset's
> descriptors are already doing in markup across all modern browsers.

This also sounds like an argument against it. The distinction between markup and transport layer data is intentional, and when there are multiple ways to achieve the same thing, that invariably causes complications.

This is like the Munchkin game - game rules say one thing, text on a card says another, someone plays a card that reverses all the rules, and conflicts are officially resolved by who yells the loudest... It's fun, but why would anyone want browsers to work like that?
Comment 8 Yoav Weiss 2015-05-28 01:36:46 PDT
> What is the specific case that concerns you? Browsers ignore the Content-DPR header field too.

If browsers would start taking the density data images have integrated into account now, current images (that have that data set to something) would have the wrong intrinsic dimensions, and current content which layout depends on image intrinsic dimensions would break, without any opt-in signals from the content's author.

Now, browsers also ignore the Content-DPR header, but since there's practically zero existing content out there with that header, that's not an issue. The Content-DPR header would be an opt-in signal for the browser to take the provided density into account.

> What is the use case for that? One can just use CSS to stretch to 100%.

Yes, CSS overrides intrinsic dimensions. That doesn't mean that intrinsic dimensions are not useful when CSS is missing (because the image was accessed directly, or because the author didn't bother, which happens quite often).

> This also sounds like an argument against it. The distinction between markup and transport layer data is intentional, and when there are multiple ways to achieve the same thing, that invariably causes complications.

Which complications would this cause? The relationship here between srcset and the Content-DPR header is well defined. srcset's descriptor is applied, unless the server-side have overridden it, using Content-DPR.

That enables automatic image resizing (with either srcset or Client-Hints) without taking styling info into account, which adds significant complexity. Content-DPR enables the server-side to adapt the intrinsic dimensions to those of the original image, so that if layout relies on them, it will not break.

> This is like the Munchkin game - game rules say one thing, text on a card says another, someone plays a card that reverses all the rules, and conflicts are officially resolved by who yells the loudest... It's fun, but why would anyone want browsers to work like that?

I'm not familiar with that game.
Comment 9 Alexey Proskuryakov 2015-05-28 09:56:55 PDT
> If browsers would start taking the density data images have integrated into account now, current images (that have that data set to something) would have the wrong intrinsic dimensions, and current content which layout depends on image intrinsic dimensions would break, without any opt-in signals from the content's author.

This is a lot easier to resolve by adding an opt-in in HTML that would tell the engine to respect image dpi.

Adding such a signal at transport level does not make sense, it's not a feature of transport. Doing this would cause conflicts with loading infrastructure - http caches, appcache, blobs, service workers all need to be analyzed for potential issues, and for impact on future design directions, which is not an easy thing to do. Some of these may be OK, but others certainly aren't, notably, one can't add Content-DPR to a Blob.
Comment 10 Geoffrey Garen 2015-05-28 10:23:04 PDT
Comment on attachment 253706 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=253706&action=review

>> Source/WebCore/ChangeLog:9
>> +        Spec: http://igrigorik.github.io/http-client-hints/#confirming-selected-dpr
> 
> Which value wins when Content-DPR doesn't agree with image's internal metadata?
> 
> I thought that most formats had that already, making duplication of the same functionality in HTTP harmful.

"In case the server returned Content-DPR value contradicts previous client-side DPR indication, the server returned value must take precedence."
Comment 11 Ilya Grigorik 2015-05-28 16:13:50 PDT
(In reply to comment #9)
> > If browsers would start taking the density data images have integrated into account now, current images (that have that data set to something) would have the wrong intrinsic dimensions, and current content which layout depends on image intrinsic dimensions would break, without any opt-in signals from the content's author.
> 
> This is a lot easier to resolve by adding an opt-in in HTML that would tell
> the engine to respect image dpi.

That would require that *all* images on the page contain correct values: if any image has this data missing, or is embedding incorrect values, then they would be rendered incorrectly. This is a non-starter given the vast volume of existing image content on the web that may or may not already have this metadata on it.. and much of which is not controlled by the origin that would enable this flag. You would have to introduce an entirely new (and consistent across all image formats) field, such that we can guarantee that its presence is a clear opt-in indicator that the display density should be adapted; we can't --safely-- piggyback on existing fields for this purpose.

Further, as Yoav already pointed out, the same image can be served for different width & density combinations.. which means that your HTTP server would now need to be able to read/write image meta-data on the fly, as opposed to writing out a simple response header -- first is expensive (read/modify image header) and requires new capability in each server, second is cheap, simple, and works on every existing server.

Last but not least, "width: 100%" is not a solution.. There are many cases where the display size is determined by the image itself, not by some outside container or explicit width value on it -- this is a valid and very common use case on the web (e.g. viewing image URL directly). In order for such images to be displayed correctly, we need an explicit "intended DPR" signal on the response.

In short, (1) the density opt-in needs to be per image, not per document (this breaks down for direct image view also), (2) it has to be "safe", (3) it needs to be cheap to generate at response time by the server, (4) we can't rely on developer-provided width / container width. As a result, the combination of all of these is, I believe, best met by the proposed Content-DPR response header.

> Adding such a signal at transport level does not make sense, it's not a
> feature of transport. Doing this would cause conflicts with loading
> infrastructure - http caches, appcache, blobs, service workers all need to
> be analyzed for potential issues, and for impact on future design
> directions, which is not an easy thing to do. 

This is not true, there are no conflicts here, we're talking about simple content negotiation [1] that has well established mechanisms: client advertises its capabilities via request headers; server selects optimal response variant based on provided request fields; server advertises properties of the response that dictate how it should be cached and processed by the client and any intermediaries. This is robust, widely used (e.g. gzip, format negotiation, etc), and nothing new.

[1] http://httpwg.github.io/specs/rfc7231.html#content.negotiation
Comment 12 Alexey Proskuryakov 2015-05-28 17:32:46 PDT
> That would require that *all* images on the page contain correct values

As you acknowledge later, there is no problem when opt-in is per image (or per class when specified in CSS).

> Further, as Yoav already pointed out, the same image can be served for different width & density combinations

The best way to handle that is via HTML/CSS sizing.

> we're talking about simple content negotiation

Content negotiation has been a huge failure on the Web, it just doesn't work. This is pretty well established common knowledge, which matches my experience too.
Comment 13 Ilya Grigorik 2015-05-28 22:20:26 PDT
(In reply to comment #12)
> > That would require that *all* images on the page contain correct values
> 
> As you acknowledge later, there is no problem when opt-in is per image (or
> per class when specified in CSS).
>
> > Further, as Yoav already pointed out, the same image can be served for different width & density combinations
> 
> The best way to handle that is via HTML/CSS sizing.

For both of the above comments, we can't rely on HTML/CSS sizing as "best" or only solution: both are optional and we need a solution that works for cases where neither is specified. Case in point, if I'm opening a direct link to the image, I expect it to be displayed correctly, and there is no HTML or CSS context there.. We need the client to advertise its desired DPR and server to confirm what is has provided.
 
> > we're talking about simple content negotiation
> 
> Content negotiation has been a huge failure on the Web, it just doesn't
> work. This is pretty well established common knowledge, which matches my
> experience too.

That's not true. Yes, there are cases where content negotiation was proposed for certain use cases but failed to gain adoption for various reasons, but this does not imply that the underlying mechanism is "broken". You're successfully using it to negotiate gzip'ed versions of text assets on this very page; it's being used (very successfully) to negotiate and deliver various new image formats (e.g. WebP, JPEG-XR), and so on. It works.

Stepping back.. the intent here is to help developers automate delivery of images. Markup based solutions are good to have, but there is no reason to force developers to write unnecessary boilerplate where client and server can use well-established mechanisms to do this on their behalf (and markup has its own limitations, as noted above). Better, automation + content-negotiation allows us to deliver an improved experience to all existing content without any code modifications - e.g. existing <img src=..> tags become DPR-aware without any changes. This is a win on all sides.
Comment 14 Alexey Proskuryakov 2015-05-28 22:36:08 PDT
> Case in point, if I'm opening a direct link to the image

Standalone images are a rare case, and it is always possible to solve any problems with those by embedding them in an HTML document. We do not need to add complicated features to the platform to enable silly approaches - making every possible silly approach fixable at some other level is not the goal.

Yes, it is always possible to advocate for addition of features with theoretical "what ifs" like this one, but new features should solve real problems, not provide an umpteenth way to solve a problem that's already solved.

> You're successfully using it to negotiate gzip'ed versions of text assets on this very page;

This is a transport level feature, so it is implemented at the correct level. There is no content negotiation going on in this case.

> it's being used (very successfully) to negotiate and deliver various new image formats (e.g. WebP, JPEG-XR), and so on. It works.

This is surprising if true. Negotiation of HTML vs. XHTML was a tragic failure.
Comment 15 Mark Nottingham 2015-05-31 18:20:30 PDT
(In reply to comment #14)
> > Case in point, if I'm opening a direct link to the image
> 
> Standalone images are a rare case, and it is always possible to solve any
> problems with those by embedding them in an HTML document. We do not need to
> add complicated features to the platform to enable silly approaches - making
> every possible silly approach fixable at some other level is not the goal.

You called the proposal "silly" twice in the same sentence -- is that really necessary? 
 
> Yes, it is always possible to advocate for addition of features with
> theoretical "what ifs" like this one, but new features should solve real
> problems, not provide an umpteenth way to solve a problem that's already
> solved.

Content-DPR is a big win for servers and services that want to support responsive images without modifying HTML/CSS. Modifying response bodies has a server-side performance hit, and also introduces bugs. 

Talking to various teams that work on Web Performance Optimisation products, this is very much a real problem; perhaps it's just not as visible to you.
Comment 16 Alexey Proskuryakov 2015-05-31 23:38:51 PDT
> You called the proposal "silly" twice in the same sentence -- is that really necessary? 

Thank you for educating me about manners, but you misread the comment. What is silly is the idea that one should add features to browsing engines that enable changing layout via HTTP header fields, not the specific proposal for how to do that.

> Modifying response bodies has a server-side performance hit, and also introduces bugs.

You don't have to modify HTML on the fly, it can be simply written to support multiple image resolutions (like authors already do it today).

We had nearly exactly identical ideas implemented before, and the results were not pretty. The implementations continue to pollute the web platform long after they became unnecessary.

Perhaps the closest historic analogy is text re-encoding, where a dumb browser could only support one charset, and then the server - or even a proxy - would re-encode text/* responses, also changing Content-Type to include the updated charset (updating the charset inside HTML would have had a performance hit, and would have introduced bugs). The right short-term answer was of course to have browsers support multiple encodings, and it was implemented nearly immediately - but the consequences of an HTTP level solution still complicate our lives through convoluted rules for charset selection in text/* vs. application/* content.
Comment 17 Ilya Grigorik 2015-06-04 12:56:22 PDT
> Standalone images are a rare case, and it is always possible to solve any problems with those by embedding them in an HTML document.

Do you have data on "rare"? I expect the browser to behave correctly when I open an image in it, and its not unusual for users to share direct image links. My server can't magically wrap every direct image request (unknowable property since we can't count on Referrer) into an HTML document with appropriate size markup such that its displayed at correct resolution.

> We had nearly exactly identical ideas implemented before, and the results were not pretty. The implementations continue to pollute the web platform long after they became unnecessary.

Are you referring to images in particular? If so, what were those ideas and what were the results?

Last but not least, the fact that a markup solution is available (with known limitations, as described above) is not an argument that automation is unnecessary or not desired. We don't force iOS or Android developers to enumerate every resolution variant when an image is used -- both platforms rely on naming conventions to avoid exactly this pain. DPR request header + Content-DPR response header confirmation is the equivalent for web developers.