Add Content-DPR header support

• Component: New Bugs
• Status: RESOLVED WONTFIX

[Yoav Weiss]

Add Content-DPR header support

[Yoav Weiss]

Created attachment 253706 [details]
Patch

[Yoav Weiss]

This is the first patch in the implementation of Client-Hints, but is also also useful in the context of srcset alone.

It implements "Content-DPR" response header support, so that the server can impact the intrinsic dimensions of the displayed image.

That enables automatic image resizing without taking into account whether the image has CSS dimensions or not. (and therefore enables automatic image resizing without the risk of breaking layout)

[Alexey Proskuryakov]

Comment on attachment 253706 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=253706&action=review

> Source/WebCore/ChangeLog:9
> +        Spec: http://igrigorik.github.io/http-client-hints/#confirming-selected-dpr

Which value wins when Content-DPR doesn't agree with image's internal metadata?

I thought that most formats had that already, making duplication of the same functionality in HTTP harmful.

[Yoav Weiss]

The image's internal meta data determines the initial image dimensions, and then Content-DPR (Just like srcset's x or w descriptors) determine the image density which then scales the original intrinsic dimensions up or down.

So Content-DPR doesn't override the image's meta data but complements it. It does OTOH override srcset's descriptors, in case they were used.

[Alexey Proskuryakov]

What is image density then? I assumed that you were talking about dpi/ppi, but that's internal image metadata. It is already known without http headers.

[Yoav Weiss]

If you are referring to the density info that images contain (e.g. http://www.w3.org/TR/PNG/#11pHYs and the Xdensity, Ydensity in JFIF's APP0 marker), these seem to be largely ignored by both browsers and desktop image viewers, which means we cannot start supporting them without breaking content. 
Also, since the same image can be served in different width & density combinations, I'd say baking this info into the image is not ideal. (e.g. a 600px wide image that is displayed over the entire viewport can be served to a 1x, 600px wide screen or to a 2x, 300px wide screen)

FWIW, the header we're discussing here does the same thing that srcset's descriptors are already doing in markup across all modern browsers.

[Alexey Proskuryakov]

(In reply to comment #6)
> If you are referring to the density info that images contain (e.g.
> http://www.w3.org/TR/PNG/#11pHYs and the Xdensity, Ydensity in JFIF's APP0
> marker), these seem to be largely ignored by both browsers and desktop image
> viewers, which means we cannot start supporting them without breaking
> content. 

What is the specific case that concerns you? Browsers ignore the Content-DPR header field too.

> Also, since the same image can be served in different width & density
> combinations, I'd say baking this info into the image is not ideal. (e.g. a
> 600px wide image that is displayed over the entire viewport can be served to
> a 1x, 600px wide screen or to a 2x, 300px wide screen)

What is the use case for that? One can just use CSS to stretch to 100%.

> FWIW, the header we're discussing here does the same thing that srcset's
> descriptors are already doing in markup across all modern browsers.

This also sounds like an argument against it. The distinction between markup and transport layer data is intentional, and when there are multiple ways to achieve the same thing, that invariably causes complications.

This is like the Munchkin game - game rules say one thing, text on a card says another, someone plays a card that reverses all the rules, and conflicts are officially resolved by who yells the loudest... It's fun, but why would anyone want browsers to work like that?

[Yoav Weiss]

> What is the specific case that concerns you? Browsers ignore the Content-DPR header field too.

If browsers would start taking the density data images have integrated into account now, current images (that have that data set to something) would have the wrong intrinsic dimensions, and current content which layout depends on image intrinsic dimensions would break, without any opt-in signals from the content's author.

Now, browsers also ignore the Content-DPR header, but since there's practically zero existing content out there with that header, that's not an issue. The Content-DPR header would be an opt-in signal for the browser to take the provided density into account.

> What is the use case for that? One can just use CSS to stretch to 100%.

Yes, CSS overrides intrinsic dimensions. That doesn't mean that intrinsic dimensions are not useful when CSS is missing (because the image was accessed directly, or because the author didn't bother, which happens quite often).

> This also sounds like an argument against it. The distinction between markup and transport layer data is intentional, and when there are multiple ways to achieve the same thing, that invariably causes complications.

Which complications would this cause? The relationship here between srcset and the Content-DPR header is well defined. srcset's descriptor is applied, unless the server-side have overridden it, using Content-DPR.

That enables automatic image resizing (with either srcset or Client-Hints) without taking styling info into account, which adds significant complexity. Content-DPR enables the server-side to adapt the intrinsic dimensions to those of the original image, so that if layout relies on them, it will not break.

> This is like the Munchkin game - game rules say one thing, text on a card says another, someone plays a card that reverses all the rules, and conflicts are officially resolved by who yells the loudest... It's fun, but why would anyone want browsers to work like that?

I'm not familiar with that game.

[Alexey Proskuryakov]

> If browsers would start taking the density data images have integrated into account now, current images (that have that data set to something) would have the wrong intrinsic dimensions, and current content which layout depends on image intrinsic dimensions would break, without any opt-in signals from the content's author.

This is a lot easier to resolve by adding an opt-in in HTML that would tell the engine to respect image dpi.

Adding such a signal at transport level does not make sense, it's not a feature of transport. Doing this would cause conflicts with loading infrastructure - http caches, appcache, blobs, service workers all need to be analyzed for potential issues, and for impact on future design directions, which is not an easy thing to do. Some of these may be OK, but others certainly aren't, notably, one can't add Content-DPR to a Blob.

[Geoffrey Garen]

Comment on attachment 253706 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=253706&action=review

>> Source/WebCore/ChangeLog:9
>> +        Spec: http://igrigorik.github.io/http-client-hints/#confirming-selected-dpr
> 
> Which value wins when Content-DPR doesn't agree with image's internal metadata?
> 
> I thought that most formats had that already, making duplication of the same functionality in HTTP harmful.

"In case the server returned Content-DPR value contradicts previous client-side DPR indication, the server returned value must take precedence."

[Ilya Grigorik]

(In reply to comment #9)
> > If browsers would start taking the density data images have integrated into account now, current images (that have that data set to something) would have the wrong intrinsic dimensions, and current content which layout depends on image intrinsic dimensions would break, without any opt-in signals from the content's author.
> 
> This is a lot easier to resolve by adding an opt-in in HTML that would tell
> the engine to respect image dpi.

That would require that *all* images on the page contain correct values: if any image has this data missing, or is embedding incorrect values, then they would be rendered incorrectly. This is a non-starter given the vast volume of existing image content on the web that may or may not already have this metadata on it.. and much of which is not controlled by the origin that would enable this flag. You would have to introduce an entirely new (and consistent across all image formats) field, such that we can guarantee that its presence is a clear opt-in indicator that the display density should be adapted; we can't --safely-- piggyback on existing fields for this purpose.

Further, as Yoav already pointed out, the same image can be served for different width & density combinations.. which means that your HTTP server would now need to be able to read/write image meta-data on the fly, as opposed to writing out a simple response header -- first is expensive (read/modify image header) and requires new capability in each server, second is cheap, simple, and works on every existing server.

Last but not least, "width: 100%" is not a solution.. There are many cases where the display size is determined by the image itself, not by some outside container or explicit width value on it -- this is a valid and very common use case on the web (e.g. viewing image URL directly). In order for such images to be displayed correctly, we need an explicit "intended DPR" signal on the response.

In short, (1) the density opt-in needs to be per image, not per document (this breaks down for direct image view also), (2) it has to be "safe", (3) it needs to be cheap to generate at response time by the server, (4) we can't rely on developer-provided width / container width. As a result, the combination of all of these is, I believe, best met by the proposed Content-DPR response header.

> Adding such a signal at transport level does not make sense, it's not a
> feature of transport. Doing this would cause conflicts with loading
> infrastructure - http caches, appcache, blobs, service workers all need to
> be analyzed for potential issues, and for impact on future design
> directions, which is not an easy thing to do. 

This is not true, there are no conflicts here, we're talking about simple content negotiation [1] that has well established mechanisms: client advertises its capabilities via request headers; server selects optimal response variant based on provided request fields; server advertises properties of the response that dictate how it should be cached and processed by the client and any intermediaries. This is robust, widely used (e.g. gzip, format negotiation, etc), and nothing new.

[1] http://httpwg.github.io/specs/rfc7231.html#content.negotiation

[Alexey Proskuryakov]

> That would require that *all* images on the page contain correct values

As you acknowledge later, there is no problem when opt-in is per image (or per class when specified in CSS).

> Further, as Yoav already pointed out, the same image can be served for different width & density combinations

The best way to handle that is via HTML/CSS sizing.

> we're talking about simple content negotiation

Content negotiation has been a huge failure on the Web, it just doesn't work. This is pretty well established common knowledge, which matches my experience too.

[Ilya Grigorik]

(In reply to comment #12)
> > That would require that *all* images on the page contain correct values
> 
> As you acknowledge later, there is no problem when opt-in is per image (or
> per class when specified in CSS).
>
> > Further, as Yoav already pointed out, the same image can be served for different width & density combinations
> 
> The best way to handle that is via HTML/CSS sizing.

For both of the above comments, we can't rely on HTML/CSS sizing as "best" or only solution: both are optional and we need a solution that works for cases where neither is specified. Case in point, if I'm opening a direct link to the image, I expect it to be displayed correctly, and there is no HTML or CSS context there.. We need the client to advertise its desired DPR and server to confirm what is has provided.
 
> > we're talking about simple content negotiation
> 
> Content negotiation has been a huge failure on the Web, it just doesn't
> work. This is pretty well established common knowledge, which matches my
> experience too.

That's not true. Yes, there are cases where content negotiation was proposed for certain use cases but failed to gain adoption for various reasons, but this does not imply that the underlying mechanism is "broken". You're successfully using it to negotiate gzip'ed versions of text assets on this very page; it's being used (very successfully) to negotiate and deliver various new image formats (e.g. WebP, JPEG-XR), and so on. It works.

Stepping back.. the intent here is to help developers automate delivery of images. Markup based solutions are good to have, but there is no reason to force developers to write unnecessary boilerplate where client and server can use well-established mechanisms to do this on their behalf (and markup has its own limitations, as noted above). Better, automation + content-negotiation allows us to deliver an improved experience to all existing content without any code modifications - e.g. existing <img src=..> tags become DPR-aware without any changes. This is a win on all sides.

[Alexey Proskuryakov]

> Case in point, if I'm opening a direct link to the image

Standalone images are a rare case, and it is always possible to solve any problems with those by embedding them in an HTML document. We do not need to add complicated features to the platform to enable silly approaches - making every possible silly approach fixable at some other level is not the goal.

Yes, it is always possible to advocate for addition of features with theoretical "what ifs" like this one, but new features should solve real problems, not provide an umpteenth way to solve a problem that's already solved.

> You're successfully using it to negotiate gzip'ed versions of text assets on this very page;

This is a transport level feature, so it is implemented at the correct level. There is no content negotiation going on in this case.

> it's being used (very successfully) to negotiate and deliver various new image formats (e.g. WebP, JPEG-XR), and so on. It works.

This is surprising if true. Negotiation of HTML vs. XHTML was a tragic failure.

[Mark Nottingham]

(In reply to comment #14)
> > Case in point, if I'm opening a direct link to the image
> 
> Standalone images are a rare case, and it is always possible to solve any
> problems with those by embedding them in an HTML document. We do not need to
> add complicated features to the platform to enable silly approaches - making
> every possible silly approach fixable at some other level is not the goal.

You called the proposal "silly" twice in the same sentence -- is that really necessary? 
 
> Yes, it is always possible to advocate for addition of features with
> theoretical "what ifs" like this one, but new features should solve real
> problems, not provide an umpteenth way to solve a problem that's already
> solved.

Content-DPR is a big win for servers and services that want to support responsive images without modifying HTML/CSS. Modifying response bodies has a server-side performance hit, and also introduces bugs. 

Talking to various teams that work on Web Performance Optimisation products, this is very much a real problem; perhaps it's just not as visible to you.

[Alexey Proskuryakov]

> You called the proposal "silly" twice in the same sentence -- is that really necessary? 

Thank you for educating me about manners, but you misread the comment. What is silly is the idea that one should add features to browsing engines that enable changing layout via HTTP header fields, not the specific proposal for how to do that.

> Modifying response bodies has a server-side performance hit, and also introduces bugs.

You don't have to modify HTML on the fly, it can be simply written to support multiple image resolutions (like authors already do it today).

We had nearly exactly identical ideas implemented before, and the results were not pretty. The implementations continue to pollute the web platform long after they became unnecessary.

Perhaps the closest historic analogy is text re-encoding, where a dumb browser could only support one charset, and then the server - or even a proxy - would re-encode text/* responses, also changing Content-Type to include the updated charset (updating the charset inside HTML would have had a performance hit, and would have introduced bugs). The right short-term answer was of course to have browsers support multiple encodings, and it was implemented nearly immediately - but the consequences of an HTTP level solution still complicate our lives through convoluted rules for charset selection in text/* vs. application/* content.

[Ilya Grigorik]

> Standalone images are a rare case, and it is always possible to solve any problems with those by embedding them in an HTML document.

Do you have data on "rare"? I expect the browser to behave correctly when I open an image in it, and its not unusual for users to share direct image links. My server can't magically wrap every direct image request (unknowable property since we can't count on Referrer) into an HTML document with appropriate size markup such that its displayed at correct resolution.

> We had nearly exactly identical ideas implemented before, and the results were not pretty. The implementations continue to pollute the web platform long after they became unnecessary.

Are you referring to images in particular? If so, what were those ideas and what were the results?

Last but not least, the fact that a markup solution is available (with known limitations, as described above) is not an argument that automation is unnecessary or not desired. We don't force iOS or Android developers to enumerate every resolution variant when an image is used -- both platforms rely on naming conventions to avoid exactly this pain. DPR request header + Content-DPR response header confirmation is the equivalent for web developers.

[Brady Eidson]

Comment on attachment 253706 [details]
Patch

This patch has been pending review since 2015 with no recent activity.
It seems unlikely that it would even still apply to trunk in its current form.

Clearing from the review queue.

Feel free to update and resubmit if the patch is still relevant.

[Noam Rosenthal]

I am taking this bug

[Noam Rosenthal]

I am taking this bug

[Yoav Weiss]

Thanks for picking this up! :)

[Noam Rosenthal]

Created attachment 382445 [details]
Patch

[Noam Rosenthal]

I've submitted a new patch.
I saw that there was some contention about this patch back in 2015, I tried to address it in the change log.

Since the time of this discussion, the content-dpr header has landed in chromium and is used in production by CDN providers for what it was intended for, allowing the image pixel size to be different from the image intrinsic size for layout by allowing image providers/delivery-mechanisms to tell the user agents which density (DPR) is used in the image.

In some cases, for example, it lets CDNs serve a lower quality image when higher network latency is detected/projected, without the risk that that lower-quality image will break layout due to a different intrinsic size. That's a cheap win for performance and user experience, since it doesn't require web authors to change anything - the work required is minor changes by the user agents (this patch for example) and image delivery networks.

If there is still no consensus about it I would like to open the discussion to involve more people, and I'll switch over to @webkit-dev. There's also an option to put in the patch with a runtime feature-flag and experiment with it. Otherwise, I would love to receive some comments on the actual patch!

[Ryosuke Niwa]

The recent thinking about client hints in general is that it adds new fingerprinting surface to tracking pixels. Those 1px images typically don't get to run JS so the amount of information it can gather is limited. With this feature, it can gather extra bit of information, and a lot more with other client hints.

For these things, client hints including but not limited to Content-DPR header is not something Apple's WebKit is interested in implementing at this point in time.

[Ryosuke Niwa]

Comment on attachment 382445 [details]
Patch

This clearly needs a feature flag.

[Alexey Proskuryakov]

There is definitely no consensus. Image resolution is not a transport level feature.

[Noam Rosenthal]

(In reply to Ryosuke Niwa from comment #24)
> The recent thinking about client hints in general is that it adds new
> fingerprinting surface to tracking pixels. Those 1px images typically don't
> get to run JS so the amount of information it can gather is limited. With
> this feature, it can gather extra bit of information, and a lot more with
> other client hints.
> 
> For these things, client hints including but not limited to Content-DPR
> header is not something Apple's WebKit is interested in implementing at this
> point in time.

Content-DPR in its current form, unlike client hints, doesn't send any request headers. Am I missing something?

[Ryosuke Niwa]

(In reply to Noam Rosenthal from comment #27)
> (In reply to Ryosuke Niwa from comment #24)
> > The recent thinking about client hints in general is that it adds new
> > fingerprinting surface to tracking pixels. Those 1px images typically don't
> > get to run JS so the amount of information it can gather is limited. With
> > this feature, it can gather extra bit of information, and a lot more with
> > other client hints.
> > 
> > For these things, client hints including but not limited to Content-DPR
> > header is not something Apple's WebKit is interested in implementing at this
> > point in time.
> 
> Content-DPR in its current form, unlike client hints, doesn't send any
> request headers. Am I missing something?

Oh, I see. Yeah, this is slightly different because the header is in the response. We probably need to do a separate privacy review of this feature and see if it poses a new threat.

[Noam Rosenthal]

Created attachment 382569 [details]
Patch

[Noam Rosenthal]

New patch protects this feature behind a settings flag.

[Noam Rosenthal]

(In reply to Ryosuke Niwa from comment #28)
> We probably need to do a separate privacy review of this feature
> and see if it poses a new threat.

Cool, what is required for this?

[Yoav Weiss]

(In reply to Ryosuke Niwa from comment #24)
> The recent thinking about client hints in general is that it adds new
> fingerprinting surface to tracking pixels. Those 1px images typically don't
> get to run JS so the amount of information it can gather is limited. With
> this feature, it can gather extra bit of information, and a lot more with
> other client hints.
> 

That used to be true, but that is no longer the case, according to the latest PRs [1]

Feature Policy is now used to control which hints can and cannot be delegated to third parties.


[1] https://github.com/yoavweiss/client-hints-infrastructure/blob/master/specification_situation.md

[Ryosuke Niwa]

(In reply to Yoav Weiss from comment #32)
> (In reply to Ryosuke Niwa from comment #24)
> > The recent thinking about client hints in general is that it adds new
> > fingerprinting surface to tracking pixels. Those 1px images typically don't
> > get to run JS so the amount of information it can gather is limited. With
> > this feature, it can gather extra bit of information, and a lot more with
> > other client hints.
> > 
> 
> That used to be true, but that is no longer the case, according to the
> latest PRs [1]
> 
> Feature Policy is now used to control which hints can and cannot be
> delegated to third parties.

I don't want to derail the discussion too much but I don't think that's sufficient. Adding an attribute is too low of a bar to gate that kind of fingerprintability.

[Noam Rosenthal]

(In reply to Ryosuke Niwa from comment #25)
> Comment on attachment 382445 [details]
> Patch
> 
> This clearly needs a feature flag.

Done (runtime flag). Does it need a build flag as well IYO?

[Noam Rosenthal]

> > Feature Policy is now used to control which hints can and cannot be
> > delegated to third parties.
> 
> I don't want to derail the discussion too much but I don't think that's
> sufficient. Adding an attribute is too low of a bar to gate that kind of
> fingerprintability.

Yes, let's leave this discussion to when client hint spec is more mature, and stay focused on content-dpr which doesn't have these issues.

[Noam Rosenthal]

(In reply to Alexey Proskuryakov from comment #26)
> There is definitely no consensus. Image resolution is not a transport level
> feature.

I would argue that it technically is a transport level feature on the server side, if you consider CDNs a transport layer. CDNs are implementing image-resolution optimizations in a similar manner to how gzip works for text, which is what content-dpr is about... 
It's true that it's an innovation on top of what transport layers and CDNs used to do, and that it is specific to images, and that blurring the lines between content and transport is somewhat experimental and new.

But isn't innovation in optimizing ways to transport images something we want to support in the web, and is generally better for everyone?(In reply to Ryosuke Niwa from comment #25)
> Comment on attachment 382445 [details]
> Patch
> 
> This clearly needs a feature flag.

@rniwa, this has been resolved. Can we re-review? would love to respond to any additional comments if it's required before landing the patch.

[Alexey Proskuryakov]

> If there is still no consensus about it I would like to open the discussion to involve more people, and I'll switch over to @webkit-dev. 

As there is no consensus, please do.

[Noam Rosenthal]

Following this feedback, discussions on the mailing list and with standard bodies, continuing with an approach of indicating image density inline, as EXIF metadata: https://bugs.webkit.org/show_bug.cgi?id=212405

[Peng Liu]

(In reply to Noam Rosenthal from comment #38)
> Following this feedback, discussions on the mailing list and with standard
> bodies, continuing with an approach of indicating image density inline, as
> EXIF metadata: https://bugs.webkit.org/show_bug.cgi?id=212405

So the patch of this bug does not need review anymore?

[Noam Rosenthal]

(In reply to Peng Liu from comment #39)
> (In reply to Noam Rosenthal from comment #38)
> > Following this feedback, discussions on the mailing list and with standard
> > bodies, continuing with an approach of indicating image density inline, as
> > EXIF metadata: https://bugs.webkit.org/show_bug.cgi?id=212405
> 
> So the patch of this bug does not need review anymore?

Correct. Thanks for clearing the flag.

[Ryosuke Niwa]

Let's close this bug then. We can file a new bug to track EXIF implementation.

[Noam Rosenthal]

There is already a new bug for the EXIF implementation, OK to close this one.

[Ryosuke Niwa]

(In reply to Noam Rosenthal from comment #42)
> There is already a new bug for the EXIF implementation, OK to close this one.

Could you add the new bug to "See Also" section of the bug?