Bug 171609 – [Cocoa] Stop performing caching of intermediate TLS certificates and/or automatic certificate downloading

• Status: RESOLVED
• Resolution: WONTFIX
• Component: WebCore Misc.

[Michael Catanzaro]

WebKitGTK+ is having a major web compatibility problem involving websites that fail to send a complete chain of TLS certificates. For example:

https://incomplete-chain.badssl.com/

This website sends only one TLS certificate with no intermediate certificate:

$ gnutls-cli incomplete-chain.badssl.com
Processed 172 CA certificate(s).
Resolving 'incomplete-chain.badssl.com:443'...
Connecting to '104.154.89.105:443'...
- Certificate type: X.509
- Got a certificate list of 1 certificates.
- Certificate[0] info:
 - subject `CN=*.badssl.com,OU=PositiveSSL Wildcard,OU=Domain Control Validated', issuer `CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB', serial 0x4c8e18714b34e75e8daefbe8f64c3a82, RSA key 2048 bits, signed using RSA-SHA256, activated `2016-07-07 00:00:00 UTC', expires `2017-09-05 23:59:59 UTC', pin-sha256="9SLklscvzMYj8f+52lp5ze/hY0CFHyLSPQzSpYYIBm8="
	Public Key ID:
		sha1:7965dfc93c6ae6fe8381ec482216ec44ef47282a
		sha256:f522e496c72fccc623f1ffb9da5a79cdefe16340851f22d23d0cd2a58608066f
	Public Key PIN:
		pin-sha256:9SLklscvzMYj8f+52lp5ze/hY0CFHyLSPQzSpYYIBm8=
	Public key's random art:
		+--[ RSA 2048]----+
		|                 |
		|     .           |
		|    o . .   o    |
		|     = o o o .o..|
		|    + + S o . .=.|
		| E . + o + o .. .|
		|  . . . + o  +o  |
		|         . .+. . |
		|            .o...|
		+-----------------+

- Status: The certificate is NOT trusted. The certificate issuer is unknown. The certificate chain uses insecure algorithm. 
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** handshake has failed: Error in the certificate.

Because legitimate certificate authorities never sign website certificates with their root certificates, all websites must send at least one intermediate certificate in order to have a valid chain of trust. Since incomplete-chain.badssl.com did not send the certificate "COMODO RSA Domain Validation Secure Server CA" along with its own certificate, GnuTLS has reasonably rejected this connection. I believe all WebKit ports except Apple ports currently reject this certificate chain, including libsoup backend ports like WebKitGTK+ and cURL backend ports like WinCairo. (Though I've only tested the cURL command line client, I would be really surprised if WinCairo implements this above cURL.) Konstantin has tested QtWebKit, which also rejects this chain. However, Safari accepts it, as do other major browsers like Firefox and Chrome.

My guess is that Safari only accepts this site because the user has previously visited some site that sent "COMODO RSA Domain Validation Secure Server CA" before and Safari has cached it for future use. This is perfectly safe and not a security problem. It also reduces certificate verification errors experienced by users. However, it means that certificate verification depends on the set of websites that the user has previously visited. For example, my old blog post [1] describes a situation where a Staples website triggers a security warning in Firefox unless you first visit Stack Overflow, after which the Staples website works properly in Firefox with no security warning. I assume Safari is likely doing the same thing here.

This is causing a web compatibility issue for other WebKit ports that do not perform this intermediate certificate caching, like WebKitGTK+, and various other command line tools. Our request is for Safari to stop performing this caching. Igalia could certainly implement this functionality for our ports, and we will if we have to, but it would be much better for the web as a whole if major browsers would start rejecting these connections. Many website administrators test to see if their website is working properly by loading it in major web browsers, instead of using an SSL test site like they should, and have no reason to expect that their website will be broken for users with fresh browser profiles. Removing this behavior from Safari would allow server administrators to reliably and predictably determine if their website will work properly independent of which websites the user has visited previously.

As an anecdote, a few years ago the IT department at my tech college was quite confused as to why an internal website did not work on newly-flashed machines. The website would start working mysteriously sooner or later. I eventually figured out that the internal website was not sending an essential intermediate certificate, and so it was only accessible once users first visited the school's public website. This confused a team of competent, technically-minded IT staffers. (In this example, we used Firefox, but the point remains.)

The downside of this proposal is that removing this caching behavior will temporarily increase certificate verification errors experienced by Safari users. I expect this problem will be quickly resolved by website administrators, as users will complain and website administrators fix their certificate chains. I'm contacting Google and Mozilla about this issue as well, but I hope Safari can take the lead here in removing this behavior.

Note: I'm not familiar with Apple's TLS stack, but I would guess that this certificate caching is most likely performed at a lower layer than WebKit, probably as part of the operating system. This could make it much more difficult to change the behavior than if it is implemented in WebKit.

[1] https://blogs.gnome.org/mcatanzaro/2015/01/30/mozilla-is-responsible-for-the-redhat-corpmerchandise-com-fiasco/

[Radar WebKit Bug Importer]

<rdar://problem/31975534>

[John Wilander]

Hi Michael!

For clarity, when you say "Igalia could certainly implement this functionality for our ports, and we will if we have to", do you mean you could implement intermediary caching to be able to accept incomplete chain connections in many cases?

[Michael Catanzaro]

(In reply to John Wilander from comment #2)
> Hi Michael!
> 
> For clarity, when you say "Igalia could certainly implement this
> functionality for our ports, and we will if we have to", do you mean you
> could implement intermediary caching to be able to accept incomplete chain
> connections in many cases?

Yeah... if we can't get any traction on having this feature removed from some major browser, as would be ideal, then we'll eventually give up and implement it. (It would actually need to be done in one of our dependencies, glib-networking.)

[John Wilander]

(In reply to Michael Catanzaro from comment #3)
> (In reply to John Wilander from comment #2)
> > Hi Michael!
> > 
> > For clarity, when you say "Igalia could certainly implement this
> > functionality for our ports, and we will if we have to", do you mean you
> > could implement intermediary caching to be able to accept incomplete chain
> > connections in many cases?
> 
> Yeah... if we can't get any traction on having this feature removed from
> some major browser, as would be ideal, then we'll eventually give up and
> implement it. (It would actually need to be done in one of our dependencies,
> glib-networking.)

Thanks!

I have now passed on your request to the engineering team behind Safari's TLS stack. If there's anything we can share from that conversation, I'll do it here.

[Michael Catanzaro]

OK, I totally forgot when I filed this earlier today, but there is another possibility for how Safari might be verifying this incomplete chain. It could be using the authority information access extension of the server certificate [1] to download the intermediate. If so, then the result is predictable, this bug report is invalid, and we should change nothing on Mac. Instead, other WebKit ports should implement the same functionality in their networking backends for compatibility. (Igalia would handle the libsoup backend.)

But if it's caching intermediates, then everything I said earlier is valid and Safari should stop doing that. It would be good to get confirmation from the relevant Apple developers as to what is happening here: is it caching intermediate certificates, or is it using authority information access?

[1] https://tools.ietf.org/html/rfc5280#section-4.2.2.1

[Michael Catanzaro]

And five minutes later, I have confirmation from Google that AIA is in fact what Chrome is doing here, with no intermediate certificate caching. Since Chrome uses authority information access, we pretty much have to implement that now.

But my Google contact is telling me they think you do *both* intermediate certificate caching *and* authority information access lookup. It would be great if you could confirm this, since if Safari is indeed caching intermediate certificates, that is independently a bad thing.

[Michael Catanzaro]

(In reply to Michael Catanzaro from comment #6) 
> But my Google contact is telling me they think you do *both* intermediate
> certificate caching *and* authority information access lookup. It would be
> great if you could confirm this, since if Safari is indeed caching
> intermediate certificates, that is independently a bad thing.

Actually I misunderstood his mail, he's saying he thinks you are doing only AIA. If your TLS people can verify that, then this bug report should be closed as invalid (and sorry for the false alarm). I have learned a lot about this today....

[Michael Catanzaro]

We're going to implement authority information access lookup for soup-based WebKit ports, despite the privacy concerns, because it seems to be the only option for web compatibility right now.

[Michael Catanzaro]

(In reply to Michael Catanzaro from comment #8)
> We're going to implement authority information access lookup for soup-based
> WebKit ports, despite the privacy concerns, because it seems to be the only
> option for web compatibility right now.

(the only option that does not result in nondeterministic certificate verification)