window.navigator.language incorrectly returns all lowercase string

• Component: Bindings
• Status: RESOLVED FIXED

[Matt Stow]

When calling window.navigator.language, Safari will return something like "en-us", whereas every other browser returns "en-US".

As per [MDN](https://developer.mozilla.org/en-US/docs/Web/API/NavigatorLanguage/language) and defined in [BCP47](http://www.ietf.org/rfc/bcp/bcp47.txt), valid Extended Language Subtags must be 2*3ALPHA (uppercase).

In the current web app I'm building, this caused havoc with our localization, as the "en-us" didn't match any languages we had localized strings for.

[Chris Dumez]

Specification:
- https://html.spec.whatwg.org/#dom-navigator-language
- https://tools.ietf.org/html/bcp47

I confirmed that the part after the '-' is uppercase in Firefox and Chrome.

[Chris Dumez]

in particular https://tools.ietf.org/html/bcp47#section-2.2.4 for Region subtag, which points to ISO3166-1 for country codes (that all appear to be uppercase).

[Chris Dumez]

As per https://developer.apple.com/reference/corefoundation/1666963-cflocale?language=objc , CFLocale is using BCP-47 language tags. Our Mac implementation is relying on CFLocaleCopyPreferredLanguages() which does return BCP-47 language tags.

However, we then call httpStyleLanguageCode() on them which alters their format (and lower cases them).

[Chris Dumez]

(In reply to comment #3)
> As per
> https://developer.apple.com/reference/corefoundation/1666963-
> cflocale?language=objc , CFLocale is using BCP-47 language tags. Our Mac
> implementation is relying on CFLocaleCopyPreferredLanguages() which does
> return BCP-47 language tags.
> 
> However, we then call httpStyleLanguageCode() on them which alters their
> format (and lower cases them).

The name of this function seems to indicate this formatting is used for HTTP. However, RFC 2616 says:
"""
3.10 Language Tags

A language tag identifies a natural language spoken, written, or otherwise conveyed by human beings for communication of information to other human beings. Computer languages are explicitly excluded. HTTP uses language tags within the Accept-Language and Content- Language fields.

The syntax and registry of HTTP language tags is the same as that defined by RFC 1766 [1]. In summary, a language tag is composed of 1 or more parts: A primary language tag and a possibly empty series of subtags:

        language-tag  = primary-tag *( "-" subtag )
        primary-tag   = 1*8ALPHA
        subtag        = 1*8ALPHA
White space is not allowed within the tag and all tags are case- insensitive. The name space of language tags is administered by the IANA. Example tags include:

       en, en-US, en-cockney, i-cherokee, x-pig-latin
where any two-letter primary-tag is an ISO-639 language abbreviation and any two-letter initial subtag is an ISO-3166 country code. (The last three tags above are not registered tags; all but the last are examples of tags which could be registered in future.)
"""

https://www.ietf.org/rfc/rfc1766.txt says that language tags are case insensitive so the fact that we no longer return lowercase would not break HTTP use-cases.

[Chris Dumez]

Created attachment 290982 [details]
Patch

[Geoffrey Garen]

+CC ap, so he can tell us why we are wrong.

[Chris Dumez]

(In reply to comment #6)
> +CC ap, so he can tell us why we are wrong.

:D

[WebKit Commit Bot]

Comment on attachment 290982 [details]
Patch

Clearing flags on attachment: 290982

Committed r206949: <http://trac.webkit.org/changeset/206949>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Alexey Proskuryakov]

> https://www.ietf.org/rfc/rfc1766.txt says that language tags are case insensitive so the fact that we no longer return lowercase would not break HTTP use-cases.

HTTP servers break for any attempted change to Accept-Language, so one needs to guess which behavior breaks the least amount of sites. Chrome and Firefox use upper case country codes in http too now, so please file a radar against CFNetwork to consider changing this.

[Thomas Steiner]

There seems to be a regression in iOS 14.6 Beta (Safari UA "Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1"). 

`navigator.language` 
// "en-us"

`navigator.languages` 
// ["en-us"]

[Chris Dumez]

(In reply to Thomas Steiner from comment #11)
> There seems to be a regression in iOS 14.6 Beta (Safari UA "Mozilla/5.0
> (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like
> Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1"). 
> 
> `navigator.language` 
> // "en-us"
> 
> `navigator.languages` 
> // ["en-us"]

This looks like a regression from Bug 200043.

[Darin Adler]

We need a new bug report, then?

[Chris Dumez]

(In reply to Darin Adler from comment #13)
> We need a new bug report, then?

Yes, I have fixed Bug 225461.

[Darin Adler]

(In reply to Chris Dumez from comment #14)
> Yes, I have fixed Bug 225461.

You meant filed.

[Chris Dumez]

(In reply to Darin Adler from comment #15)
> (In reply to Chris Dumez from comment #14)
> > Yes, I have fixed Bug 225461.
> 
> You meant filed.

Oops, Indeed.

[Chris Dumez]

(In reply to Thomas Steiner from comment #11)
> There seems to be a regression in iOS 14.6 Beta (Safari UA "Mozilla/5.0
> (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like
> Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1"). 
> 
> `navigator.language` 
> // "en-us"
> 
> `navigator.languages` 
> // ["en-us"]

@Thomas: Are you sure this is a regression in iOS 14.6 Beta? Looking at the change that regressed it, it doesn't seem super recent.

[Chris Dumez]

(In reply to Chris Dumez from comment #17)
> (In reply to Thomas Steiner from comment #11)
> > There seems to be a regression in iOS 14.6 Beta (Safari UA "Mozilla/5.0
> > (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like
> > Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1"). 
> > 
> > `navigator.language` 
> > // "en-us"
> > 
> > `navigator.languages` 
> > // ["en-us"]
> 
> @Thomas: Are you sure this is a regression in iOS 14.6 Beta? Looking at the
> change that regressed it, it doesn't seem super recent.

FYI, I have just confirmed that this was already broken in iOS 14.5. So not a super recent regression.

[Thomas Steiner]

All my devices are always on the latest beta (#YOLO), and I first noticed it on 14.6. But, as was pointed out in the comment before, the bug may have been re-introduced before.

[Chris Dumez]

(In reply to Chris Dumez from comment #18)
> (In reply to Chris Dumez from comment #17)
> > (In reply to Thomas Steiner from comment #11)
> > > There seems to be a regression in iOS 14.6 Beta (Safari UA "Mozilla/5.0
> > > (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like
> > > Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1"). 
> > > 
> > > `navigator.language` 
> > > // "en-us"
> > > 
> > > `navigator.languages` 
> > > // ["en-us"]
> > 
> > @Thomas: Are you sure this is a regression in iOS 14.6 Beta? Looking at the
> > change that regressed it, it doesn't seem super recent.
> 
> FYI, I have just confirmed that this was already broken in iOS 14.5. So not
> a super recent regression.

Was already broken on 14.0 too. Anyway, I have am fixing this (again) via Bug 225461.