Bug 142074 – [GTK] Improve user agent construction

• Status: RESOLVED
• Resolution: FIXED
• Component: WebKitGTK

[Michael Catanzaro]

For Fedora we are adding our branding to the default WebKitGTK+ user agent. Since this in non-intrusive and might be useful for other distributors, I will post the patch here. If you don't want it then please close this bug.

We are going to try this:

Mozilla/5.0 (Fedora; Linux x86_64) AppleWebKit/601.1 (KHTML, like Gecko) Version/8.0 Safari/601.1

If removing X11 causes problems, then we'll do this:

Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/601.1 (KHTML, like Gecko) Version/8.0 Safari/601.1

We're not using Epiphany's branding.conf file so that the change affects other browsers and so that we can get it in the parenthetical at the start of the user agent rather than at the end (because that's where we put it in Firefox).

[Michael Catanzaro]

Created attachment 247493 [details]
[GTK] Consider allowing vendor branding in the default user agent

[Martin Robinson]

Why is this an improvement over webkit_settings_set_user_agent?

[Martin Robinson]

(In reply to comment #2)
> Why is this an improvement over webkit_settings_set_user_agent?

I suppose this is useful for distributions, but browsers should probably be overriding this anyway.

[Michael Catanzaro]

(In reply to comment #3) 
> I suppose this is useful for distributions, but browsers should probably be
> overriding this anyway.

I think most browsers probably want to leave the default user agent alone, and only append to the user agent with webkit_settings_set_user_agent_with_application_details(). At least that's what we do with Epiphany. If they do that, then the distributor branding remains intact.

[Carlos Garcia Campos]

User agent is unfortunately very weak thing, I would not change anything here, we already have API to set any user agent. Has this been requested by any distributor?

[Michael Catanzaro]

Yes, it was requested by the Red Hat desktop team.

I think adding the Fedora branding will be safe. If some odd site proves that wrong, we'll add quirks to remove it for individual problematic sites in response to bug reports, or experiment with moving it to a different position in the user agent. But I think it will be fine where it is.

Removing X11 seems riskier, but we have Wayland user sessions working and plan to default to Wayland later this year, and I don't want to lie about running in X11 unless we determine that it's actually needed to make sites work. If this causes bugs, I'll put it back.

[Michael Catanzaro]

Comment on attachment 247493 [details]
[GTK] Consider allowing vendor branding in the default user agent

One year later, the only site I'm aware has been broken by this is Google Calendar. We will add a user agent quirk for them and continue to use this patch.

But I am removing r? because I should update this patch to check the platform at runtime.

[Michael Catanzaro]

Created attachment 269191 [details]
Patch

[Michael Catanzaro]

Created attachment 269192 [details]
Patch

[Carlos Garcia Campos]

Comment on attachment 269192 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=269192&action=review

Thanks for working on this, user agent string is always a nightmare. So, I think we should do the changes carefully. First, I would split this patch in several ones, so that it's easier to review, and also easier to roll out individual changes if needed eventually. One patch to improve the detection of platforms, CPU, etc. another one to add the distro name, another to remove the yahoo quirk and then one patch for every new quirk. Every new quirk should add a test case to Tools/TestWebKitAPI/Tests/WebCore/gtk/UserAgentQuirks.cpp. Regarding the google domains, in WebKit1 we used to detect google sites to use a different UA, see bug #134631, we backported that to 2.4 branch in r170882, but didn't do the same in trunk because google sites worked at that time, but maybe that approach would work now in trunk as well.

> Source/WebCore/platform/gtk/UserAgentGtk.cpp:109
> +    GdkDisplay* display = gdk_display_manager_get_default_display(gdk_display_manager_get());
> +#if PLATFORM(X11)
> +    if (GDK_IS_X11_DISPLAY(display))
> +        return "X11";
> +#endif

Use PlatformDisplay which also don't require to build this twice for the plugin process.

> Source/WebCore/platform/gtk/UserAgentGtk.cpp:122
> +#warning OS X has an incomplete implementation of platformVersionForUAString. Consider fixing it.

Do we really need a compile warning for this? Leave the FIXME, open a bug report and add the bug URL to the FIXME comment.

> Source/WebCore/platform/gtk/UserAgentGtk.cpp:129
> +    // TODO: For Windows, return a version string of the form "Windows NT 6.1".

In WebKit we don't use TODO, but FIXME.

> Source/WebCore/platform/gtk/UserAgentGtk.cpp:148
> +        if (std::strlen(platform)) {

If you return nullptr instead of "" you don't need strlen here, but if uou still want to return "" you could just do if *platform or if platform[0] instead of strlen. You might also consider using CString and check is platforn.isNull().

> Source/WebCore/platform/gtk/UserAgentGtk.cpp:207
> +    const char* baseDomain = soup_tld_get_base_domain(url.host().utf8().data(), nullptr);
> +    if (!baseDomain)
> +        return false;

Use topPrivatelyControlledDomain() from PublicSuffix.h

> Source/WebCore/platform/gtk/UserAgentGtk.cpp:220
> +    const char* baseDomain = soup_tld_get_base_domain(url.host().utf8().data(), nullptr);
> +    if (!baseDomain)
> +        return false;

Ditto.

> Source/WebCore/platform/gtk/UserAgentGtk.cpp:231
> +    // Various subdomains display "unsupported browser" warnings when using our standard UA. Google
> +    // used to complain loudly about this when Chrome was new. Now they are the biggest offender.
> +    // Note in particular that calendar.google.com falls back to an atrocious fallback page with our
> +    // standard user agent if it does not include the string "X11" so be careful if removing this.
> +    if (!std::strcmp(baseDomain, "google.com"))
> +        return true;

See http://trac.webkit.org/changeset/170882 we have better ways to detect google sites.

[Michael Catanzaro]

OK to all except:

(In reply to comment #10)
> Use PlatformDisplay which also don't require to build this twice for the
> plugin process.

I couldn't figure out how to link TestWebCore when I added PlatformDisplay; linker errors are really hard. :(

[Michael Catanzaro]

Created attachment 271470 [details]
Screenshot of calendar.google.com

For posterity, a screenshot of the breakage on calendar.google.com

[Martin Robinson]

It seems that this patch still doesn't fix maps.google.com, which has force-disables Google Earth.

[Michael Catanzaro]

(In reply to comment #13)
> It seems that this patch still doesn't fix maps.google.com, which has
> force-disables Google Earth.

Also, as Google has not responded to my previous user-agent related bug report, and they're throwing unsupported browser warnings on so many domains, and I have zero remaining patience for them, I think the only reasonable solution is to add Chromium to our user agent on all Google domains. If Chromium in the UA breaks some Google site, we'll just have to add more user agent hackarounds to remove it just for those domains.

I'm confident that we should use the Chromium UA only on Google domains, as we know using it on the entire Internet would be quite risky.

[Michael Catanzaro]

Also that will shut up their Chrome advertisements, fitting punishment for all the harm they've caused to the web via their broken user agent parsing.

[Michael Catanzaro]

(In reply to comment #10)
> Regarding the
> google domains, in WebKit1 we used to detect google sites to use a different
> UA, see bug #134631, we backported that to 2.4 branch in r170882, but didn't
> do the same in trunk because google sites worked at that time, but maybe
> that approach would work now in trunk as well.

That approach is a bit extreme and it will break for any new country-specific domains Google sets up. I would just say: if the second-level domain is "google" or if it's gmail.com, then we should use the Chromium UA. I would avoid UA hacks for youtube.com, gstatic.com, and ytimg.com unless we know it's needed to fix something.

[Michael Catanzaro]

A couple Chinese sites to try fixing:

 * taobao.com (redirecting to mobile version)
 * news.163.com (ditto)

Let's try pretending to be Safari on OS X and see if that works.

[Michael Catanzaro]

Also:

* web.whatsapp.com (requires Chromium UA)
* framateam.org (requires Safari 9 UA)

[Carlos Alberto Lopez Perez]

Comment on attachment 269192 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=269192&action=review

> Source/WebCore/ChangeLog:26
> +        this is not supported in GTK+ 3. Return X11 only if actually running in X11. This can be
> +        expected to break sites similarly to replacing X11 with a distributor name, and is the
> +        riskiest change in this patch, but I think we can get away with it.

It seems not returning the X11 string on the UA string breaks (at least) Google maps.
I think the X11 string should be always present (even if running on Wayland).
The branding can be after the X11 string (like: "Mozilla/5.0 (X11; Fedora; Linux x86_64)") but it can't be instead of the X11 string.

[Michael Catanzaro]

Yeah, we already discovered that removing X11 breaks Google Calendar.

Since this is so far only an issue on Google domains, and we are going to send a fake UA to Google domains regardless, we don't necessarily *have* to bring back X11. I don't want to, but it does seem safer to do so....

[Carlos Alberto Lopez Perez]

(In reply to comment #20)
> Yeah, we already discovered that removing X11 breaks Google Calendar.
> 
> Since this is so far only an issue on Google domains, and we are going to
> send a fake UA to Google domains regardless, we don't necessarily *have* to
> bring back X11. I don't want to, but it does seem safer to do so....

Firefox on fedora does not replace the X11 string, but adds the Fedora string after it.

I think is better to play safe here. There are trillions of Web sites out there. Is impossible to know if we are going to break something or not. Taking the most conservative approach (that is, just do what the other browsers do) seems the best option to me.

[Michael Catanzaro]

Olivier Blin reports that we also need a Chrome user agent hack for YouTube, to get YouTube 360 to work: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"

Test video: https://www.youtube.com/watch?v=FRTnWZxTeQs

I'm seriously considering adding Chrome by default, like Martin wants to do, and adding workarounds for sites that break with the Chrome UA, rather than vice-versa.

[Michael Catanzaro]

(In reply to comment #22)
> Olivier Blin reports that we also need a Chrome user agent hack for YouTube,
> to get YouTube 360 to work: "Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"

Iulian Radu reports it's needed for Slack as well. Same Chrome user agent.

(In reply to comment #22)
> I'm seriously considering adding Chrome by default, like Martin wants to do,
> and adding workarounds for sites that break with the Chrome UA, rather than
> vice-versa.

[Michael Catanzaro]

I'll try to get to this at the web engines hackfest.

[Ting-Wei Lan]

Google Calendar is still broken on FreeBSD. Here are some test results:

WebKitGTK+ default + FreeBSD amd64 => OK
WebKitGTK+ default + FreeBSD x86_64 => OK
WebKitGTK+ default + Linux x86_64 => OK
WebKitGTK+ default + Linux amd64 => Broken
WebKitGTK+ default + Linux abc => Broken

Chrome quirk + FreeBSD amd64 => Broken (This is the default used by WebKitGTK+)
Chrome quirk + FreeBSD x86_64 => Broken
Chrome quirk + Linux x86_64 => OK
Chrome quirk + Linux amd64 => Broken
Chrome quirk + Linux abc => Broken

It seems that Google Calendar checks not only the browser but also the operating system. It also tests the name of the CPU architecture. Using anything started with 'a' can possibly trigger the mobile version.

I cannot find a way to change user agent in WebKitGTK+ for websites in the quirk list, so I got these results by copying user agent string in WebKit inspector to Firefox.

[Michael Catanzaro]

(In reply to comment #25)
> Google Calendar is still broken on FreeBSD.

Yes, on Fedora too. I must have done something wrong when testing it, or maybe Google changed. Anyway, we are having other problems with using the Chrome quirk on Google domains; it seems to have broken Hangouts and Google notifications at least. I'm planning to switch to the Safari on macOS quirk, that should hopefully fix this.

[Michael Catanzaro]

(In reply to comment #25)
> Google Calendar is still broken on FreeBSD. Here are some test results:
> 
> WebKitGTK+ default + FreeBSD amd64 => OK
> WebKitGTK+ default + FreeBSD x86_64 => OK
> WebKitGTK+ default + Linux x86_64 => OK
> WebKitGTK+ default + Linux amd64 => Broken
> WebKitGTK+ default + Linux abc => Broken
> 
> Chrome quirk + FreeBSD amd64 => Broken (This is the default used by
> WebKitGTK+)
> Chrome quirk + FreeBSD x86_64 => Broken
> Chrome quirk + Linux x86_64 => OK
> Chrome quirk + Linux amd64 => Broken
> Chrome quirk + Linux abc => Broken

Are you still able to reproduce these results? In my testing, user agent doesn't make any difference at all anymore. I always get the good calendar.google.com in my development Epiphany/WebKit, never in my system Epiphany with WebKitGTK+ 2.14.2, using fresh profiles, regardless of user agent....

So that's confusing.

We need to not send the Chrome quirk to make google.com work anyway, so I guess just removing the quirk seems to be an option...? Can anybody else test this to confirm my result?

[Ting-Wei Lan]

(In reply to comment #27)
> (In reply to comment #25)
> > Google Calendar is still broken on FreeBSD. Here are some test results:
> > 
> > WebKitGTK+ default + FreeBSD amd64 => OK
> > WebKitGTK+ default + FreeBSD x86_64 => OK
> > WebKitGTK+ default + Linux x86_64 => OK
> > WebKitGTK+ default + Linux amd64 => Broken
> > WebKitGTK+ default + Linux abc => Broken
> > 
> > Chrome quirk + FreeBSD amd64 => Broken (This is the default used by
> > WebKitGTK+)
> > Chrome quirk + FreeBSD x86_64 => Broken
> > Chrome quirk + Linux x86_64 => OK
> > Chrome quirk + Linux amd64 => Broken
> > Chrome quirk + Linux abc => Broken
> 
> Are you still able to reproduce these results? In my testing, user agent
> doesn't make any difference at all anymore. I always get the good
> calendar.google.com in my development Epiphany/WebKit, never in my system
> Epiphany with WebKitGTK+ 2.14.2, using fresh profiles, regardless of user
> agent....

I can reproduce the results with Firefox by setting general.useragent.override in about:config. I don't know how to do the test with Epiphany/WebKit because webkit_settings_set_user_agent is useless on websites with quirks.

WebKitGTK+ default is:
Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/603.1 (KHTML, like Gecko)  Version/10.0 Safari/603.1

Chrome quirk is:
Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/603.1 (KHTML, like Gecko) Chrome/56.0.2891.4  Version/10.0 Safari/603.1

> 
> So that's confusing.
> 
> We need to not send the Chrome quirk to make google.com work anyway, so I
> guess just removing the quirk seems to be an option...? Can anybody else
> test this to confirm my result?

I tested Google Calendar (with Firefox) and Google Maps (with Chromium).
Google Calendar works with WebKitGTK+ default on both Linux and FreeBSD.
Google Maps works with WebKitGTK+ default on Linux.
I didn't test whether adding Epiphany/3.23.3 can break things.

Google Maps automatically switches to lite mode on FreeBSD, and it doesn't provide an interface to switch it back to full mode. It seems we needs a quirk such as NeedsLinuxPlatform to workaround it.

Do you know how to set user agent on websites with quirks in WebKit? Doing user agent test with other browsers is inconvenient and probably leads to incorrect results.

[Michael Catanzaro]

(In reply to comment #28) 
> I can reproduce the results with Firefox by setting
> general.useragent.override in about:config. I don't know how to do the test
> with Epiphany/WebKit because webkit_settings_set_user_agent is useless on
> websites with quirks.

That was my mistake. I was changing the gsetting, but as you observe it has no effect. Good catch! That was really confusing me!

> WebKitGTK+ default is:
> Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/603.1 (KHTML, like Gecko) 
> Version/10.0 Safari/603.1
> 
> Chrome quirk is:
> Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/603.1 (KHTML, like Gecko)
> Chrome/56.0.2891.4  Version/10.0 Safari/603.1

OK, that's helpful to see. We might need to add a Linux x86_64 quirk.

> > 
> > So that's confusing.
> > 
> > We need to not send the Chrome quirk to make google.com work anyway, so I
> > guess just removing the quirk seems to be an option...? Can anybody else
> > test this to confirm my result?
> 
> I tested Google Calendar (with Firefox) and Google Maps (with Chromium).
> Google Calendar works with WebKitGTK+ default on both Linux and FreeBSD.

The problem with Calender, is we have a requirement to add Fedora branding, which is currently a downstream patch, and that is what breaks it. Sigh.

> Google Maps works with WebKitGTK+ default on Linux.
> I didn't test whether adding Epiphany/3.23.3 can break things.

I don't know what doesn't work without the UA tweak, some 3D feature or something.

> Google Maps automatically switches to lite mode on FreeBSD, and it doesn't
> provide an interface to switch it back to full mode. It seems we needs a
> quirk such as NeedsLinuxPlatform to workaround it.

OK, noted. If I commit any more user agent changes here without fixing this, then I forgot and you should yell at me.

> Do you know how to set user agent on websites with quirks in WebKit? Doing
> user agent test with other browsers is inconvenient and probably leads to
> incorrect results.

No, we don't currently have any way to override user agent quirks, sorry. That requires future work.

[Michael Catanzaro]

I think all we need to do here is switch Google domains to use the macOS quirk. That should fix FreeBSD.

After disabling the quirk and changing my user agent to match FreeBSD, I still failed to reproduce the Calendar issue, but was able to reproduce the Maps issue. (I did see a little lightning bolt in the bottom-right that I could use to switch  out of Lite mode, though). The macOS quirk works fine for it, so I think we don't need a Linux platform quirk at all.

[Michael Catanzaro]

Created attachment 297774 [details]
Patch

[Carlos Garcia Campos]

Comment on attachment 297774 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=297774&action=review

> Source/WebCore/platform/UserAgentQuirks.cpp:65
> +    // Avoid receiving a terrible fallback version of calendar.google.com with
> +    // custom distributor name or on FreeBSD. Avoid receiving Lite version of
> +    // maps.google.com on FreeBSD. Google is the worst. Bug #142074.

You mention FreeBSD here but it's applied unconditionally, making the comment a bit confusing. I don't think "Google is the worst" adds anything useful to the comment, to be honest. And I would add a bz url instead of just the bug number, you can a short version.

[Michael Catanzaro]

Committed r210168: <http://trac.webkit.org/changeset/210168>

[Ting-Wei Lan]

I found a way to change user agent on websites with quirks:
MiniBrowser --enable-site-specific-quirks=false --user-agent='<user_agent_string>'

I tested r210197 on FreeBSD. Google Calendar issue is fixed now, and Google Maps uses full mode by default.

However, the bottom-left of the Google Maps page says 'Satellite'. (It is 'Earth' on Firefox and Chrome.) I am not sure whether this causes functionality difference, but some features work differently. In 'Earth' mode, it is possible to see the street view by zooming in. In 'Satellite' mode, zooming in doesn't work, but there are other ways to get street view mode. If I add 'Chrome/55.0.2883.87' to user agent, it can change to 'Earth'.

[Michael Catanzaro]

Thanks Ting-Wei, this is a great help. The Earth mode is awesome and seems to work fine so long as we don't remove the Chrome quirk like I just did yesterday. I think the best solution is to use both the Chrome and macOS quirks for Google Maps, that way we avoid any possibility of breakage on FreeBSD.

Carlos: Google is really the worst.

[Michael Catanzaro]

But crap, Google is now redirecting from maps.google.com to www.google.com/maps, so we can't use the host portion of the URL to decide whether to apply the quirk... it's like they're trying to make it as hard as possible for us to make their website work properly.

[Michael Catanzaro]

I think what we should do is always use BOTH quirks on all Google domains, except hangouts.google.com.

[Michael Catanzaro]

(In reply to comment #37)
> I think what we should do is always use BOTH quirks on all Google domains,
> except hangouts.google.com.

Unfortunately the macOS quirk makes it impossible to download the Hangouts plugin, Google only offers the macOS download. And unfortunately the download comes from google.com and not hangouts.google.com. So with Hangouts and Maps both using the same host, we can't differentiate between the two except by using the path component of the URL, and as they require incompatible platform quirks... this is quite tricky.

[Michael Catanzaro]

I guess Hangouts cannot work on FreeBSD anyway, because there's no appropriate download. So the fix that works as well as possible for everyone would be a Linux  x86_64 platform quirk. We don't really need the Mac platform, we just need some hardcoded platform string that doesn't include "FreeBSD" or "Fedora" or "amd64". That avoids any quirk problems with Hangouts on Linux, and it's unfixable for FreeBSD so won't cause any additional problem there.

[Michael Catanzaro]

No, even that doesn't work. If we use the Chrome UA that breaks Hangouts, it won't try to download the plugin. A Firefox quirk fixes Maps without breaking Hangouts. So need to add both a Linux quirk (for FreeBSD) and a Firefox quirk.

[Michael Catanzaro]

OK I got it working. This was really hard! I'm fairly confident that the Firefox quirk is the safest approach to take with Google domains, that avoids Chrome-specific crap like Hangouts while ensuring we also get the nicest versions of sites like Google Maps. The Linux platform quirk seems odd but is really needed for FreeBSD. I also fixed a teensy bug that caused an extra space to appear in the user agent before the Safari version.

[Michael Catanzaro]

Created attachment 297819 [details]
Patch

[Carlos Garcia Campos]

Could we use new bug reports for new issues instead of keep reopening this one?

[WebKit Commit Bot]

Comment on attachment 297819 [details]
Patch

Clearing flags on attachment: 297819

Committed r210207: <http://trac.webkit.org/changeset/210207>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.