Bug 215845 – [GLIB] Stop using firefox user agent quirk for google docs

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Sergio Villar Senin]

Steps to reproduce:
1. Go to gmail.com and enter credentials to login
2. An animated envelope is shown along with an horizontal progress bar

Expected outcome:
The load finishes and the inbox is shown

Actual outcome:
The inbox is never shown, the progress bar is stuck at the very end but never completes

Misc:
Clicking on reload does load the inbox but without it, it is never shown.

[Carlos Garcia Campos]

I've noticed this too. Reloading while the animation is present works for me and shows the inbox. WPE doesn't even show the animation...

[Carlos Garcia Campos]

It has to do with PSON, just disabling it makes it work. Weird thing is that PSON is enabled in WPE too. I'll continue investigating tomorrow.

[Carlos Garcia Campos]

I have finally found the issue after lot of debugging. We fail to load the page because we are rejecting several inline scripts from CSP. And the cause of this is our user agent. This is what happens when PSON is disabled (urls are truncated and only relevant headers shown):

> GET /accounts/SetOSID? HTTP/1.1
> Soup-Debug: SoupSession 1 (0x558d044c2220), SoupMessage 28 (0x558d048ce190), SoupSocket 17 (0x558d04879fa0)
> Host: mail.google.com
> Referer: https://accounts.google.com/signin/v2/challenge/pwd?> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15

< HTTP/1.1 302 Found
< Soup-Debug: SoupMessage 28 (0x558d048ce190)
< Location: https://accounts.youtube.com/accounts/SetSID?...&continue=https://mail.google.com/mail/&dbus=ES

> GET /accounts/SetSID? HTTP/1.1
> Soup-Debug: SoupSession 1 (0x558d044c2220), SoupMessage 29 (0x558d048ce280), SoupSocket 10 (0x558d04808f90)
> Host: accounts.youtube.com
> Referer: https://accounts.google.com/signin/v2/challenge/pwd?
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15

< HTTP/1.1 302 Found
< Soup-Debug: SoupMessage 29 (0x558d048ce280)
< Location: https://accounts.google.es/accounts/SetSID?...&continue=https://mail.google.com/mail/

> GET /accounts/SetSID? HTTP/1.1
> Soup-Debug: SoupSession 1 (0x558d044c2220), SoupMessage 30 (0x558d048ce190), SoupSocket 18 (0x558d048fc100)
> Host: accounts.google.es
> Referer: https://accounts.google.com/signin/v2/challenge/pwd?
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15

< HTTP/1.1 302 Found
< Soup-Debug: SoupMessage 30 (0x558d048ce190)
< Location: https://mail.google.com/mail/

> GET /mail/ HTTP/1.1
> Soup-Debug: SoupSession 1 (0x558d044c2220), SoupMessage 31 (0x558d048ce280), SoupSocket 17 (0x558d04879fa0)
> Host: mail.google.com
> Referer: https://accounts.google.com/signin/v2/challenge/pwd?
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15

< HTTP/1.1 302 Moved Temporarily
< Soup-Debug: SoupMessage 31 (0x558d048ce280)
< Location: https://mail.google.com/mail/u/0/

> GET /mail/u/0/ HTTP/1.1
> Soup-Debug: SoupSession 1 (0x558d044c2220), SoupMessage 32 (0x558d048ce190), SoupSocket 17 (0x558d04879fa0)
> Host: mail.google.com
> Referer: https://accounts.google.com/signin/v2/challenge/pwd?
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15

< HTTP/1.1 200 OK
< Soup-Debug: SoupMessage 32 (0x558d048ce190)
< Content-Security-Policy: script-src https://clients4.google.com/insights/consumersurveys/ https://www.google.com/js/bg/ 'self' 'unsafe-inline' 'unsafe-eval' https://mail.google.com/_/scs/\
mail-static/ https://hangouts.google.com/ https://talkgadget.google.com/ https://*.talkgadget.google.com/ https://www.googleapis.com/appsmarket/v2/installedApps/ https://www-gm-opensocial.g\
oogleusercontent.com/gadgets/js/ https://docs.google.com/static/doclist/client/js/ https://www.google.com/tools/feedback/ https://s.ytimg.com/yts/jsbin/ https://www.youtube.com/iframe_api h\
ttps://apis.google.com/_/scs/abc-static/ https://apis.google.com/js/ https://clients1.google.com/complete/ https://apis.google.com/_/scs/apps-static/_/js/ https://ssl.gstatic.com/inputtools\
/js/ https://inputtools.google.com/request https://ssl.gstatic.com/cloudsearch/static/o/js/ https://www.gstatic.com/feedback/js/ https://www.gstatic.com/common_sharing/static/client/js/ htt\
ps://www.gstatic.com/og/_/js/ https://pagead2.googlesyndication.com/pagead/gadgets/gmail_ads/leadgen/ https://www.gstatic.com/mail/ads/leadgen/;frame-src https://clients4.google.com/insight\
s/consumersurveys/ https://calendar.google.com/accounts/ https://ogs.google.com https://onegoogle-autopush.sandbox.google.com 'self' https://accounts.google.com/ https://apis.google.com/u/ \
https://apis.google.com/_/streamwidgets/ https://clients6.google.com/static/ https://content.googleapis.com/static/ https://mail-attachment.googleusercontent.com/ https://www.google.com/cal\
endar/ https://calendar.google.com/calendar/ https://docs.google.com/ https://drive.google.com https://*.googleusercontent.com/docs/securesc/ https://feedback.googleusercontent.com/resource\
s/ https://www.google.com/tools/feedback/ https://support.google.com/inapp/ https://*.googleusercontent.com/gadgets/ifr https://hangouts.google.com/ https://talkgadget.google.com/ https://*\
.talkgadget.google.com/ https://www-gm-opensocial.googleusercontent.com/gadgets/ https://plus.google.com/ https://wallet.google.com/gmail/ https://www.youtube.com/embed/ https://clients5.go\
ogle.com/pagead/drt/dn/ https://clients5.google.com/ads/measurement/jn/ https://www.gstatic.com/mail/ww/ https://www.gstatic.com/mail/intl/ https://clients5.google.com/webstore/wall/ https:\
//ci3.googleusercontent.com/ https://gsuite.google.com/u/ https://gsuite.google.com/marketplace/appfinder https://www.gstatic.com/mail/promo/ https://notifications.google.com/ https://trace\
depot-pa.clients6.google.com/static/ https://staging-taskassist-pa-googleapis.sandbox.google.com https://taskassist-pa.clients6.google.com https://*.prod.amp4mail.googleusercontent.com/ htt\
ps://*.client-channel.google.com/client-channel/client https://clients4.google.com/invalidation/lcs/client https://tasks.google.com/embed/ https://keep.google.com/companion https://addons.g\
suite.google.com https://contacts.google.com/widget/hovercard/v/2 https://*.googleusercontent.com/confidential-mail/attachments/;report-uri https://mail.google.com/mail/cspreport;object-src\
 https://mail-attachment.googleusercontent.com/attachment/


So, there's a redirection chain mail.google.com -> accounts.youtube.com -> accounts.google.es -> mail.google.com. If you see the user agent, the same one is always used, the one including the linux platform that we use for google sites. This is because in case of redirection we copy the user agent from the previous request without applying quirks. This is the actual bug, but also the reason why it works with PSON disabled. In the last response we can see there's a single Content-Security-Policy header wiht the rules that allow to run the inline scripts. In the case of PSON, after the redirection to accounts.youtube.com, we switch to a different process, because it's a cross-site navigation, and start a new request on accounts.youtube.com. The user agent quirks are applied in this case, and we end up using the firefox user agent instead, see:

> GET /accounts/SetSID? HTTP/1.1
> Soup-Debug: SoupSession 1 (0x55bece5b2220), SoupMessage 29 (0x55bece5c0490), SoupSocket 10 (0x55bece8f7790)
> Host: accounts.youtube.com
> Referer: https://accounts.google.com/signin/v2/challenge/pwd?
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0

Next redirections don't switch process again because the new process hasn't committed any load yet, so we keep using the firefox user agent.

> GET /mail/u/0/ HTTP/1.1
> Soup-Debug: SoupSession 1 (0x55bece5b2220), SoupMessage 32 (0x55bece5c01c0), SoupSocket 17 (0x55bece9666c0)
> Host: mail.google.com
> Referer: https://accounts.google.com/signin/v2/challenge/pwd?
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0


And for some reason with the firefox user agent, the server responds with two content security headers, see:

< HTTP/1.1 200 OK
< Soup-Debug: SoupMessage 32 (0x55bece5c01c0)
< Content-Security-Policy: script-src https://clients4.google.com/insights/consumersurveys/ https://www.google.com/js/bg/ 'self' 'unsafe-inline' 'unsafe-eval' https://mail.google.com/_/scs/\
mail-static/ https://hangouts.google.com/ https://talkgadget.google.com/ https://*.talkgadget.google.com/ https://www.googleapis.com/appsmarket/v2/installedApps/ https://www-gm-opensocial.g\
oogleusercontent.com/gadgets/js/ https://docs.google.com/static/doclist/client/js/ https://www.google.com/tools/feedback/ https://s.ytimg.com/yts/jsbin/ https://www.youtube.com/iframe_api h\
ttps://apis.google.com/_/scs/abc-static/ https://apis.google.com/js/ https://clients1.google.com/complete/ https://apis.google.com/_/scs/apps-static/_/js/ https://ssl.gstatic.com/inputtools\
/js/ https://inputtools.google.com/request https://ssl.gstatic.com/cloudsearch/static/o/js/ https://www.gstatic.com/feedback/js/ https://www.gstatic.com/common_sharing/static/client/js/ htt\
ps://www.gstatic.com/og/_/js/ https://pagead2.googlesyndication.com/pagead/gadgets/gmail_ads/leadgen/ https://www.gstatic.com/mail/ads/leadgen/;frame-src https://clients4.google.com/insight\
s/consumersurveys/ https://calendar.google.com/accounts/ https://ogs.google.com https://onegoogle-autopush.sandbox.google.com 'self' https://accounts.google.com/ https://apis.google.com/u/ \
https://apis.google.com/_/streamwidgets/ https://clients6.google.com/static/ https://content.googleapis.com/static/ https://mail-attachment.googleusercontent.com/ https://www.google.com/cal\
endar/ https://calendar.google.com/calendar/ https://docs.google.com/ https://drive.google.com https://*.googleusercontent.com/docs/securesc/ https://feedback.googleusercontent.com/resource\
s/ https://www.google.com/tools/feedback/ https://support.google.com/inapp/ https://*.googleusercontent.com/gadgets/ifr https://hangouts.google.com/ https://talkgadget.google.com/ https://*\
.talkgadget.google.com/ https://www-gm-opensocial.googleusercontent.com/gadgets/ https://plus.google.com/ https://wallet.google.com/gmail/ https://www.youtube.com/embed/ https://clients5.go\
ogle.com/pagead/drt/dn/ https://clients5.google.com/ads/measurement/jn/ https://www.gstatic.com/mail/ww/ https://www.gstatic.com/mail/intl/ https://clients5.google.com/webstore/wall/ https:\
//ci3.googleusercontent.com/ https://gsuite.google.com/u/ https://gsuite.google.com/marketplace/appfinder https://www.gstatic.com/mail/promo/ https://notifications.google.com/ https://trace\
depot-pa.clients6.google.com/static/ https://wallet.google.com/payments/ https://staging-taskassist-pa-googleapis.sandbox.google.com https://taskassist-pa.clients6.google.com https://*.prod\
.amp4mail.googleusercontent.com/ https://*.client-channel.google.com/client-channel/client https://clients4.google.com/invalidation/lcs/client https://tasks.google.com/embed/ https://keep.g\
oogle.com/companion https://addons.gsuite.google.com https://contacts.google.com/widget/hovercard/v/2 https://*.googleusercontent.com/confidential-mail/attachments/;report-uri https://mail.\
google.com/mail/cspreport;object-src https://mail-attachment.googleusercontent.com/attachment/
< Content-Security-Policy: script-src 'nonce-nkv9lvbrORE/miZ2Lu7SWg' 'unsafe-inline' 'strict-dynamic' https: http: 'unsafe-eval';object-src 'none';base-uri 'self';report-uri https://mail.go\
ogle.com/mail/cspreport

The first one is the same as when using the linux platform user agent which allows to run the inline scripts, but the second one rejects them. When reloading the page from this point a new request to mail.google.com is started with the right user agent and then it works. The reason why it works in WPE is because WPE doesn't use user agent quirks. 

So, I think the fix would be to apply user agent quirks on redirections, but maybe we should also re-consider using the firefox user agent for accounts.youtube.com

[Michael Catanzaro]

Wow, good debugging.

I'm unable to reproduce this gmail bug in Tech Preview (2.29.91). I wonder why it works for me....

(In reply to Carlos Garcia Campos from comment #3)
> So, I think the fix would be to apply user agent quirks on redirections, but
> maybe we should also re-consider using the firefox user agent for
> accounts.youtube.com

Agreed on both counts.

We actually have an old bug for this problem with redirections: bug #191858.

I investigated and found the Firefox quirk is not really for YouTube at all, but for Google Docs and Google Drive: https://trac.webkit.org/changeset/257128/webkit. I guess they're loading a browser detection script from youtube.com. That quirk is already broken btw: Google Docs has started displaying unsupported browser warnings yet again even with updated Firefox version quirk in 2.29.91, and I haven't attempted to debug that yet. So we should definitely drop that quirk to get GMail working again, since loading GMail is obviously more important than avoiding unsupported browser warnings. Bonus points if we can find some other quirk that avoids the warnings without breaking gmail, but that might be hard. I hate the user agent header, it's always so hard to find something that fixes a problem without breaking other things. :/

[Carlos Garcia Campos]

Ok, then lets remove the quirk for youtube to easily fix this bug. I'll fix the user agent on redirections in bug #191858

[Carlos Garcia Campos]

The gmail issue is now fixed by r266576, but it shown that using firefox user agent quirk on google services causes other issues related to csp, so it's better to stop using it. Michael confirmed that the quirk no longer works for google docs, so let's just remove it.

[Carlos Garcia Campos]

Created attachment 407952 [details]
Patch

[Carlos Garcia Campos]

Created attachment 407953 [details]
Patch for landing

[Carlos Garcia Campos]

Committed r266584: <https://trac.webkit.org/changeset/266584>