Bug 155754

Summary: REGRESSION (r191180): Safari does not send Referer Header to iframe src in certain situations
Product: WebKit Reporter: Adam Podolnick <adam>
Component: New BugsAssignee: Chris Dumez <cdumez>
Status: RESOLVED FIXED    
Severity: Normal CC: adrian+webkit, aestes, beidson, cdumez, commit-queue, dbates, ddkilzer, esprehn+autocc, gbarros, gyuyoung.kim, iamcraigcampbell, jiewen_tan, koivisto, me, paulking247, rossideas, ruudibear, webkit-bug-importer, yuichiohkawa
Priority: P1 Keywords: InRadar, Regression
Version: Safari 9   
Hardware: iPhone / iPad   
OS: iOS 9.3   
Bug Depends on: 150097    
Bug Blocks:    
Attachments:
Description Flags
Roll out
none
Patch none

Description Adam Podolnick 2016-03-22 10:23:53 PDT
Overview:
Safari seems to no longer send a Referer header when loading a page through an iframe in some situations. I'm still trying to isolate the exact cause, but it seems to happen reliably when an external reference, such as a javascript file or css file, is loaded in the document head. Also, the Accept header seems to switch to */* instead of text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Steps to Reproduce:
1) I set up a test case here: https://sproutvideo-examples.s3.amazonaws.com/safari_headers_b.html. View this page in Safari on a device running iOS 9.3
2) Refresh the page if the HTTP_REFERER header is still displayed. It should disappear after a reload.


Actual Results:
HTTP_REFERER header is missing
HTTP_ACCEPT header is */*

Expected Results:
HTTP_REFERER header should be https://sproutvideo-examples.s3.amazonaws.com/safari_headers_b.html
HTTP_ACCEPT header should be text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Platform:
iOS 9.3

Other Platforms:
Works as expected on iOS 9.2 and below
Works as expected on Safari, Chrome, Opera, and Firefox on Mac OSX 10.11.3
Works as expected on Safari, Chrome, Firefox, Opera, Edge, and IE on Windows 10
Comment 1 David Kilzer (:ddkilzer) 2016-03-22 12:04:24 PDT
Does this happen to be a cross-origin request?
Comment 2 Adam Podolnick 2016-03-22 12:05:16 PDT
(In reply to comment #1)
> Does this happen to be a cross-origin request?

Yes, it is a cross-origin request.
Comment 3 Radar WebKit Bug Importer 2016-03-22 12:05:46 PDT
<rdar://problem/25296445>
Comment 4 Adrian Holovaty 2016-03-24 07:01:16 PDT
We're seeing this same bug with soundslice.com, which offers an embedded iframe sheet-music viewer that optionally does referrer checking. Referrer checking is broken in iOS 9.3, which means our iframes are broken.

This bug also affects Vimeo Pro video embeds, which apparently do the same whitelist referrer checking.
Comment 5 Adrian Holovaty 2016-03-24 07:09:52 PDT
Here are people complaining about it on Vimeo's forums: https://vimeo.com/forums/help/topic:281389
Comment 6 Andy Estes 2016-03-25 03:51:59 PDT
I believe this was caused by http://trac.webkit.org/r191180
Comment 7 Brady Eidson 2016-03-25 14:11:01 PDT
(In reply to comment #6)
> I believe this was caused by http://trac.webkit.org/r191180

I was thinking the same thing, but the reporter says they see this in iOS 9.3

Did the preload scanner ship in 9.3?
Comment 8 Chris Dumez 2016-03-25 14:12:52 PDT
(In reply to comment #7)
> (In reply to comment #6)
> > I believe this was caused by http://trac.webkit.org/r191180
> 
> I was thinking the same thing, but the reporter says they see this in iOS 9.3
> 
> Did the preload scanner ship in 9.3?

The preload scanner shipped in 9.3 but iframe preloading did not ship in 9.3: this is a fairly recent addition.
Comment 9 Brady Eidson 2016-03-25 14:15:33 PDT
(In reply to comment #8)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > I believe this was caused by http://trac.webkit.org/r191180
> > 
> > I was thinking the same thing, but the reporter says they see this in iOS 9.3
> > 
> > Did the preload scanner ship in 9.3?
> 
> The preload scanner shipped in 9.3 but iframe preloading did not ship in
> 9.3: this is a fairly recent addition.

I was thinking of a different thing that we've seen recent issues with (speculative validation)

Okay, good.
Comment 10 Chris Dumez 2016-03-25 14:21:30 PDT
(In reply to comment #6)
> I believe this was caused by http://trac.webkit.org/r191180

I have just double-checked: http://trac.webkit.org/r191180 / <rdar://problem/23094475> did not ship yet.
Comment 11 Adam Podolnick 2016-03-25 14:22:22 PDT
This might be helpful. If there are no external resources loaded in the <head> or, more generally, before the iframe, the correct headers are sent. Example: https://sproutvideo-examples.s3.amazonaws.com/safari_headers_a.html

It also sends the correct headers if the external resource is included AFTER the iframe: https://sproutvideo-examples.s3.amazonaws.com/safari_headers_c.html
Comment 12 Chris Dumez 2016-03-25 15:42:34 PDT
(In reply to comment #10)
> (In reply to comment #6)
> > I believe this was caused by http://trac.webkit.org/r191180
> 
> I have just double-checked: http://trac.webkit.org/r191180 /
> <rdar://problem/23094475> did not ship yet.

Never mind, Andy proved me wrong. r191180 did indeed ship in 9.3 and is likely to cause of this regression.
Comment 13 Ohkawa Yuichi 2016-03-30 21:39:23 PDT
This bug also happen on OS X 10.11.4(Safari 9.1/11601.5.17.1 and Technology Preview 9.1.1).
In addition, this also happen with same-origin request.

Sample page:
https://www.ei.tohoku.ac.jp/demo/sample.html

Steps:
1) Access above page.
2) Reload the page.

A sample page has style tag and script tag.
If there is no style tag(*1) or no script tag(*2) in page, safari sends referer header correctly.

Additional samples:
*1 https://www.ei.tohoku.ac.jp/demo/sample3.html
*2 https://www.ei.tohoku.ac.jp/demo/sample2.html

In these sample, I used php page for iframe src.
If I used static page for iframe src, the behavior changed.
When pages are cached in safari, safari sends referer header.
However, if I clear page caches, safari became not to send.
Comment 14 Chris Dumez 2016-03-31 12:53:19 PDT
Created attachment 275313 [details]
Roll out
Comment 15 Chris Dumez 2016-03-31 13:27:14 PDT
Created attachment 275315 [details]
Patch
Comment 16 WebKit Commit Bot 2016-03-31 14:11:38 PDT
The commit-queue encountered the following flaky tests while processing attachment 275313 [details]:

transitions/default-timing-function.html bug 138901 (author: simon.fraser@apple.com)
The commit-queue is continuing to process your patch.
Comment 17 Chris Dumez 2016-03-31 14:12:51 PDT
Comment on attachment 275315 [details]
Patch

Clearing flags on attachment: 275315

Committed r198917: <http://trac.webkit.org/changeset/198917>
Comment 18 Chris Dumez 2016-03-31 14:12:57 PDT
All reviewed patches have been landed.  Closing bug.
Comment 19 Adam Podolnick 2016-03-31 14:17:09 PDT
Thanks for looking into this! I see that this has been marked as "RESOLVED FIXED". Has this been tested against the test cases that Ohkawa Yuichi and I provided to make sure that the iframe preloading was the source of the problem?
Comment 20 Chris Dumez 2016-03-31 15:49:32 PDT
(In reply to comment #19)
> Thanks for looking into this! I see that this has been marked as "RESOLVED
> FIXED". Has this been tested against the test cases that Ohkawa Yuichi and I
> provided to make sure that the iframe preloading was the source of the
> problem?

Yes, I have confirmed that both test cases now work on iOS with latest WebKit.
Comment 21 Adam Podolnick 2016-03-31 15:51:22 PDT
Excellent! Thanks for clarifying.
Comment 22 cp30 2016-04-02 09:52:38 PDT
Hi,

We're still seeing the same issue with iOS 9.3.1 if the page is refreshed.  First page load is ok but subsequent refreshes drop the referrer once again.  Doesn't seem that this one is fixed yet.

Thanks
Comment 23 Chris Dumez 2016-04-02 10:18:20 PDT
(In reply to comment #22)
> Hi,
> 
> We're still seeing the same issue with iOS 9.3.1 if the page is refreshed. 
> First page load is ok but subsequent refreshes drop the referrer once again.
> Doesn't seem that this one is fixed yet.
> 
> Thanks

It is fixed, just not in iOS 9.3.1 yet.
Comment 24 cp30 2016-04-02 10:20:14 PDT
Ah - please accept my apologies - not familiar with the way these things work with Apple updates etc.  Thanks for confirming, I will let our users know.

Kind regards
Comment 25 Ruudi 2016-04-06 04:46:47 PDT
(In reply to comment #23)
> (In reply to comment #22)
> > Hi,
> > 
> > We're still seeing the same issue with iOS 9.3.1 if the page is refreshed. 
> > First page load is ok but subsequent refreshes drop the referrer once again.
> > Doesn't seem that this one is fixed yet.
> > 
> > Thanks
> 
> It is fixed, just not in iOS 9.3.1 yet.

This problem is still affecting our website after iOS 9.3.1 is installed on device. Please can you advise when you think this bug fix will be sent out in an update by Apple. If the fix is in 9.3.1 then the problem still exists and is not resolved. Thanks
Comment 26 Chris Dumez 2016-04-06 08:49:23 PDT
(In reply to comment #25)
> (In reply to comment #23)
> > (In reply to comment #22)
> > > Hi,
> > > 
> > > We're still seeing the same issue with iOS 9.3.1 if the page is refreshed. 
> > > First page load is ok but subsequent refreshes drop the referrer once again.
> > > Doesn't seem that this one is fixed yet.
> > > 
> > > Thanks
> > 
> > It is fixed, just not in iOS 9.3.1 yet.
> 
> This problem is still affecting our website after iOS 9.3.1 is installed on
> device. Please can you advise when you think this bug fix will be sent out
> in an update by Apple. If the fix is in 9.3.1 then the problem still exists
> and is not resolved. Thanks

The bug *is* fixed in WebKit trunk. Yes, the bug still exists in iOS 9.3.1.
Apple does not comment on the timing or content of future releases.
Comment 27 Ruudi 2016-04-06 09:28:43 PDT
(In reply to comment #26)
> (In reply to comment #25)
> > (In reply to comment #23)
> > > (In reply to comment #22)
> > > > Hi,
> > > > 
> > > > We're still seeing the same issue with iOS 9.3.1 if the page is refreshed. 
> > > > First page load is ok but subsequent refreshes drop the referrer once again.
> > > > Doesn't seem that this one is fixed yet.
> > > > 
> > > > Thanks
> > > 
> > > It is fixed, just not in iOS 9.3.1 yet.
> > 
> > This problem is still affecting our website after iOS 9.3.1 is installed on
> > device. Please can you advise when you think this bug fix will be sent out
> > in an update by Apple. If the fix is in 9.3.1 then the problem still exists
> > and is not resolved. Thanks
> 
> The bug *is* fixed in WebKit trunk. Yes, the bug still exists in iOS 9.3.1.
> Apple does not comment on the timing or content of future releases.


Thank you - sorry for asking basic questions - I'm not familiar with this. You can understand why I ask, as our site is subscription based and don't want to remove the domain level privacy on our videos. We are trying to understand the time implications so we can manage our customer expectations.

If you could indulge me whilst I ask a couple more basic questions:

How does the process work once you have resolved the bug fix?
Do you submit your bug fix to Apple? 
DO they normally accept it and use it or are we likely to have this problem in the long term, until they decide the problem is big enough to issue a fix?

Thank you in advance for your time - it's most appreciated
Comment 28 Chris Dumez 2016-04-06 11:19:17 PDT
(In reply to comment #27)
> (In reply to comment #26)
> > (In reply to comment #25)
> > > (In reply to comment #23)
> > > > (In reply to comment #22)
> > > > > Hi,
> > > > > 
> > > > > We're still seeing the same issue with iOS 9.3.1 if the page is refreshed. 
> > > > > First page load is ok but subsequent refreshes drop the referrer once again.
> > > > > Doesn't seem that this one is fixed yet.
> > > > > 
> > > > > Thanks
> > > > 
> > > > It is fixed, just not in iOS 9.3.1 yet.
> > > 
> > > This problem is still affecting our website after iOS 9.3.1 is installed on
> > > device. Please can you advise when you think this bug fix will be sent out
> > > in an update by Apple. If the fix is in 9.3.1 then the problem still exists
> > > and is not resolved. Thanks
> > 
> > The bug *is* fixed in WebKit trunk. Yes, the bug still exists in iOS 9.3.1.
> > Apple does not comment on the timing or content of future releases.
> 
> 
> Thank you - sorry for asking basic questions - I'm not familiar with this.
> You can understand why I ask, as our site is subscription based and don't
> want to remove the domain level privacy on our videos. We are trying to
> understand the time implications so we can manage our customer expectations.
> 
> If you could indulge me whilst I ask a couple more basic questions:
> 
> How does the process work once you have resolved the bug fix?
> Do you submit your bug fix to Apple? 
> DO they normally accept it and use it or are we likely to have this problem
> in the long term, until they decide the problem is big enough to issue a fix?
> 
> Thank you in advance for your time - it's most appreciated

(In reply to comment #27)
> (In reply to comment #26)
> > (In reply to comment #25)
> > > (In reply to comment #23)
> > > > (In reply to comment #22)
> > > > > Hi,
> > > > > 
> > > > > We're still seeing the same issue with iOS 9.3.1 if the page is refreshed. 
> > > > > First page load is ok but subsequent refreshes drop the referrer once again.
> > > > > Doesn't seem that this one is fixed yet.
> > > > > 
> > > > > Thanks
> > > > 
> > > > It is fixed, just not in iOS 9.3.1 yet.
> > > 
> > > This problem is still affecting our website after iOS 9.3.1 is installed on
> > > device. Please can you advise when you think this bug fix will be sent out
> > > in an update by Apple. If the fix is in 9.3.1 then the problem still exists
> > > and is not resolved. Thanks
> > 
> > The bug *is* fixed in WebKit trunk. Yes, the bug still exists in iOS 9.3.1.
> > Apple does not comment on the timing or content of future releases.
> 
> 
> Thank you - sorry for asking basic questions - I'm not familiar with this.
> You can understand why I ask, as our site is subscription based and don't
> want to remove the domain level privacy on our videos. We are trying to
> understand the time implications so we can manage our customer expectations.
> 
> If you could indulge me whilst I ask a couple more basic questions:
> 
> How does the process work once you have resolved the bug fix?
> Do you submit your bug fix to Apple? 
> DO they normally accept it and use it or are we likely to have this problem
> in the long term, until they decide the problem is big enough to issue a fix?
> 
> Thank you in advance for your time - it's most appreciated

Have you tried iOS 9.3.2 beta that was seeded to developers last week?
Comment 29 Chris Dumez 2016-04-06 11:34:03 PDT
(In reply to comment #28)
> (In reply to comment #27)
> > (In reply to comment #26)
> > > (In reply to comment #25)
> > > > (In reply to comment #23)
> > > > > (In reply to comment #22)
> > > > > > Hi,
> > > > > > 
> > > > > > We're still seeing the same issue with iOS 9.3.1 if the page is refreshed. 
> > > > > > First page load is ok but subsequent refreshes drop the referrer once again.
> > > > > > Doesn't seem that this one is fixed yet.
> > > > > > 
> > > > > > Thanks
> > > > > 
> > > > > It is fixed, just not in iOS 9.3.1 yet.
> > > > 
> > > > This problem is still affecting our website after iOS 9.3.1 is installed on
> > > > device. Please can you advise when you think this bug fix will be sent out
> > > > in an update by Apple. If the fix is in 9.3.1 then the problem still exists
> > > > and is not resolved. Thanks
> > > 
> > > The bug *is* fixed in WebKit trunk. Yes, the bug still exists in iOS 9.3.1.
> > > Apple does not comment on the timing or content of future releases.
> > 
> > 
> > Thank you - sorry for asking basic questions - I'm not familiar with this.
> > You can understand why I ask, as our site is subscription based and don't
> > want to remove the domain level privacy on our videos. We are trying to
> > understand the time implications so we can manage our customer expectations.
> > 
> > If you could indulge me whilst I ask a couple more basic questions:
> > 
> > How does the process work once you have resolved the bug fix?
> > Do you submit your bug fix to Apple? 
> > DO they normally accept it and use it or are we likely to have this problem
> > in the long term, until they decide the problem is big enough to issue a fix?
> > 
> > Thank you in advance for your time - it's most appreciated
> 
> (In reply to comment #27)
> > (In reply to comment #26)
> > > (In reply to comment #25)
> > > > (In reply to comment #23)
> > > > > (In reply to comment #22)
> > > > > > Hi,
> > > > > > 
> > > > > > We're still seeing the same issue with iOS 9.3.1 if the page is refreshed. 
> > > > > > First page load is ok but subsequent refreshes drop the referrer once again.
> > > > > > Doesn't seem that this one is fixed yet.
> > > > > > 
> > > > > > Thanks
> > > > > 
> > > > > It is fixed, just not in iOS 9.3.1 yet.
> > > > 
> > > > This problem is still affecting our website after iOS 9.3.1 is installed on
> > > > device. Please can you advise when you think this bug fix will be sent out
> > > > in an update by Apple. If the fix is in 9.3.1 then the problem still exists
> > > > and is not resolved. Thanks
> > > 
> > > The bug *is* fixed in WebKit trunk. Yes, the bug still exists in iOS 9.3.1.
> > > Apple does not comment on the timing or content of future releases.
> > 
> > 
> > Thank you - sorry for asking basic questions - I'm not familiar with this.
> > You can understand why I ask, as our site is subscription based and don't
> > want to remove the domain level privacy on our videos. We are trying to
> > understand the time implications so we can manage our customer expectations.
> > 
> > If you could indulge me whilst I ask a couple more basic questions:
> > 
> > How does the process work once you have resolved the bug fix?
> > Do you submit your bug fix to Apple? 
> > DO they normally accept it and use it or are we likely to have this problem
> > in the long term, until they decide the problem is big enough to issue a fix?
> > 
> > Thank you in advance for your time - it's most appreciated
> 
> Have you tried iOS 9.3.2 beta that was seeded to developers last week?

Ok, I have verified that I cannot reproduce the bug anymore on the public iOS 9.3.2 beta that was seeded to developers last week. I confirmed using both test cases (the one from Adam and the one from Ohkawa) on this bug report. Hopefully this helps. Feel free to confirm it yourself as well.
Comment 30 Ruudi 2016-04-06 12:24:20 PDT
Thank you !
Comment 31 Gabriel Barros 2016-05-23 17:36:04 PDT
I know there is no comment on timing or content of future releases, but where can I subscribe/be informed when the release for OSX happens?

There will be no changelog mentioning this bug number, I assume since i did not see references to bug numbers in any other changelog.

Will this page be updated when it happens?

Sorry for not being familiar with the process. Feel free to just point me to a FAQ if there is one I failed to find.
Comment 32 Brady Eidson 2016-05-23 17:48:14 PDT
(In reply to comment #31)
> I know there is no comment on timing or content of future releases, but
> where can I subscribe/be informed when the release for OSX happens?

No such notification exists for OS X Safari releases.

If you're interested in Safari Technology Preview releases, the release notes for those have - so far - included ChangeLogs referencing bugzillas.

> Will this page be updated when it happens?

No.
Comment 33 Gabriel Barros 2016-05-23 18:18:42 PDT
> If you're interested in Safari Technology Preview releases, the release notes
> for those have - so far - included ChangeLogs referencing bugzillas.


Thank you. Can I assume the reference on the release-notes[1] will be the exact (or similar) title here?

[1] https://developer.apple.com/safari/technology-preview/release-notes/
Comment 34 Chris Dumez 2016-05-23 18:27:04 PDT
(In reply to comment #31)
> I know there is no comment on timing or content of future releases, but
> where can I subscribe/be informed when the release for OSX happens?
> 
> There will be no changelog mentioning this bug number, I assume since i did
> not see references to bug numbers in any other changelog.
> 
> Will this page be updated when it happens?
> 
> Sorry for not being familiar with the process. Feel free to just point me to
> a FAQ if there is one I failed to find.

While there is no comment on the content of future releases. I am pretty sure the fix already shipped in 10.11.5 on May 16.