WKWebView should expose navigator.mediaDevices when content is loaded from app bundle

• Component: WebRTC
• Status: RESOLVED FIXED

[Laurent Denoue]

WKWebView now exposes getUserMedia but only when loaded over HTTPS.
For native apps embedding WKWebView, it would be very handy to allow mediaDevices.getUserMedia for pages loaded from the app bundle, or maybe custom handlers.

[Radar WebKit Bug Importer]

<rdar://problem/72792032>

[youenn fablet]

Created attachment 417006 [details]
Patch

[Laurent Denoue]

Why can't we allow file protocol in addition to http or https?

Something like:
 if (origin.protocol() != "http" && origin.protocol() != "https" && origin.protocol() != "file") {
...
}

That way we could get getUserMedia working for resources loaded from the app bundle?

[youenn fablet]

(In reply to Laurent Denoue from comment #3)
> Why can't we allow file protocol in addition to http or https?
> 
> Something like:
>  if (origin.protocol() != "http" && origin.protocol() != "https" &&
> origin.protocol() != "file") {
> ...
> }
> 
> That way we could get getUserMedia working for resources loaded from the app
> bundle?

The uploaded patch would handle this case as well.

[Laurent Denoue]

(In reply to youenn fablet from comment #4)
> (In reply to Laurent Denoue from comment #3)
> > Why can't we allow file protocol in addition to http or https?
> > 
> > Something like:
> >  if (origin.protocol() != "http" && origin.protocol() != "https" &&
> > origin.protocol() != "file") {
> > ...
> > }
> > 
> > That way we could get getUserMedia working for resources loaded from the app
> > bundle?
> 
> The uploaded patch would handle this case as well.

Yes I saw it now, sorry about that. You removed the test on origin.protocol (that used to enforce http or https) altogether, thus allowing ANY origin to access the media. Neat. Do you know when this becomes available?

[EWS]

Committed r271229: <https://trac.webkit.org/changeset/271229>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 417006 [details].

[Eric]

Congrats Youenn, great fix 👏

[youenn fablet]

*** Bug 221031 has been marked as a duplicate of this bug. ***

[Marc Bonsels]

Sorry for the possibly stupid question, as I'm not really familiar with the involved procedures.

Has this fix been shipped with iOS 14.4? As mentioned in #221031, I'm still getting errors using getUserMedia and am wondering whether the issue is on my end or if the patch is not there yet.

[Sami BDK]

Thanks for the patch. This would be greatly apprecied by cordova/ionic/phonegap users.


Any idea on what version of iOS this will be shipped with ?

Will this be shipped with 14.5? Or in a beta version?

FYI, for anyone that comes across this page, this fix appears to be integrated into the iOS 14.5 beta.

[Marvin]

I cant see it working on iOS 14.5(@michael said it's shipped with iOS 14.5).
Anybody else able to work with it on "ionic://localhost"?

[youenn fablet]

(In reply to Marvin from comment #13)
> I cant see it working on iOS 14.5(@michael said it's shipped with iOS 14.5).
> Anybody else able to work with it on "ionic://localhost"?

I would expect it to work in the latest iOS 14.5 beta.
I would also expect the prompt to show the name of the app instead of localhost.
If that is not the case, please provide repro steps.

[Pascal Ethier]

(In reply to Marvin from comment #13)
> I cant see it working on iOS 14.5(@michael said it's shipped with iOS 14.5).
> Anybody else able to work with it on "ionic://localhost"?

Hello,

New to this thread but this fix looks great!

I have just tried an Ionic/Capacitor app on iOS 14.5 beta 6.  I is based on Cordova and loads content from the App module. It makes a call to WebRTC mediaDevices.getUserMedia().

So far, mediaDevices.getUserMedia is still not exposed. 

Any clue which Beta this fix will be rolled-into?

[Brian]

I've tested a capacitor app with iOS 14.5 beta 6 and it works fine calling navigator.mediaDevices.getUserMedia({video: true, audio: true})

[Pascal Ethier]

(In reply to Brian from comment #16)
> I've tested a capacitor app with iOS 14.5 beta 6 and it works fine calling
> navigator.mediaDevices.getUserMedia({video: true, audio: true})

Hi Brian,

That's interesting.
I am doing navigator.mediaDevices.getUserMedia({video: true, audio: false})

I also have overridden the "hostname" to be something else than "localhost".

I wonder if one of those is why I am still seeing the error:
TypeError: undefined is not an object (evaluating 'navigator.mediaDevices.addEventListener')


Or Maybe it is in the way the build is configured?  
Did you have to add anything special in your PodFile or app configuration in XCode?

Any idea would be helpful.  Thank you!

[youenn fablet]

> I also have overridden the "hostname" to be something else than "localhost".
> 
> I wonder if one of those is why I am still seeing the error:
> TypeError: undefined is not an object (evaluating
> 'navigator.mediaDevices.addEventListener')

localhost is required in the latest iOS version.
WebKit ToT has a fix that allows any custom scheme page to be considered SecureContext, and then have getUserMedia access.

[Pascal Ethier]

(In reply to youenn fablet from comment #18)
> > I also have overridden the "hostname" to be something else than "localhost".
> > 
> > I wonder if one of those is why I am still seeing the error:
> > TypeError: undefined is not an object (evaluating
> > 'navigator.mediaDevices.addEventListener')
> 
> localhost is required in the latest iOS version.
> WebKit ToT has a fix that allows any custom scheme page to be considered
> SecureContext, and then have getUserMedia access.

Thank you youenn!  Yes, this worked.

[skmbr]

(In reply to Pascal Ethier from comment #19)
> (In reply to youenn fablet from comment #18)
> > > I also have overridden the "hostname" to be something else than "localhost".
> > > 
> > > I wonder if one of those is why I am still seeing the error:
> > > TypeError: undefined is not an object (evaluating
> > > 'navigator.mediaDevices.addEventListener')
> > 
> > localhost is required in the latest iOS version.
> > WebKit ToT has a fix that allows any custom scheme page to be considered
> > SecureContext, and then have getUserMedia access.
> 
> Thank you youenn!  Yes, this worked.


I've been struggling to get this working in a Quasar/Cordova app.

Can any one point me in the direction of how I set up the SecureContext mentioned above?

And forgive my ignorance, but what is ToT ?

[Louis Romero]

Created attachment 427698 [details]
Sample app to test getUserMedia.

I can't get it to work in a simple WKWebView (i.e. not using cordova, etc.).

Example:

```
<!doctype html>

<meta name="viewport" content="width=device-width, initial-scale=0.83, maximum-scale=3.0, minimum-scale=0.83">

<html>
<body>
<video></video>
</body>
</html>

<script>
const video = document.querySelector("video")
video.setAttribute('autoplay', '')
video.setAttribute('playsinline', '')
navigator.mediaDevices.getUserMedia( { video: true }).then((stream) => {
  video.srcObject = stream;
});
</script>
```

This works in Safari iOS 14.5, where the camera gets loaded, but in a WKWebView, the call to `getUserMedia` succeeds, and the video element appears, but all black, without the camera images.
Please see the xcodeproj attached that reproduces the issue.

[Louis Romero]

(In reply to skmbr from comment #20)
> (In reply to Pascal Ethier from comment #19)
> > (In reply to youenn fablet from comment #18)
> > > > I also have overridden the "hostname" to be something else than "localhost".
> > > > 
> > > > I wonder if one of those is why I am still seeing the error:
> > > > TypeError: undefined is not an object (evaluating
> > > > 'navigator.mediaDevices.addEventListener')
> > > 
> > > localhost is required in the latest iOS version.
> > > WebKit ToT has a fix that allows any custom scheme page to be considered
> > > SecureContext, and then have getUserMedia access.
> > 
> > Thank you youenn!  Yes, this worked.
> 
> 
> I've been struggling to get this working in a Quasar/Cordova app.
> 
> Can any one point me in the direction of how I set up the SecureContext
> mentioned above?
> 
> And forgive my ignorance, but what is ToT ?

ToT is tip of trunk or tip of tree, i.e. at the latest revision in the repo. It will take some time before it gets in an iOS release.

[Louis Romero]

(In reply to Louis Romero from comment #21)
> Created attachment 427698 [details]
> Sample app to test getUserMedia.
> 
> I can't get it to work in a simple WKWebView (i.e. not using cordova, etc.).
> 
> Example:
> 
> ```
> <!doctype html>
> 
> <meta name="viewport" content="width=device-width, initial-scale=0.83,
> maximum-scale=3.0, minimum-scale=0.83">
> 
> <html>
> <body>
> <video></video>
> </body>
> </html>
> 
> <script>
> const video = document.querySelector("video")
> video.setAttribute('autoplay', '')
> video.setAttribute('playsinline', '')
> navigator.mediaDevices.getUserMedia( { video: true }).then((stream) => {
>   video.srcObject = stream;
> });
> </script>
> ```
> 
> This works in Safari iOS 14.5, where the camera gets loaded, but in a
> WKWebView, the call to `getUserMedia` succeeds, and the video element
> appears, but all black, without the camera images.
> Please see the xcodeproj attached that reproduces the issue.

I just found the solution thanks to https://github.com/react-native-webview/react-native-webview/issues/1672.

I needed to set `allowsInlineMediaPlayback` on the WKWebViewConfiguration.

To summarize, the requirements for it to work are:
1. Use iOS 14.5+.
2. Use the following two attributes: <video autoplay playsinline>.
3. Request camera permission with AVCaptureDevice.authorizationStatus(for:) **before** creating the WKWebView (and thus have a NSCameraUsageDescription in your Info.plist)
4. Set allowsInlineMediaPlayback on the WKWebViewConfiguration.

[skmbr]

I think I'm going slowly insane chasing this issue.

I actually had it working in my Quasar/Cordova app today.

The key was custom scheme and hostname support in Cordova ios 6:

https://cordova.apache.org/announcements/2020/06/01/cordova-ios-release-6.0.0.html

So I added the following to my config.xml...

<preference name="scheme" value="app" />
<preference name="hostname" value="localhost" />

... to put the app in a secure context, and the next time I built, it worked!

Then due to a separate issue with my app's local storage I completely removed the app from my phone before building again.

Now it doesn't work any more, and despite a console.log(window.isSecureContext) still returning true, navigator.mediaDevices is now undefined again.

I have no idea what's going on! 😭