Bug 186294 – Implement getDisplayMedia, the Screen Capture API

• Component: WebRTC
• Status: RESOLVED FIXED

[Lee Ho]

This is a request for implementing getDisplayMedia, the Screen Capture API that is an extension to the Media Capture API. This allows a display surface to be used as a source for a media stream.

Examples of display surfaces that may be supported include: monitor, window, application (entire collection of windows for an application), or browser (single browser window). Having the ability to capture and share a monitor display would be a great start.

The specification is here: https://w3c.github.io/mediacapture-screen-share/

Microsoft Edge (EdgeHTML 17) added support for the Screen Capture API as part of the Windows 10 April 2018 update: https://blogs.windows.com/msedgedev/2018/05/02/bringing-screen-capture-to-microsoft-edge-media-capture-api/
Google Chrome and Mozilla Firefox have already had support for non-standard screen capture APIs for years. They have bugs tracking migrations to the standard getDisplayMedia API (Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=326740, Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1321221).

[Eric Carlson]

getDisplayMedia is implemented (for displays only, not window, or application) on macOS behind the ScreenCapture runtime flag (off by default). See bugs 181070, 181291, and 181333.

[Lee Ho]

That's great to hear! How do we go about enabling this ScreenCapture runtime flag? What (minimum) version of Safari is required?

[Eric Carlson]

There is no UI to enable this setting yet because the (complex) user prompt for getDisplayMedia hasn't been added to Safari yet.

Are there any plans to extend the implementation to include window or application? If possible, it would also be really valuable to get any sort of estimate about when the UI work for Safari is planned.

[Adam Szmyd]

(In reply to Eric Carlson from comment #1)
> getDisplayMedia is implemented (for displays only, not window, or
> application) on macOS behind the ScreenCapture runtime flag (off by
> default). See bugs 181070, 181291, and 181333.

How to enable this `ScreenCapture` runtime flag to prepare code for this implementation and see how it works for now?

[Eric Carlson]

(In reply to Adam Szmyd from comment #5)
> How to enable this `ScreenCapture` runtime flag to prepare code for this
> implementation and see how it works for now?

It can only be enabled at build time currently.

[Raship Shah]

+1

[octavn]

+1

Please ship this!

Chrome 72 supports it by default since yesterday: https://addpipe.com/blog/standards-compliant-screen-capture-in-chrome-72/

Firefox 66 supports this too:
https://bugzilla.mozilla.org/show_bug.cgi?id=1321221

Edge supports this already since EdgeHTML 17: https://blogs.windows.com/msedgedev/2018/05/02/bringing-screen-capture-to-microsoft-edge-media-capture-api/#Amr4BHB73ZQfDKLq.97

[Skip]

+1

Even Edge has this functionality. Please implement!

[Jon Lee]

It is now available as an experimental feature in Safari Technology Preview. Please give it a try. We welcome your feedback!

[netemowihe]

I tried it out. Works great for sharing the entire screen! 

Would be nice if the feature allowed the user to select whether they want to share their entire screen(s), a particular window or a safari browser tab. Similar functionality exists in Chrome, Firefox and Edge.

[netemowihe]

Some of the functionality appears to have been broken in Tech Preview 83. Granting an application permission to capture the screen works, however, even after stopping the sharing indicator continues to appear in the web address as well as in the menu bar. Attempting to "Pause", which is the only available action, does not do anything.

[youenn fablet]

(In reply to netemowihe from comment #14)
> Some of the functionality appears to have been broken in Tech Preview 83.
> Granting an application permission to capture the screen works, however,
> even after stopping the sharing indicator continues to appear in the web
> address as well as in the menu bar. Attempting to "Pause", which is the only
> available action, does not do anything.

netemowihe, would you be able to file a separate bug?
Testing locally on STP83/MacOS10.14.5, I do not see any menu/address bar icons showing up for either getDisplayMedia or getUserMedia. I can see the tab icons though.

Hi,

I noticed that a MediaStream returned by

getDisplayMedia({ audio: true, video: true })

does not contain an audio track.

[Eric Carlson]

(In reply to philipp.weissensteiner from comment #16)
> Hi,
> 
> I noticed that a MediaStream returned by
> 
> getDisplayMedia({ audio: true, video: true })
> 
> does not contain an audio track.

That is by design, and is something the spec allows:

Unlike getUserMedia with regards to audio+video, the user agent is allowed not to return audio even if the audio constraint is present. If the user agent knows no audio will be shared for the lifetime of the stream it MUST NOT include an audio track in the resulting stream. The user agent MAY accept a request for audio and video by only returning a video track in the resulting stream, or it MAY accept the request by returning both an audio track and a video track in the resulting stream.

https://w3c.github.io/mediacapture-screen-share/#mediadevices-additions

Thanks for the clarification, Eric.

Since the spec also mentions:

> In the case of audio, the user agent MAY present the end-user with audio sources to share. Which choices are available to choose from is up to the user agent, and the audio source(s) are not necessarily the same as the video source(s)

I'm wondering are there plans to introduce a selectable audio source for gDM?

As of right now, if you were developing a screen sharing application
you would have to add the audio track manually, correct?

displayStream = gDM({ video: true });
audioStream = gUM({ audio: true });

displayStream.addTrack(audioStream.getAudioTracks()[0]);

[octavn]

> As of right now, if you were developing a screen sharing application
> you would have to add the audio track manually, correct?
> 
> displayStream = gDM({ video: true });
> audioStream = gUM({ audio: true });
> 
> displayStream.addTrack(audioStream.getAudioTracks()[0]);

That's how we're doing it in Chrome and Firefox but I've never tested it with Safari.

https://blog.addpipe.com/screen-recording/

[Eric Carlson]

(In reply to philipp.weissensteiner from comment #18)
> Thanks for the clarification, Eric.
> 
> Since the spec also mentions:
> 
> > In the case of audio, the user agent MAY present the end-user with audio sources to share. Which choices are available to choose from is up to the user agent, and the audio source(s) are not necessarily the same as the video source(s)
> 
> I'm wondering are there plans to introduce a selectable audio source for gDM?
> 
> As of right now, if you were developing a screen sharing application
> you would have to add the audio track manually, correct?
> 
> displayStream = gDM({ video: true });
> audioStream = gUM({ audio: true });
> 
> displayStream.addTrack(audioStream.getAudioTracks()[0]);

We aren't currently planning to add support for capturing microphone audio to getDisplayMedia. I see two problems, 1) microphone capture and screen capture are conceptually quite different, and 2) adding microphone capture to gDM would require something in the user prompt to make it clear to the user that the page was asking for both, and I'm afaid that it would be easy for a user to overlook the extra request. This will be especially in the much more complicated user prompt required once we support screen capture from a monitor or window.

It *would* make sense to allow screen+audio capture if the audio captured was the  audio output from the machine (for screen capture) or application (for window capture), but we don't have system API for that.

Thanks for the feedback! We're always happy to hear about feature requests, bugs, etc - even if we aren't able to do anything as is the case here.

[Raship Shah]

Now that Safari on iPadOS claims to support desktop class browsing, would we expect to see this available there as well?

[Laurent Denoue]

> We aren't currently planning to add support for capturing microphone audio
> to getDisplayMedia.

Are people working on window sharing?
All other engines support picking an individual window to share instead of the entire screen, which I find useful for privacy reasons.

[Sam Sneddon [:gsnedders]]

Closing this as fixed given this shipped in Safari 13 in 2019.