Bug 186294 - Implement getDisplayMedia, the Screen Capture API
Summary: Implement getDisplayMedia, the Screen Capture API
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: Safari 11
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-06-04 17:52 PDT by Lee Ho
Modified: 2022-02-25 10:43 PST (History)
23 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Lee Ho 2018-06-04 17:52:05 PDT
This is a request for implementing getDisplayMedia, the Screen Capture API that is an extension to the Media Capture API. This allows a display surface to be used as a source for a media stream.

Examples of display surfaces that may be supported include: monitor, window, application (entire collection of windows for an application), or browser (single browser window). Having the ability to capture and share a monitor display would be a great start.

The specification is here: https://w3c.github.io/mediacapture-screen-share/

Microsoft Edge (EdgeHTML 17) added support for the Screen Capture API as part of the Windows 10 April 2018 update: https://blogs.windows.com/msedgedev/2018/05/02/bringing-screen-capture-to-microsoft-edge-media-capture-api/
Google Chrome and Mozilla Firefox have already had support for non-standard screen capture APIs for years. They have bugs tracking migrations to the standard getDisplayMedia API (Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=326740, Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1321221).
Comment 1 Eric Carlson 2018-06-05 13:50:37 PDT
getDisplayMedia is implemented (for displays only, not window, or application) on macOS behind the ScreenCapture runtime flag (off by default). See bugs 181070, 181291, and 181333.
Comment 2 Lee Ho 2018-06-05 21:50:39 PDT
That's great to hear! How do we go about enabling this ScreenCapture runtime flag? What (minimum) version of Safari is required?
Comment 3 Eric Carlson 2018-06-06 09:28:59 PDT
There is no UI to enable this setting yet because the (complex) user prompt for getDisplayMedia hasn't been added to Safari yet.
Comment 4 wilhelm.wanecek 2018-08-15 01:22:38 PDT
Are there any plans to extend the implementation to include window or application? If possible, it would also be really valuable to get any sort of estimate about when the UI work for Safari is planned.
Comment 5 Adam Szmyd 2018-11-01 00:36:12 PDT
(In reply to Eric Carlson from comment #1)
> getDisplayMedia is implemented (for displays only, not window, or
> application) on macOS behind the ScreenCapture runtime flag (off by
> default). See bugs 181070, 181291, and 181333.

How to enable this `ScreenCapture` runtime flag to prepare code for this implementation and see how it works for now?
Comment 6 Eric Carlson 2018-11-01 04:58:00 PDT
(In reply to Adam Szmyd from comment #5)
> How to enable this `ScreenCapture` runtime flag to prepare code for this
> implementation and see how it works for now?

It can only be enabled at build time currently.
Comment 7 Raship Shah 2018-11-10 22:47:25 PST
+1
Comment 8 Octavian Naicu 2018-11-23 07:47:57 PST
+1
Comment 9 m.kurz+webkitbugs 2019-01-30 12:59:00 PST
Please ship this!

Chrome 72 supports it by default since yesterday: https://addpipe.com/blog/standards-compliant-screen-capture-in-chrome-72/

Firefox 66 supports this too:
https://bugzilla.mozilla.org/show_bug.cgi?id=1321221

Edge supports this already since EdgeHTML 17: https://blogs.windows.com/msedgedev/2018/05/02/bringing-screen-capture-to-microsoft-edge-media-capture-api/#Amr4BHB73ZQfDKLq.97
Comment 10 Skip 2019-03-25 22:29:53 PDT
+1
Comment 11 netemowihe 2019-04-11 05:02:24 PDT
Even Edge has this functionality. Please implement!
Comment 12 Jon Lee 2019-04-12 13:08:30 PDT
It is now available as an experimental feature in Safari Technology Preview. Please give it a try. We welcome your feedback!
Comment 13 netemowihe 2019-04-12 14:31:51 PDT
I tried it out. Works great for sharing the entire screen! 

Would be nice if the feature allowed the user to select whether they want to share their entire screen(s), a particular window or a safari browser tab. Similar functionality exists in Chrome, Firefox and Edge.
Comment 14 netemowihe 2019-05-30 11:07:21 PDT
Some of the functionality appears to have been broken in Tech Preview 83. Granting an application permission to capture the screen works, however, even after stopping the sharing indicator continues to appear in the web address as well as in the menu bar. Attempting to "Pause", which is the only available action, does not do anything.
Comment 15 youenn fablet 2019-05-30 11:25:43 PDT
(In reply to netemowihe from comment #14)
> Some of the functionality appears to have been broken in Tech Preview 83.
> Granting an application permission to capture the screen works, however,
> even after stopping the sharing indicator continues to appear in the web
> address as well as in the menu bar. Attempting to "Pause", which is the only
> available action, does not do anything.

netemowihe, would you be able to file a separate bug?
Testing locally on STP83/MacOS10.14.5, I do not see any menu/address bar icons showing up for either getDisplayMedia or getUserMedia. I can see the tab icons though.
Comment 16 philipp.weissensteiner 2019-06-11 07:37:14 PDT
Hi,

I noticed that a MediaStream returned by

getDisplayMedia({ audio: true, video: true })

does not contain an audio track.
Comment 17 Eric Carlson 2019-06-11 08:59:48 PDT
(In reply to philipp.weissensteiner from comment #16)
> Hi,
> 
> I noticed that a MediaStream returned by
> 
> getDisplayMedia({ audio: true, video: true })
> 
> does not contain an audio track.

That is by design, and is something the spec allows:

Unlike getUserMedia with regards to audio+video, the user agent is allowed not to return audio even if the audio constraint is present. If the user agent knows no audio will be shared for the lifetime of the stream it MUST NOT include an audio track in the resulting stream. The user agent MAY accept a request for audio and video by only returning a video track in the resulting stream, or it MAY accept the request by returning both an audio track and a video track in the resulting stream.

https://w3c.github.io/mediacapture-screen-share/#mediadevices-additions
Comment 18 philipp.weissensteiner 2019-06-12 01:28:39 PDT
Thanks for the clarification, Eric.

Since the spec also mentions:

> In the case of audio, the user agent MAY present the end-user with audio sources to share. Which choices are available to choose from is up to the user agent, and the audio source(s) are not necessarily the same as the video source(s)

I'm wondering are there plans to introduce a selectable audio source for gDM?

As of right now, if you were developing a screen sharing application
you would have to add the audio track manually, correct?

displayStream = gDM({ video: true });
audioStream = gUM({ audio: true });

displayStream.addTrack(audioStream.getAudioTracks()[0]);
Comment 19 Octavian Naicu 2019-06-12 04:35:16 PDT
> As of right now, if you were developing a screen sharing application
> you would have to add the audio track manually, correct?
> 
> displayStream = gDM({ video: true });
> audioStream = gUM({ audio: true });
> 
> displayStream.addTrack(audioStream.getAudioTracks()[0]);

That's how we're doing it in Chrome and Firefox but I've never tested it with Safari.

https://blog.addpipe.com/screen-recording/
Comment 20 Eric Carlson 2019-06-12 09:32:02 PDT
(In reply to philipp.weissensteiner from comment #18)
> Thanks for the clarification, Eric.
> 
> Since the spec also mentions:
> 
> > In the case of audio, the user agent MAY present the end-user with audio sources to share. Which choices are available to choose from is up to the user agent, and the audio source(s) are not necessarily the same as the video source(s)
> 
> I'm wondering are there plans to introduce a selectable audio source for gDM?
> 
> As of right now, if you were developing a screen sharing application
> you would have to add the audio track manually, correct?
> 
> displayStream = gDM({ video: true });
> audioStream = gUM({ audio: true });
> 
> displayStream.addTrack(audioStream.getAudioTracks()[0]);

We aren't currently planning to add support for capturing microphone audio to getDisplayMedia. I see two problems, 1) microphone capture and screen capture are conceptually quite different, and 2) adding microphone capture to gDM would require something in the user prompt to make it clear to the user that the page was asking for both, and I'm afaid that it would be easy for a user to overlook the extra request. This will be especially in the much more complicated user prompt required once we support screen capture from a monitor or window.

It *would* make sense to allow screen+audio capture if the audio captured was the  audio output from the machine (for screen capture) or application (for window capture), but we don't have system API for that.

Thanks for the feedback! We're always happy to hear about feature requests, bugs, etc - even if we aren't able to do anything as is the case here.
Comment 21 Raship Shah 2020-12-18 22:02:28 PST
Now that Safari on iPadOS claims to support desktop class browsing, would we expect to see this available there as well?
Comment 22 Laurent Denoue 2021-12-14 07:24:15 PST
> We aren't currently planning to add support for capturing microphone audio
> to getDisplayMedia.

Are people working on window sharing?
All other engines support picking an individual window to share instead of the entire screen, which I find useful for privacy reasons.
Comment 23 Sam Sneddon [:gsnedders] 2022-02-25 10:43:12 PST
Closing this as fixed given this shipped in Safari 13 in 2019.