Attachment 353965 Details for Bug 190684 – Patch for landing

[patch] Patch for landing

bug-190684-20181106105725.patch (text/plain), 30.35 KB, created by Thibault Saunier on 2018-11-06 05:57:26 PST

(hide)

Description:

Filename:

MIME Type:

Creator: Thibault Saunier

Created: 2018-11-06 05:57:26 PST

Size: 30.35 KB

patch

obsolete

>Subversion Revision: 237804
>diff --git a/Source/WebCore/ChangeLog b/Source/WebCore/ChangeLog
>index 0dacdf736a0c921ecf6649981cc5bc94963b1886..a414b46c8d7c0d0a94db011c483c12d36df1fff3 100644
>--- a/Source/WebCore/ChangeLog
>+++ b/Source/WebCore/ChangeLog
>@@ -1,3 +1,76 @@
>+2018-10-17  Thibault Saunier  <tsaunier@igalia.com>
>+
>+        [GStreamer][WebRTC] Implement black frame generation
>+        https://bugs.webkit.org/show_bug.cgi?id=190684
>+
>+        Reviewed by Xabier Rodriguez-Calvar.
>+
>+        Avoiding webrtc/video-mute-vp8.html to crash but it still fails because
>+        we are missing the canvas bridging code.
>+
>+        * platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp:
>+        (WebCore::RealtimeOutgoingVideoSourceLibWebRTC::createBlackFrame):
>+        * platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h:
>+
>+2018-10-17  Thibault Saunier  <tsaunier@igalia.com>
>+
>+        [GStreamer][WebRTC] Do not try to handle framerate modulation in the encoder
>+        https://bugs.webkit.org/show_bug.cgi?id=190683
>+
>+        Reviewed by Xabier Rodriguez-Calvar.
>+
>+        This has to already be handled in capturing pipeline or in libwebrtc itself.
>+
>+        No other encoder implementation do that, and libwebrtc is not happy with encoder that do not output the exact number of frames that have been passed in.
>+
>+        No regressions detected and libwebrtc is happier this way, less warning output and no more frame corruption in H264 streams found.
>+
>+        * platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp:
>+        (WebCore::GStreamerVideoEncoder::InitEncode):
>+
>+2018-10-17  Thibault Saunier  <tsaunier@igalia.com>
>+
>+        [GStreamer][WebRTC] Handle setting max number of frame between keyframes
>+        https://bugs.webkit.org/show_bug.cgi?id=190682
>+
>+        Reviewed by Xabier Rodriguez-Calvar.
>+
>+        That has been manually checked.
>+
>+        * platform/mediastream/libwebrtc/GStreamerVideoEncoder.cpp:
>+        (gst_webrtc_video_encoder_get_property):
>+        (gst_webrtc_video_encoder_set_property):
>+        (register_known_encoder):
>+        (gst_webrtc_video_encoder_class_init):
>+        * platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp:
>+        (WebCore::GStreamerVideoEncoder::InitEncode):
>+
>+2018-10-17  Thibault Saunier  <tsaunier@igalia.com>
>+
>+        [GStreamer] Fix EncodedImage timestamps to match what libWebRTC expects
>+        https://bugs.webkit.org/show_bug.cgi?id=190035
>+
>+        Reviewed by Xabier Rodriguez-Calvar.
>+
>+        We can't rely on GStreamer timestamps to pass to EncodedImages after encoding
>+        because libWebRTC doesn't use the timestamp we fed it but does
>+        some computation on the input timestamp in the images we pass in before it passes
>+        them back to the encoder. Then internally LibWebRTC relies on those exact timestamps
>+        passed into the encoder to do checks and compute RTP timestamps so we need to carefully
>+        pass the exact timestamps to LibWebRTC (in practice we still use GStreamer timestamps in
>+        all the GStreamer processing pipelines as the WebRTC object basically wraps the "same"
>+        `GstSample` all around, but we are not synced on the clock anyway).
>+
>+        * platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:
>+        * platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.cpp:
>+        (WebCore::LibWebRTCVideoFrameFromGStreamerSample):
>+        * platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.h:
>+        * platform/mediastream/libwebrtc/GStreamerVideoDecoderFactory.cpp:
>+        (WebCore::GStreamerVideoDecoder::newSampleCallback):
>+        * platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp:
>+        (WebCore::GStreamerVideoEncoder::GStreamerVideoEncoder):
>+        (WebCore::GStreamerVideoEncoder::newSampleCallback):
>+
> 2018-11-05  Thibault Saunier  <tsaunier@igalia.com>
> 
>         [GStreamer][WebRTC] Error out when simulcast is activated
>diff --git a/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp b/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp
>index ffdae4eb508ee625d282ca22a9987c5cc2fe30d6..09a43e60c4c4bf63091842a86696f9e2501b648f 100644
>--- a/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp
>+++ b/Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp
>@@ -154,7 +154,6 @@ struct _WebKitMediaStreamSrc {
>     GstClockTime firstFramePts;
> 
>     std::unique_ptr<WebKitMediaStreamTrackObserver> observer;
>-    String videoTrackID;
>     volatile gint npads;
>     gulong probeid;
>     RefPtr<MediaStreamPrivate> stream;
>diff --git a/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.cpp b/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.cpp
>index 57c48d2e081de1d4123a23c2d15eecdfae88fbf4..120f5f3cdb5a8dde9765ab1938ed9c1077cd7821 100644
>--- a/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.cpp
>+++ b/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.cpp
>@@ -69,12 +69,13 @@ rtc::scoped_refptr<webrtc::VideoFrameBuffer> GStreamerVideoFrameLibWebRTC::creat
>     return rtc::scoped_refptr<webrtc::VideoFrameBuffer>(new GStreamerVideoFrameLibWebRTC(sample, info));
> }
> 
>-std::unique_ptr<webrtc::VideoFrame> LibWebRTCVideoFrameFromGStreamerSample(GstSample* sample, webrtc::VideoRotation rotation)
>+std::unique_ptr<webrtc::VideoFrame> LibWebRTCVideoFrameFromGStreamerSample(GstSample* sample, webrtc::VideoRotation rotation,
>+    int64_t timestamp, int64_t renderTimeMs)
> {
>     auto frameBuffer(GStreamerVideoFrameLibWebRTC::create(sample));
> 
>-    auto buffer = gst_sample_get_buffer(sample);
>-    return std::unique_ptr<webrtc::VideoFrame>(new webrtc::VideoFrame(frameBuffer, GST_BUFFER_DTS(buffer), GST_BUFFER_PTS(buffer), rotation));
>+    return std::unique_ptr<webrtc::VideoFrame>(
>+        new webrtc::VideoFrame(frameBuffer, timestamp, renderTimeMs, rotation));
> }
> 
> webrtc::VideoFrameBuffer::Type GStreamerVideoFrameLibWebRTC::type() const
>diff --git a/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.h b/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.h
>index 53ada42361bc803ee3a42574a458a0a279d1c821..9e53e96da2b62c04154209fee7f5d82716689b61 100644
>--- a/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.h
>+++ b/Source/WebCore/platform/mediastream/gstreamer/GStreamerVideoFrameLibWebRTC.h
>@@ -32,7 +32,7 @@
> namespace WebCore {
> 
> const GRefPtr<GstSample> GStreamerSampleFromLibWebRTCVideoFrame(const webrtc::VideoFrame&);
>-std::unique_ptr<webrtc::VideoFrame> LibWebRTCVideoFrameFromGStreamerSample(GstSample*, webrtc::VideoRotation);
>+std::unique_ptr<webrtc::VideoFrame> LibWebRTCVideoFrameFromGStreamerSample(GstSample*, webrtc::VideoRotation, int64_t timestamp, int64_t renderTimeMs);
> 
> class GStreamerVideoFrameLibWebRTC : public rtc::RefCountedObject<webrtc::VideoFrameBuffer> {
> public:
>diff --git a/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp b/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp
>index 91314221c6ae25f51e5885ac5fcadabd3a98f2dc..f178eed584011ee17746ffed1ac692a6d6554b82 100644
>--- a/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp
>+++ b/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp
>@@ -80,6 +80,21 @@ void RealtimeOutgoingVideoSourceLibWebRTC::sampleBufferUpdated(MediaStreamTrackP
>     sendFrame(WTFMove(frameBuffer));
> }
> 
>+rtc::scoped_refptr<webrtc::VideoFrameBuffer> RealtimeOutgoingVideoSourceLibWebRTC::createBlackFrame(size_t  width, size_t  height)
>+{
>+    GstVideoInfo info;
>+
>+    gst_video_info_set_format(&info, GST_VIDEO_FORMAT_RGB, width, height);
>+
>+    GRefPtr<GstBuffer> buffer = adoptGRef(gst_buffer_new_allocate(nullptr, info.size, nullptr));
>+    GRefPtr<GstCaps> caps = adoptGRef(gst_video_info_to_caps(&info));
>+
>+    GstMappedBuffer map(buffer.get(), GST_MAP_WRITE);
>+    memset(map.data(), 0, info.size);
>+
>+    return GStreamerVideoFrameLibWebRTC::create(gst_sample_new(buffer.get(), caps.get(), NULL, NULL));
>+}
>+
> } // namespace WebCore
> 
> #endif // USE(LIBWEBRTC)
>diff --git a/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h b/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h
>index 1f01a254c156587bda5d6ad2e90db08ac9ab69f0..9e1758df9f451b470af2f7d4e653ea358541c04c 100644
>--- a/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h
>+++ b/Source/WebCore/platform/mediastream/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.h
>@@ -39,7 +39,7 @@ public:
> private:
>     explicit RealtimeOutgoingVideoSourceLibWebRTC(Ref<MediaStreamTrackPrivate>&&);
> 
>-    rtc::scoped_refptr<webrtc::VideoFrameBuffer> createBlackFrame(size_t, size_t) final { return nullptr; }
>+    rtc::scoped_refptr<webrtc::VideoFrameBuffer> createBlackFrame(size_t, size_t) final;
> 
>     // MediaStreamTrackPrivate::Observer API
>     void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) final;
>diff --git a/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoDecoderFactory.cpp b/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoDecoderFactory.cpp
>index dc330b96c90e9dbcddbca5fa24aba16d60728bb0..f0ac8df6f6adbbe59f29ab133ce95c21453584d9 100644
>--- a/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoDecoderFactory.cpp
>+++ b/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoDecoderFactory.cpp
>@@ -44,6 +44,11 @@ GST_DEBUG_CATEGORY(webkit_webrtcdec_debug);
> 
> namespace WebCore {
> 
>+typedef struct {
>+    uint64_t timestamp;
>+    int64_t renderTimeMs;
>+} InputTimestamps;
>+
> class GStreamerVideoDecoder : public webrtc::VideoDecoder {
> public:
>     GStreamerVideoDecoder()
>@@ -167,7 +172,8 @@ public:
>         GST_BUFFER_PTS(buffer.get()) = (static_cast<guint64>(renderTimeMs) * GST_MSECOND) - m_firstBufferPts;
>         {
>             auto locker = holdLock(m_bufferMapLock);
>-            m_dtsPtsMap[GST_BUFFER_PTS(buffer.get())] = inputImage.Timestamp();
>+            InputTimestamps timestamps = {inputImage.Timestamp(), renderTimeMs};
>+            m_dtsPtsMap[GST_BUFFER_PTS(buffer.get())] = timestamps;
>         }
> 
>         GST_LOG_OBJECT(pipeline(), "%ld Decoding: %" GST_PTR_FORMAT, renderTimeMs, buffer.get());
>@@ -228,14 +234,16 @@ public:
>         auto sample = gst_app_sink_pull_sample(GST_APP_SINK(sink));
>         auto buffer = gst_sample_get_buffer(sample);
> 
>-        {
>-            auto locker = holdLock(m_bufferMapLock);
>-            // Make sure that the frame.timestamp == previsouly input_frame._timeStamp
>-            // as it is required by the VideoDecoder baseclass.
>-            GST_BUFFER_DTS(buffer) = m_dtsPtsMap[GST_BUFFER_PTS(buffer)];
>-            m_dtsPtsMap.erase(GST_BUFFER_PTS(buffer));
>-        }
>-        auto frame(LibWebRTCVideoFrameFromGStreamerSample(sample, webrtc::kVideoRotation_0));
>+        m_bufferMapLock.lock();
>+        // Make sure that the frame.timestamp == previsouly input_frame._timeStamp
>+        // as it is required by the VideoDecoder baseclass.
>+        auto timestamps = m_dtsPtsMap[GST_BUFFER_PTS(buffer)];
>+        m_dtsPtsMap.erase(GST_BUFFER_PTS(buffer));
>+        m_bufferMapLock.unlock();
>+
>+        auto frame(LibWebRTCVideoFrameFromGStreamerSample(sample, webrtc::kVideoRotation_0,
>+            timestamps.timestamp, timestamps.renderTimeMs));
>+
>         GST_BUFFER_DTS(buffer) = GST_CLOCK_TIME_NONE;
>         GST_LOG_OBJECT(pipeline(), "Output decoded frame! %d -> %" GST_PTR_FORMAT,
>             frame->timestamp(), buffer);
>@@ -267,7 +275,7 @@ private:
>     webrtc::DecodedImageCallback* m_imageReadyCb;
> 
>     Lock m_bufferMapLock;
>-    StdMap<GstClockTime, GstClockTime> m_dtsPtsMap;
>+    StdMap<GstClockTime, InputTimestamps> m_dtsPtsMap;
>     GstClockTime m_firstBufferPts;
>     GstClockTime m_firstBufferDts;
> };
>diff --git a/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoder.cpp b/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoder.cpp
>index e0f7daf4ba14a9596e86c76b0831475d591b64c0..d6093b9fdda3a62a6514fed5a80983d57d87b49f 100644
>--- a/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoder.cpp
>+++ b/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoder.cpp
>@@ -54,6 +54,7 @@ typedef struct
>   SetBitrateFunc setBitrate;
>   SetupEncoder setupEncoder;
>   const gchar *bitrate_propname;
>+  const gchar *keyframe_interval_propname;
> } EncoderDefinition;
> 
> typedef enum
>@@ -74,6 +75,7 @@ EncoderDefinition encoders[ENCODER_LAST] = {
>   NULL,
>   NULL,
>   NULL,
>+  NULL,
> };
> 
> typedef struct
>@@ -97,6 +99,7 @@ enum
>   PROP_FORMAT,
>   PROP_ENCODER,
>   PROP_BITRATE,
>+  PROP_KEYFRAME_INTERVAL,
>   N_PROPS
> };
> 
>@@ -126,6 +129,11 @@ gst_webrtc_video_encoder_get_property (GObject * object,
>     case PROP_BITRATE:
>       g_value_set_uint (value, priv->bitrate);
>       break;
>+    case PROP_KEYFRAME_INTERVAL:
>+      if (priv->encoder)
>+        g_object_get_property (G_OBJECT (priv->encoder),
>+            encoders[priv->encoderId].keyframe_interval_propname, value);
>+      break;
>     default:
>       G_OBJECT_WARN_INVALID_PROPERTY_ID (object, prop_id, pspec);
>   }
>@@ -212,6 +220,7 @@ gst_webrtc_video_encoder_set_property (GObject * object,
>     guint prop_id, const GValue * value, GParamSpec * pspec)
> {
>   GstWebrtcVideoEncoder *self = GST_WEBRTC_VIDEO_ENCODER (object);
>+  GstWebrtcVideoEncoderPrivate *priv = PRIV (self);
> 
>   switch (prop_id) {
>     case PROP_FORMAT:
>@@ -219,6 +228,13 @@ gst_webrtc_video_encoder_set_property (GObject * object,
>       break;
>     case PROP_BITRATE:
>       gst_webrtc_video_encoder_set_bitrate (self, g_value_get_uint (value));
>+      break;
>+    case PROP_KEYFRAME_INTERVAL:
>+      if (priv->encoder)
>+        g_object_set (priv->encoder,
>+            encoders[priv->encoderId].keyframe_interval_propname,
>+            g_value_get_uint (value), NULL);
>+
>       break;
>     default:
>       G_OBJECT_WARN_INVALID_PROPERTY_ID (object, prop_id, pspec);
>@@ -229,7 +245,7 @@ static void
> register_known_encoder (EncoderId encId, const gchar * name,
>     const gchar * parser_name, const gchar * caps, const gchar * encoded_format,
>     SetupEncoder setupEncoder, const gchar * bitrate_propname,
>-    SetBitrateFunc setBitrate)
>+    SetBitrateFunc setBitrate, const gchar * keyframe_interval_propname)
> {
>   GstPluginFeature *feature =
>       gst_registry_lookup_feature (gst_registry_get (), name);
>@@ -252,6 +268,7 @@ register_known_encoder (EncoderId encId, const gchar * name,
>   encoders[encId].setupEncoder = setupEncoder;
>   encoders[encId].bitrate_propname = bitrate_propname;
>   encoders[encId].setBitrate = setBitrate;
>+  encoders[encId].keyframe_interval_propname = keyframe_interval_propname;
> }
> 
> static void
>@@ -314,15 +331,22 @@ gst_webrtc_video_encoder_class_init (GstWebrtcVideoEncoderClass * klass)
>           (GParamFlags) (G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS |
>               G_PARAM_CONSTRUCT)));
> 
>+  g_object_class_install_property (object_class, PROP_KEYFRAME_INTERVAL,
>+      g_param_spec_uint ("keyframe-interval", "Keyframe interval",
>+          "The interval between keyframes", 0, G_MAXINT, 0,
>+          (GParamFlags) (G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS |
>+              G_PARAM_CONSTRUCT)));
>+
>   register_known_encoder (ENCODER_X264, "x264enc", "h264parse", "video/x-h264",
>       "video/x-h264,alignment=au,stream-format=byte-stream,profile=baseline",
>-      setup_x264enc, "bitrate", set_bitrate_kbit_per_sec);
>+      setup_x264enc, "bitrate", set_bitrate_kbit_per_sec, "key-int-max");
>   register_known_encoder (ENCODER_OPENH264, "openh264enc", "h264parse",
>       "video/x-h264",
>       "video/x-h264,alignment=au,stream-format=byte-stream,profile=baseline",
>-      setup_openh264enc, "bitrate", set_bitrate_kbit_per_sec);
>+      setup_openh264enc, "bitrate", set_bitrate_kbit_per_sec, "gop-size");
>   register_known_encoder (ENCODER_VP8, "vp8enc", NULL, "video/x-vp8", NULL,
>-      setup_vp8enc, "target-bitrate", set_bitrate_bit_per_sec);
>+      setup_vp8enc, "target-bitrate", set_bitrate_bit_per_sec,
>+      "keyframe-max-dist");
> }
> 
> static void
>diff --git a/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp b/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp
>index d58f3e56d2d8be80e62cb1b119fb9e89cd41f754..b4421137e1c2edfc440e7f45c5108e78843e00ea 100644
>--- a/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp
>+++ b/Source/WebCore/platform/mediastream/libwebrtc/GStreamerVideoEncoderFactory.cpp
>@@ -40,8 +40,9 @@
> #undef GST_USE_UNSTABLE_API
> #include <gst/pbutils/encoding-profile.h>
> #include <gst/video/video.h>
>-
>-#include <mutex>
>+#include <wtf/HashMap.h>
>+#include <wtf/Lock.h>
>+#include <wtf/StdMap.h>
> 
> // Required for unified builds
> #ifdef GST_CAT_DEFAULT
>@@ -55,21 +56,24 @@ GST_DEBUG_CATEGORY(webkit_webrtcenc_debug);
> 
> namespace WebCore {
> 
>-typedef void (*BitrateSetter)(GstElement* encoder, uint32_t bitrate);
>-static GRefPtr<GRegex> targetBitrateBitPerSec;
>-static GRefPtr<GRegex> bitrateBitPerSec;
>-static GRefPtr<GRegex> bitrateKBitPerSec;
>+typedef struct {
>+    uint64_t rtpTimestamp;
>+    int64_t captureTimeMs;
>+    webrtc::CodecSpecificInfo codecInfo;
>+} FrameData;
> 
> class GStreamerVideoEncoder : public webrtc::VideoEncoder {
> public:
>     GStreamerVideoEncoder(const webrtc::SdpVideoFormat&)
>         : m_firstFramePts(GST_CLOCK_TIME_NONE)
>         , m_restrictionCaps(adoptGRef(gst_caps_new_empty_simple("video/x-raw")))
>+        , m_adapter(adoptGRef(gst_adapter_new()))
>     {
>     }
>     GStreamerVideoEncoder()
>         : m_firstFramePts(GST_CLOCK_TIME_NONE)
>         , m_restrictionCaps(adoptGRef(gst_caps_new_empty_simple("video/x-raw")))
>+        , m_adapter(adoptGRef(gst_adapter_new()))
>     {
>     }
> 
>@@ -79,7 +83,6 @@ public:
>             newBitrate, frameRate);
> 
>         auto caps = adoptGRef(gst_caps_copy(m_restrictionCaps.get()));
>-        gst_caps_set_simple(caps.get(), "framerate", GST_TYPE_FRACTION, frameRate, 1, nullptr);
> 
>         SetRestrictionCaps(WTFMove(caps));
> 
>@@ -128,11 +131,12 @@ public:
>         ASSERT(encoder);
>         m_encoder = encoder.get();
> 
>+        g_object_set(m_encoder, "keyframe-interval", KeyframeInterval(codecSettings), nullptr);
>+
>         m_src = makeElement("appsrc");
>         g_object_set(m_src, "is-live", true, "format", GST_FORMAT_TIME, nullptr);
> 
>         auto videoconvert = makeElement("videoconvert");
>-        auto videorate = makeElement("videorate");
>         auto sink = makeElement("appsink");
>         gst_app_sink_set_emit_signals(GST_APP_SINK(sink), TRUE);
>         g_signal_connect(sink, "new-sample", G_CALLBACK(newSampleCallbackTramp), this);
>@@ -142,8 +146,8 @@ public:
>         if (m_restrictionCaps)
>             g_object_set(m_capsFilter, "caps", m_restrictionCaps.get(), nullptr);
> 
>-        gst_bin_add_many(GST_BIN(m_pipeline.get()), m_src, videorate, videoconvert, m_capsFilter, encoder.leakRef(), sink, nullptr);
>-        if (!gst_element_link_many(m_src, videorate, videoconvert, m_capsFilter, m_encoder, sink, nullptr))
>+        gst_bin_add_many(GST_BIN(m_pipeline.get()), m_src, videoconvert, m_capsFilter, encoder.leakRef(), sink, nullptr);
>+        if (!gst_element_link_many(m_src, videoconvert, m_capsFilter, m_encoder, sink, nullptr))
>             ASSERT_NOT_REACHED();
> 
>         gst_element_set_state(m_pipeline.get(), GST_STATE_PLAYING);
>@@ -187,7 +191,7 @@ public:
>     }
> 
>     int32_t Encode(const webrtc::VideoFrame& frame,
>-        const webrtc::CodecSpecificInfo*,
>+        const webrtc::CodecSpecificInfo* codecInfo,
>         const std::vector<webrtc::FrameType>* frameTypes) final
>     {
>         if (!m_imageReadyCb) {
>@@ -211,11 +215,19 @@ public:
>             gst_pad_set_offset(pad.get(), -m_firstFramePts);
>         }
> 
>+        webrtc::CodecSpecificInfo localCodecInfo;
>+        FrameData frameData = { frame.timestamp(), frame.render_time_ms(), codecInfo ? *codecInfo : localCodecInfo };
>+        {
>+            auto locker = holdLock(m_bufferMapLock);
>+            m_framesData.append(frameData);
>+        }
>+
>         for (auto frame_type : *frameTypes) {
>             if (frame_type == webrtc::kVideoFrameKey) {
>                 auto pad = adoptGRef(gst_element_get_static_pad(m_src, "src"));
>                 auto forceKeyUnit = gst_video_event_new_downstream_force_key_unit(GST_CLOCK_TIME_NONE,
>                     GST_CLOCK_TIME_NONE, GST_CLOCK_TIME_NONE, FALSE, 1);
>+                GST_INFO_OBJECT(m_pipeline.get(), "Requesting KEYFRAME!");
> 
>                 if (!gst_pad_push_event(pad.get(), forceKeyUnit))
>                     GST_WARNING_OBJECT(pipeline(), "Could not send ForceKeyUnit event");
>@@ -240,6 +252,29 @@ public:
>         auto buffer = gst_sample_get_buffer(sample.get());
>         auto caps = gst_sample_get_caps(sample.get());
> 
>+        webrtc::CodecSpecificInfo localCodecInfo;
>+        FrameData frameData = { 0, 0, localCodecInfo};
>+        {
>+            auto locker = holdLock(m_bufferMapLock);
>+            if (!m_framesData.size()) {
>+                gst_adapter_push(m_adapter.get(), gst_buffer_ref(buffer));
>+
>+                return GST_FLOW_OK;
>+            }
>+
>+            if (gst_adapter_available(m_adapter.get()) > 0) {
>+                uint flags = GST_BUFFER_FLAGS(buffer);
>+
>+                GST_INFO_OBJECT(m_pipeline.get(), "Got more buffer than pushed frame, trying to deal with it.");
>+                gst_adapter_push(m_adapter.get(), gst_buffer_ref(buffer));
>+
>+                buffer = gst_adapter_take_buffer(m_adapter.get(), gst_adapter_available(m_adapter.get()));
>+                GST_BUFFER_FLAGS(buffer) = flags;
>+            }
>+            frameData = m_framesData[0];
>+            m_framesData.remove(static_cast<size_t>(0));
>+        }
>+
>         webrtc::RTPFragmentationHeader fragmentationInfo;
>         Fragmentize(&m_encodedFrame, &m_encodedImageBuffer, &m_encodedImageBufferSize, buffer, &fragmentationInfo);
>         if (!m_encodedFrame._size)
>@@ -251,15 +286,16 @@ public:
>             nullptr);
> 
>         m_encodedFrame._frameType = GST_BUFFER_FLAG_IS_SET(buffer, GST_BUFFER_FLAG_DELTA_UNIT) ? webrtc::kVideoFrameDelta : webrtc::kVideoFrameKey;
>-        m_encodedFrame._completeFrame = true;
>-        m_encodedFrame.capture_time_ms_ = GST_TIME_AS_MSECONDS(GST_BUFFER_PTS(buffer));
>-        m_encodedFrame.SetTimestamp(GST_TIME_AS_MSECONDS(GST_BUFFER_DTS(buffer)));
>-        GST_LOG_OBJECT(m_pipeline.get(), "Got buffer TS: %" GST_TIME_FORMAT, GST_TIME_ARGS(GST_BUFFER_PTS(buffer)));
>+        m_encodedFrame._completeFrame = false;
>+        m_encodedFrame.capture_time_ms_ = frameData.captureTimeMs;
>+        m_encodedFrame.SetTimestamp(frameData.rtpTimestamp);
> 
>-        webrtc::CodecSpecificInfo codecSpecifiInfos;
>-        PopulateCodecSpecific(&codecSpecifiInfos, buffer);
>+        GST_LOG_OBJECT(m_pipeline.get(), "Got buffer capture_time_ms: %ld _timestamp: %ld",
>+            m_encodedFrame.capture_time_ms_, m_encodedFrame.Timestamp());
> 
>-        webrtc::EncodedImageCallback::Result result = m_imageReadyCb->OnEncodedImage(m_encodedFrame, &codecSpecifiInfos, &fragmentationInfo);
>+        PopulateCodecSpecific(&frameData.codecInfo, buffer);
>+
>+        webrtc::EncodedImageCallback::Result result = m_imageReadyCb->OnEncodedImage(m_encodedFrame, &frameData.codecInfo, &fragmentationInfo);
>         if (result.error != webrtc::EncodedImageCallback::Result::OK)
>             GST_ERROR_OBJECT(m_pipeline.get(), "Encode callback failed: %d", result.error);
> 
>@@ -294,11 +330,6 @@ public:
>         }
>     }
> 
>-    virtual const gchar* ProfileName()
>-    {
>-        return nullptr;
>-    }
>-
>     virtual const gchar* Caps()
>     {
>         return nullptr;
>@@ -346,6 +377,7 @@ public:
>     }
> 
>     virtual const gchar* Name() = 0;
>+    virtual int KeyframeInterval(const webrtc::VideoCodec* codecSettings) = 0;
> 
>     void SetRestrictionCaps(GRefPtr<GstCaps> caps)
>     {
>@@ -372,6 +404,10 @@ private:
>     webrtc::EncodedImage m_encodedFrame;
>     std::unique_ptr<uint8_t[]> m_encodedImageBuffer;
>     size_t m_encodedImageBufferSize;
>+
>+    Lock m_bufferMapLock;
>+    GRefPtr<GstAdapter> m_adapter;
>+    Vector<FrameData> m_framesData;
> };
> 
> class H264Encoder : public GStreamerVideoEncoder {
>@@ -388,6 +424,11 @@ public:
>             packetizationMode = webrtc::H264PacketizationMode::NonInterleaved;
>     }
> 
>+    int KeyframeInterval(const webrtc::VideoCodec* codecSettings) final
>+    {
>+        return codecSettings->H264().keyFrameInterval;
>+    }
>+
>     // FIXME - MT. safety!
>     void Fragmentize(webrtc::EncodedImage* encodedImage, std::unique_ptr<uint8_t[]>* encodedImageBuffer, size_t *bufferSize,
>         GstBuffer* gstbuffer, webrtc::RTPFragmentationHeader* fragmentationHeader) final
>@@ -474,7 +515,11 @@ public:
>     const gchar* Caps() final { return "video/x-vp8"; }
>     const gchar* Name() final { return cricket::kVp8CodecName; }
>     webrtc::VideoCodecType CodecType() final { return webrtc::kVideoCodecVP8; }
>-    virtual const gchar* ProfileName() { return "Profile Realtime"; }
>+
>+    int KeyframeInterval(const webrtc::VideoCodec* codecSettings) final
>+    {
>+        return codecSettings->VP8().keyFrameInterval;
>+    }
> 
>     void PopulateCodecSpecific(webrtc::CodecSpecificInfo* codecSpecifiInfos, GstBuffer* buffer) final
>     {
>diff --git a/Tools/ChangeLog b/Tools/ChangeLog
>index 46ca4fbd1cc24755488e209924465a3a5f937488..8b34d09d4bb5f986126436c10085f0240112aaee 100644
>--- a/Tools/ChangeLog
>+++ b/Tools/ChangeLog
>@@ -1,3 +1,27 @@
>+2018-10-17  Thibault Saunier  <tsaunier@igalia.com>
>+
>+        [Flatpak] Build libx264 as it is the h264 encoder we support in WebRTC
>+        https://bugs.webkit.org/show_bug.cgi?id=190680
>+
>+        Reviewed by Xabier Rodriguez-Calvar.
>+
>+        * flatpak/org.webkit.WebKit.yaml:
>+
>+2018-10-17  Thibault Saunier  <tsaunier@igalia.com>
>+
>+        [Flatpak] Do not disable signals when running "usual" commands
>+        https://bugs.webkit.org/show_bug.cgi?id=190679
>+
>+        Reviewed by Xabier Rodriguez-Calvar.
>+
>+        Otherwise <ctr>c doesn't work while compiling. It is not the best fix
>+        as there are cases we want to let only children receive signals, but
>+        it is not clear how to determine when (when running gdb, we still
>+        disable those).
>+
>+        * flatpak/flatpakutils.py:
>+        (WebkitFlatpak.run_in_sandbox):
>+
> 2018-11-05  Thibault Saunier  <tsaunier@igalia.com>
> 
>         [GStreamer][WebRTC] Add webrtcencoder bin to cleanup and refactor the way we set encoders
>diff --git a/Tools/flatpak/flatpakutils.py b/Tools/flatpak/flatpakutils.py
>index df048cbe196b5afd70841004623f85a4f367b025..621566d462f7b05f1f4c125cba84eb79cdeb189f 100644
>--- a/Tools/flatpak/flatpakutils.py
>+++ b/Tools/flatpak/flatpakutils.py
>@@ -742,12 +742,11 @@ class WebkitFlatpak:
>             _log.debug('Running in sandbox: "%s" %s\n' % ('" "'.join(flatpak_command), shell_string))
>             flatpak_command.extend(['sh', "/run/host/" + tmpscript.name])
> 
>-            with disable_signals():
>-                try:
>-                    subprocess.check_call(flatpak_command, stdout=stdout)
>-                except subprocess.CalledProcessError as e:
>-                    sys.stderr.write(str(e) + "\n")
>-                    return e.returncode
>+            try:
>+                subprocess.check_call(flatpak_command, stdout=stdout)
>+            except subprocess.CalledProcessError as e:
>+                sys.stderr.write(str(e) + "\n")
>+                return e.returncode
> 
>         return 0
> 
>diff --git a/Tools/flatpak/org.webkit.WebKit.yaml b/Tools/flatpak/org.webkit.WebKit.yaml
>index 59bdc92d7c8986c6c43bbb8d11ce75f6ea369539..8b2915e1aa9ee6924fd7f33a89ae5b7a42269467 100644
>--- a/Tools/flatpak/org.webkit.WebKit.yaml
>+++ b/Tools/flatpak/org.webkit.WebKit.yaml
>@@ -173,6 +173,17 @@ modules:
>         path: ../gstreamer/patches/gst-plugins-good-0003-qtdemux-Clarify-field-name-about-stream-encryption-s.patch
>     config-opts:
>       - -Ddisable_gtkdoc=true
>+  - name: x264
>+    config-opts:
>+    - "--enable-shared"
>+    - "--enable-static"
>+    - "--enable-pic"
>+    - "--disable-lavf"
>+    sources:
>+      - type: archive
>+        url: http://download.videolan.org/pub/x264/snapshots/x264-snapshot-20140212-2245-stable.tar.bz2
>+        sha256: 5d98e9e4faf6dd55e7193ed379aff477b8acbda6777758956ef7e5f05067be18
>+
>   - name: gst-plugins-ugly
>     buildsystem: meson
>     sources:
>@@ -181,6 +192,7 @@ modules:
>         branch: 1.14.1
>     config-opts:
>       - -Ddisable_gtkdoc=true
>+
>   - name: gst-plugins-bad
>     buildsystem: meson
>     sources:
>diff --git a/LayoutTests/ChangeLog b/LayoutTests/ChangeLog
>index 7589be208125d4e07c5f52516b428d7884982832..545c26a8581870c9e1d1738ba079b5da75785503 100644
>--- a/LayoutTests/ChangeLog
>+++ b/LayoutTests/ChangeLog
>@@ -1,3 +1,16 @@
>+2018-10-17  Thibault Saunier  <tsaunier@igalia.com>
>+
>+        [GStreamer][WebRTC] Implement black frame generation
>+        https://bugs.webkit.org/show_bug.cgi?id=190684
>+
>+        Reviewed by Xabier Rodriguez-Calvar.
>+
>+        Avoiding webrtc/video-mute-vp8.html to crash but it still fails because
>+        we are missing the canvas bridging code.
>+
>+        * platform/gtk/TestExpectations:
>+        * platform/wpe/TestExpectations:
>+
> 2018-11-05  Thibault Saunier  <tsaunier@igalia.com>
> 
>         [GStreamer][WebRTC] Error out when simulcast is activated
>diff --git a/LayoutTests/platform/gtk/TestExpectations b/LayoutTests/platform/gtk/TestExpectations
>index 39b4c13b55a6818c73ff389557c9dbe95ec22792..9946e36f9c61d27d028de8721d31bb1c2e667ddf 100644
>--- a/LayoutTests/platform/gtk/TestExpectations
>+++ b/LayoutTests/platform/gtk/TestExpectations
>@@ -2392,7 +2392,7 @@ webkit.org/b/190738 media/encrypted-media/clearKey/clearKey-webm-video-playback-
> 
> webkit.org/b/191008 webanimations/accelerated-animation-suspension.html [ Timeout ]
> 
>-webkit.org/b/191009 webrtc/video-mute-vp8.html [ Timeout ]
>+webkit.org/b/191009 webrtc/video-mute-vp8.html [ Failure ]
> 
> #////////////////////////////////////////////////////////////////////////////////////////
> # End of Tests timing out
>diff --git a/LayoutTests/platform/wpe/TestExpectations b/LayoutTests/platform/wpe/TestExpectations
>index 4b2214dd804b4c850ab95ddafa8b496b8dbed135..cdd8a920321b66933e0029e98dcfa87bd1d6cca3 100644
>--- a/LayoutTests/platform/wpe/TestExpectations
>+++ b/LayoutTests/platform/wpe/TestExpectations
>@@ -47,6 +47,7 @@
> 
> # The webrtc implementation is not fully completed yet
> webkit.org/b/187064 webrtc/simulcast-h264.html
>+webkit.org/b/191009 webrtc/video-mute-vp8.html [ Failure ]
> webkit.org/b/189567 webrtc/video-addLegacyTransceiver.html [ Failure ]
> webkit.org/b/187064 webrtc/audio-peer-connection-g722.html
> webkit.org/b/187064 webrtc/video-with-receiver.html

Actions: View | Formatted Diff | Diff

Attachments on bug 190684: 352624 | 353871 | 353884 | 353965 | 353966 | 353967