[MSE][GStreamer] Incorrect notification of a source buffer about AppSrc being ready to more data

• Component: WebKitGTK
• Status: RESOLVED INVALID

[Sergey Krutikov]

The bug happens on our proprietary platform, which is based on WebKitGTK. We have added an extra GStreamer pipeline that consumes data from the output of PlayBin pipeline. There is a MpegTsMux element which generates an encoded stream using this data in a synchronous manner. The symptom of the bug is a playback getting stuck due to the lack of video data, which makes a muxer in the extra pipeline to not produce any stream data. We have found this bug playing movies in Hulu application.

The problem is caused by incorrect notification of a video source buffer about video AppSrc in PlayBin pipeline needs more data. The notification is made in the corresponding callback of GstAppSrc element (https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/mse/WebKitMediaSourceGStreamer.cpp#L138). However, there is a problem in how this notification is made. In fact, it should be dispatched to the main thread using the corresponding method of the notifier (https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/MainThreadNotifier.h#L44). Careful examination of this code reveals a hidden problem: if a notification of a particular type has already been added (i.e. it is pending currently) but has not been handled yet, then a new notification of the same type will be lost. This might happen because both audio and video AppSrc share the same notification type (https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/mse/WebKitMediaSourceGStreamerPrivate.h#L99).

If we assume that a buffer in audio AppSink of PlayBin pipeline is quite small, then 'need data' event will happen much more often for audio AppSrc than video AppSrc. This is exactly what sometimes happens in our platform. The consequence is that at some moment after handling a pending notification a new one is added, and it is always for audio since there are far more events for audio than for video. So 'need data' events for video get lost. This leads to 'm_isReadyForMoreSamples' flag staying in False state (https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/mse/SourceBufferPrivateGStreamer.cpp#L138), which prevents video source buffer from pushing data to PlayBin pipeline (https://github.com/WebKit/webkit/blob/master/Source/WebCore/Modules/mediasource/SourceBuffer.cpp#L2016).

In our platform we have fixed the bug by introducing a special flag to a stream structure (https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/mse/WebKitMediaSourceGStreamerPrivate.h#L49). Notification function (https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/mse/WebKitMediaSourceGStreamer.cpp#L678) has been changed so that it iterates over all streams and notifies those which have this flag set.

[Enrique Ocaña]

The implementation of WebKitMediaSourceGStreamer has changed radically since https://trac.webkit.org/changeset/277031/webkit and the issues and suggestions reported in this bug, although valid at the time of reporting, aren't applicable anymore now.

Fixed as invalid.