Summary: | [GStreamer][Playbin3] Stream collection handling fixes | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | Philippe Normand <pnormand> | ||||||
Component: | Platform | Assignee: | Philippe Normand <pnormand> | ||||||
Status: | RESOLVED FIXED | ||||||||
Severity: | Normal | CC: | aboya, calvaris, cgarcia, eric.carlson, ews-watchlist, glenn, gustavo, hta, jer.noble, menard, philipj, sergio, tommyw, vjaquez, webkit-bug-importer | ||||||
Priority: | P2 | Keywords: | InRadar | ||||||
Version: | WebKit Nightly Build | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Attachments: |
|
Description
Philippe Normand
2021-02-23 10:20:37 PST
Created attachment 421326 [details]
Patch
Comment on attachment 421326 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=421326&action=review > Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:1567 > + GST_DEBUG_OBJECT(pipeline(), "phil Collection %p len: %u", collection.get(), numStreams); I'll reword this one. Comment on attachment 421326 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=421326&action=review > Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:1465 > +void MediaPlayerPrivateGStreamer::updateTracks(const GRefPtr<GstStreamCollection> streamCollection) Did you mean `const GRefPtr<GstStreamCollection>?` > Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:1477 > + // consistent with the collection contents, we can't reuse our old tracks. Can you elaborate? Comment on attachment 421326 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=421326&action=review > Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:448 > + void updateTracks(const GRefPtr<GstStreamCollection>); This should be a const GRefPtr& > Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:513 > + const char* sourceType = ""; We could better initialize to "unknown". DEBUG below could be more informative in case we don't hit the assert in a release build. > Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:556 > -static void webkitMediaStreamSrcPostStreamCollection(WebKitMediaStreamSrc* self) > +void webkitMediaStreamSrcPostStreamCollection(WebKitMediaStreamSrc* self) Do you need to drop the static? I does not look to me at a first glance. > Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:648 > + if (!g_strcmp0(streamId, track.id().utf8().data())) { There are String operators for this: inline bool operator==(const String& a, const char* b) { return equal(a.impl(), reinterpret_cast<const LChar*>(b)); } inline bool operator==(const char* a, const String& b) { return equal(reinterpret_cast<const LChar*>(a), b.impl()); } Comment on attachment 421326 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=421326&action=review >> Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:1477 >> + // consistent with the collection contents, we can't reuse our old tracks. > > Can you elaborate? Sure! Taking the example of http/tests/media/hls/hls-audio-tracks.html, the player would receive various stream-collection messages and actually handle them from different threads, as this log shows: 0:00:00.943368778 340 0x210cde0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1597:handleSyncMessage:<media-player-0> Received STREAM_COLLECTION message with upstream id "(null)" defining the following streams: 0:00:00.943407182 340 0x7fe1e8008cc0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1597:handleSyncMessage:<media-player-0> Received STREAM_COLLECTION message with upstream id "decodebin3" defining the following streams: 0:00:00.943432352 340 0x210cde0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1601:handleSyncMessage:<media-player-0> #0 audio f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_2 0:00:00.943467669 340 0x7fe1e8008cc0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1601:handleSyncMessage:<media-player-0> #0 audio f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_1 0:00:00.943412535 340 0x1c85610 LOG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1800:handleMessage:<media-player-0> Message state-changed received from element audio-concat 0:00:00.943504040 340 0x7fe1e8008cc0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1601:handleSyncMessage:<media-player-0> #1 audio f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_2 ... 0:00:00.943771346 340 0x1c85610 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1500:updateTracks:<media-player-0> Processing a stream collection with 1 streams 0:00:00.943858510 340 0x1c85610 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1521:updateTracks:<media-player-0> Inspecting audio track with ID f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_2 0:00:00.943910641 340 0x1c85610 DEBUG webkitmediaplayer TrackPrivateBaseGStreamer.cpp:113:tagsChanged: Inspecting track at index 0 with tags: (NULL) ... 0:00:00.955157540 340 0x21094c0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1597:handleSyncMessage:<media-player-0> Received STREAM_COLLECTION message with upstream id "decodebin3" defining the following streams: 0:00:00.955224337 340 0x1c85610 LOG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1800:handleMessage:<media-player-0> Message state-changed received from element aconv 0:00:00.955236157 340 0x21094c0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1601:handleSyncMessage:<media-player-0> #0 video f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_0:1/00000101 0:00:00.955290618 340 0x1c85610 LOG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1800:handleMessage:<media-player-0> Message state-changed received from element aqueue 0:00:00.955316558 340 0x21094c0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1601:handleSyncMessage:<media-player-0> #1 audio f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_0:1/00000102 0:00:00.955372572 340 0x1c85610 LOG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1800:handleMessage:<media-player-0> Message state-changed received from element scaletempo0 0:00:00.955377016 340 0x21094c0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1601:handleSyncMessage:<media-player-0> #2 audio f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_1 0:00:00.955426783 340 0x1c85610 LOG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1800:handleMessage:<media-player-0> Message state-changed received from element filter-convert 0:00:00.955457383 340 0x21094c0 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1601:handleSyncMessage:<media-player-0> #3 audio f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_2 0:00:00.955776957 340 0x1c85610 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1521:updateTracks:<media-player-0> Inspecting video track with ID f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_0:1/00000101 0:00:00.955839477 340 0x1c85610 DEBUG webkitmediaplayer TrackPrivateBaseGStreamer.cpp:113:tagsChanged: Inspecting track at index 0 with tags: taglist, video-codec=(string)H.264; 0:00:00.956053700 340 0x1c85610 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1521:updateTracks:<media-player-0> Inspecting audio track with ID f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_0:1/00000102 0:00:00.956105627 340 0x1c85610 DEBUG webkitmediaplayer TrackPrivateBaseGStreamer.cpp:113:tagsChanged: Inspecting track at index 0 with tags: taglist, audio-codec=(string)"MPEG-4\ AAC"; 0:00:00.956134517 340 0x1c85610 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1463:playbin3SendSelectStreamsIfAppropriate:<media-player-0> Checking if to send SELECT_STREAMS, m_waitingForStreamsSelectedEvent = false, haveDifferentStreamIds = true, m_currentState = READY... shouldSendSelectStreams = false 0:00:00.956173911 340 0x1c85610 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1521:updateTracks:<media-player-0> Inspecting audio track with ID f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_1 0:00:00.956333746 340 0x1c85610 DEBUG webkitmediaplayer TrackPrivateBaseGStreamer.cpp:113:tagsChanged: Inspecting track at index 1 with tags: taglist, private-data=(sample)00000000000daaca:None:c2VnbWVudCwgZmxhZ3M9KEdzdFNlZ21lbnRGbGFncylHU1RfU0VHTUVOVF9GTEFHX05PTkUsIHJhdGU9KGRvdWJsZSkxLCBhcHBsaWVkLXJhdGU9KGRvdWJsZSkxLCBmb3JtYXQ9KEdzdEZvcm1hdCl0aW1lLCBiYXNlPShndWludDY0KTAsIG9mZnNldD0oZ3VpbnQ2NCkwLCBzdGFydD0oZ3VpbnQ2NCkwLCBzdG9wPShndWludDY0KTE4NDQ2NzQ0MDczNzA5NTUxNjE1LCB0aW1lPShndWludDY0KTAsIHBvc2l0aW9uPShndWludDY0KTAsIGR1cmF0aW9uPShndWludDY0KTE4NDQ2NzQ0MDczNzA5NTUxNjE1OwA_:SUQzUHJpdmF0ZUZyYW1lLCBvd25lcj0oc3RyaW5nKWNvbS5hcHBsZS5zdHJlYW1pbmcudHJhbnNwb3J0U3RyZWFtVGltZXN0YW1wOwA_, nominal-bitrate=(uint)125659, audio-codec=(string)"MPEG-4\ AAC", minimum-bitrate=(uint)85788, maximum-bitrate=(uint)103703, bitrate=(uint)92303; 0:00:00.956402430 340 0x1c85610 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1521:updateTracks:<media-player-0> Inspecting audio track with ID f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_2 0:00:00.956425207 340 0x1c85610 DEBUG webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1525:updateTracks:<media-player-0> audio track with ID f0963b9f548a9f7bb591f70ba2b10f0c444b9d5f65181c654403b2a206edaf76/src_2 already exists, skipping The track orphaning code would then kick-in and try to remove the tracks no longer present in the collection, but leaving indexes unchanged, so for instance if we received an initial collection with 2 audio streams (A1, A2) and then later on A2 is removed, A3 added we would end up with a final list of (A1, A3) where A1 has an index of 0 as expected, but A3 would have an index of 2 instead of 1. The current MediaPlayer interface for track management allows only to append a new track, you can't insert at a given index, which actually is an issue for another test I started debugging. Comment on attachment 421326 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=421326&action=review >> Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:1465 >> +void MediaPlayerPrivateGStreamer::updateTracks(const GRefPtr<GstStreamCollection> streamCollection) > > Did you mean `const GRefPtr<GstStreamCollection>?` const GRefPtr<T>& indeed, that slipped through the crack :) >> Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:513 >> + const char* sourceType = ""; > > We could better initialize to "unknown". DEBUG below could be more informative in case we don't hit the assert in a release build. Yes good point! >> Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:556 >> +void webkitMediaStreamSrcPostStreamCollection(WebKitMediaStreamSrc* self) > > Do you need to drop the static? I does not look to me at a first glance. No need, good spot. >> Source/WebCore/platform/mediastream/gstreamer/GStreamerMediaStreamSource.cpp:648 >> + if (!g_strcmp0(streamId, track.id().utf8().data())) { > > There are String operators for this: > > inline bool operator==(const String& a, const char* b) { return equal(a.impl(), reinterpret_cast<const LChar*>(b)); } > inline bool operator==(const char* a, const String& b) { return equal(reinterpret_cast<const LChar*>(a), b.impl()); } Indeed! Good cleanup. Created attachment 421522 [details]
Patch
> for instance if we received an initial collection with 2 audio streams (A1, A2) and then later on A2 is removed, A3 added we would end up with a final list of (A1, A3) where A1 has an index of 0 as expected, but A3 would have an index of 2 instead of 1.
Is this something happening in actual streams or a consequence of bugs in GStreamer's handling of stream collections?
afaik any demuxer can send a variable amount of stream-collection messages and decodebin3 is expected to merge them together and the "app" is expected to clean-up internal mappings of the collection whenever a new one is received. Committed r273644: <https://commits.webkit.org/r273644> All reviewed patches have been landed. Closing bug and clearing flags on attachment 421522 [details]. |