| Differences between
and this patch
- a/Source/WebCore/ChangeLog +35 lines
Lines 1-3 a/Source/WebCore/ChangeLog_sec1
1
2021-09-27  Alicia Boya García  <aboya@igalia.com>
2
3
        [MSE][GStreamer] Honor MP4 edit lists
4
        https://bugs.webkit.org/show_bug.cgi?id=231019
5
6
        Reviewed by NOBODY (OOPS!).
7
8
        This patch takes into consideration the GstSegment attached to a
9
        sample to offset the PTS and DTS. This ensures accurate timestamps are
10
        obtained for MP4 files containing edit lists (commonly necessary for
11
        files containing video with B frames to have PTS starting at zero).
12
13
        Before this was implemented, a workaround was in place based on a
14
        heuristic (DTS = 0 && PTS > 0 && PTS < 0.1). The workaround is
15
        preserved for the sake of content without proper edit lists, but
16
        any edit list takes preference.
17
18
        The time fudge factor has been modified from 0.083 seconds up to
19
        0.100 seconds to accomodate the size of the empty edit in test.mp4
20
        used by Web Platform Tests.
21
22
        This test fixes improves expectation results and fixes two subtests in
23
        imported/w3c/web-platform-tests/media-source/mediasource-remove.html.
24
25
        * Modules/mediasource/MediaSource.cpp:
26
        (WebCore::MediaSource::currentTimeFudgeFactor):
27
        * platform/graphics/SourceBufferPrivate.h:
28
        (WebCore::SourceBufferPrivate::timeFudgeFactor const):
29
        * platform/graphics/gstreamer/MediaSampleGStreamer.cpp:
30
        (WebCore::MediaSampleGStreamer::extendToTheBeginning): Deleted.
31
        * platform/graphics/gstreamer/MediaSampleGStreamer.h:
32
        * platform/graphics/gstreamer/mse/AppendPipeline.cpp:
33
        (WebCore::bufferTimeToStreamTimeClamped):
34
        (WebCore::AppendPipeline::appsinkNewSample):
35
1
2021-09-27  Alicia Boya García  <aboya@igalia.com>
36
2021-09-27  Alicia Boya García  <aboya@igalia.com>
2
37
3
        [MSE][GStreamer] Don't create MediaSourceTrackGStreamer objects twice for the same track
38
        [MSE][GStreamer] Don't create MediaSourceTrackGStreamer objects twice for the same track
- a/Source/WebCore/Modules/mediasource/MediaSource.cpp -2 / +2 lines
Lines 322-329 ExceptionOr<void> MediaSource::clearLiveSeekableRange() a/Source/WebCore/Modules/mediasource/MediaSource.cpp_sec1
322
322
323
const MediaTime& MediaSource::currentTimeFudgeFactor()
323
const MediaTime& MediaSource::currentTimeFudgeFactor()
324
{
324
{
325
    // Allow hasCurrentTime() to be off by as much as the length of two 24fps video frames
325
    // Allow hasCurrentTime() to be off by as much as 100ms.
326
    static NeverDestroyed<MediaTime> fudgeFactor(2002, 24000);
326
    static NeverDestroyed<MediaTime> fudgeFactor(1, 10);
327
    return fudgeFactor;
327
    return fudgeFactor;
328
}
328
}
329
329
- a/Source/WebCore/platform/graphics/SourceBufferPrivate.h -1 / +1 lines
Lines 150-156 public: a/Source/WebCore/platform/graphics/SourceBufferPrivate.h_sec1
150
protected:
150
protected:
151
    // The following method should never be called directly and be overridden instead.
151
    // The following method should never be called directly and be overridden instead.
152
    WEBCORE_EXPORT virtual void append(Vector<unsigned char>&&);
152
    WEBCORE_EXPORT virtual void append(Vector<unsigned char>&&);
153
    virtual MediaTime timeFudgeFactor() const { return {2002, 24000}; }
153
    virtual MediaTime timeFudgeFactor() const { return {1, 10}; }
154
    virtual bool isActive() const { return false; }
154
    virtual bool isActive() const { return false; }
155
    virtual bool isSeeking() const { return false; }
155
    virtual bool isSeeking() const { return false; }
156
    virtual MediaTime currentMediaTime() const { return { }; }
156
    virtual MediaTime currentMediaTime() const { return { }; }
- a/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp -9 lines
Lines 183-197 RefPtr<JSC::Uint8ClampedArray> MediaSampleGStreamer::getRGBAImageData() const a/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp_sec1
183
    return JSC::Uint8ClampedArray::tryCreate(WTFMove(bufferStorage), 0, byteLength);
183
    return JSC::Uint8ClampedArray::tryCreate(WTFMove(bufferStorage), 0, byteLength);
184
}
184
}
185
185
186
void MediaSampleGStreamer::extendToTheBeginning()
187
{
188
    // Only to be used with the first sample, as a hack for lack of support for edit lists.
189
    // See AppendPipeline::appsinkNewSample()
190
    ASSERT(m_dts == MediaTime::zeroTime());
191
    m_duration += m_pts;
192
    m_pts = MediaTime::zeroTime();
193
}
194
195
void MediaSampleGStreamer::setTimestamps(const MediaTime& presentationTime, const MediaTime& decodeTime)
186
void MediaSampleGStreamer::setTimestamps(const MediaTime& presentationTime, const MediaTime& decodeTime)
196
{
187
{
197
    m_pts = presentationTime;
188
    m_pts = presentationTime;
- a/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h -1 lines
Lines 42-48 public: a/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h_sec1
42
    static Ref<MediaSampleGStreamer> createFakeSample(GstCaps*, MediaTime pts, MediaTime dts, MediaTime duration, const FloatSize& presentationSize, const AtomString& trackId);
42
    static Ref<MediaSampleGStreamer> createFakeSample(GstCaps*, MediaTime pts, MediaTime dts, MediaTime duration, const FloatSize& presentationSize, const AtomString& trackId);
43
    static Ref<MediaSampleGStreamer> createImageSample(PixelBuffer&&, const IntSize& destinationSize = { }, double frameRate = 1);
43
    static Ref<MediaSampleGStreamer> createImageSample(PixelBuffer&&, const IntSize& destinationSize = { }, double frameRate = 1);
44
44
45
    void extendToTheBeginning();
46
    MediaTime presentationTime() const override { return m_pts; }
45
    MediaTime presentationTime() const override { return m_pts; }
47
    MediaTime decodeTime() const override { return m_dts; }
46
    MediaTime decodeTime() const override { return m_dts; }
48
    MediaTime duration() const override { return m_duration; }
47
    MediaTime duration() const override { return m_duration; }
- a/Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp -27 / +60 lines
Lines 375-393 void AppendPipeline::handleEndOfAppend() a/Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp_sec1
375
    sourceBufferPrivate().didReceiveAllPendingSamples();
375
    sourceBufferPrivate().didReceiveAllPendingSamples();
376
}
376
}
377
377
378
static GstClockTime bufferTimeToStreamTimeClamped(const GstSegment* segment, GstClockTime bufferTime)
379
{
380
    guint64 streamTime;
381
    int result = gst_segment_to_stream_time_full(segment, GST_FORMAT_TIME, bufferTime, &streamTime);
382
    if (!result) {
383
        GST_ERROR("Couldn't map buffer time %" GST_TIME_FORMAT " to segment %" GST_PTR_FORMAT, GST_TIME_ARGS(bufferTime), segment);
384
        return bufferTime;
385
    }
386
    if (result < 0)
387
        return 0; // Clamp negative timestamps down to zero.
388
    return streamTime;
389
}
390
378
void AppendPipeline::appsinkNewSample(const Track& track, GRefPtr<GstSample>&& sample)
391
void AppendPipeline::appsinkNewSample(const Track& track, GRefPtr<GstSample>&& sample)
379
{
392
{
380
    ASSERT(isMainThread());
393
    ASSERT(isMainThread());
381
394
382
    if (UNLIKELY(!gst_sample_get_buffer(sample.get()))) {
395
    {
383
        GST_WARNING("Received sample without buffer from appsink.");
396
        GstBuffer* buffer = gst_sample_get_buffer(sample.get());
384
        return;
397
        if (UNLIKELY(!buffer)) {
385
    }
398
            GST_WARNING("Received sample without buffer from appsink.");
399
            return;
400
        }
386
401
387
    if (!GST_BUFFER_PTS_IS_VALID(gst_sample_get_buffer(sample.get()))) {
402
        if (!GST_BUFFER_PTS_IS_VALID(buffer)) {
388
        // When demuxing Vorbis, matroskademux creates several PTS-less frames with header information. We don't need those.
403
            // When demuxing Vorbis, matroskademux creates several PTS-less frames with header information. We don't need those.
389
        GST_DEBUG("Ignoring sample without PTS: %" GST_PTR_FORMAT, gst_sample_get_buffer(sample.get()));
404
            GST_DEBUG("Ignoring sample without PTS: %" GST_PTR_FORMAT, gst_sample_get_buffer(sample.get()));
390
        return;
405
            return;
406
        }
407
408
        GstSegment* segment = gst_sample_get_segment(sample.get());
409
        bool hasMappedTime = false;
410
        GstClockTime pts = GST_BUFFER_PTS(buffer);
411
        GstClockTime dts = GST_BUFFER_DTS(buffer);
412
        GstClockTime duration = GST_BUFFER_DURATION(buffer);
413
        if (segment && (segment->time || segment->start)) {
414
            // MP4 has the concept of edit lists, where some buffer time needs to be offsetted, often very slightly,
415
            // to get exact timestamps.
416
            pts = bufferTimeToStreamTimeClamped(segment, GST_BUFFER_PTS(buffer));
417
            dts = bufferTimeToStreamTimeClamped(segment, GST_BUFFER_DTS(buffer));
418
            GST_TRACE_OBJECT(track.appsinkPad.get(), "Mapped buffer to segment, PTS %" GST_TIME_FORMAT " -> %" GST_TIME_FORMAT " DTS %" GST_TIME_FORMAT " -> %" GST_TIME_FORMAT,
419
                GST_TIME_ARGS(GST_BUFFER_PTS(buffer)), GST_TIME_ARGS(pts), GST_TIME_ARGS(GST_BUFFER_DTS(buffer)), GST_TIME_ARGS(dts));
420
            hasMappedTime = true;
421
        } else if (!dts && pts > 0 && pts <= 100'000'000) {
422
            // Because a track presentation time starting at some close to zero, but not exactly zero time can cause unexpected
423
            // results for applications, we extend the duration of this first sample to the left so that it starts at zero.
424
            // This is relevant for files that should have an edit list but don't, or when using GStreamer < 1.16, where
425
            // edit lists are not parsed in push-mode.
426
427
            GST_DEBUG("Extending first sample of track '%s' to make it start at PTS=0 %" GST_PTR_FORMAT, track.trackId.string().utf8().data(), buffer);
428
            duration += pts;
429
            pts = 0;
430
            hasMappedTime = true;
431
        }
432
433
        if (hasMappedTime) {
434
            sample = adoptGRef(gst_sample_make_writable(sample.leakRef()));
435
            GRefPtr<GstBuffer> newBuffer = gst_sample_get_buffer(sample.get());
436
            gst_sample_set_buffer(sample.get(), nullptr);
437
            newBuffer = adoptGRef(gst_buffer_make_writable(newBuffer.leakRef()));
438
            GST_BUFFER_PTS(newBuffer.get()) = pts;
439
            GST_BUFFER_DTS(newBuffer.get()) = dts;
440
            GST_BUFFER_DURATION(newBuffer.get()) = duration;
441
            gst_sample_set_buffer(sample.get(), newBuffer.leakRef());
442
        }
391
    }
443
    }
392
444
393
    auto mediaSample = MediaSampleGStreamer::create(WTFMove(sample), track.presentationSize, track.trackId);
445
    auto mediaSample = MediaSampleGStreamer::create(WTFMove(sample), track.presentationSize, track.trackId);
Lines 399-423 void AppendPipeline::appsinkNewSample(const Track& track, GRefPtr<GstSample>&& s a/Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp_sec2
399
        mediaSample->duration().toString().utf8().data(),
451
        mediaSample->duration().toString().utf8().data(),
400
        mediaSample->presentationSize().width(), mediaSample->presentationSize().height());
452
        mediaSample->presentationSize().width(), mediaSample->presentationSize().height());
401
453
402
    // Hack, rework when GStreamer >= 1.16 becomes a requirement:
403
    // We're not applying edit lists. GStreamer < 1.16 doesn't emit the correct segments to do so.
404
    // GStreamer fix in https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/commit/c2a0da8096009f0f99943f78dc18066965be60f9
405
    // Also, in order to apply them we would need to convert the timestamps to stream time, which we're not currently
406
    // doing for consistency between GStreamer versions.
407
    //
408
    // In consequence, the timestamps we're handling here are unedited track time. In track time, the first sample is
409
    // guaranteed to have DTS == 0, but in the case of streams with B-frames, often PTS > 0. Edit lists fix this by
410
    // offsetting all timestamps by that amount in movie time, but we can't do that if we don't have access to them.
411
    // (We could assume the track PTS of the sample with track DTS = 0 is the offset, but we don't have any guarantee
412
    // we will get appended that sample first, or ever).
413
    //
414
    // Because a track presentation time starting at some close to zero, but not exactly zero time can cause unexpected
415
    // results for applications, we extend the duration of this first sample to the left so that it starts at zero.
416
    if (mediaSample->decodeTime() == MediaTime::zeroTime() && mediaSample->presentationTime() > MediaTime::zeroTime() && mediaSample->presentationTime() <= MediaTime(1, 10)) {
417
        GST_DEBUG("Extending first sample to make it start at PTS=0");
418
        mediaSample->extendToTheBeginning();
419
    }
420
421
    m_sourceBufferPrivate.didReceiveSample(mediaSample.get());
454
    m_sourceBufferPrivate.didReceiveSample(mediaSample.get());
422
}
455
}
423
456
- a/LayoutTests/ChangeLog +12 lines
Lines 1-3 a/LayoutTests/ChangeLog_sec1
1
2021-09-27  Alicia Boya García  <aboya@igalia.com>
2
3
        [MSE][GStreamer] Honor MP4 edit lists
4
        https://bugs.webkit.org/show_bug.cgi?id=231019
5
6
        Reviewed by NOBODY (OOPS!).
7
8
        Update expectations for mediasource-remove.html in the GStreamer
9
        ports, as a couple subtests get fixed.
10
11
        * platform/glib/imported/w3c/web-platform-tests/media-source/mediasource-remove-expected.txt:
12
1
2021-09-22  Jean-Yves Avenard  <jya@apple.com>
13
2021-09-22  Jean-Yves Avenard  <jya@apple.com>
2
14
3
        Make SharedBuffer inherit from ThreadSafeRefCounted
15
        Make SharedBuffer inherit from ThreadSafeRefCounted
- a/LayoutTests/platform/glib/imported/w3c/web-platform-tests/media-source/mediasource-remove-expected.txt -4 / +4 lines
Lines 11-18 PASS Test remove while update pending. a/LayoutTests/platform/glib/imported/w3c/web-platform-tests/media-source/mediasource-remove-expected.txt_sec1
11
PASS Test aborting a remove operation.
11
PASS Test aborting a remove operation.
12
PASS Test remove with a start at the duration.
12
PASS Test remove with a start at the duration.
13
PASS Test remove transitioning readyState from 'ended' to 'open'.
13
PASS Test remove transitioning readyState from 'ended' to 'open'.
14
FAIL Test removing all appended data. assert_equals: Initial buffered range. expected "{ [0.095, 6.548) }" but got "{ [0.000, 6.548) }"
14
PASS Test removing all appended data.
15
FAIL Test removing beginning of appended data. assert_equals: Initial buffered range. expected "{ [0.095, 6.548) }" but got "{ [0.000, 6.548) }"
15
PASS Test removing beginning of appended data.
16
FAIL Test removing the middle of appended data. assert_equals: Initial buffered range. expected "{ [0.095, 6.548) }" but got "{ [0.000, 6.548) }"
16
FAIL Test removing the middle of appended data. assert_equals: Buffered ranges after remove(). expected "{ [0.095, 0.997) [3.298, 6.548) }" but got "{ [0.095, 0.975) [3.298, 6.548) }"
17
FAIL Test removing the end of appended data. assert_equals: Initial buffered range. expected "{ [0.095, 6.548) }" but got "{ [0.000, 6.548) }"
17
FAIL Test removing the end of appended data. assert_equals: Buffered ranges after remove(). expected "{ [0.095, 1.022) }" but got "{ [0.095, 0.995) }"
18
18

Return to Bug 231019