Bug 276421 – [MSE] MediaSource playback rapidly loops after changing audio track

• Component: Media
• Status: RESOLVED FIXED

[Theodore Abshire]

Summary:

I'm a member of the Shaka Player team, and have been investigating a bug where an MPEG-DASH asset being played gets stuck after changing audio language.
This only happens with Safari (tested on version 17.5, tested on macOS Sonoma 14.5). I was not able to reproduce it on Chromium-based browsers or Firefox.
At first I assumed this was a Shaka Player bug, but after testing with the dash.js reference player a similar problem happens there, so it looks like it might be a Safari issue instead.


Reproduction steps:

1. Navigate to the following URL:
https://shaka-player-demo.appspot.com/demo/#audiolang=en-US;textlang=en-US;uilang=en-US;asset=https://content.uplynk.com/2e881c1059bd47cfbd2844c40ec9931b.mpd?sstart=5;panel=CUSTOM%20CONTENT;build=uncompiled

2. Press the overflow menu button (three dots) on the player.

3. Press the audio language button (globe icon) in the overflow menu.

4. Choose a different language.


What happens:

The time on the seek bar rapidly jitters back and forth. Inspecting the currentTime value closely, it looks like the time is going forward as expected, but being reset backwards after a fraction of a second.
Even if you seek elsewhere in the presentation, this behavior continues.
The buffered value of the video shows that there is content buffered for a reasonable amount of time ahead, so it's not reaching the end of buffered content.
Some internal testing I have done on the Shaka Player side shows that our player is not changing the currentTime value of the video while it is in this state, it's seemingly being done by the browser.

[Radar WebKit Bug Importer]

<rdar://problem/131577794>

[Karl Dubost]

Theodore,

Thanks for the report. 
Do you remember if it happened before 17.5?

[Karl Dubost]

Created attachment 471941 [details]
screen recording

I can definitely reproduce in an internal recent build of Safari.

The video player is 

* not starting at all on Firefox Nightly (130.0a1 (2024-07-17) (64-bit)).
* working on Chrome and we change the language without issues while playing


On STP 199, I have even a kind of back and forth on one sec after switching to Japanese.

[Karl Dubost]

<div 
   class="shaka-range-container shaka-seek-bar-container" 
   style="background: linear-gradient(to right, rgb(255, 255, 255) 8.335368%, rgb(255, 255, 255) 8.335368%, rgb(255, 255, 255) 20.456359%, rgba(255, 255, 255, 0.54) 20.456359%, rgba(255, 255, 255, 0.54) 41.668701%, rgba(255, 255, 255, 0.3) 41.668701%);">
    <div class="shaka-ad-markers"></div><input class="shaka-range-element shaka-seek-bar shaka-no-propagation shaka-show-controls-on-mouse-over" type="range" step="any" min="0" max="49.152" aria-label="Seek"><div id="shaka-player-ui-thumbnail-container" style="visibility: hidden;"><img id="shaka-player-ui-thumbnail-image" draggable="false"><div id="shaka-player-ui-thumbnail-time"></div></div><div id="shaka-player-ui-time-container" style="width: auto; height: 20px; top: -30px; left: 104.703125px; visibility: visible;">0:10</div>
</div>

The value in the linear-gradient is constantly changing.

This is updated by 

```
  static setDisplay(element, display) {
    if (!element) {
      return;
    }

    if (display) {
      // Removing a non-existent class doesn't throw, so, even if
      // the element is not hidden, this should be fine.
      element.classList.remove('shaka-hidden');
    } else {
      element.classList.add('shaka-hidden');
    }
  }
```

called by 

```
  update() {
    const colors = this.config_.seekBarColors;
    const currentTime = this.getValue();
    const bufferedLength = this.video.buffered.length;
    const bufferedStart = bufferedLength ? this.video.buffered.start(0) : 0;
    const bufferedEnd =
        bufferedLength ? this.video.buffered.end(bufferedLength - 1) : 0;

    const seekRange = this.player.seekRange();
    const seekRangeSize = seekRange.end - seekRange.start;

    this.setRange(seekRange.start, seekRange.end);

    if (!this.shouldBeDisplayed_()) {
      shaka.ui.Utils.setDisplay(this.container, false);
    } else {
      shaka.ui.Utils.setDisplay(this.container, true);

      if (bufferedLength == 0) {
        this.container.style.background = colors.base;
      } else {
        const clampedBufferStart = Math.max(bufferedStart, seekRange.start);
        const clampedBufferEnd = Math.min(bufferedEnd, seekRange.end);
        const clampedCurrentTime = Math.min(
            Math.max(currentTime, seekRange.start),
            seekRange.end);

        const bufferStartDistance = clampedBufferStart - seekRange.start;
        const bufferEndDistance = clampedBufferEnd - seekRange.start;
        const playheadDistance = clampedCurrentTime - seekRange.start;

        // NOTE: the fallback to zero eliminates NaN.
        const bufferStartFraction = (bufferStartDistance / seekRangeSize) || 0;
        const bufferEndFraction = (bufferEndDistance / seekRangeSize) || 0;
        const playheadFraction = (playheadDistance / seekRangeSize) || 0;

        const unbufferedColor =
            this.config_.showUnbufferedStart ? colors.base : colors.played;

        const gradient = [
          'to right',
          this.makeColor_(unbufferedColor, bufferStartFraction),
          this.makeColor_(colors.played, bufferStartFraction),
          this.makeColor_(colors.played, playheadFraction),
          this.makeColor_(colors.buffered, playheadFraction),
          this.makeColor_(colors.buffered, bufferEndFraction),
          this.makeColor_(colors.base, bufferEndFraction),
        ];
        this.container.style.background =
            'linear-gradient(' + gradient.join(',') + ')';
      }
    }
  }
```

which was called by:

```
  updateTimeAndSeekRange_() {
    if (this.seekBar_) {
      this.seekBar_.setValue(this.video_.currentTime);
      this.seekBar_.update();

      if (this.seekBar_.isShowing()) {
        for (const menu of this.menus_) {
          menu.classList.remove('shaka-low-position');
        }
      } else {
        for (const menu of this.menus_) {
          menu.classList.add('shaka-low-position');
        }
      }
    }

    this.dispatchEvent(new shaka.util.FakeEvent('timeandseekrangeupdated'));
  }
```

called by 

```
    this.timeAndSeekRangeTimer_ = new shaka.util.Timer(() => {
      // Suppress timer-based updates if the controls are hidden.
      if (this.isOpaque()) {
        this.updateTimeAndSeekRange_();
      }
    });
```

called by

```
  tickEvery(seconds) {
    this.stop();

    if (goog.DEBUG) {
      // Capture the stack trace by making a fake error.
      const stackTrace = Error('Timer created').stack;
      shaka.util.Timer.activeTimers.set(this, stackTrace);
    }
    this.ticker_ = new shaka.util.DelayedTick(() => {
      // Schedule the timer again first. |onTick_| could cancel the timer and
      // rescheduling first simplifies the implementation.
      this.ticker_.tickAfter(seconds);
      this.onTick_();
    }).tickAfter(seconds);

    return this;
  }
```

called by 

```
    // For some reason, a timeout may still execute after we have cleared it in
    // our tests. We will wrap the callback so that we can double-check our
    // |alive| flag.
    const onTick = () => {
      if (alive) {
        this.onTick_();
      }
    };

    timeoutId = window.setTimeout(onTick, delayInSeconds * 1000);

    return this;
  }
```


and this is happening in a loop.

[Jean-Yves Avenard [:jya]]

Theodore, could you check on an iPad, the behaviour is different and rather interesting: when you change the language, you then see the poster instead, the sourcebuffers are cleared and no new data is being loaded (buffered range stays empty)

I'm curious on what Shaka is doing different here.

[Jean-Yves Avenard [:jya]]

Ok I have a simple reproduction case.
https://jyavenard.github.io/htmltests/tests/mse_mp4/276421/

it does exactly what the Shaka player does.
Load a bit of audio and video in two distinct source buffer with sourcebuffer.timestampOffset set to -20.478999999999999

then it calls `audiosb.remove(0, 49.152000000000001)`
and add the new audio segment.

It's the remove operation that cause the playback engine to become broken.

[Jean-Yves Avenard [:jya]]

To make the video play:
```
diff --git a/tests/mse_mp4/276421/index.html b/tests/mse_mp4/276421/index.html
index e96bdb7..e185cca 100644
--- a/tests/mse_mp4/276421/index.html
+++ b/tests/mse_mp4/276421/index.html
@@ -43,8 +43,8 @@ runWithMSE(async function(ms, el) {
   promises.push(once(el, 'loadedmetadata'));
   await Promise.all(promises);
   ok(true, "got all required event");
-  audiosb.remove(0, 49.152000000000001);
-  await once(audiosb, 'updateend');
+  // audiosb.remove(0, 49.152000000000001);
+  // await once(audiosb, 'updateend');
   fetchAndLoad(audiosb, 'buffer-audio-1530DC430-', range(4, 8), '.bin');
   ok(true, "loaded extra audio");
 });
```

[Jean-Yves Avenard [:jya]]

Pull request: https://github.com/WebKit/WebKit/pull/32938

[EWS]

Committed 283049@main (4aa58c3908a0): <https://commits.webkit.org/283049@main>

Reviewed commits have been landed. Closing PR #32938 and removing active labels.

[Theodore Abshire]

Thanks for looking into this! Sorry I didn't respond earlier, I've been busy with other duties.

When should we expect this fix to be in a release?