Video buffering and degraded playback performance

• Component: Media
• Status: NEW ---

[Dustin Kerstein]

Below are two examples (with 4 variations total - directly loading via video.src and Blob as src) that show degraded playback in Safari (tested on 2019 Macbook Pro and iPad Pro 1st Gen). Both examples, across all 4 variations, playback with no issues in Firefox and Chrome. The issue appears to be related to the network requests being made in an attempt to buffer. Note that with the Fetch() variations it may take a little while to download the example videos.

Example #1
- 4k60fps with 100% I-Frames (and only 243 frames total).
- Try setting src directly and not using fetch() and note they behave the same way (which is different from Example #2)
- This broken playback can even be replicated by directly going to https://files.panomoments.com/uhd60fps.mp4 in the browser

var src = "https://files.panomoments.com/uhd60fps.mp4"
var video = document.createElement("VIDEO");
document.body.appendChild(video);
video.preload = 'none';
video.autoplay = true;
video.muted = true;
video.loop = true;
//video.src = src;  // Only use this when Fetch() is disabled.
fetch(src).then(function(response) {
  response.blob().then(function(myBlob) {
    video.src = window.URL.createObjectURL(myBlob);
  });
});


Example #2 
- 4k60fps with normal I-Frame / GOP encoding. Source - http://bbb3d.renderfarming.net/download.html
- Note that in this example, playback is better than the example above with 100% I-Frames. Only when using Blob as src does it slow down. When using Blob as src, each buffering requests' Resource Size is near the full size of the file. When not using Blob as src, each request is much smaller.

var src = "https://files.panomoments.com/bbb_sunflower_2160p_60fps_normal.mp4"
var video = document.createElement("VIDEO");
document.body.appendChild(video);
video.preload = 'none';
video.autoplay = true;
video.muted = true;
video.loop = true;
// video.src = src; // Only use this when Fetch() is disabled.
fetch(src).then(function(response) {
  response.blob().then(function(myBlob) {
    video.src = window.URL.createObjectURL(myBlob);
  });
});


Feel free to play around with a live version here - https://jsfiddle.net/dustinkerstein/5cmdy3p9/

To me it seems like there could be two possibly related issues - both potentially having to do with the buffering strategy. The first having more to do with videos that have many I-Frames (Example #1) and second, loading videos using Blob as src (more clearly evident in Example #2). Please let me know if I can get any further info / debug.

[Radar WebKit Bug Importer]

<rdar://problem/62735455>

[Jer Noble]

Looks like this is entirely due to copying bytes in and out of XPC messages (rather than just using a shared memory SharedBuffer to transfer data).

[Dustin Kerstein]

K, do you think this would be a simple fix? Are there any possible workarounds in the meantime? Lastly, do you think a fix could eventually make its way to older iOS devices that are stuck on iOS 12? Let me know if there's anything else I can help with. Thanks!

[Dustin Kerstein]

One potentially interesting note is that Safari has no issues playing uhd60fps.mp4 using the local file system - file:///Users/XYZ/Downloads/uhd60fps.mp4 (and I see no 206 Partial Content requests in the network tab)

[Dustin Kerstein]

Created attachment 399600 [details]
Blob.png

[Dustin Kerstein]

Created attachment 399601 [details]
Src.png

[Dustin Kerstein]

I just did a little more testing comparing setting the uhd60fps.mp4 video (164MB with 100% I-Frames) directly as URL src vs. using a blob, and it appears there may be more than one issue at play here. 

With the following code:

      var src = "https://files.panomoments.com/uhd60fps.mp4"
      var video = document.createElement("video");
      document.body.appendChild(video);
      video.controls = true;
      video.autoplay = true;
      video.src = src;

I see a total of 273MB of network requests after the video finished (sluggishly) playing. See the Src.png attachment.

And with this code:

      var src = "https://files.panomoments.com/uhd60fps.mp4"
      var video = document.createElement("video");
      document.body.appendChild(video);
      video.controls = true;
      video.autoplay = true;
      fetch(src).then(function(response) {
        response.blob().then(function(myBlob) {
          video.src = window.URL.createObjectURL(myBlob);
        });
      });

I see a total of 36.1GB of network requests after the video has finished (very sluggishly) playing. See the Blob.png attachment.

In summary, I think I'm seeing these four behaviors in Safari when testing using the uhd60fps.mp4 video linked above: 

1. Play from file:///Users/XYZ/Downloads/uhd60fps.mp4 (ie. dragging the video into the URL bar) - Playback is perfect at 60fps
2. Play via setting src directly to a URL - Playback is degraded (possibly the XPC memory issue)
3. Play with src as blob - Playback is severely degraded (possibly XPC memory issue and additionally the drastically increased 206 network requests)
4. When setting playbackRate < 1 and setting src directly to a URL - Buffering and Playback stall (this feels a bit unrelated compared to 1-3)

Let me know if there's anything else I can get. Are you able to replicate the behaviors above?

[youenn fablet]

(In reply to Jer Noble from comment #2)
> Looks like this is entirely due to copying bytes in and out of XPC messages
> (rather than just using a shared memory SharedBuffer to transfer data).

I am not sure, we should be using SharedBuffer.

From Dustin experiments, it seems we are getting lots of requests from the same data, that got cancelled. In the case of blobs, we are able to provide the exact requested data, which triggers excessive processing. The video backend should probably schedule loads in a simpler manner.
Or we could try to optimise this further and use a file-based approach for blobs.

[Dustin Kerstein]

Is there any other debug I can help get? This issue is severely impacting our services, particularly on older iOS devices. Do you happen to have any ideas on possible temporary workarounds? Thanks!

[Dustin Kerstein]

FYI, this issue seems to have gotten quite a bit worse on recent versions of Safari.

[Anton Zhuravsky]

Oh this seems to be related to https://bugs.webkit.org/show_bug.cgi?id=232076 - data URIs and Blobs can no longer act as a source for <video /> element on iOS 15 due to the underlying issue.

[Dustin Kerstein]

Just try going to this URL using Safari 15 (desktop, iPadOS, iOS, etc.) - 
https://s3.amazonaws.com/data.panomoments.com/processed/5849a51bc26d08000b62f9f8/58a5dbc8cd5f48000bdbbe1f/uhd_dashinit.mp4 - This blob behavior is significantly worse in recent versions, though there have been performance issues for a while before this recent regression.

Note this link correctly returns 206 partial range responses. See additional notes at the bottom of - https://bugs.webkit.org/show_bug.cgi?id=229413

[Dustin Kerstein]

Here's a non-dashed version of the above url - https://s3.amazonaws.com/data.panomoments.com/processed/5849a51bc26d08000b62f9f8/58a5dbc8cd5f48000bdbbe1f/uhd.mp4 - The dashed version has a duration = 0 to simulate a live stream, and that may be affecting WebKit's buffer logic. And here's a version of the link in the first post that bypasses the CDN - https://s3.amazonaws.com/files.panomoments.com/uhd60fps.mp4