Unable to reliably grab video frame from Blob immediately after loading

• Component: JavaScriptCore
• Status: RESOLVED FIXED

[Teodor]

Problem: Generating video preview/thumbnail locally (on iOS device).

Approach: File is loaded (via URL.createObjectURL) into <video> element, the time is set to middle of the file, video frame is drawn onto canvas, canvas contents are exported as JPEG.

All browsers, except Safari, either make video frame available before firing events, or wait until a frame is available when a call is made to capture it.

Reproduction: https://codepen.io/thexeos/pen/GROEYrN
Open on iPhone (not tested on macOS) and select video file. You are expecting to see 3 video stills. If only one is shown - it is broken.
Now, comment out the resolve(video) in video.onseeked handler and use setTimeout version - it works "realiably" (larger file, older device, multiple tabs open in Safari all affect the minimum delay needed).

seeked event is used in codepen, but waiting until after dataloaded, canplay, canplaythrough, or any combination of those followed by or preceding seeked all lead to flaky results. Most often it does not work. On rare occasions I was able to grab a frame successfully within 16ms after last event in a chain, but usually the delay is much larger. The video used when testing was less than 1 minute long and about 15 megabytes in size.

Maybe related: https://chromium.googlesource.com/chromium/src/+/f2a8353e592bc8ef5827a378adc919eb1075f5a3/third_party/WebKit/Source/core/html/HTMLVideoElement.cpp#367

[Radar WebKit Bug Importer]

<rdar://problem/88943512>

[Jer Noble]

The most reliable way of achieving your goal is to use the new requestVideoFrameCallback() API. Installing this callback before loading the video will result in your callback being called once a video frame is available.  This guarantees you can paint as early as possible, without having to wait for a particular event to fire.

[Jer Noble]

Please take a look at <https://jernoble.github.io/samples/bin/video-to-canvas/blob.html>. The rVFC() technique can be used at startup & seeking, doesn't require specific events, will be triggered immediately when the frame becomes available (which may occur before the triggering event fires), and provides metadata about the available frame.

[Teodor]

Thank you for the link. I was not aware of this new API. I have no way of testing it just yet on 15.4, but I will report back when I have. If it works, it will be the right solution for this issue. I'm closing the issue as resolved for now.

[Teodor]

The proposed solution is confirmed to be working.