Bug 264893 – VideoEncoder outputs mostly key frames

• Component: Media
• Status: RESOLVED DUPLICATE

[Jacob Bandes-Storch]

VideoEncoder with `avc1.42001f` seems to output mostly `key` frames, and rarely any `delta` frames, using the following configuration:

  {
    codec: "avc1.42001f", // Baseline profile (42 00) with level 3.1 (1f)
    width,
    height,
    avc: { format: "annexb" },
  }

Steps to reproduce:
1. Visit https://codesandbox.io/s/crimson-http-nvddmr?file=/src/index.mjs
2. Open the console tab and observe the output is "{keyFrames: 20, deltaFrames: 0}".
3. Try changing the config's `framerate` and `bitrate`, but no changes seem to produce delta frames.

This occurs on macOS 14.0 (23A344). Same behavior in Safari 17.0 (19616.1.27.211.1) and also in Safari Technology Preview 182 (Safari 17.4, WebKit 19618.1.4.1).


Full code below:

```
async function main() {
  const width = 200;
  const height = 100;
  const img = new ImageData(width, height);
  
  for (let r=0;r<height;r++) {
    for (let c=0;c<width;c++) {
      img.data[(r*width+c)*4+0] = Math.floor(r/height*255);
      img.data[(r*width+c)*4+1] = Math.floor(c/width*255);
      img.data[(r*width+c)*4+2] = 127;
      img.data[(r*width+c)*4+3] = 255;
    }
  }

  const bitmap = await createImageBitmap(img);

  let keyFrames=0;
  let deltaFrames=0;
  const encoder = new VideoEncoder({
    output: (chunk, metadata) => {
      if (chunk.type==="key") {
        ++keyFrames;
      } else if (chunk.type === "delta") {
        ++deltaFrames;
      }
      console.log('output chunk',chunk,metadata);
    },
    error: (err) => {
      console.error('encode error', err)
    }
  })

  encoder.addEventListener("dequeue", (event) => {
    console.log('dequeued')
  })

  const fps = 30;

  encoder.configure({
    codec: "avc1.42001f", // Baseline profile (42 00) with level 3.1 (1f)
    width,
    height,
    // latencyMode: "realtime",
    // framerate: fps,
    avc: { format: "annexb" },
    // hardwareAcceleration:"prefer-software"
    // bitrate:1000000,
    // bitrate: 10000000,
  });

  for (let i=0; i < 20; i++) {
    const frame = new VideoFrame(bitmap, {
      timestamp:i * 1/fps * 1e6,
      duration:1/fps * 1e6,
    });
    encoder.encode(frame, {keyFrame: i === 0});
    frame.close();
  }

  bitmap.close();
  console.log("sent frames");
  
  await encoder.flush();
  console.log("flushed", {keyFrames,deltaFrames})
  encoder.close();
  console.log("closed", {keyFrames,deltaFrames})
}

main().catch((e) => console.error(e,e.message));
```

[Radar WebKit Bug Importer]

<rdar://problem/118725469>

[youenn fablet]

I canot reproduce on my local AS device with STP 182. I got 19 delta frames and 1 key frame. This might be below WebKit.

I have created a video transpiler that is complete in a web browser and am facing the same problem.

https://github.com/tamaina/venc2/issues/4

https://venc2.vercel.app

macOS 14.2（23C64)
Safari TP Release 184 (Safari 17.4, WebKit 19618.1.7.1.1)

I forked the reproduction and modified it slightly.

https://codesandbox.io/p/sandbox/crimson-http-nvddmr?file=%2Fsrc%2Findex.mjs%3A58%2C5

My 14" MacBook Pro with M1Pro and iPhone 13 both have keyFrames of 20.

[Jacob Bandes-Storch]

I believe the URL you shared is the same as the original repro I posted. You may need to explicitly "save" the codesandbox for it to give you a new URL. Can you share what modifications you needed to make?

Wow, sorry. The modifications are to display the logs on the browser screen (since opening the console on a mobile device is a pain) and to delay the encode dispatch.

https://codesandbox.io/p/sandbox/trusting-elgamal-zyks56

I know it's probably taboo to post this, but is there any update?

[youenn fablet]

Would it be possible to send to me ([e-mail]) a sysdiagnose with the timestamp where the issue happens?

[Jacob Bandes-Storch]

It's a bit large to send by email, but I just created FB13674236 and attached the sysdiagnose there. After starting the sysdiagnose I refreshed the demo page several times in Safari while watching for the `keyFrames: 20` message -- this was between 7:03 and 7:04 PM.

[Michael Hagar]

I'm also seeing this on Safari 17.4 on Mac OS 14.4.

Opening the original demo page, Safari shows 20 keyframes. Chrome Shows 1 keyframe, 19 delta frames.

[youenn fablet]

Can anyone that was able to reproduce the issue validate this is now working properly on STP 191?

*** This bug has been marked as a duplicate of bug 270159 ***

[Michael Hagar]

I confirmed this is fixed in Safari Technology Preview 191, with the demo now showing 1 keyframe and 19 delta frames (just like Chrome and Firefox).

I too saw the generation of delta frames and a significant reduction in file size after encoding.