Bug 264894 – VideoEncoder produces no frames with latencyMode "realtime" when framerate/bitrate are not given

• Component: Media
• Status: RESOLVED FIXED

[Jacob Bandes-Storch]

VideoEncoder with `latencyMode: "realtime"` but no `bitrate` or `framerate` specified does not output any frames, nor error messages.

Related: https://bugs.webkit.org/show_bug.cgi?id=264893 & https://bugs.webkit.org/show_bug.cgi?id=258669

Steps to reproduce:
1. Visit https://codesandbox.io/s/cocky-ioana-pjyz5v?file=/src/index.mjs
2. Open the console tab and observe the output is "{keyFrames: 0, deltaFrames: 0}" -- i.e. no frames were output whatsoever. No error messages are present.
3. Uncomment `framerate: fps` and `bitrate: 100000` and observe that the output changes to "{keyFrames: 1, deltaFrames: 19}".

Expected behavior:
If encoding without these parameters is not supported, then the browser should not claim to support the configuration (via isConfigSupported and by not calling the error callback).


Full code below:

```
import "./styles.css";

async function main() {
  const width = 200;
  const height = 100;
  const img = new ImageData(width, height);
  
  for (let r=0;r<height;r++) {
    for (let c=0;c<width;c++) {
      img.data[(r*width+c)*4+0] = Math.floor(r/height*255);
      img.data[(r*width+c)*4+1] = Math.floor(c/width*255);
      img.data[(r*width+c)*4+2] = 127;
      img.data[(r*width+c)*4+3] = 255;
    }
  }

  const bitmap = await createImageBitmap(img);

  let keyFrames=0;
  let deltaFrames=0;
  const encoder = new VideoEncoder({
    output: (chunk, metadata) => {
      if (chunk.type==="key") {
        ++keyFrames;
      } else if (chunk.type === "delta") {
        ++deltaFrames;
      }
      console.log('output chunk',chunk,metadata);
    },
    error: (err) => {
      console.error('encode error', err)
    }
  })

  encoder.addEventListener("dequeue", (event) => {
    console.log('dequeued')
  })

  const fps = 30;

  encoder.configure({
    codec: "avc1.42001f", // Baseline profile (42 00) with level 3.1 (1f)
    width,
    height,
    latencyMode: "realtime",
    // framerate: fps,
    avc: { format: "annexb" },
    // hardwareAcceleration:"prefer-software"
    // bitrate:100000,
    // bitrate: 10000000,
  });

  for (let i=0; i < 20; i++) {
    const frame = new VideoFrame(bitmap, {
      timestamp:i * 1/fps * 1e6,
      duration:1/fps * 1e6,
    });
    encoder.encode(frame, {keyFrame: i === 0});
    frame.close();
  }

  bitmap.close();
  console.log("sent frames");
  
  await encoder.flush();
  console.log("flushed", {keyFrames,deltaFrames})
  encoder.close();
  console.log("closed", {keyFrames,deltaFrames})
}

main().catch((e) => console.error(e,e.message));
```

[Jacob Bandes-Storch]

Tested in Safari Technology Preview 182 (Safari 17.4, WebKit 19618.1.4.1). Note that due to https://bugs.webkit.org/show_bug.cgi?id=258669, step 3 does not work in Safari 17.0.

[Radar WebKit Bug Importer]

<rdar://problem/118725549>

[youenn fablet]

We might need to set a default bitrate allocation in this case, or bail out.
I wonder what other browsers are doing here.

[youenn fablet]

Pull request: https://github.com/WebKit/WebKit/pull/20842

[EWS]

Committed 271087@main (ef16bd3f9b1c): <https://commits.webkit.org/271087@main>

Reviewed commits have been landed. Closing PR #20842 and removing active labels.

[Jacob Bandes-Storch]

Thanks for your quick response on this ticket.

Would there be any way of ensuring that other similar configuration problems in the future produce an isConfigSupported=false response?