Bug 264894 - VideoEncoder produces no frames with latencyMode "realtime" when framerate/bitrate are not given
Summary: VideoEncoder produces no frames with latencyMode "realtime" when framerate/bi...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Media (show other bugs)
Version: Safari Technology Preview
Hardware: Mac (Apple Silicon) macOS 14
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2023-11-15 12:26 PST by Jacob Bandes-Storch
Modified: 2023-11-23 13:10 PST (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jacob Bandes-Storch 2023-11-15 12:26:50 PST
VideoEncoder with `latencyMode: "realtime"` but no `bitrate` or `framerate` specified does not output any frames, nor error messages.

Related: https://bugs.webkit.org/show_bug.cgi?id=264893 & https://bugs.webkit.org/show_bug.cgi?id=258669

Steps to reproduce:
1. Visit https://codesandbox.io/s/cocky-ioana-pjyz5v?file=/src/index.mjs
2. Open the console tab and observe the output is "{keyFrames: 0, deltaFrames: 0}" -- i.e. no frames were output whatsoever. No error messages are present.
3. Uncomment `framerate: fps` and `bitrate: 100000` and observe that the output changes to "{keyFrames: 1, deltaFrames: 19}".

Expected behavior:
If encoding without these parameters is not supported, then the browser should not claim to support the configuration (via isConfigSupported and by not calling the error callback).


Full code below:

```
import "./styles.css";

async function main() {
  const width = 200;
  const height = 100;
  const img = new ImageData(width, height);
  
  for (let r=0;r<height;r++) {
    for (let c=0;c<width;c++) {
      img.data[(r*width+c)*4+0] = Math.floor(r/height*255);
      img.data[(r*width+c)*4+1] = Math.floor(c/width*255);
      img.data[(r*width+c)*4+2] = 127;
      img.data[(r*width+c)*4+3] = 255;
    }
  }

  const bitmap = await createImageBitmap(img);

  let keyFrames=0;
  let deltaFrames=0;
  const encoder = new VideoEncoder({
    output: (chunk, metadata) => {
      if (chunk.type==="key") {
        ++keyFrames;
      } else if (chunk.type === "delta") {
        ++deltaFrames;
      }
      console.log('output chunk',chunk,metadata);
    },
    error: (err) => {
      console.error('encode error', err)
    }
  })

  encoder.addEventListener("dequeue", (event) => {
    console.log('dequeued')
  })

  const fps = 30;

  encoder.configure({
    codec: "avc1.42001f", // Baseline profile (42 00) with level 3.1 (1f)
    width,
    height,
    latencyMode: "realtime",
    // framerate: fps,
    avc: { format: "annexb" },
    // hardwareAcceleration:"prefer-software"
    // bitrate:100000,
    // bitrate: 10000000,
  });

  for (let i=0; i < 20; i++) {
    const frame = new VideoFrame(bitmap, {
      timestamp:i * 1/fps * 1e6,
      duration:1/fps * 1e6,
    });
    encoder.encode(frame, {keyFrame: i === 0});
    frame.close();
  }

  bitmap.close();
  console.log("sent frames");
  
  await encoder.flush();
  console.log("flushed", {keyFrames,deltaFrames})
  encoder.close();
  console.log("closed", {keyFrames,deltaFrames})
}

main().catch((e) => console.error(e,e.message));
```
Comment 1 Jacob Bandes-Storch 2023-11-15 12:29:09 PST
Tested in Safari Technology Preview 182 (Safari 17.4, WebKit 19618.1.4.1). Note that due to https://bugs.webkit.org/show_bug.cgi?id=258669, step 3 does not work in Safari 17.0.
Comment 2 Radar WebKit Bug Importer 2023-11-22 12:27:13 PST
<rdar://problem/118725549>
Comment 3 youenn fablet 2023-11-22 13:03:06 PST
We might need to set a default bitrate allocation in this case, or bail out.
I wonder what other browsers are doing here.
Comment 4 youenn fablet 2023-11-23 00:34:00 PST
Pull request: https://github.com/WebKit/WebKit/pull/20842
Comment 5 EWS 2023-11-23 12:01:18 PST
Committed 271087@main (ef16bd3f9b1c): <https://commits.webkit.org/271087@main>

Reviewed commits have been landed. Closing PR #20842 and removing active labels.
Comment 6 Jacob Bandes-Storch 2023-11-23 13:10:42 PST
Thanks for your quick response on this ticket.

Would there be any way of ensuring that other similar configuration problems in the future produce an isConfigSupported=false response?