Bug 216832 - REGRESSION (iOS/Safari 14): MediaRecorder produces invalid video files
Summary: REGRESSION (iOS/Safari 14): MediaRecorder produces invalid video files
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: Safari 14
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks: 85851
  Show dependency treegraph
 
Reported: 2020-09-22 09:54 PDT by AndyDeveloper
Modified: 2020-10-15 16:01 PDT (History)
12 users (show)

See Also:


Attachments
Patch (6.58 KB, patch)
2020-09-23 13:39 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
Patch for landing (6.32 KB, patch)
2020-09-23 23:14 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
MediaRecorder recorded video with STP Release 113 (Safari 14.0.1, WebKit 15610.2.3.1) (2.39 MB, video/mp4)
2020-09-30 06:56 PDT, d2vid
no flags Details
FFMPEG output video - dropping frames/stuttery but playable on Chrome (184.93 KB, video/mp4)
2020-09-30 06:57 PDT, d2vid
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description AndyDeveloper 2020-09-22 09:54:16 PDT
Since iOS 14 and Safari 14 on MacOS. Most media players can't play the recorded files. It seems the metadata (duration, frame rate...) are not set properly. If I re-encode the file assuming the frame rate, then I get a valid file.

Steps to reproduce:
1. Record the camera stream using the getUserMedia and MediaRecorder APIs
2. Play back the recorded chunks in a video element within Safari
OBSERVATION: The recorded video plays back properly in Safari
3. Save the recorded chunks in a file
4. Play back the saved file
OBSERVATIONS:
- Safari and Quicktime play the file properly. 
- Other video players cannot play the file (Chrome, VLC)
- The file's duration is incorrect (usually very large)
- ffmpeg cannot determine the file's duration or frame rate
EXPECTED: The file should play properly and have valid metadata. This was the case in iOS 13.
Comment 1 Radar WebKit Bug Importer 2020-09-22 10:13:28 PDT
<rdar://problem/69377550>
Comment 2 youenn fablet 2020-09-23 09:28:29 PDT
Did some tests and repro the issue.
On both Chrome and VLC, audio-only is rendered ok, video-only is rendered ok, except that video duration is not correctly computed.
In case of audio-video, there is probably an issue in trying to sync audio and video, which makes it fail to display the video.
Comment 3 youenn fablet 2020-09-23 09:29:41 PDT
Pre Safari 14, the moov box was probably giving the duration of the media, which is no longer the case now that we are supporting chunk-based outputs
Comment 4 youenn fablet 2020-09-23 13:39:59 PDT
Created attachment 409500 [details]
Patch
Comment 5 Eric Carlson 2020-09-23 16:51:54 PDT
Comment on attachment 409500 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=409500&action=review

> Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:249
> +    [m_writer.get() startSessionAtSourceTime: kCMTimeZero];

Nit: extra space
Comment 6 youenn fablet 2020-09-23 23:14:54 PDT
Created attachment 409534 [details]
Patch for landing
Comment 7 EWS 2020-09-24 00:10:30 PDT
Committed r267521: <https://trac.webkit.org/changeset/267521>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 409534 [details].
Comment 8 AndyDeveloper 2020-09-28 08:53:32 PDT
bug is still present in iOS 14.0.1 
(just noting this for future reference as I don't know which version the fix is targeted for)
Comment 9 d2vid 2020-09-30 06:56:40 PDT
Created attachment 410116 [details]
MediaRecorder recorded video with STP Release 113 (Safari 14.0.1, WebKit 15610.2.3.1)
Comment 10 d2vid 2020-09-30 06:57:02 PDT
Created attachment 410117 [details]
FFMPEG output video - dropping frames/stuttery but playable on Chrome
Comment 11 d2vid 2020-09-30 06:58:11 PDT
(In reply to AndyDeveloper from comment #8)
> bug is still present in iOS 14.0.1 
> (just noting this for future reference as I don't know which version the fix
> is targeted for)

Can you share the ffmpeg command you used to transcode a valid video file? And is it literally only a "valid" file but has transcoding issues, or are you getting a quality video out?

Here's the command I'm using to create a Chrome-playable video from a Safari MediaRecorder recorded video, but it's dropping frames/stuttery:
  $ ffmpeg -r 30 -i in.mp4 -y -threads 8 -analyzeduration 2147483647 -probesize 2147483647 -max_muxing_queue_size 9999 out.mp4

(I've attached for input/output video files for anyone's reference)
Comment 12 AndyDeveloper 2020-09-30 07:58:15 PDT
@d2vid are you using the latest version of ffmpeg?  I remember I did have to upgrade. I just tried the exact command on your invalid input file and got a good output video that doesn't skip. My version is ffmpeg version N-99213-ga265e6604e-tessus.
The command I normally use is a bit different and requires you to know the actual duration
ffmpeg -r 29.97 -t 6000ms -i in.mp4 -r 29.97 -profile:v main  -max_muxing_queue_size 4096 out.mp4
Comment 13 AndyDeveloper 2020-09-30 08:10:22 PDT
@d2vid are you using the latest version of ffmpeg?  I remember I did have to upgrade. I just tried YOUR exact command on your invalid input file and got a good output video that doesn't skip. My version is ffmpeg version N-99213-ga265e6604e-tessus.

ffmpeg -r 30 -i in.mp4 -y -threads 8 -analyzeduration 2147483647 -probesize 2147483647 -max_muxing_queue_size 9999 out.mp4

Note that the command I normally use is a bit different and requires you to know the actual duration, although it often works without it.
ffmpeg -r 29.97 -t 6000ms -i in.mp4 -r 29.97 -profile:v main -pix_fmt yuv420p -max_muxing_queue_size 4096 out.mp4
Comment 14 shaunbowe 2020-09-30 08:18:12 PDT
Maybe a stupid question but is there any way to determine the duration if the metadata in the file is invalid?
Comment 15 AndyDeveloper 2020-09-30 10:52:23 PDT
> is there any way to determine the duration if the metadata in the file is invalid?

Great question!  I just tried this and it seems to work:

ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 in.mp4 | { read num_frames; echo "$num_frames * 1000 / 29.97"; } | bc
Comment 16 d2vid 2020-10-15 16:01:23 PDT
(In reply to AndyDeveloper from comment #13)
> @d2vid are you using the latest version of ffmpeg?  I remember I did have to
> upgrade. I just tried YOUR exact command on your invalid input file and got
> a good output video that doesn't skip. My version is ffmpeg version
> N-99213-ga265e6604e-tessus.
> 
> ffmpeg -r 30 -i in.mp4 -y -threads 8 -analyzeduration 2147483647 -probesize
> 2147483647 -max_muxing_queue_size 9999 out.mp4
> 
> Note that the command I normally use is a bit different and requires you to
> know the actual duration, although it often works without it.
> ffmpeg -r 29.97 -t 6000ms -i in.mp4 -r 29.97 -profile:v main -pix_fmt
> yuv420p -max_muxing_queue_size 4096 out.mp4

@AndyDeveloper Thank you so much, that did the trick! I'm now on FFMPEG version N-99557-g6bdfea8 and the command `ffmpeg -r 30 -i in.mp4 -y -threads 8 -analyzeduration 2147483647 -probesize 2147483647 -max_muxing_queue_size 9999 out.mp4` is working.

For the few sample videos I have from users/myself I can confirm that the transcoding command works without the `-t` time argument, but I'll report back if I encounter any where that argument is necessary.