[iOS] MediaRecorder incorrect screen orientation handling

• Component: WebRTC
• Status: RESOLVED FIXED

[Igor Ievsiukov]

When using MediaRecorder to recording a file in portrait mode, for ex. recording a video message using the frontal camera, Safari automatically switches width/height (640x480 => 480x640) but does not applies necessary transformation which results the video track to be rotated 90deg thus stretching the video making it hardly usable, please see [1].

Similar kind of distortions, upside-down video, etc. could be observed while holding the phone in any other orientation except landscape with camera on the right side.

After looking at the source code I discover recording is performed using AVAssetWriterInput[2]. According to the dev docs [3] the issue could be resolved using the "transform" property to rotate the image according to the current device orientation [5]. You can find an example of transformation for each screen orientation via the following link [6].

I would be glad to provide a patch, unfortunately iOS Simulator does not supports webcams nor it is possible to deploy a debug version of WebKit to an iOS device.

Please let me know if I can help with any further debugging.

Thank you very much in advance,
Igor

[1] https://www.dropbox.com/s/3kdx77bmw2mghvi/Vid%C3%A9o%2017.06.19%2011%2052%2040.mp4?dl=0
[2] https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h#L70
[3] https://developer.apple.com/library/archive/qa/qa1744/_index.html#avassetwriter
[4] https://developer.apple.com/documentation/avfoundation/avassetwriterinput/1390183-transform
[5] https://github.com/remirobert/CameraEngine/blob/master/CameraEngine/CameraEngineVideoEncoder.swift#L130
[6] https://github.com/remirobert/CameraEngine/blob/master/CameraEngine/CameraEngineVideoEncoder.swift#L83-L86

[Radar WebKit Bug Importer]

<rdar://problem/51802521>

[borisstodic]

I am facing same issue. Created new ticket with steps to reproduce bug:

https://bugs.webkit.org/show_bug.cgi?id=201137

[youenn fablet]

*** Bug 201137 has been marked as a duplicate of this bug. ***

[youenn fablet]

Created attachment 409265 [details]
Patch

[youenn fablet]

Created attachment 409269 [details]
Patch

[youenn fablet]

Created attachment 409270 [details]
Patch

[youenn fablet]

Created attachment 409272 [details]
Patch

[youenn fablet]

Created attachment 409276 [details]
Patch

[Eric Carlson]

Comment on attachment 409276 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=409276&action=review

> Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:430
> +        if (sample.videoMirrored())
> +            videoTransform = CGAffineTransformScale(videoTransform, -1, 1);

It would be nice to have internals API to test mirrored mode, like we have setCameraMediaStreamTrackOrientation to test rotation.

[youenn fablet]

(In reply to Eric Carlson from comment #9)
> Comment on attachment 409276 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=409276&action=review
> 
> > Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:430
> > +        if (sample.videoMirrored())
> > +            videoTransform = CGAffineTransformScale(videoTransform, -1, 1);
> 
> It would be nice to have internals API to test mirrored mode, like we have
> setCameraMediaStreamTrackOrientation to test rotation.

I'll file a bug for that.

As you noted, I'll remove setting the transform in case of no rotation or no mirroring.

[youenn fablet]

no rotation and no mirroring.

[youenn fablet]

Created attachment 409286 [details]
Patch for landing

[EWS]

Committed r267366: <https://trac.webkit.org/changeset/267366>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 409286 [details].

[youenn fablet]

Reopening since this is not properly working

[youenn fablet]

Created attachment 410099 [details]
Patch

[Eric Carlson]

Comment on attachment 410099 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=410099&action=review

> Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:433
> +            m_videoTransform = CGAffineTransformScale(*m_videoTransform, -1, 1);

Do we always want to flip the transform?

[youenn fablet]

(In reply to Eric Carlson from comment #16)
> Comment on attachment 410099 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=410099&action=review
> 
> > Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:433
> > +            m_videoTransform = CGAffineTransformScale(*m_videoTransform, -1, 1);
> 
> Do we always want to flip the transform?

Indeed, will fix that.

[youenn fablet]

Created attachment 410201 [details]
Patch for landing

[EWS]

Committed r267828: <https://trac.webkit.org/changeset/267828>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 410201 [details].