Bug 198053

Summary: [GStreamer][MSE] Desync between audio/video
Product: WebKit Reporter: Michael Catanzaro <mcatanzaro>
Component: MediaAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: bugs-noreply, calvaris, cturner, mcatanzaro, philn, pnormand
Priority: P2    
Version: WebKit Nightly Build   
Hardware: PC   
OS: Linux   
See Also: https://bugs.webkit.org/show_bug.cgi?id=200175
Attachments:
Description Flags
screencast none

Description Michael Catanzaro 2019-05-20 15:11:56 PDT
Visit any YouTube video featuring a talking head (to make the problem more obvious), e.g. https://www.youtube.com/watch?v=jTBNHkCjgnc

Click to either fullscreen the video, or enlarge into theater mode. Video playback will hang for a second or two while audio continues (unlike in other browsers, where playback is smooth).

Then when playback resumes, notice that the audio and video are slightly desynced. The audio is playing properly (it was never stalled). The video is noticeably a quarter of a second or so behind the audio. This is our fault; they should be perfectly synced.
Comment 1 Michael Catanzaro 2019-05-24 10:27:30 PDT
Actually I've seen this several times even when not changing video size.

Another example where it can be really obvious, at the 3m20s mark:

https://www.youtube.com/watch?v=le1goFKE2B8

Note that seeking directly there will cause the bug to not occur.
Comment 2 Michael Catanzaro 2019-05-24 10:49:34 PDT
We get up to something like 5s of delay between the audio and the video... I'm not sure if the delay increases as the video progresses, or if it just starts out really bad.
Comment 3 Xabier Rodríguez Calvar 2019-05-27 01:36:25 PDT
I can't reproduce the desync in trunk, neither in release or debug mode.
Comment 4 Philippe Normand 2019-05-27 10:14:49 PDT
That's probably because AC is deactivated and you use software decoders. How's your CPU usage?
Comment 5 Michael Catanzaro 2019-05-27 16:46:51 PDT
(In reply to Philippe Normand from comment #4)
> That's probably because AC is deactivated and you use software decoders.
> How's your CPU usage?

<1% CPU usage... even if there was a performance issue, there should not be any desync.
Comment 6 Philippe Normand 2019-06-11 03:02:18 PDT
In Ephy TP it starts to be noticeable around 2:45. I can't reproduce the issue in MiniBrowser from ToT...
Comment 7 Michael Catanzaro 2019-06-17 15:46:55 PDT
(In reply to Michael Catanzaro from comment #2)
> I'm not sure if the delay increases as the video progresses, or if it just
> starts out really bad.

The former. I'm pretty sure the video is correct. Perhaps the audio is playing just slightly too fast and so the difference between the video and audio increases incrementally as playback continues. I agree that around 2-3m it becomes noticeable. Later it becomes more and more obvious. Just reproduced on https://www.youtube.com/watch?v=zxT8CM8XntA.

(In reply to Philippe Normand from comment #6)
> In Ephy TP it starts to be noticeable around 2:45. I can't reproduce the
> issue in MiniBrowser from ToT...

Using our flatpak environment that's based on the GNOME runtime? We seem to have a frequently-recurring problem with multimedia bugs that occur for our users but aren't reproducible in our development environment. I think we're shooting ourselves in the foot by building our own GStreamer in org.webkit.CommonModules.yaml. I suspect removing the custom multimedia stuff and getting it from the upstream runtime would go a long way towards helping to reproduce such issues.
Comment 8 Charlie Turner 2019-07-11 07:27:36 PDT
I reproduced this today testing my deadlock work on Twitter. This tweet seems to reproduce it everytime when I switch back and forth to fullscreen: https://twitter.com/espn/status/1149133320107646977

When you fullscreen, the audio gets way ahead of the video. When you minimize, the video starts playing much faster and then catches up and syncs. Very strange!
Comment 9 Philippe Normand 2023-01-05 09:48:34 PST
(In reply to Michael Catanzaro from comment #0)
> Visit any YouTube video featuring a talking head (to make the problem more
> obvious), e.g. https://www.youtube.com/watch?v=jTBNHkCjgnc
> 
> Click to either fullscreen the video, or enlarge into theater mode. Video
> playback will hang for a second or two while audio continues (unlike in
> other browsers, where playback is smooth).
> 
> Then when playback resumes, notice that the audio and video are slightly
> desynced. The audio is playing properly (it was never stalled). The video is
> noticeably a quarter of a second or so behind the audio. This is our fault;
> they should be perfectly synced.

I can't reproduce this with Ephy TP. The openh264 decoder was choking a bit at the beginning and introducing stuttering but then it settled... Attaching a screencast (without audio though).
Comment 10 Philippe Normand 2023-01-05 09:49:11 PST
Created attachment 464352 [details]
screencast
Comment 11 Michael Catanzaro 2023-01-05 10:27:14 PST
This bug was fixed a long time ago. Let's close it.