Bug 260455 – [GStreamer][MSE] Crash in webKitMediaSrcStreamFlush

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Kdwk]

Created attachment 467354 [details]
gdb (bt full; c) output.txt

On Nvidia RTX 4070 (driver version 535)

1. Set WEBKIT_DMABUF_RENDERER_DISABLE_GBM=1
2. Set WEBKIT_GST_DMABUF_SINK_DISABLED=1
3. Visit apple.com/apple-watch-series-8
4. Crash

[Kdwk]

Created attachment 467384 [details]
gdb (bt full; c) 2.txt

Here's a similar one encountered on https://www.apple.com/apple-watch-ultra/

[Michael Catanzaro]

*** Bug 260984 has been marked as a duplicate of this bug. ***

[Michael Catanzaro]

*** Bug 261872 has been marked as a duplicate of this bug. ***

[Michael Catanzaro]

This isn't an NVIDIA-related issue, because in my bug #260984 I hit the same crash with AMD graphics.

[Kdwk]

This seems to be a regression. This website has always worked before this bug was reported and Apple didn't change the website design

[Michael Catanzaro]

Are you able to reproduce the crash reliably? I'm not able to trigger it by visiting those apple.com websites.

[Kdwk]

Yes I am able to trigger it reliably

[Michael Catanzaro]

If you could figure out which WebKitGTK release it broke in, that would help.

If you could bisect it, that would help even more (but if you're not familiar with building WebKit, this may not be easy).

[Kdwk]

I do know how to build WebKit, but as I've said on the Matrix channel, all downloads from the Igalia repo fail so it's not convenient (I have to use system libraries in a toolbox), and the resultant MiniBrowser can't get GPU acceleration (I am unable to work around this issue). Beyond that, I also don't know how to use git bisect so instructions are welcome

[Michael Catanzaro]

(In reply to Kdwk from comment #9)
> I do know how to build WebKit, but as I've said on the Matrix channel, all
> downloads from the Igalia repo fail so it's not convenient (I have to use
> system libraries in a toolbox), 

System libraries in a toolbox is the way to go.

That said, is there a bug report for the problem with the Igalia repo? We need to either fix it or change build-webkit to stop depending on it.

> and the resultant MiniBrowser can't get GPU
> acceleration (I am unable to work around this issue). Beyond that, I also
> don't know how to use git bisect so instructions are welcome

Is that GPU acceleration problem caused by toolbx, perhaps? I wonder if GPU acceleration is required for you to reproduce this bug reliably?

git bisect is really easy to use, if you have a regression you can confidently reproduce to determine whether a particular commit is bad or good. Example tutorial: https://stackoverflow.com/a/37306623/1120203

[Michael Catanzaro]

*** Bug 263509 has been marked as a duplicate of this bug. ***

[Michael Catanzaro]

*** Bug 266535 has been marked as a duplicate of this bug. ***

[Philippe Normand]

Here with Canary and WEBKIT_GST_DMABUF_SINK_DISABLED=1 and va decoders ranked to 0 I get scrambled output.

[Philippe Normand]

Created attachment 469480 [details]
screenshot

[Philippe Normand]

With default dmabuf sink enabled and va decoders up-ranked, all is fine here...

Anyway, is this crash in webKitMediaSrcStreamFlush still happening?

[Kdwk]

It is still reliably happening in WebKitGTK 2.42.4. The original pages no longer exist so here's another one: apple.com/apple-watch-ultra-2

[Philippe Normand]

I got a critical warning here in Ephy TP

                                                                                                                                                                                                                    
      Commit: 0e3a544be3ab1e038c36260379e91163f7f02a7d3eb8d62d3a4ae9ff3a91d626                                                                                                                                      
      Parent: b0d6f546c77add052c453376afa9afacb0f0ba77ce055b57bf7644015564c650                                                                                                                                      
     Subject: Export org.gnome.Epiphany.Devel                                                                                                                                                                       
        Date: 2024-01-21 05:07:56 +0000

(WebKitWebProcess:2): GStreamer-CRITICAL **: 17:44:13.183: gst_caps_remove_structure: assertion 'IS_WRITABLE (caps)' failed

(gdb) bt                                                                                                                                                                                                            
#0  g_logv (log_domain=0x7f490fb229ae "GStreamer", log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, args=args@entry=0x7f483bdd8fb0) at ../glib/gmessages.c:1277                                              
#1  0x00007f490eec2233 in g_log (log_domain=<optimized out>, log_level=<optimized out>, format=<optimized out>) at ../glib/gmessages.c:1315                                                                         
#2  0x00007f4843ea5f14 in gst_vp9_parse_negotiate (in_align=<optimized out>, in_caps=0x7f483000aae0 [GstCaps], self=0x559120b5b320 [GstVp9Parse|V0_parser]) at ../gst/videoparsers/gstvp9parse.c:299                
#3  gst_vp9_parse_set_sink_caps (parse=0x559120b5b320 [GstVp9Parse|V0_parser], caps=<optimized out>) at ../gst/videoparsers/gstvp9parse.c:816                                                                       
#4  0x00007f490fbaa7f8 in gst_base_parse_sink_event_default (parse=0x559120b5b320 [GstVp9Parse|V0_parser], event=0x7f4830007580 [GstEvent]) at ../libs/gst/base/gstbaseparse.c:1244                                 
#5  0x00007f490fac466d in gst_pad_send_event_unchecked (pad=pad@entry=0x559120b4ded0 [GstPad|sink], event=event@entry=0x7f4830007580 [GstEvent], type=<optimized out>,                                              
    type@entry=GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM) at ../gst/gstpad.c:5939                                                                                                                                         
#6  0x00007f490fac4d53 in gst_pad_push_event_unchecked (pad=pad@entry=0x7f483000dde0 [GstPad|video_0], event=0x7f4830007580 [GstEvent], type=<optimized out>, type@entry=GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM)       
    at ../gst/gstpad.c:5572                                                                                                                                                                                         
#7  0x00007f490fac5518 in push_sticky (pad=pad@entry=0x7f483000dde0 [GstPad|video_0], ev=ev@entry=0x7f483bdd9410, user_data=user_data@entry=0x7f483bdd9480) at ../gst/gstpad.c:4057                                 
#8  0x00007f490fab9c85 in events_foreach (pad=0x7f483000dde0 [GstPad|video_0], func=0x7f490fac5470 <push_sticky>, user_data=0x7f483bdd9480) at ../gst/gstpad.c:613                                                  
#9  0x00007f490fac85f1 in check_sticky (event=0x7f483000baa0 [GstEvent], pad=0x7f483000dde0 [GstPad|video_0]) at ../gst/gstpad.c:4116                                                                               
#10 gst_pad_push_event (pad=0x7f483000dde0 [GstPad|video_0], event=0x7f483000baa0 [GstEvent]) at ../gst/gstpad.c:5705                                                                                               
#11 0x00007f488c0d4a0e in gst_matroska_demux_send_tags (demux=demux@entry=0x559120b54320 [GstMatroskaDemux|matroskademux0]) at ../gst/matroska/matroska-demux.c:1993                                                
#12 0x00007f488c0dfef1 in gst_matroska_demux_parse_id (demux=0x559120b54320 [GstMatroskaDemux|matroskademux0], id=<optimized out>, length=<optimized out>, needed=6) at ../gst/matroska/matroska-demux.c:5655       
#13 0x00007f488c0e7df4 in gst_matroska_demux_chain (pad=pad@entry=0x559120b54920 [GstPad|sink], parent=parent@entry=0x559120b54320 [GstMatroskaDemux|matroskademux0], buffer=<optimized out>,                       
    buffer@entry=0x5591209dbf30 [GstBuffer]) at ../gst/matroska/matroska-demux.c:6202                                                                                                                               
#14 0x00007f490fac2eec in gst_pad_chain_data_unchecked (pad=pad@entry=0x559120b54920 [GstPad|sink], type=type@entry=4112, data=data@entry=0x5591209dbf30) at ../gst/gstpad.c:4463                                   
#15 0x00007f490fac628e in gst_pad_push_data (pad=pad@entry=0x559120b59800 [GstPad|src], type=type@entry=4112, data=data@entry=0x5591209dbf30) at ../gst/gstpad.c:4739                                               
#16 0x00007f490fac68c4 in gst_pad_push (pad=0x559120b59800 [GstPad|src], buffer=0x5591209dbf30 [GstBuffer]) at ../gst/gstpad.c:4858                                                                                 
#17 0x00007f490fbbda5c in gst_base_transform_chain (pad=pad@entry=0x559120b56520 [GstPad|sink], parent=parent@entry=0x559120b56110 [GstIdentity|identity0], buffer=buffer@entry=0x5591209dbf30 [GstBuffer])         
    at ../libs/gst/base/gstbasetransform.c:2391                                                                                                                                                                     
#18 0x00007f490fac2eec in gst_pad_chain_data_unchecked (pad=pad@entry=0x559120b56520 [GstPad|sink], type=type@entry=4112, data=data@entry=0x5591209dbf30) at ../gst/gstpad.c:4463                                   
#19 0x00007f490fac628e in gst_pad_push_data (pad=pad@entry=0x559120b56aa0 [GstPad|src], type=type@entry=4112, data=data@entry=0x5591209dbf30) at ../gst/gstpad.c:4739                                               
#20 0x00007f490fac68c4 in gst_pad_push (pad=pad@entry=0x559120b56aa0 [GstPad|src], buffer=0x5591209dbf30 [GstBuffer]) at ../gst/gstpad.c:4858                                                                       
#21 0x00007f490fbc1efb in gst_base_src_loop (pad=0x559120b56aa0 [GstPad|src]) at ../libs/gst/base/gstbasesrc.c:3035                                                                                                 
#22 0x00007f490faf3204 in gst_task_func (task=0x559120b5ae00 [GstTask|appsrc0:src]) at ../gst/gsttask.c:384                                                                                                         
#23 0x00007f490eee92c2 in g_thread_pool_thread_proxy (data=<optimized out>) at ../glib/gthreadpool.c:336                                                                                                            
#24 0x00007f490eee86c9 in g_thread_proxy (data=0x7f48fc0019d0) at ../glib/gthread.c:821                                                                                                                             
#25 0x00007f49132a1e39 in start_thread (arg=<optimized out>) at pthread_create.c:444                                                                                                                                
#26 0x00007f4913329904 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

[Philippe Normand]

Apart from that, unable to reproduce the issue (but I'm on Intel and AMD).

[Kdwk]

I am able to reproduce this on non-Nvidia hardware. On my MacBook, visiting apple.com/apple-watch-ultra-2 with software decoding and scrolling all the way down crashes the WebProcess.

[Philippe Normand]

(In reply to Kdwk from comment #19)
> I am able to reproduce this on non-Nvidia hardware. On my MacBook, visiting
> apple.com/apple-watch-ultra-2 with software decoding and scrolling all the
> way down crashes the WebProcess.

Can you share the backtrace?

The one I shared earlier results from a warning, so it shouldn't trigger crashes, unless you set this env var G_DEBUG=fatal-criticals

[Philippe Normand]

(In reply to Philippe Normand from comment #17)
> I got a critical warning here in Ephy TP
> 
> (WebKitWebProcess:2): GStreamer-CRITICAL **: 17:44:13.183:
> gst_caps_remove_structure: assertion 'IS_WRITABLE (caps)' failed
> 

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5975

[Kdwk]

Created attachment 469531 [details]
gdb (MacBook; software decoding).txt

(In reply to Philippe Normand from comment #20)
> (In reply to Kdwk from comment #19)
> > I am able to reproduce this on non-Nvidia hardware. On my MacBook, visiting
> > apple.com/apple-watch-ultra-2 with software decoding and scrolling all the
> > way down crashes the WebProcess.
> 
> Can you share the backtrace?
> 
> The one I shared earlier results from a warning, so it shouldn't trigger
> crashes, unless you set this env var G_DEBUG=fatal-criticals

[Philippe Normand]

Can you collect gst logs? Do you remember how?

[Michael Catanzaro]

I haven't been able to reproduce the crash on apple.com, but I hit this crash 100% of the time in Ephy Tech Preview (WebKitGTK 2.43.3, GStreamer 1.22.5) when loading https://www.newsweek.com/missouri-republican-senators-duel-nick-schroer-1863838 and scrolling down the page. Backtrace is basically the same as what I posted in bug #260984.

I will attach a gst.log following the instructions https://trac.webkit.org/wiki/WebKitGTK/Debugging#Debuggingmultimediastuff. We should really move all the instructions you care about to https://docs.webkit.org/ so we can have some updated link to point to instructions.

[Michael Catanzaro]

Created attachment 469550 [details]
Debug log

BTW, since streamByName may return nullptr, I suggest webKitMediaSrcFlush should either handle that case or assert that it returns non-null.

[Philippe Normand]

That log looks incomplete.

I can't reproduce this issue here in Ephy TP.

[Philippe Normand]

(In reply to Michael Catanzaro from comment #24)
> I will attach a gst.log following the instructions
> https://trac.webkit.org/wiki/WebKitGTK/Debugging#Debuggingmultimediastuff.
> We should really move all the instructions you care about to
> https://docs.webkit.org/ so we can have some updated link to point to
> instructions.

https://github.com/WebKit/Documentation/pull/78

[Michael Catanzaro]

(In reply to Philippe Normand from comment #26)
> That log looks incomplete.

I reproduced the issue today and got a second log. It looks the same as the first. That's really all there is before the crash occurs. I'll attach my second debug log and the dots requested by your new documentation.

[Michael Catanzaro]

Created attachment 469566 [details]
Debug

[Michael Catanzaro]

BTW to reproduce in Tech Preview, just keep scrolling up and down the page. It seems to crash about 70% of the time, not the 100% that I claimed earlier. If it refuses to crash, then I press Ctrl+R and try again and it will probably crash.

[Philippe Normand]

I can reproduce the crash now, for the record, you need to start playing the video (auto play doesn't kick in here) and scroll down until the player moves to PiP state, then scroll up until it goes back to non-PiP state, and scroll down again and so on...

So it seems the MSE src element tears down its streams and later on a seek triggers a flush on the same src element...

One of the issues is that m_hasAllTracks in MediaSourcePrivateGStreamer doesn't seem to be set back to false after the streams have been removed...

[Carlos Bentzen]

Created attachment 470445 [details]
region in the page where the crash happens

I can reproduce the crash when scrolling down on https://apple.com/apple-watch-ultra-2 until it hits this area where the watch side view is a video element that is played and seeked as you scroll through it (you may need to scroll past it and return).

Bisected it down to https://commits.webkit.org/265206@main.

Before the user agent quirk, the video element played this MP4 file, without MSE: https://www.apple.com/105/media/us/apple-watch-ultra-2/2023/4d9e62e1-fe94-4bb9-abbe-0b8c9626a304/anim/schematic_rotation-2/large.mp4

After the user agent quirk, it now plays this WebM file instead, that has alpha channel signaled in the WebM container, and it's played via MSE: https://www.apple.com/105/media/us/apple-watch-ultra-2/2023/4d9e62e1-fe94-4bb9-abbe-0b8c9626a304/anim/schematic_rotation-2/large.webm

(In reply to Michael Catanzaro from comment #25)
> Created attachment 469550 [details]
> Debug log
> 
> BTW, since streamByName may return nullptr, I suggest webKitMediaSrcFlush
> should either handle that case or assert that it returns non-null.

Indeed. Though checking for a null Stream pointer and bailing early doesn't seem sufficient, as it does fixes the crash in webKitMediaSrcFlush, but then then video element is broken, not displaying anything.

I'm investing this further.

[Carlos Bentzen]

(In reply to Carlos Bentzen from comment #32)
> 
> I'm investing this further.

investigating*, obviously.

[Michael Catanzaro]

(In reply to Carlos Bentzen from comment #32)
> Bisected it down to https://commits.webkit.org/265206@main.

Good job!

[Michael Catanzaro]

Found another reproducer for this crash. Try to play this video on nbcnews.com:

https://www.nbcnews.com/news/us-news/toddler-dies-pinned-tire-uber-suv-dropped-houston-rcna144187

[Michael Catanzaro]

(Um, although maybe I should have picked a different video for bug report purposes. Presumably that one contains disturbing content if it doesn't trigger the crash.)

[Philippe Normand]

(In reply to Michael Catanzaro from comment #34)
> (In reply to Carlos Bentzen from comment #32)
> > Bisected it down to https://commits.webkit.org/265206@main.
> 
> Good job!

Well, I'm not sure this can be flagged as regression, unless we update the bug title again to be specific to Apple website.

[Carlos Bentzen]

(In reply to Philippe Normand from comment #37)
> (In reply to Michael Catanzaro from comment #34)
> > (In reply to Carlos Bentzen from comment #32)
> > > Bisected it down to https://commits.webkit.org/265206@main.
> > 
> > Good job!
> 
> Well, I'm not sure this can be flagged as regression, unless we update the
> bug title again to be specific to Apple website.

Yeah, we get different content served with the user-agent quirk, but the new content seems valid on Firefox and Chrome, so the GStreamer MSE code is the one broken IMO, and was broken before already. Had the same page been served with WebM + MSE before r265206, it would also crash (I checked with the test below).

Reduced the test case down to https://people.igalia.com/cadubentzen/webkit/bug260455.
Scrolling past the video area and back, I get a crash reliably. (the video area is blank in webkitgtk, it doesn't play).

In https://people.igalia.com/cadubentzen/webkit/bug260455_2, on the other hand, the video plays and I get no crash anymore. The only difference is the web page starts with the video in the viewport.

Philippe pointed out to me that we have a setting via the environment variable WEBKIT_GST_ALLOW_PLAYBACK_OF_INVISIBLE_VIDEOS. Setting that to 1, the video plays and I get no crashes, so it's definitely related. Continuing to investigate...

[Carlos Bentzen]

Pull request: https://github.com/WebKit/WebKit/pull/26472

[EWS]

Committed 276798@main (f91aeb92bd8e): <https://commits.webkit.org/276798@main>

Reviewed commits have been landed. Closing PR #26472 and removing active labels.

[Michael Catanzaro]

Reopened Bugzilla.
Causes excessive CPU usage of cached web process and web process failure to render web content after cache restore, tracking revert in https://bugs.webkit.org/show_bug.cgi?id=274329.

[Michael Catanzaro]

I wound up using bug #274261 to track the revert. This will need a second try, sorry. :(

[Michael Catanzaro]

Fortunately it looks like this was not backported to 2.44 since nobody requested it.

[Philippe Normand]

(In reply to Michael Catanzaro from comment #43)
> Fortunately it looks like this was not backported to 2.44 since nobody
> requested it.

It was backported to 2.44. See https://github.com/WebKit/WebKit/commit/30ad9a720e6b12a6c958fcef0d7dd3f52da485bd

[Michael Catanzaro]

OK, will revert there too. I must have gotten very confused when I checked for the backport....

[Michael Catanzaro]

Found yet another reproducer: visit https://www.msnbc.com/opinion/msnbc-opinion/trump-hush-money-verdict-biden-campaign-reaction-rcna154560 and just scroll down the page

[Kdwk]

Created attachment 472096 [details]
gdb.txt

I have hit this again on one of Apple's product pages. It seems this bug is crashing a lot of websites

[Jeff Fortin]

*** Bug 278569 has been marked as a duplicate of this bug. ***

[Michael Catanzaro]

Hit this just now on https://www.nbcnews.com/nbc-out/out-news/probe-closed-owners-pulse-nightclub-49-died-mass-shooting-rcna168778

[Philippe Normand]

I think by now, we do know how to reproduce this.

[Philippe Normand]

http/tests/media/media-source/mediasource-rvfc.html is affected by this ASSERT.

[Philippe Normand]

*** Bug 270733 has been marked as a duplicate of this bug. ***

[Caden Mitchell]

Created attachment 473732 [details]
A back trace from MacBook Air landing page

Encountered this issue just opening the webpage. Has been an issue for at least a year for me consistently. https://apple.com/macbook-air

[Michael Catanzaro]

*** Bug 285390 has been marked as a duplicate of this bug. ***

[Philippe Normand]

*** Bug 287773 has been marked as a duplicate of this bug. ***

[Alicia Boya García]

Pull request: https://github.com/WebKit/WebKit/pull/41623

[EWS]

Committed 291490@main (6b895b68961a): <https://commits.webkit.org/291490@main>

Reviewed commits have been landed. Closing PR #41623 and removing active labels.