Bug 225765 – [GStreamer] Another crash under gst_element_add_pad

• Status: RESOLVED
• Resolution: FIXED
• Component: Media

[Michael Catanzaro]

Reminds me of bug #222763, but this one is different. Visit https://www.warbyparker.com/eyeglasses/lenses in Epiphany Tech Preview with WebKitGTK 2.32.1. You'll see a couple warnings:

** (WebKitWebProcess:1339): WARNING **: 09:03:23.762: Warning: 11, not negotiated. Debug output: ../libs/gst/base/gstbasetransform.c(1423): gst_base_transform_reconfigure (): /GstPipeline:image-decoder-0/GstDecodebin3:decodebin3-0/GstParseBin:parsebin0/GstCapsFilter:capsfilter0:
not negotiated

** (WebKitWebProcess:1339): WARNING **: 09:03:23.763: Error: 1, Internal data stream error.. Debug output: ../gst/isomp4/qtdemux.c(6619): gst_qtdemux_loop (): /GstPipeline:image-decoder-0/GstDecodebin3:decodebin3-0/GstParseBin:parsebin0/GstQTDemux:qtdemux0:
streaming stopped, reason not-negotiated (-4)

Then it will crash:

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f05e6596855 in __GI_abort () at abort.c:79
#2  0x00007f05e6da0bb1 in  () at /usr/lib/x86_64-linux-gnu/libwebkit2gtk-4.0.so.37
#3  0x00007f05e2170f75 in ffi_call_unix64 () at ../src/x86/unix64.S:101
#4  0x00007f05e2170369 in ffi_call_int
    (cif=<optimized out>, fn=<optimized out>, rvalue=<optimized out>, avalue=<optimized out>, closure=<optimized out>) at ../src/x86/ffi64.c:669
#5  0x00007f05e6230a9c in g_cclosure_marshal_generic () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#6  0x00007f05e622ffcf in g_closure_invoke () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#7  0x00007f05e6242ddb in signal_emit_unlocked_R.isra.0 () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#8  0x00007f05e62498f1 in g_signal_emit_valist () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#9  0x00007f05e6249a53 in g_signal_emit () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#10 0x00007f05e35745a0 in gst_element_add_pad (element=element@entry=0x7f040c010030, pad=0x7f040c013600)
    at ../gst/gstelement.c:714
#11 0x00007f052c489433 in reconfigure_output_stream (output=0x7f03f00024d0, slot=0x7f04040591e0)
    at ../gst/playback/gstdecodebin3.c:2254
#12 0x00007f052c489b4f in multiqueue_src_probe
    (pad=pad@entry=0x7f0404012f20, info=info@entry=0x7f03ff7fd950, slot=0x7f04040591e0)
    at ../gst/playback/gstdecodebin3.c:1791
#13 0x00007f05e35902ee in probe_hook_marshal (hook=0x7f04040116c0, data=0x7f03ff7fd820) at ../gst/gstpad.c:3565
#14 0x00007f05e61287f6 in g_hook_list_marshal () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#15 0x00007f05e358f9d9 in do_probe_callbacks
    (pad=pad@entry=0x7f0404012f20, info=<optimized out>, defaultval=defaultval@entry=GST_FLOW_OK)
    at ../gst/gstpad.c:3728
#16 0x00007f05e35931c5 in gst_pad_push_event_unchecked
    (pad=pad@entry=0x7f0404012f20, event=0x7f040405b870, type=type@entry=GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM)
    at ../gst/gstpad.c:5376
#17 0x00007f05e3593758 in push_sticky
    (pad=pad@entry=0x7f0404012f20, ev=ev@entry=0x7f03ff7fda30, user_data=user_data@entry=0x7f03ff7fdaa0)
    at ../gst/gstevent.h:438
#18 0x00007f05e35910b0 in events_foreach
    (pad=pad@entry=0x7f0404012f20, func=func@entry=0x7f05e3593700 <push_sticky>, user_data=user_data@entry=0x7f03ff7fdaa0) at ../gst/gstpad.c:608
#19 0x00007f05e359c400 in check_sticky (event=0x7f040405b870, pad=0x7f0404012f20) at ../gst/gstpad.c:3986
#20 gst_pad_push_event (pad=0x7f0404012f20, event=event@entry=0x7f040405b870) at ../gst/gstpad.c:5542
#21 0x00007f052c2c4474 in gst_single_queue_push_one
    (allow_drop=<synthetic pointer>, object=0x7f040405b870, sq=0x7f040405ddb0, mq=0x7f040c018000)
    at ../plugins/elements/gstmultiqueue.c:1688
#22 gst_multi_queue_loop (pad=<optimized out>) at ../plugins/elements/gstmultiqueue.c:1959
#23 0x00007f05e35ca017 in gst_task_func (task=0x7f040002a050) at ../gst/gsttask.c:328
#24 0x00007f05e61643b4 in g_thread_pool_thread_proxy () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#25 0x00007f05e6163ab1 in g_thread_proxy () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#26 0x00007f05e28be4d2 in start_thread (arg=<optimized out>) at pthread_create.c:477
#27 0x00007f05e6672323 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Michael Catanzaro]

Created attachment 428513 [details]
GStreamer log

[Michael Catanzaro]

Created attachment 428514 [details]
Full backtrace (all threads)

[Philippe Normand]

I think this happens because your openh264dec decoder can't handle the progressive-high profile:

:00:00.024920765  1419 0x7f20ac001aa0 WARN           basetransform gstbasetransform.c:1362:gst_base_transform_setcaps:<capsfilter0> transform could not transform video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.2, profile=(string)progressive-high, width=(int)1366, height=(int)684, framerate=(fraction)24000/1001, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true in anything we support

Plugin Details:
  Name                     openh264
  Description              OpenH264 encoder/decoder plugin
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstopenh264.so
  Version                  1.16.3
  License                  BSD
  Source module            gst-plugins-bad
  Binary package           GStreamer Bad Plug-ins source release
  Origin URL               freedesktop-sdk

...

  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: byte-stream
              alignment: au
                profile: { (string)constrained-baseline, (string)baseline, (string)main, (string)high }

[Philippe Normand]

Also this shouldn't happen:

0:00:00.033920128  1419 0x7f20ac001aa0 DEBUG     webkitimagedecoder ImageDecoderGStreamer.cpp:242:connectDecoderPad:<image-decoder-1> New decodebin pad <decodebin3-1:audio_0> caps: audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)[ 8000, 96000 ], channels=(int)[ 1, 8 ]

I'll try to reproduce the issue but I already suspect this might be a bug in your old gst 1.16.3 :)

[Philippe Normand]

Can you backport this patch in your runtime?
https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/commit/8bf7816790aa4e963319f3333edec9646a558765

Without it I think the stream selection in the image decoder might not work because the collection owner might not be decodebin3 so the decoder is not sending the select-streams event aimed to select only video streams.

If you can modify WebKit in your runtime, in ImageDecoderGStreamer.cpp line 332 after the gst_message_parse_stream_collection() call add some logging:

gst_printerrln("collection: %p owner: %s", collection.get(), GST_MESSAGE_SRC_NAME(message));

[Philippe Normand]

I can't reproduce this with the WebKit SDK...

Also this page is weird, it uses a <video> element that has the src and poster attributes set to the same (video) URL. Why on earth would you do that?

[Michael Catanzaro]

(In reply to Philippe Normand from comment #5)
> Can you backport this patch in your runtime?
> https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/commit/
> 8bf7816790aa4e963319f3333edec9646a558765
> 
> Without it I think the stream selection in the image decoder might not work
> because the collection owner might not be decodebin3 so the decoder is not
> sending the select-streams event aimed to select only video streams.

Will do.

(In reply to Philippe Normand from comment #5)
> If you can modify WebKit in your runtime, in ImageDecoderGStreamer.cpp line
> 332 after the gst_message_parse_stream_collection() call add some logging:
> 
> gst_printerrln("collection: %p owner: %s", collection.get(),
> GST_MESSAGE_SRC_NAME(message));

It's possible, but it's a real pain. Will see if I find time for it next week....

[Michael Catanzaro]

(In reply to Philippe Normand from comment #5)
> Can you backport this patch in your runtime?
> https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/commit/
> 8bf7816790aa4e963319f3333edec9646a558765

Done. Sadly it didn't fix this crash.

(In reply to Michael Catanzaro from comment #7) 
> It's possible, but it's a real pain. Will see if I find time for it next
> week....

Still TODO for me.

[Michael Catanzaro]

This was... hard. I built a modified runtime with WebKit, following the instructions at https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/README.rst. (Yes, it appears that I wrote those instructions, but I really just copied them from somewhere.) I couldn't figure out how to run it with an application, so had to rebuild the runtime to enable -DMINIBROWSER=ON because we have that off by default for some reason. Eventually I figured out the incantation to make MiniBrowser work:

$ flatpak run --command=/bin/bash -d --socket=wayland --device=dri --share=ipc --share=network --socket=pulseaudio --filesystem=home org.gnome.Platform
[📦 org.gnome.Platform ~]$ export GST_DEBUG="3,webkit*:6" GST_DEBUG_FILE="$HOME/gst.log" GST_DEBUG_NO_COLOR=1 WEBKIT_FORCE_SANDBOX=0
[📦 org.gnome.Platform ~]$ /usr/libexec/webkit2gtk-4.0/MiniBrowser https://www.warbyparker.com/eyeglasses/lenses
collection: 0x7f4e8c01c030 owner: decodebin3-0

** (WebKitWebProcess:160): WARNING **: 19:45:08.074: Warning: 11, not negotiated. Debug output: ../libs/gst/base/gstbasetransform.c(1423): gst_base_transform_reconfigure (): /GstPipeline:image-decoder-0/GstDecodebin3:decodebin3-0/GstParseBin:parsebin0/GstCapsFilter:capsfilter0:
not negotiated

** (WebKitWebProcess:160): WARNING **: 19:45:08.075: Error: 1, Internal data stream error.. Debug output: ../gst/isomp4/qtdemux.c(6619): gst_qtdemux_loop (): /GstPipeline:image-decoder-0/GstDecodebin3:decodebin3-0/GstParseBin:parsebin0/GstQTDemux:qtdemux0:
streaming stopped, reason not-negotiated (-4)

** (MiniBrowser:3): WARNING **: 19:45:09.510: WebProcess CRASHED

I'll attach the gst.log it produced as well, though I guess you probably don't need it, since in theory it should match the original log more or less? Except note that OpenH264 isn't present here (it's only present in my system install, but I followed the instructions to use my user install). Anyway, the most important point was the debug you requested: collection: 0x7f4e8c01c030 owner: decodebin3-0. It took several hours to get that, so I hope it was worth it. :P

[Michael Catanzaro]

Created attachment 430460 [details]
GStreamer log from custom-built runtime

(Forgot to attach the log.)

[Philippe Normand]

After locally reverting these 2 in my -bad checkout:

https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1634
https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1789

I can reproduce the warnings (caps negotiation failing), but not the crash.


In any case, I would advise to backport these 2 MRs in your runtime, if you can. 

I'll try to reproduce the crash...

[Philippe Normand]

I could reproduce the crash after reverting:

https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/commit/b41b87522f59355bb21c001e9e2df96dc6956928

[Philippe Normand]

This is not a WebKit bug. Please update your runtime :)

[Michael Catanzaro]

I've confirmed the GNOME master runtime currently has gst-plugins-bad and gst-plugins-base 1.16.3, which is the latest 1.16 release. This is what GNOME will stick with until we update to freedesktop-sdk 21.08. That'

[Michael Catanzaro]

Whoops. "That's probably coming soon."

I'm a little concerned that the GStreamer release cycle has become quite disconnected with GNOME's, but that's not an issue to be solved on WebKit Bugzilla.

[Philippe Normand]

One thing we could do though, avoid the RELEASE_ASSERT in case gst < 1.18 is found...

[Philippe Normand]

(In reply to Philippe Normand from comment #16)
> One thing we could do though, avoid the RELEASE_ASSERT in case gst < 1.18 is
> found...

Yeah let's do that, I suppose Debian could run into the same crash.

[Philippe Normand]

Created attachment 431452 [details]
Patch

[EWS]

Committed r278892 (238834@main): <https://commits.webkit.org/238834@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 431452 [details].

[Radar WebKit Bug Importer]

<rdar://problem/79357665>