Bug 245876 – [GStreamer][MSE] FLAC Audio Format Support

• Component: Media
• Status: NEW

[Tamas Cservenak]

Does WebKitGTK support FLAC audio format for playback? If not, this would be really good thing to have, as various streaming services (Deezer for example) streams in FLAC but their service is not usable with WebKitGTK as there is no support for it.

[Tamas Cservenak]

Reported initially as https://gitlab.gnome.org/GNOME/epiphany/-/issues/1865

[Philippe Normand]

Flac should already be supported, but you need a decoder for it...

Please provide a gst log, GST_DEBUG="3,webkit*:8" GST_DEBUG_FILE=gst.log app ...

[Tamas Cservenak]

I cannot make it produce log file as requested. Tried this (has much more than requested, but gst.log file is NOT produced):

export PIPEWIRE_DEBUG=5 && export EPHY_LOG_MODULES="all" && export GST_DEBUG=8 && export GST_DEBUG_FILE=gst.log && epiphany https://www.deezer.com/us/

On the other hand, on console these lines appears in the moment of error:

(WebKitWebProcess:2): GLib-GObject-WARNING **: 11:04:12.563: invalid (NULL) pointer instance
(WebKitWebProcess:2): GLib-GObject-CRITICAL **: 11:04:12.563: g_signal_connect_data: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
(WebKitWebProcess:2): GStreamer-CRITICAL **: 11:04:12.563: gst_element_link_many: assertion 'GST_IS_ELEMENT (element_2)' failed


Also, gst-inspect-1.0 do report flac as present:
flac:  flacdec: FLAC audio decoder
flac:  flacenc: FLAC audio encoder
flac:  flactag: FLAC tagger

[Tamas Cservenak]

Also, GStream does work with FLAC:

$ gst-launch-1.0 filesrc location=~/Downloads/Symphony\ No.6\ \(1st\ movement\).flac ! flacparse ! flacdec ! pulsesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstPulseSinkClock
^Chandling interrupt. (1,4 %)
Interrupt: Stopping pipeline ...
Execution ended after 0:00:10.397468132
Setting pipeline to NULL ...
Freeing pipeline ...
$

[Philippe Normand]

How do you get flac with deezer? Here it's playing with mp3...

[Philippe Normand]

> export PIPEWIRE_DEBUG=5 && export EPHY_LOG_MODULES="all" && export GST_DEBUG=8 && export GST_DEBUG_FILE=gst.log && epiphany https://www.deezer.com/us/

Maybe some sandboxing stuff is preventing files writing in whatever dir you are... Try GST_DEBUG_FILE=$HOME/Downloads/gst.log

[Philippe Normand]

(In reply to Philippe Normand from comment #6)
> > export PIPEWIRE_DEBUG=5 && export EPHY_LOG_MODULES="all" && export GST_DEBUG=8 && export GST_DEBUG_FILE=gst.log && epiphany https://www.deezer.com/us/
> 
> Maybe some sandboxing stuff is preventing files writing in whatever dir you
> are... Try GST_DEBUG_FILE=$HOME/Downloads/gst.log

and you don't need the export and &&

FOO=bar epiphany ... will set the FOO env var in the epiphany process.

If your shell is fish, you need to prefix with "env", env FOO=bar command...

[Tamas Cservenak]

(In reply to Philippe Normand from comment #5)
> How do you get flac with deezer? Here it's playing with mp3...

Mp3 is the default quality, the "Better".

To get flac you need to change in Settings (lower right icon, left to the rightmost Queue) Audio Quality from "Better" (mp3 320kb.s) to "High Fidelity (lossless)". But, am not sure, it may require subscription...

[Philippe Normand]

(In reply to Tamas Cservenak from comment #8)
> (In reply to Philippe Normand from comment #5)
> > How do you get flac with deezer? Here it's playing with mp3...
> 
> Mp3 is the default quality, the "Better".
> 
> To get flac you need to change in Settings (lower right icon, left to the
> rightmost Queue) Audio Quality from "Better" (mp3 320kb.s) to "High Fidelity
> (lossless)". But, am not sure, it may require subscription...

It's a premium feature. Sorry I won't test this then.

[Tamas Cservenak]

Created attachment 462769 [details]
GST_DEBUG log

GST_DEBUG_FILE did not work (see "Could not open log file '/home/cstamas/Downloads/gst.log' for writing: No such file or directory", so probably some sandboxing, as that directory do exists), but this time it went to console, so grabbed it and attaching here.

[Tamas Cservenak]

> It's a premium feature. Sorry I won't test this then.

As I am interested in this bug getting fixed, and have family subscription with a free seat, I would be grateful to offer that for testing.

Am not experienced in building (and testing) native stuff, am just a lame Java dev ;)

[Philippe Normand]

Can you set G_DEBUG=fatal_warnings ? Then epiphany should crash during the player initialization. If you could attach the gdb backtrace here it'd be great.

[Philippe Normand]

(In reply to Tamas Cservenak from comment #11)
> > It's a premium feature. Sorry I won't test this then.
> 
> As I am interested in this bug getting fixed, and have family subscription
> with a free seat, I would be grateful to offer that for testing.
> 
> Am not experienced in building (and testing) native stuff, am just a lame
> Java dev ;)

Ah ok! Hm yeah if you can share a seat with me somehow that would be great, my email is [e-mail]

[Tamas Cservenak]

> Ah ok! Hm yeah if you can share a seat with me somehow that would be great,
> my email is [e-mail]

First it's you who need to create Deezer account, and then tell me the email you used for the account, so I can take "onboard".

[Tamas Cservenak]

> Ah ok! Hm yeah if you can share a seat with me somehow that would be great,
> my email is [e-mail]

Sorry, was wrong: sent you an invite email

[Tamas Cservenak]

(In reply to Philippe Normand from comment #12)
> Can you set G_DEBUG=fatal_warnings ? Then epiphany should crash during the
> player initialization. If you could attach the gdb backtrace here it'd be
> great.

I did set it, and epiphany crashes all right (actually 2 processes: WebKitWebProcess and epiphany), they are here all right:

[cstamas@urnebes ~]$ coredumpctl 
TIME                           PID  UID  GID SIG     COREFILE  EXE                                          SIZE
Mon 2022-10-03 12:40:43 CEST 74413 1000 1000 SIGTRAP truncated /usr/libexec/webkit2gtk-4.0/WebKitWebProcess 2.6M
Mon 2022-10-03 12:40:46 CEST 74360 1000 1000 SIGTRAP truncated /usr/bin/epiphany                            2.3M
Mon 2022-10-03 13:09:39 CEST 78363 1000 1000 SIGTRAP truncated /usr/libexec/webkit2gtk-4.0/WebKitWebProcess 3.1M
Mon 2022-10-03 13:09:42 CEST 78305 1000 1000 SIGTRAP truncated /usr/bin/epiphany                            2.3M
Mon 2022-10-03 13:21:09 CEST 80383 1000 1000 SIGTRAP truncated /usr/libexec/webkit2gtk-4.0/WebKitWebProcess 2.9M
Mon 2022-10-03 13:21:12 CEST 80333 1000 1000 SIGTRAP truncated /usr/bin/epiphany                            3.5M
[cstamas@urnebes ~]$ 


But from this point on (following this guide https://fedoramagazine.org/file-better-bugs-coredumpctl/), and after installing deguginfo (for webkit2gtk3 and epiphany, as coredumpctl gdb requested), I still have no luck to produce meaningful backtrace :(

[Philippe Normand]

have you tried this? coredumpctl gdb 80383

[Philippe Normand]

Ah, truncated coredumps.

Here I had to tune systemd to lift some size limitations:

cat /etc/systemd/coredump.conf.d/tune.conf
[Coredump]
ProcessSizeMax=12G
ExternalSizeMax=12G

[Philippe Normand]

(In reply to Philippe Normand from comment #18)
> Ah, truncated coredumps.
> 
> Here I had to tune systemd to lift some size limitations:
> 
> cat /etc/systemd/coredump.conf.d/tune.conf
> [Coredump]
> ProcessSizeMax=12G
> ExternalSizeMax=12G

12G is clearly a lot, likely smaller limits would be more reasonable, but at least my coredumps are whole :)

[Tamas Cservenak]

(In reply to Philippe Normand from comment #18)
> Ah, truncated coredumps.

Much better, now I have this (last 2)

Mon 2022-10-03 13:21:09 CEST 80383 1000 1000 SIGTRAP truncated /usr/libexec/webkit2gtk-4.0/WebKitWebProcess  2.9M
Mon 2022-10-03 13:21:12 CEST 80333 1000 1000 SIGTRAP truncated /usr/bin/epiphany                             3.5M
Mon 2022-10-03 13:38:34 CEST 83267 1000 1000 SIGTRAP present   /usr/libexec/webkit2gtk-4.0/WebKitWebProcess 57.2M
Mon 2022-10-03 13:38:44 CEST 83217 1000 1000 SIGTRAP present   /usr/bin/epiphany                             6.0M


But now "thread apply all bt full" runs like is rendering ToyStory 10... will chime back with results.

[Philippe Normand]

Did you disable pagination as mentioned in https://fedoramagazine.org/file-better-bugs-coredumpctl/ ?

[Tamas Cservenak]

(In reply to Philippe Normand from comment #21)
> Did you disable pagination as mentioned in
> https://fedoramagazine.org/file-better-bugs-coredumpctl/ ?

yup

(gdb) set logging file webkitwebprocess.txt
(gdb) set logging on
Warning: 'set logging on', an alias for the command 'set logging enabled', is deprecated.
Use 'set logging enabled on'.

Copying output to webkitwebprocess.txt.
Copying debug output to webkitwebprocess.txt.
(gdb) set pagination off
(gdb) thread apply all bt full

Thread 44 (Thread 0x7ffa7c937640 (LWP 31)):
#0  0x00007ffb2ff1d88f in poll () at /lib64/libc.so.6
#1  0x00007ffb2c76223d in g_main_context_iterate.constprop () at /lib64/libglib-2.0.so.0
#2  0x00007ffb2c70c6cf in g_main_loop_run () at /lib64/libglib-2.0.so.0
#3  0x00007ffb2f79c4b0 in WTF::RunLoop::run() () at /lib64/libjavascriptcoregtk-4.0.so.18
#4  0x00007ffb2f7a69c1 in WTF::wtfThreadEntryPoint(void*) [clone .lto_priv.0] () at /lib64/libjavascriptcoregtk-4.0.so.18
#5  0x00007ffb2fea4e2d in start_thread () at /lib64/libc.so.6
#6  0x00007ffb2ff29364 in clone () at /lib64/libc.so.6
...


It advances, but slowly. Current ouput is Thread 2 I guess Thread 1 to go and done.

[Tamas Cservenak]

Incredible, but gdb is still working (100% on one thread on CPU)...

[Philippe Normand]

I don't like "full" backtraces... can you check if "thread apply all bt" would run faster?

[Tamas Cservenak]

(In reply to Philippe Normand from comment #24)
> I don't like "full" backtraces... can you check if "thread apply all bt"
> would run faster?

full backtrace still runs :)

non-full is def faster, is already on thread 1 (where full is since hour or more).

Let's wait for it's end then...

[Tamas Cservenak]

Created attachment 462771 [details]
GDB backtrace (non-full)

GDB backtrace "non-full" attached.

[Philippe Normand]

Trace looks a bit odd though:

Thread 1 (Thread 0x7ffb292bea80 (LWP 2)):
#0  0x00007ffb2c711e51 in g_logv () at /lib64/libglib-2.0.so.0
#1  0x00007ffb2c7120d3 in g_log () at /lib64/libglib-2.0.so.0
#2  0x00007ffb2c82df6a in g_type_check_instance () at /lib64/libgobject-2.0.so.0
#3  0x00007ffb2c8220bf in g_signal_connect_data () at /lib64/libgobject-2.0.so.0
#4  0x00007ffb30c7a66f in WebCore::AppendPipeline::AppendPipeline(WebCore::SourceBufferPrivateGStreamer&, WebCore::MediaPlayerPrivateGStreamerMSE&) (playerPrivate=<optimized out>, sourceBufferPrivate=..., this=0x7ffa55aaf2e0) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/redhat-linux-build/WTF/Headers/wtf/glib/GRefPtr.h:110
#5  WTF::makeUniqueRefWithoutFastMallocCheck<WebCore::AppendPipeline, WebCore::SourceBufferPrivateGStreamer&, WebCore::MediaPlayerPrivateGStreamerMSE&>(WebCore::SourceBufferPrivateGStreamer&, WebCore::MediaPlayerPrivateGStreamerMSE&) () at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/redhat-linux-build/WTF/Headers/wtf/UniqueRef.h:40
#6  WTF::makeUniqueRef<WebCore::AppendPipeline, WebCore::SourceBufferPrivateGStreamer&, WebCore::MediaPlayerPrivateGStreamerMSE&>(WebCore::SourceBufferPrivateGStreamer&, WebCore::MediaPlayerPrivateGStreamerMSE&) () at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/redhat-linux-build/WTF/Headers/wtf/UniqueRef.h:47
#7  WebCore::SourceBufferPrivateGStreamer::SourceBufferPrivateGStreamer(WebCore::MediaSourcePrivateGStreamer*, WebCore::ContentType const&, WebCore::MediaPlayerPrivateGStreamerMSE&) (playerPrivate=<optimized out>, contentType=<optimized out>, mediaSource=0x7ffa76488c80, this=<optimized out>) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/Source/WebCore/platform/graphics/gstreamer/mse/SourceBufferPrivateGStreamer.cpp:73
#8  WebCore::SourceBufferPrivateGStreamer::create(WebCore::MediaSourcePrivateGStreamer*, WebCore::ContentType const&, WebCore::MediaPlayerPrivateGStreamerMSE&) (playerPrivate=<optimized out>, contentType=<optimized out>, mediaSource=0x7ffa76488c80) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/Source/WebCore/platform/graphics/gstreamer/mse/SourceBufferPrivateGStreamer.cpp:65
#9  WebCore::MediaSourcePrivateGStreamer::addSourceBuffer(WebCore::ContentType const&, bool, WTF::RefPtr<WebCore::SourceBufferPrivate, WTF::RawPtrTraits<WebCore::SourceBufferPrivate>, WTF::DefaultRefDerefTraits<WebCore::SourceBufferPrivate> >&) (this=this@entry=0x7ffa76488c80, contentType=..., sourceBufferPrivate=...) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/Source/WebCore/platform/graphics/gstreamer/mse/MediaSourcePrivateGStreamer.cpp:87
#10 0x00007ffb31049739 in WebCore::MediaSource::createSourceBufferPrivate(WebCore::ContentType const&) (incomingType=..., this=0x7ffaaa0101e0) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/Source/WebCore/Modules/mediasource/MediaSource.cpp:1067
#11 WebCore::MediaSource::addSourceBuffer(WTF::String const&) (type=..., this=0x7ffaaa0101e0) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/Source/WebCore/Modules/mediasource/MediaSource.cpp:692
#12 WebCore::jsMediaSourcePrototypeFunction_addSourceBufferBody(JSC::JSGlobalObject*, JSC::CallFrame*, WebCore::IDLOperation<WebCore::JSMediaSource>::ClassParameter) (lexicalGlobalObject=0x7ffb119f0e68, callFrame=<optimized out>, castedThis=0x7ffb19f42768) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/redhat-linux-build/WebCore/DerivedSources/JSMediaSource.cpp:471
#13 0x00007ffb31049f98 in WebCore::IDLOperation<WebCore::JSMediaSource>::call<WebCore::jsMediaSourcePrototypeFunction_addSourceBufferBody> (operationName=<optimized out>, callFrame=<optimized out>, lexicalGlobalObject=<optimized out>) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/Source/WebCore/bindings/js/JSDOMOperation.h:63
#14 WebCore::jsMediaSourcePrototypeFunction_addSourceBuffer(JSC::JSGlobalObject*, JSC::CallFrame*) (lexicalGlobalObject=<optimized out>, callFrame=<optimized out>) at /usr/src/debug/webkit2gtk3-2.38.0-2.fc36.x86_64/redhat-linux-build/WebCore/DerivedSources/JSMediaSource.cpp:476
#15 0x00007ffac4008338 in  ()
#16 0x00007ffc615681d0 in  ()
#17 0x00007ffb2eaba925 in op_call_slow_return_location () at /lib64/libjavascriptcoregtk-4.0.so.18
#18 0x0000000000000000 in  ()

There's anyway at least one issue in AppendPipeline, no demuxer is created for audio/flac SourceBuffers:

0:00:07.673421922     2 0x55bcf5856350 DEBUG              webkitmse AppendPipeline.cpp:143:AppendPipeline: SourceBuffer containerType: audio/flac

    GST_DEBUG("SourceBuffer containerType: %s", type.utf8().data());
    bool hasDemuxer = true;
    if (type.endsWith("mp4"_s) || type.endsWith("aac"_s)) {
        m_demux = makeGStreamerElement("qtdemux", nullptr);
        m_typefind = makeGStreamerElement("identity", nullptr);
    } else if (type.endsWith("webm"_s)) {
        m_demux = makeGStreamerElement("matroskademux", nullptr);
        m_typefind = makeGStreamerElement("identity", nullptr);
    } else if (type == "audio/mpeg"_s) {
        m_demux = makeGStreamerElement("identity", nullptr);
        m_typefind = makeGStreamerElement("typefind", nullptr);
        hasDemuxer = false;
    } else
        ASSERT_NOT_REACHED();

If you had a Debug build you would see an ASSERT+crash there.

[Michael Catanzaro]

(In reply to Philippe Normand from comment #6)
> Maybe some sandboxing stuff is preventing files writing in whatever dir you
> are... Try GST_DEBUG_FILE=$HOME/Downloads/gst.log

You're saving the gst.log inside the sandbox, not on the host system. Our instructions https://trac.webkit.org/wiki/WebKitGTK/Debugging#Debuggingmultimediastuff show you how to disable that and also disable the color codes so the resulting log is readable.

[Philippe Normand]

Pull request: https://github.com/WebKit/WebKit/pull/4927

[Philippe Normand]

(In reply to Philippe Normand from comment #29)
> Pull request: https://github.com/WebKit/WebKit/pull/4927

I'm not sure this will fix your issues, but at least it should prevent warnings...

[Tamas Cservenak]

> It's a premium feature. Sorry I won't test this then.

I think I found a reproducer, at least it crashes my epiphany with very same result:

Open this page:
https://support.deezer.com/hc/en-gb/articles/115004588345-High-Fidelity-HiFi-

and if you scroll below, there will be a section called "Hifi classics" (for me with Daft punk, might be randomized). Click on it, and it will allow you to play "sample" w/o premium subscription of flac.... and epihpany with G_DEBUG=fatal_warnings crashes for me.

[Tamas Cservenak]

It seems that whole album of Daft Punk is "open" for listening in FLAC:
https://www.deezer.com/en/album/6575789

[Tamas Cservenak]

meh, sorry, seems I was wrong, I was signed in with my account in Epiphany, while I assumed am not :(

[Philippe Normand]

Tamas, in which browsers exactly are you able to play premium flac deezer content?

[Tamas Cservenak]

(In reply to Philippe Normand from comment #34)
> Tamas, in which browsers exactly are you able to play premium flac deezer
> content?

My main browser is Google Chrome (on Linux but also Mac), and "secondary" browsers are Firefox and (on Fedora36) Gnome Web (epiphany/webkit). Out of these, only epiphany fails to play it (this bug), the two others are okay with it.

[Philippe Normand]

Does this demo works in your browsers? Do you hear something?

https://brionv.com/misc/msetest4/flac2.html

[Tamas Cservenak]

Found some interesting related links:
https://chromestatus.com/feature/5713014258925568
https://bugzilla.mozilla.org/show_bug.cgi?id=1195723

And from this mozilla bug this one:
https://github.com/audiocogs/flac.js/issues/8

With conclusion "All major browsers support FLAC now..."

Interesting thing: this test page http://hpr.dogphilosophy.net/test/index.php WORKS in epiphany (the same one that does not work with Deezer, am beginning to think they - Deezer - do something differently?

[Tamas Cservenak]

(In reply to Philippe Normand from comment #36)
> Does this demo works in your browsers? Do you hear something?
> 
> https://brionv.com/misc/msetest4/flac2.html

No, it does not, and when epiphany started from terminal, same errors are printed to stderr:

(WebKitWebProcess:2): GLib-GObject-WARNING **: 16:23:23.734: invalid (NULL) pointer instance

(WebKitWebProcess:2): GLib-GObject-CRITICAL **: 16:23:23.734: g_signal_connect_data: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed

(WebKitWebProcess:2): GStreamer-CRITICAL **: 16:23:23.734: gst_element_link_many: assertion 'GST_IS_ELEMENT (element_2)' failed

[Philippe Normand]

IIUC there are two options:

- Mux flac in isobmff container : That should already be supported in 2.38.
- Pass a flac bitstream directly, without muxing : That's what Deezer does, IIUC, and is not currently supported (but I do have a patch)

[Philippe Normand]

(In reply to Tamas Cservenak from comment #38)
> (In reply to Philippe Normand from comment #36)
> > Does this demo works in your browsers? Do you hear something?
> > 
> > https://brionv.com/misc/msetest4/flac2.html
> 
> No, it does not, and when epiphany started from terminal, same errors are
> printed to stderr:
> 

Yeah I know that, my patch fixes those warnings and playback works :)

I was asking about the other browsers, like Chrome, FF... Because here I tested that demo on Linux in Chromium, Chrome and FF and neither of those was able to playback the audio track.

[Tamas Cservenak]

(In reply to Philippe Normand from comment #40)
> I was asking about the other browsers, like Chrome, FF... Because here I
> tested that demo on Linux in Chromium, Chrome and FF and neither of those
> was able to playback the audio track.

It works in Chrome and also FF without a problem (sounds is okay as well, just like the Gecko). Have to note, that there is a slight delay, and then you need to press Play button, but it plays...

[Philippe Normand]

I've got flac working on Deezer with this branch:

https://github.com/philn/WebKit/tree/mse-flac

Still WIP because:

- seek is bumpy
- need layout test

You can try though if you feel like building WebKit.

[Tamas Cservenak]

Thanks, but see https://bugs.webkit.org/show_bug.cgi?id=245876#c11  :)