Bug 210264 – [EME][GStreamer] web-platform-tests/encrypted-media/clearkey-events-session-closed-event.https.html random crashes

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Charlie Turner]

This is crashing in a call to gcry_cipher_setctr for unknown reasons.

Maybe the GCrypt library is not being correctly initialised. The initialisation rules are complex[1]. I wonder if we can use the classes in PAL::Crypto instead. The AES-CTR decrypt method looked very costly there however, making me think it might not be appropriate for real-time use cases. Testing needed.

This is not reproducible in isolated test runs unfortunately, in seems threading contention triggers it.

#0  SIGSEGV ??
#1  0x00007f90bccd2349 in gcry_cipher_setctr () at /lib/x86_64-linux-gnu/libgcrypt.so.20
#2  0x00007f90cf97bcd3 in WebCore::CDMProxyClearKey::cencSetCounterVector(WebCore::CDMProxyClearKey::cencDecryptContext const&) (this=0x7f905c452410, input=...) at /home/cht/igalia/sources/WebKit/Source/WebCore/platform/graphics/gstreamer/eme/CDMProxyClearKey.cpp:98
#3  0x00007f90cf97c2cf in WebCore::CDMProxyClearKey::cencDecrypt(WebCore::CDMProxyClearKey::cencDecryptContext&) (this=0x7f905c452410, input=...) at /home/cht/igalia/sources/WebKit/Source/WebCore/platform/graphics/gstreamer/eme/CDMProxyClearKey.cpp:203
#4  0x00007f90cf963b93 in decrypt(_WebKitMediaCommonEncryptionDecrypt*, _GstBuffer*, _GstBuffer*, _GstBuffer*, unsigned int, _GstBuffer*) (self=0x7f77d4011ca0, ivBuffer=0x7f77d4258c60, keyIDBuffer=0x7f77e003c5a0, buffer=0x7f9050017b40, subsampleCount=2, subsamplesBuffer=0x7f77dc26e000) at /home/cht/igalia/sources/WebKit/Source/WebCore/platform/graphics/gstreamer/eme/WebKitClearKeyDecryptorGStreamer.cpp:165
#5  0x00007f90cf9654ea in transformInPlace(_GstBaseTransform*, _GstBuffer*) (base=0x7f77d4011ca0, buffer=0x7f9050017b40) at /home/cht/igalia/sources/WebKit/Source/WebCore/platform/graphics/gstreamer/eme/WebKitCommonEncryptionDecryptorGStreamer.cpp:260
#6  0x00007f90bcc7bac7 in default_generate_output (trans=0x7f77d4011ca0, outbuf=0x7f76d5ffa5b8) at ../subprojects/gstreamer/libs/gst/base/gstbasetransform.c:2175
#7  0x00007f90bcc7b207 in gst_base_transform_chain (pad=<optimized out>, parent=0x7f77d4011ca0, buffer=<optimized out>) at ../subprojects/gstreamer/libs/gst/base/gstbasetransform.c:2333
#8  0x00007f90bcb702e2 in gst_pad_chain_data_unchecked (data=0x7f9050017b40, type=4112, pad=0x7f778835d2c0) at ../subprojects/gstreamer/gst/gstpad.c:4399
#9  0x00007f90bcb702e2 in gst_pad_push_data (pad=pad@entry=0x7f777c35eaa0, type=type@entry=4112, data=data@entry=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstpad.c:4655
#10 0x00007f90bcb76da2 in gst_pad_push (pad=0x7f777c35eaa0, buffer=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstpad.c:4774
#11 0x00007f90bcb702e2 in gst_pad_chain_data_unchecked (data=0x7f9050017b40, type=4112, pad=0x7f778835cbd0) at ../subprojects/gstreamer/gst/gstpad.c:4399
#12 0x00007f90bcb702e2 in gst_pad_push_data (pad=pad@entry=0x7f77dc253d30, type=type@entry=4112, data=data@entry=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstpad.c:4655
#13 0x00007f90bcb76da2 in gst_pad_push (pad=pad@entry=0x7f77dc253d30, buffer=buffer@entry=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstpad.c:4774
#14 0x00007f90bcb5b11b in gst_proxy_pad_chain_default (pad=<optimized out>, parent=<optimized out>, buffer=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstghostpad.c:127
#15 0x00007f90bcb702e2 in gst_pad_chain_data_unchecked (data=0x7f9050017b40, type=4112, pad=0x7f77dc261750) at ../subprojects/gstreamer/gst/gstpad.c:4399
#16 0x00007f90bcb702e2 in gst_pad_push_data (pad=pad@entry=0x7f77dc1f8560, type=type@entry=4112, data=data@entry=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstpad.c:4655
#17 0x00007f90bcb76da2 in gst_pad_push (pad=pad@entry=0x7f77dc1f8560, buffer=buffer@entry=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstpad.c:4774
#18 0x00007f90bcb5b11b in gst_proxy_pad_chain_default (pad=<optimized out>, parent=<optimized out>, buffer=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstghostpad.c:127
#19 0x00007f90cf9689ff in webkitMediaSrcChain(_GstPad*, _GstObject*, _GstBuffer*) (pad=0x7f77883333d0, parent=0x7f77dc1f8560, buffer=0x7f9050017b40) at /home/cht/igalia/sources/WebKit/Source/WebCore/platform/graphics/gstreamer/mse/WebKitMediaSourceGStreamer.cpp:250
#20 0x00007f90bcb702e2 in gst_pad_chain_data_unchecked (data=0x7f9050017b40, type=4112, pad=0x7f77883333d0) at ../subprojects/gstreamer/gst/gstpad.c:4399
#21 0x00007f90bcb702e2 in gst_pad_push_data (pad=pad@entry=0x7f772835c760, type=type@entry=4112, data=data@entry=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstpad.c:4655
#22 0x00007f90bcb76da2 in gst_pad_push (pad=pad@entry=0x7f772835c760, buffer=0x7f9050017b40) at ../subprojects/gstreamer/gst/gstpad.c:4774
#23 0x00007f90bcc767fd in gst_base_src_loop (pad=0x7f772835c760) at ../subprojects/gstreamer/libs/gst/base/gstbasesrc.c:2974
#24 0x00007f90bcba4e11 in gst_task_func (task=0x7f7780265dd0) at ../subprojects/gstreamer/gst/gsttask.c:328


[1] https://www.gnupg.org/documentation/manuals/gcrypt-devel/Multi_002dThreading.html#Multi_002dThreading

[Lauro Moura]

This segfault also happened once for imported/w3c/web-platform-tests/encrypted-media/clearkey-mp4-playback-temporary-multikey.https.html in build 14196[1] (r263334) of the GTK Release bot.

It was the first failure for this test ever, so I think we can just keep an eye on it and add it as a flaky failure to this bug if it repeats in the future.

[1] https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/14196

[Xabier Rodríguez Calvar]

I tried to reproduce this bug in trunk both in release and debug, with both tests reported here and I can't reproduce it.

[Philippe Normand]

FWIW you could download coredumps from the bots and debug locally from there. If this crashes on Release only and not Debug it might be tricky to debug, but could still be worth a try.

[Philippe Normand]

For reference, as crash logs urls are not permanent:


Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f7768caa585 in _gcry_cipher_setctr (hd=hd@entry=0x0, ctr=ctr@entry=0x7f76f9efe160, ctrlen=ctrlen@entry=16) at cipher.c:1185
1185	  if (ctr && ctrlen == hd->spec->blocksize)

Thread 1 (Thread 0x7f76f9eff700 (LWP 88455)):
#0  0x00007f7768caa585 in _gcry_cipher_setctr (hd=hd@entry=0x0, ctr=ctr@entry=0x7f76f9efe160, ctrlen=ctrlen@entry=16) at cipher.c:1185
#1  0x00007f7768c9e362 in gcry_cipher_setctr (hd=0x0, ctr=0x7f76f9efe160, ctrlen=16) at visibility.c:748
#2  0x00007f776f4ca48c in _ZN7WebCore16CDMProxyClearKey20cencSetCounterVectorERKNS0_18cencDecryptContextE () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#3  0x00007f776f4ca782 in _ZN7WebCore16CDMProxyClearKey11cencDecryptERNS0_18cencDecryptContextE () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#4  0x00007f776f4c07c4 in _ZL7decryptP35_WebKitMediaCommonEncryptionDecryptP10_GstBufferS2_S2_jS2_ () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#5  0x00007f776f4c1dd6 in _ZL16transformInPlaceP17_GstBaseTransformP10_GstBuffer () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#6  0x00007f7768c46bf6 in default_generate_output (trans=0x7f75ac016290 [WebKitMediaClearKeyDecrypt|webkitmediaclearkeydecrypt7], outbuf=0x7f76f9efe3f0) at ../libs/gst/base/gstbasetransform.c:2136
#7  0x00007f7768c462f6 in gst_base_transform_chain (pad=<optimized out>, parent=0x7f75ac016290 [WebKitMediaClearKeyDecrypt|webkitmediaclearkeydecrypt7], buffer=<optimized out>) at ../libs/gst/base/gstbasetransform.c:2294
#8  0x00007f7768b3c43f in gst_pad_chain_data_unchecked (pad=pad@entry=0x7f75980e3380 [GstPad|sink], type=type@entry=4112, data=data@entry=0x7f75980bfd80) at ../gst/gstpad.c:4327
#9  0x00007f7768b3e5a1 in gst_pad_push_data (pad=pad@entry=0x7f75980e3130 [GstPad|src], type=type@entry=4112, data=data@entry=0x7f75980bfd80) at ../gst/gstpad.c:4583
#10 0x00007f7768b45493 in gst_pad_push (pad=0x7f75980e3130 [GstPad|src], buffer=0x7f75980bfd80 [GstBuffer]) at ../gst/gstpad.c:4702
#11 0x00007f7768b3c43f in gst_pad_chain_data_unchecked (pad=pad@entry=0x7f75980e2ee0 [GstPad|sink], type=type@entry=4112, data=data@entry=0x7f75980bfd80) at ../gst/gstpad.c:4327
#12 0x00007f7768b3e5a1 in gst_pad_push_data (pad=pad@entry=0x7f75b80f6340 [GstProxyPad|proxypad201], type=type@entry=4112, data=data@entry=0x7f75980bfd80) at ../gst/gstpad.c:4583
#13 0x00007f7768b45493 in gst_pad_push (pad=pad@entry=0x7f75b80f6340 [GstProxyPad|proxypad201], buffer=buffer@entry=0x7f75980bfd80 [GstBuffer]) at ../gst/gstpad.c:4702
#14 0x00007f7768b2969b in gst_proxy_pad_chain_default (pad=<optimized out>, parent=<optimized out>, buffer=0x7f75980bfd80 [GstBuffer]) at ../gst/gstghostpad.c:127
#15 0x00007f7768b3c43f in gst_pad_chain_data_unchecked (pad=pad@entry=0x7f7620152130 [GstGhostPad|sink], type=type@entry=4112, data=data@entry=0x7f75980bfd80) at ../gst/gstpad.c:4327
#16 0x00007f7768b3e5a1 in gst_pad_push_data (pad=pad@entry=0x7f7620153530 [GstGhostPad|src_1], type=type@entry=4112, data=data@entry=0x7f75980bfd80) at ../gst/gstpad.c:4583
#17 0x00007f7768b45493 in gst_pad_push (pad=pad@entry=0x7f7620153530 [GstGhostPad|src_1], buffer=buffer@entry=0x7f75980bfd80 [GstBuffer]) at ../gst/gstpad.c:4702
#18 0x00007f7768b2969b in gst_proxy_pad_chain_default (pad=<optimized out>, parent=<optimized out>, buffer=0x7f75980bfd80 [GstBuffer]) at ../gst/gstghostpad.c:127
#19 0x00007f776f4c4a69 in _ZL19webkitMediaSrcChainP7_GstPadP10_GstObjectP10_GstBuffer () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#20 0x00007f7768b3c43f in gst_pad_chain_data_unchecked (pad=pad@entry=0x7f7620009840 [GstProxyPad|proxypad200], type=type@entry=4112, data=data@entry=0x7f75980bfd80) at ../gst/gstpad.c:4327
#21 0x00007f7768b3e5a1 in gst_pad_push_data (pad=pad@entry=0x7f75b80f0a80 [GstPad|src], type=type@entry=4112, data=data@entry=0x7f75980bfd80) at ../gst/gstpad.c:4583
#22 0x00007f7768b45493 in gst_pad_push (pad=pad@entry=0x7f75b80f0a80 [GstPad|src], buffer=0x7f75980bfd80 [GstBuffer]) at ../gst/gstpad.c:4702
#23 0x00007f7768c41885 in gst_base_src_loop (pad=0x7f75b80f0a80 [GstPad|src]) at ../libs/gst/base/gstbasesrc.c:2974
#24 0x00007f7768b73be7 in gst_task_func (task=0x7f75b4314290 [GstTask|appsrc35:src]) at ../gst/gsttask.c:328
#25 0x00007f7768246004 in g_thread_pool_thread_proxy (data=<optimized out>) at ../glib/gthreadpool.c:354
#26 0x00007f7768245761 in g_thread_proxy (data=0x7f75ac007700) at ../glib/gthread.c:807
#27 0x00007f77692985e2 in start_thread (arg=<optimized out>) at pthread_create.c:479
#28 0x00007f7766f29473 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Xabier Rodríguez Calvar]

Created attachment 403770 [details]
Patch

(In reply to Xabier Rodríguez Calvar from comment #2)
> I tried to reproduce this bug in trunk both in release and debug, with both
> tests reported here and I can't reproduce it.

I tried again, 1000 runs both tests, both debug and release. All clear.

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fencrypted-media%2Fclearkey-mp4-playback-temporary-multikey.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fencrypted-media%2Fclearkey-events-session-closed-event.https.html

As you can see, results.webkit.org shows tests are green for a long time so I'm closing this if you agree. If you do, please r+ cq+.

[EWS]

Committed r264097: <https://trac.webkit.org/changeset/264097>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 403770 [details].