Bug 199355 – [SOUP] WebSocket crashes

• Status: RESOLVED
• Resolution: FIXED
• Component: Platform

[Philippe Normand]

In the WPE release test bot:
https://build.webkit.org/results/WPE%20Linux%2064-bit%20Release%20(Tests)/r246965%20(14222)/results.html

Regressions: Unexpected crashes (17)
  fast/mediastream/RTCPeerConnection-media-setup-single-dialog.html [ Crash ]
  fast/mediastream/change-tracks-media-stream-being-played.html [ Crash ]
  http/tests/adClickAttribution/attribution-conversion-through-image-redirect-with-priority.html [ Crash ]
  http/tests/adClickAttribution/second-attribution-converted-with-higher-priority.html [ Crash ]
  http/tests/adClickAttribution/second-attribution-converted-with-lower-priority.html [ Crash ]
  http/tests/adClickAttribution/store-ad-click-attribution.html [ Crash ]
  imported/w3c/web-platform-tests/cors/redirect-preflight.htm [ Crash ]
  imported/w3c/web-platform-tests/fetch/api/policies/referrer-origin-when-cross-origin-worker.html [ Crash ]
  imported/w3c/web-platform-tests/fetch/security/embedded-credentials.tentative.sub.html [ Crash ]
  imported/w3c/web-platform-tests/html/dom/interfaces.worker.html [ Crash ]
  imported/w3c/web-platform-tests/websockets/constructor/009.html [ Crash ]
  imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-defineProperty-setter.html [ Crash ]
  imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/extensions/001.html [ Crash ]
  imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/readyState/005.html [ Crash ]
  imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/006.html [ Crash ]
  imported/w3c/web-platform-tests/websockets/opening-handshake/005.html [ Crash ]
  imported/w3c/web-platform-tests/xhr/send-redirect-bogus-sync.htm [ Crash ]

For instance imported/w3c/web-platform-tests/xhr/send-redirect-bogus-sync.htm:

#0  0x00007fe8c9bfcb97 in g_param_spec_get_name_quark (pspec=pspec@entry=0x1d) at ../../Source/glib-2.58.1/gobject/gparam.c:1593
1593	  return priv->name_quark;
Thread 1 (Thread 0x7fe8c810f9c0 (LWP 20592)):
#0  0x00007fe8c9bfcb97 in g_param_spec_get_name_quark (pspec=pspec@entry=0x1d) at ../../Source/glib-2.58.1/gobject/gparam.c:1593
#1  0x00007fe8c9bf414f in g_object_dispatch_properties_changed (object=0x55c350014e10, n_pspecs=<optimized out>, pspecs=<optimized out>) at ../../Source/glib-2.58.1/gobject/gobject.c:1088
#2  0x00007fe8c9bf3a6e in g_object_notify_queue_thaw (object=object@entry=0x55c350014e10, nqueue=<optimized out>) at ../../Source/glib-2.58.1/gobject/gobject.c:296
#3  0x00007fe8c9bf5754 in g_object_new_internal (class=class@entry=0x55c3503825c0, params=params@entry=0x7ffdc0497120, n_params=n_params@entry=2) at ../../Source/glib-2.58.1/gobject/gobject.c:1862
#4  0x00007fe8c9bf7194 in g_object_new_valist (object_type=<optimized out>, first_property_name=first_property_name@entry=0x7fe8c9e81a3e "connectable", var_args=var_args@entry=0x7ffdc0497268) at ../../Source/glib-2.58.1/gobject/gobject.c:2128
#5  0x00007fe8c9bf74bc in g_object_new (object_type=<optimized out>, first_property_name=first_property_name@entry=0x7fe8c9e81a3e "connectable") at ../../Source/glib-2.58.1/gobject/gobject.c:1648
#6  0x00007fe8c9e3cf37 in soup_address_connectable_proxy_enumerate (connectable=0x7fe84c0023a0) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-address.c:1276
#7  0x00007fe8c9ccde5f in g_socket_client_connect_async (client=0x7fe83c001840, connectable=0x7fe84c0023a0, cancellable=0x55c350538f30, callback=0x7fe8c9e72240 <async_connected>, user_data=0x55c3503fa9e0) at ../../Source/glib-2.58.1/gio/gsocketclient.c:1669
#8  0x00007fe8c9e73955 in soup_socket_connect_async_internal (sock=<optimized out>, cancellable=<optimized out>, callback=<optimized out>, user_data=0x55c3505722c0) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-socket.c:957
#9  0x00007fe8c9e4bbb2 in soup_connection_connect_async (conn=0x55c350396870, cancellable=0x55c350538f30, callback=0x7fe8c9e6fe10 <connect_async_complete>, user_data=0x55c350532600) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-connection.c:418
#10 0x00007fe8c9e6f80e in get_connection (should_cleanup=<optimized out>, item=0x55c350532600) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-session.c:1950
#11 soup_session_process_queue_item (loop=<optimized out>, should_cleanup=<optimized out>, item=<optimized out>, session=<optimized out>) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-session.c:1977
#12 soup_session_process_queue_item (session=<optimized out>, item=0x55c350532600, should_cleanup=<optimized out>, loop=<optimized out>) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-session.c:1964
#13 0x00007fe8c9e6fd7a in async_run_queue (session=session@entry=0x55c35003e220) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-session.c:2082
#14 0x00007fe8c9e6fdf6 in idle_run_queue (user_data=<optimized out>) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-session.c:2109
#15 0x00007fe8c9b0f6b8 in g_main_dispatch (context=0x55c35000d5b0) at ../../Source/glib-2.58.1/glib/gmain.c:3182
#16 g_main_context_dispatch (context=context@entry=0x55c35000d5b0) at ../../Source/glib-2.58.1/glib/gmain.c:3847
#17 0x00007fe8c9b0fa78 in g_main_context_iterate (context=0x55c35000d5b0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../../Source/glib-2.58.1/glib/gmain.c:3920
#18 0x00007fe8c9b0fd62 in g_main_loop_run (loop=0x55c35000e7b0) at ../../Source/glib-2.58.1/glib/gmain.c:4116
#19 0x00007fe8d20c3520 in WTF::RunLoop::run() () from /home/buildbot/wpe/wpe-linux-64-release/build/WebKitBuild/Release/lib/libWPEWebKit-1.0.so.3
#20 0x00007fe8cf97e20a in int WebKit::AuxiliaryProcessMain<WebKit::NetworkProcess, WebKit::NetworkProcessMain>(int, char**) () from /home/buildbot/wpe/wpe-linux-64-release/build/WebKitBuild/Release/lib/libWPEWebKit-1.0.so.3
#21 0x00007fe8c93df09b in __libc_start_main (main=0x55c34ea46a60 <main>, argc=3, argv=0x7ffdc0497808, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdc04977f8) at ../csu/libc-start.c:308
#22 0x000055c34ea46aea in _start ()

imported/w3c/web-platform-tests/websockets/constructor/009.html:

#0  magazine_chain_pop_head (magazine_chunks=<optimized out>) at ../../Source/glib-2.58.1/glib/gslice.c:538
538	      (*magazine_chunks)->data = chunk->next;
[Current thread is 1 (Thread 0x7f9e3b6c19c0 (LWP 4979))]
Thread 1 (Thread 0x7f9e3b6c19c0 (LWP 4979)):
#0  magazine_chain_pop_head (magazine_chunks=<optimized out>) at ../../Source/glib-2.58.1/glib/gslice.c:538
#1  thread_memory_magazine1_alloc (tmem=<optimized out>, ix=0) at ../../Source/glib-2.58.1/glib/gslice.c:841
#2  g_slice_alloc (mem_size=mem_size@entry=16) at ../../Source/glib-2.58.1/glib/gslice.c:1015
#3  0x00007f9e3d0df6a6 in g_slist_prepend (list=0x0, data=data@entry=0x5625da449060) at ../../Source/glib-2.58.1/glib/gslist.c:259
#4  0x00007f9e3d1a650c in g_object_notify_queue_add (pspec=0x5625da449060, nqueue=<optimized out>, nqueue=<optimized out>, object=<optimized out>) at ../../Source/glib-2.58.1/gobject/gobject.c:311
#5  0x00007f9e3d1a9c72 in object_set_property (nqueue=0x7f9db8001800, value=0x7fff7be751b0, pspec=<optimized out>, object=0x5625da79b590) at ../../Source/glib-2.58.1/gobject/gobject.c:1456
#6  g_object_set_valist (object=object@entry=0x5625da79b590, first_property_name=first_property_name@entry=0x7f9e3d4368ab "tls-certificate", var_args=var_args@entry=0x7fff7be75278) at ../../Source/glib-2.58.1/gobject/gobject.c:2308
#7  0x00007f9e3d1aa4bf in g_object_set (_object=_object@entry=0x5625da79b590, first_property_name=first_property_name@entry=0x7f9e3d4368ab "tls-certificate") at ../../Source/glib-2.58.1/gobject/gobject.c:2473
#8  0x00007f9e3d40e0da in soup_message_set_https_status (msg=0x5625da79b590, conn=<optimized out>) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-message.c:1914
#9  0x00007f9e3d41dc6e in connect_complete (item=0x5625da44ad10, conn=0x5625da794710, error=0x7f9db4001880) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-session.c:1750
#10 0x00007f9e3d421e5d in connect_async_complete (object=<optimized out>, result=0x5625da7b0680, user_data=user_data@entry=0x5625da44ad10) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-session.c:1778
#11 0x00007f9e3d2897a9 in g_task_return_now (task=0x5625da7b0680) at ../../Source/glib-2.58.1/gio/gtask.c:1148
#12 0x00007f9e3d28a226 in g_task_return (task=0x5625da7b0680, type=<optimized out>) at ../../Source/glib-2.58.1/gio/gtask.c:1206
#13 0x00007f9e3d3fdbef in socket_connect_finished (task=0x5625da7b0680, error=0x7f9db4001880, sock=<optimized out>) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-connection.c:341
#14 0x00007f9e3d3fdd1d in socket_connect_complete (object=<optimized out>, result=0x7f9df00046f0, user_data=user_data@entry=0x5625da7b0680) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-connection.c:366
#15 0x00007f9e3d2897a9 in g_task_return_now (task=0x7f9df00046f0) at ../../Source/glib-2.58.1/gio/gtask.c:1148
#16 0x00007f9e3d28a226 in g_task_return (task=0x7f9df00046f0, type=<optimized out>) at ../../Source/glib-2.58.1/gio/gtask.c:1206
#17 0x00007f9e3d4242cd in async_connected (client=0x5625da786c40, result=0x5625da7ae6e0, data=data@entry=0x7f9df00046f0) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-socket.c:925
#18 0x00007f9e3d2897a9 in g_task_return_now (task=0x5625da7ae6e0) at ../../Source/glib-2.58.1/gio/gtask.c:1148
#19 0x00007f9e3d28a226 in g_task_return (task=0x5625da7ae6e0, type=<optimized out>) at ../../Source/glib-2.58.1/gio/gtask.c:1206
#20 0x00007f9e3d27f696 in g_socket_client_enumerator_callback (object=<optimized out>, result=<optimized out>, user_data=user_data@entry=0x7f9d98007a30) at ../../Source/glib-2.58.1/gio/gsocketclient.c:1602
#21 0x00007f9e3d2897a9 in g_task_return_now (task=0x5625da7ae540) at ../../Source/glib-2.58.1/gio/gtask.c:1148
#22 0x00007f9e3d28a226 in g_task_return (task=0x5625da7ae540, type=<optimized out>) at ../../Source/glib-2.58.1/gio/gtask.c:1206
#23 0x00007f9e3d270a35 in complete_async (task=0x5625da7ae540) at ../../Source/glib-2.58.1/gio/gproxyaddressenumerator.c:319
#24 0x00007f9e3d2897a9 in g_task_return_now (task=0x5625da781b40) at ../../Source/glib-2.58.1/gio/gtask.c:1148
#25 0x00007f9e3d28a226 in g_task_return (task=0x5625da781b40, type=<optimized out>) at ../../Source/glib-2.58.1/gio/gtask.c:1206
#26 0x00007f9e3d3ee375 in got_addresses (addr=<optimized out>, status=<optimized out>, user_data=0x5625da781b40) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-address.c:1184
#27 0x00007f9e3d3ee081 in complete_resolve_async (res_data=0x5625da767340, status=2) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-address.c:716
#28 0x00007f9e3d3ee4ac in lookup_resolved (source=<optimized out>, result=<optimized out>, user_data=user_data@entry=0x5625da767340) at /home/buildbot/wpe/wpe-linux-64-release-tests/build/WebKitBuild/DependenciesWPE/Source/libsoup-2.64.2/libsoup/soup-address.c:757
#29 0x00007f9e3d2897a9 in g_task_return_now (task=0x5625da7929d0) at ../../Source/glib-2.58.1/gio/gtask.c:1148
#30 0x00007f9e3d28a226 in g_task_return (task=0x5625da7929d0, type=<optimized out>) at ../../Source/glib-2.58.1/gio/gtask.c:1206
#31 0x00007f9e42f29995 in webkitCachedResolverLookupByNameAsync(_GResolver*, char const*, _GCancellable*, void (*)(_GObject*, _GAsyncResult*, void*), void*)::{lambda(_GObject*, _GAsyncResult*, void*)#1}::_FUN(_GObject*, _GAsyncResult*, void*) () from /home/buildbot/wpe/wpe-linux-64-release/build/WebKitBuild/Release/lib/libWPEWebKit-1.0.so.3
#32 0x00007f9e3d2897a9 in g_task_return_now (task=0x5625da7818d0) at ../../Source/glib-2.58.1/gio/gtask.c:1148
#33 0x00007f9e3d2897e9 in complete_in_idle_cb (task=0x5625da7818d0) at ../../Source/glib-2.58.1/gio/gtask.c:1162
#34 0x00007f9e3d0c16b8 in g_main_dispatch (context=0x5625da40d5b0) at ../../Source/glib-2.58.1/glib/gmain.c:3182
#35 g_main_context_dispatch (context=context@entry=0x5625da40d5b0) at ../../Source/glib-2.58.1/glib/gmain.c:3847
#36 0x00007f9e3d0c1a78 in g_main_context_iterate (context=0x5625da40d5b0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../../Source/glib-2.58.1/glib/gmain.c:3920
#37 0x00007f9e3d0c1d62 in g_main_loop_run (loop=0x5625da40e7b0) at ../../Source/glib-2.58.1/glib/gmain.c:4116
#38 0x00007f9e45675520 in WTF::RunLoop::run() () from /home/buildbot/wpe/wpe-linux-64-release/build/WebKitBuild/Release/lib/libWPEWebKit-1.0.so.3
#39 0x00007f9e42f3020a in int WebKit::AuxiliaryProcessMain<WebKit::NetworkProcess, WebKit::NetworkProcessMain>(int, char**) () from /home/buildbot/wpe/wpe-linux-64-release/build/WebKitBuild/Release/lib/libWPEWebKit-1.0.so.3
#40 0x00007f9e3c99109b in __libc_start_main (main=0x5625da206a60 <main>, argc=3, argv=0x7fff7be759f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff7be759e8) at ../csu/libc-start.c:308
#41 0x00005625da206aea in _start ()

[Michael Catanzaro]

The second backtrace is definitely memory corruption. We need to catch it in valgrind or asan to get a useful backtrace that would point to the issue.

I suspect the first backtrace is too, since that's a very strange place for a crash to occur, but I can't be sure.

[Carlos Garcia Campos]

Is this still happening? I think this could be due bug #199572 too.

[Michael Catanzaro]

(In reply to Carlos Garcia Campos from comment #2)
> Is this still happening? I think this could be due bug #199572 too.

Well it very well could be, since it's a memory corruption bug.

But that bug was introduced after the 2.25.2. Meanwhile, I'm experiencing regular UI process crashes due to memory corruption in 2.25.2, before that issue was introduced. Just hit two in the past hour. That means we have at least one more issue here. (I hope only one, because until recently such crashes were extremely rare in my experience.) The symptoms will be identical. Could be bug #199295, or something we don't know about yet.

[Michael Catanzaro]

Recent WPE results look good and I don't see any expectations added for these tests. Close?

[Carlos Garcia Campos]

(In reply to Michael Catanzaro from comment #3)
> (In reply to Carlos Garcia Campos from comment #2)
> > Is this still happening? I think this could be due bug #199572 too.
> 
> Well it very well could be, since it's a memory corruption bug.
> 
> But that bug was introduced after the 2.25.2. Meanwhile, I'm experiencing
> regular UI process crashes due to memory corruption in 2.25.2, before that
> issue was introduced. Just hit two in the past hour. That means we have at
> least one more issue here. (I hope only one, because until recently such
> crashes were extremely rare in my experience.) The symptoms will be
> identical. Could be bug #199295, or something we don't know about yet.

This is about network process crashes, nothing to do with UI process.