Bug 178149 - [GTK] Debug test bot ran out of semaphores
Summary: [GTK] Debug test bot ran out of semaphores
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Local Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-10-10 16:56 PDT by Carlos Alberto Lopez Perez
Modified: 2017-10-11 00:12 PDT (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Carlos Alberto Lopez Perez 2017-10-10 16:56:17 PDT
I have been debugging this: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Debug%20%28Tests%29/builds/1791/steps/layout-test/logs/stdio

It looks like the server ran out of disk space, but in reality there is plenty of free space available.

It seems it ran out of semaphores!!

$ strace -f /usr/sbin/apache2 -f "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/layout-test-results/httpd.conf" -C 'DocumentRoot "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/http/tests"' -c 'TypesConfig "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/http/conf/mime.types"' -c 'PHPINIDir "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/http/conf"' -c 'CustomLog "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/layout-test-results/access_log.txt" common' -c 'ErrorLog "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/layout-test-results/error_log.txt"' -c 'PidFile /tmp/WebKit/httpd.pid' -k start -c 'Alias /js-test-resources "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/resources"' -c 'Alias /media-resources "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/media"' -c 'Alias /modern-media-controls "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/../Source/WebCore/Modules/modern-media-controls"' -c 'Alias /resources/testharness.css "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/resources/testharness.css"' -c 'Alias /resources/testharness.js "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/resources/testharness.js"' -c 'Alias /resources/testharnessreport.js "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/resources/testharnessreport.js"' -C 'User "slave"' -C 'Listen 127.0.0.1:8000' -C 'Listen [::1]:8000' -C 'Listen 127.0.0.1:8080' -C 'Listen [::1]:8080' -C 'Listen 127.0.0.1:8443' -C 'Listen [::1]:8443' -c 'SSLCertificateFile /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/http/conf/webkit-httpd.pem'"" 2>&1 | grep ENOSPC
semget(IPC_PRIVATE, 1, IPC_CREAT|0600)  = -1 ENOSPC (No space left on device)


$ ipcs -s|wc -l
132

Fixed it by manually running:
$ ipcrm --all=sem


So I think we should run this command as part of the kill-old-process script on the bots


Related: https://www.liquidweb.com/kb/apache-error-semget-no-space-left-on-device/
Comment 1 Adrian Perez 2017-10-11 00:12:28 PDT
(In reply to Carlos Alberto Lopez Perez from comment #0)
> I have been debugging this:
> https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Debug%20%28Tests%29/
> builds/1791/steps/layout-test/logs/stdio
> 
> It looks like the server ran out of disk space, but in reality there is
> plenty of free space available.
> 
> It seems it ran out of semaphores!!
> [...]
> semget(IPC_PRIVATE, 1, IPC_CREAT|0600)  = -1 ENOSPC (No space left on device)

For a split second I was about to comment that we shouldn't let WebKit
leak semaphores, and then I noticed that Apache is to blame here  8-)

> $ ipcs -s|wc -l
> 132
> 
> Fixed it by manually running:
> $ ipcrm --all=sem
> 
> 
> So I think we should run this command as part of the kill-old-process script
> on the bots

What if there is something else that uses semaphores running in the system?
I suppose it's fine in most cases — for example for the GTK+ and WPE bots we
run them in containers and it would affect only the semaphores which belong
to the container, but it wouldn't hurt removing only the semaphores which
belong to the current user for extra safety (the “ipcs” command lists the
owners).