I have been debugging this: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Debug%20%28Tests%29/builds/1791/steps/layout-test/logs/stdio It looks like the server ran out of disk space, but in reality there is plenty of free space available. It seems it ran out of semaphores!! $ strace -f /usr/sbin/apache2 -f "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/layout-test-results/httpd.conf" -C 'DocumentRoot "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/http/tests"' -c 'TypesConfig "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/http/conf/mime.types"' -c 'PHPINIDir "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/http/conf"' -c 'CustomLog "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/layout-test-results/access_log.txt" common' -c 'ErrorLog "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/layout-test-results/error_log.txt"' -c 'PidFile /tmp/WebKit/httpd.pid' -k start -c 'Alias /js-test-resources "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/resources"' -c 'Alias /media-resources "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/media"' -c 'Alias /modern-media-controls "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/../Source/WebCore/Modules/modern-media-controls"' -c 'Alias /resources/testharness.css "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/resources/testharness.css"' -c 'Alias /resources/testharness.js "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/resources/testharness.js"' -c 'Alias /resources/testharnessreport.js "/home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/resources/testharnessreport.js"' -C 'User "slave"' -C 'Listen 127.0.0.1:8000' -C 'Listen [::1]:8000' -C 'Listen 127.0.0.1:8080' -C 'Listen [::1]:8080' -C 'Listen 127.0.0.1:8443' -C 'Listen [::1]:8443' -c 'SSLCertificateFile /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/LayoutTests/http/conf/webkit-httpd.pem'"" 2>&1 | grep ENOSPC semget(IPC_PRIVATE, 1, IPC_CREAT|0600) = -1 ENOSPC (No space left on device) $ ipcs -s|wc -l 132 Fixed it by manually running: $ ipcrm --all=sem So I think we should run this command as part of the kill-old-process script on the bots Related: https://www.liquidweb.com/kb/apache-error-semget-no-space-left-on-device/
(In reply to Carlos Alberto Lopez Perez from comment #0) > I have been debugging this: > https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Debug%20%28Tests%29/ > builds/1791/steps/layout-test/logs/stdio > > It looks like the server ran out of disk space, but in reality there is > plenty of free space available. > > It seems it ran out of semaphores!! > [...] > semget(IPC_PRIVATE, 1, IPC_CREAT|0600) = -1 ENOSPC (No space left on device) For a split second I was about to comment that we shouldn't let WebKit leak semaphores, and then I noticed that Apache is to blame here 8-) > $ ipcs -s|wc -l > 132 > > Fixed it by manually running: > $ ipcrm --all=sem > > > So I think we should run this command as part of the kill-old-process script > on the bots What if there is something else that uses semaphores running in the system? I suppose it's fine in most cases — for example for the GTK+ and WPE bots we run them in containers and it would affect only the semaphores which belong to the container, but it wouldn't hurt removing only the semaphores which belong to the current user for extra safety (the “ipcs” command lists the owners).