Bug 216068 - [GLIB] kill-old-processes: Remove stale Apache2 shm segments
Summary: [GLIB] kill-old-processes: Remove stale Apache2 shm segments
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Diego Pino
URL:
Keywords: InRadar
Depends on: 216135
Blocks:
  Show dependency treegraph
 
Reported: 2020-09-01 23:09 PDT by Diego Pino
Modified: 2020-09-09 13:44 PDT (History)
6 users (show)

See Also:


Attachments
Patch (3.81 KB, patch)
2020-09-01 23:35 PDT, Diego Pino
no flags Details | Formatted Diff | Diff
Patch (3.45 KB, patch)
2020-09-01 23:53 PDT, Diego Pino
no flags Details | Formatted Diff | Diff
Patch (3.48 KB, patch)
2020-09-02 14:18 PDT, Diego Pino
no flags Details | Formatted Diff | Diff
Patch (3.32 KB, patch)
2020-09-03 20:51 PDT, Diego Pino
no flags Details | Formatted Diff | Diff
Refactored patch. (4.03 KB, patch)
2020-09-08 14:00 PDT, Lauro Moura
no flags Details | Formatted Diff | Diff
Patch for landing (3.98 KB, patch)
2020-09-08 21:30 PDT, Lauro Moura
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Diego Pino 2020-09-01 23:09:34 PDT
kill-old-processes: Remove stale Apache shm segments
Comment 1 Diego Pino 2020-09-01 23:35:13 PDT
Created attachment 407742 [details]
Patch
Comment 2 Diego Pino 2020-09-01 23:47:18 PDT
This issue was happening in several EWS workers running in the UAT. For instance:

https://ews-build.webkit-uat.org/#/builders/34/builds/4926/steps/10/logs/stdio

Apache2 doesn't start because it believes there's already an Apache2 instance running, when actually what's happening is that there's an orphan shm segment which pid matches the pid of the new Apache2 instance. In that case layout-test-results/error-log shows an error like this:

[Sat Aug 29 20:02:16.013912 2020] [core:crit] [pid 26] (17)File exists: AH00001: unable to create or access scoreboard "/tmp/WebKit/httpd.scoreboard" (name-based shared memory failure)

And here's a post with the description of the problem and solution:
https://makandracards.com/fabio-silva/49967-apache-unable-to-create-scoreboard-shared-memory-failure
Comment 3 Diego Pino 2020-09-01 23:53:01 PDT
Created attachment 407744 [details]
Patch
Comment 4 Alexey Proskuryakov 2020-09-02 10:31:14 PDT
Comment on attachment 407744 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=407744&action=review

> Tools/BuildSlaveSupport/kill-old-processes:27
> +import subprocess as sp

Why rename the module? WebKit coding style is to avoid abbreviations.
Comment 5 Diego Pino 2020-09-02 14:18:04 PDT
Created attachment 407812 [details]
Patch
Comment 6 Diego Pino 2020-09-02 14:20:24 PDT
(In reply to Alexey Proskuryakov from comment #4)
> Comment on attachment 407744 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=407744&action=review
> 
> > Tools/BuildSlaveSupport/kill-old-processes:27
> > +import subprocess as sp
> 
> Why rename the module? WebKit coding style is to avoid abbreviations.

Sorry, I didn't know about this code convention. I pushed a new patch undoing the renaming.
Comment 7 EWS 2020-09-03 01:07:53 PDT
Committed r266507: <https://trac.webkit.org/changeset/266507>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 407812 [details].
Comment 8 Radar WebKit Bug Importer 2020-09-03 01:08:13 PDT
<rdar://problem/68262415>
Comment 9 Truitt Savell 2020-09-03 11:15:01 PDT
It looks like the changes in https://trac.webkit.org/changeset/266507/webkit 

broke our JSC EWS bot:
https://ews-build.webkit.org/#/builders/38/builds/1408

We will need to roll this out unless it can be fixed very quickly.


Traceback (most recent call last):
  File "Tools/BuildSlaveSupport/kill-old-processes", line 192, in <module>
    sys.exit(main(user))
  File "Tools/BuildSlaveSupport/kill-old-processes", line 184, in main
    removeOrphanShmSegmentsOf("apache2")
  File "Tools/BuildSlaveSupport/kill-old-processes", line 39, in removeOrphanShmSegmentsOf
    pid = futurePidOf(task)
  File "Tools/BuildSlaveSupport/kill-old-processes", line 56, in futurePidOf
    ps = subprocess.Popen(task, stderr=subprocess.STDOUT, stdout=devnull)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Comment 10 WebKit Commit Bot 2020-09-03 11:40:46 PDT
Re-opened since this is blocked by bug 216135
Comment 11 Diego Pino 2020-09-03 20:51:38 PDT
Created attachment 407941 [details]
Patch
Comment 12 Lauro Moura 2020-09-08 14:00:48 PDT
Created attachment 408266 [details]
Refactored patch.
Comment 13 EWS 2020-09-08 20:55:35 PDT
ChangeLog entry in Tools/ChangeLog is not at the top of the file.
Comment 14 Lauro Moura 2020-09-08 21:30:52 PDT
Created attachment 408309 [details]
Patch for landing
Comment 15 EWS 2020-09-08 22:14:20 PDT
Committed r266775: <https://trac.webkit.org/changeset/266775>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 408309 [details].