Bug 216536

Summary:

Add further temporary debug logging for wptserve

Product:

WebKit

Reporter:

Sam Sneddon [:gsnedders] <gsnedders>

Component:

Tools / Tests

Assignee:

Sam Sneddon [:gsnedders] <gsnedders>

Status:

RESOLVED WONTFIX

Severity:

Normal

CC:

aakash_jain, ap, clopez, ews-watchlist, youennf

Priority:

Version:

WebKit Nightly Build

Hardware:

Unspecified

OS:

Unspecified

See Also:

https://bugs.webkit.org/show_bug.cgi?id=215829

Attachments:

Description	Flags
Patch	none
Patch	none
Patch	none
Patch	aakash_jain: review-, ews-feeder: commit-queue-

Sam Sneddon [:gsnedders]

Reported 2020-09-15 06:06:11 PDT

This just checks what processes and open files we have, to try and see why we're running into "too many open files"

Attachments
Patch (1.69 KB, patch) 2020-09-15 06:11 PDT, Sam Sneddon [:gsnedders]	no flags	Details Formatted Diff Diff
Patch (1.70 KB, patch) 2020-09-15 09:15 PDT, Sam Sneddon [:gsnedders]	no flags	Details Formatted Diff Diff
Patch (1.76 KB, patch) 2020-09-15 11:06 PDT, Sam Sneddon [:gsnedders]	no flags	Details Formatted Diff Diff
Patch (1.82 KB, patch) 2020-09-15 11:30 PDT, Sam Sneddon [:gsnedders]	aakash_jain: review- ews-feeder: commit-queue-	Details Formatted Diff Diff
Show Obsolete (3) View All Add attachment proposed patch, testcase, etc.

Sam Sneddon [:gsnedders]

Comment 1 2020-09-15 06:11:24 PDT

Created attachment 408814 [details] Patch

EWS Watchlist

Comment 2 2020-09-15 06:12:29 PDT

This patch modifies the imported WPT tests. Please ensure that any changes on the tests (not coming from a WPT import) are exported to WPT. Please see https://trac.webkit.org/wiki/WPTExportProcess

Aakash Jain

Comment 3 2020-09-15 08:15:18 PDT

Comment on attachment 408814 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=408814&action=review > LayoutTests/imported/w3c/web-platform-tests/tools/wptserve/wptserve/response.py:254 > + self.logger.debug(subprocess.check_output(["ps", "aux"]) missing parenthesis

Sam Sneddon [:gsnedders]

Comment 4 2020-09-15 09:15:52 PDT

Created attachment 408824 [details] Patch

Aakash Jain

Comment 5 2020-09-15 09:43:27 PDT

Where are we running into "too many open files"? Any bug/radar with the details?

Sam Sneddon [:gsnedders]

Comment 6 2020-09-15 09:50:24 PDT

See the see also bug: 215829 :)

Sam Sneddon [:gsnedders]

Comment 7 2020-09-15 11:06:43 PDT

Created attachment 408834 [details] Patch

Sam Sneddon [:gsnedders]

Comment 8 2020-09-15 11:30:59 PDT

Created attachment 408838 [details] Patch

Sam Sneddon [:gsnedders]

Comment 9 2020-09-15 14:25:54 PDT

This entire approach is doomed to failure, because lsof can block and make everything worse.

Aakash Jain

Comment 10 2020-09-15 14:27:07 PDT

Comment on attachment 408838 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=408838&action=review > LayoutTests/imported/w3c/web-platform-tests/tools/wptserve/wptserve/response.py:257 > + self.logger.debug(subprocess.check_output(["ps", "aux"])) This ('ps aux' and 'lsof -P +c0') will produce huge amount of output (tens of thousands of lines). Are you sure you want to add that much output in the logs? Maybe you just need a count of number of lines of these. Also, when the machine is already running out of file handles, these commands might just slow down/hang the machine further. This might explain why https://ews-build.webkit.org/#/builders/30/builds/17561/steps/11/logs/stdio hanged with this patch.

Alexey Proskuryakov

Comment 11 2020-09-16 16:40:18 PDT

We can manually run a job on one of the machines dumping lsof every few seconds, this way we could catch it right before the condition occurs. As long as it's not a super quick explosion, that could be good enough.

Note You need to log in before you can comment on or make changes to this bug.