Summary: | W3C Web Platform Tests occasionally flake on the bots (Web Platform Test server fails?) | ||||||
---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | Brady Eidson <beidson> | ||||
Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> | ||||
Status: | RESOLVED FIXED | ||||||
Severity: | Normal | CC: | ap, cdumez, gsnedders, lforschler, ryanhaddad, youennf | ||||
Priority: | P2 | ||||||
Version: | WebKit Nightly Build | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
See Also: | https://bugs.webkit.org/show_bug.cgi?id=158183 | ||||||
Bug Depends on: | 158253 | ||||||
Bug Blocks: | |||||||
Attachments: |
|
Description
Brady Eidson
2016-05-26 16:57:15 PDT
Are you sure this is what's happening? If you look at the attachment at: https://bugs.webkit.org/attachment.cgi?id=279916 There is a wptwk_process_log.out.txt file which seems to be the log file for the W3C Web Platform test server. It is pretty big and mostly 200 Found lines. (In reply to comment #1) > Are you sure this is what's happening? No. But it's the only theory we have so far. I updated the bug summary. Created attachment 279997 [details] wptwk process log from ews103 Associated EWS log: https://webkit-queues.webkit.org/results/1393177 The error seems indeed located in wpt server. Looking at the log, the server starts at some point to return 404 to URLs pointing to files that should exist (resources/testharnessreport.js, existing IDB test files). The request handler is FileHandler (https://github.com/w3c/wptserve/blob/22c2f947fafa07ce26d381bf5215714067c41ae6/wptserve/handlers.py) Whenever this part of code is raising OSError or IOError, it is converted to a 404 HTTP response. Logging these errors may help getting additional information. I can prepare a patch to add that logging, something like "self.logger.error(traceback.format_exc())" after line 148 in "handlers.py" (In reply to comment #4) > The error seems indeed located in wpt server. > > Looking at the log, the server starts at some point to return 404 to URLs > pointing to files that should exist (resources/testharnessreport.js, > existing IDB test files). > > The request handler is FileHandler > (https://github.com/w3c/wptserve/blob/ > 22c2f947fafa07ce26d381bf5215714067c41ae6/wptserve/handlers.py) > Whenever this part of code is raising OSError or IOError, it is converted to > a 404 HTTP response. > > Logging these errors may help getting additional information. > I can prepare a patch to add that logging, something like > "self.logger.error(traceback.format_exc())" after line 148 in "handlers.py" Filed bug 158183 for that purpose. I am also wondering whether this issue happens for all bots. It seems that wptserver cannot serve some files due to too many opened files at the same time Here is a log excerpt found as part of a mac bot processing a bug 158222 patch. DEBUG:web-platform-tests:Traceback (most recent call last): File "/Volumes/Data/EWS/WebKit/LayoutTests/imported/w3c/web-platform-tests/tools/wptserve/wptserve/handlers.py", line 133, in __call__ data = self.get_data(response, path, byte_ranges) File "/Volumes/Data/EWS/WebKit/LayoutTests/imported/w3c/web-platform-tests/tools/wptserve/wptserve/handlers.py", line 183, in get_data return open(path, 'rb') IOError: [Errno 24] Too many open files: '/Volumes/Data/EWS/WebKit/LayoutTests/imported/w3c/web-platform-tests/resources/testharness.js' We used to sometimes have runaway DumpRenderTree processes, which would cause running out of system resources, but this doesn't seem to be the case on any of the bots hitting this issue that I checked. Perhaps wtpserve itself leaks file handles? (In reply to comment #7) > We used to sometimes have runaway DumpRenderTree processes, which would > cause running out of system resources, but this doesn't seem to be the case > on any of the bots hitting this issue that I checked. Perhaps wtpserve > itself leaks file handles? It might be the case that wptserve is relying on garbage collector to close the file handles. AFAIS, only mac ews yosemite bots are hitting this error. What is the max number of file handles for these bots? It's the default (and since there are multiple limits at play, there is no simple answer). We can't change it anyway, as tests need to work on engineers' machines too. I filed https://bugs.webkit.org/show_bug.cgi?id=158253 to close file handles explicitly as a temporary patch. I also filed https://github.com/w3c/wptserve/pull/83 so that it gets handled directly in w3c wptserve. Is the problem still happening? I haven't seen it after the patch landed. The above seemingly fixed it? |