Bug 30869 – commit-queue keeps crashing

• Status: RESOLVED
• Resolution: DUPLICATE
• Component: Tools / Tests

[Eric Seidel (no email)]

commit-queue keeps crashing

Exception while checking queue and bots: unknown encoding: idna. Sleeping until 2009-10-27 23:07:56 (5 mins).
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/Users/eseidel/Projects/WebKit/WebKitTools/Scripts/bugzilla-tool", line 33, in <module>
    import os
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/os.py", line 49, in <module>
    import posixpath as path
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/posixpath.py", line 14, in <module>
    import stat
ImportError: No module named stat

I think that's happening on re-exec.  I think it may only happen when I have the patch from bug 30098 applied locally.  Either way, it's disturbing.  It's happened 3 times at seeming random.  The commit-queue used to be rock-solid!

[Eric Seidel (no email)]

This appears to be a python bug.  I've made a reduced test case.

[Eric Seidel (no email)]

Created attachment 42122 [details]
Script demonstrating python 2.5.1 (or mac os x?) bug

[Evan Martin]

On Linux (the "import site" bits are when I'm repeatedly hitting ctl-C).
Python 2.5.2


% ./t
...............................................................................'import site' failed; use -v for traceback
.........'import site' failed; use -v for traceback
............'import site' failed; use -v for traceback
...........'import site' failed; use -v for traceback
....................'import site' failed; use -v for traceback
..........

[Eric Seidel (no email)]

Bah.  I just don't know how exec works.  I didn't realize it carried file handles through.

[Eric Seidel (no email)]

Created attachment 42123 [details]
Patch v1

[Adam Barth]

Comment on attachment 42123 [details]
Patch v1

File handle leaks = bad.

[Eric Seidel (no email)]

Comment on attachment 42123 [details]
Patch v1

Going to make a better patch.

[Eric Seidel (no email)]

It would appear that even with my fix, Python itself is still leaking 3 file descriptors on every exec():

Python    57958 eseidel   58r     REG       14,2    144580    466329 /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/HITo
olbox.rsrc
Python    57958 eseidel   59r     REG       14,2    490410   9627854 /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/Engl
ish.lproj/Localized.rsrc
Python    57958 eseidel   60r     REG       14,2     86770    481232 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac/errors.rsrc.df.rsrc

I expect this may be a mac-only problem.

[Eric Seidel (no email)]

I wrote a function:

def check_for_file_leak(message):
    log("before %s" % message)
    lsof_output = SCM.run_command(['lsof', '-p', str(os.getpid())])
    if lsof_output.count('HIToolbox'):
        log(message)
        error(lsof_output)
    log("after %s" % message)

and sprinkled it throughout the code.

It seems these 3 leaking files are opened by mechanize during:

self.browser = Browser()

in Bugzilla.__init__()

I've not tried to trace through the mechanize code to see where the leaks come from.

[Adam Barth]

Why don't we fork and exec and then throw away the old version of ourselves?  If that doesn't work, can we use a trampoline executable?  Tracking down every leak seems fragile.

[Eric Seidel (no email)]

Here is a stand-alone version of the same function:

def check_for_file_leak(message):
    import os
    import sys
    import subprocess
    print >> sys.stderr, "before %s" % message
    process = subprocess.Popen(['lsof', '-p', str(os.getpid())], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    lsof_output = process.communicate()[0].rstrip()
    if lsof_output.count('HIToolbox'):
        print >> sys.stderr, message
        print >> sys.stderr, lsof_output
        exit(1)
    print >> sys.stderr, "after %s" % message

[Eric Seidel (no email)]

We could definitely use a trampoline executable.  That may be the simplest solution.

[Eric Seidel (no email)]

bugzilla-tool commit-queue
is already a trampoline executable of sorts.  It's just no longer a shell script, instead it's implemented in bugzilla-tool itself.

I think I might just get rid of the "don't need to restart bugzilla-tool to notice committers.py changes" feature for now.

[Eric Seidel (no email)]

Rolled out the change and re-opened bug 30084.

*** This bug has been marked as a duplicate of bug 30084 ***