There are many ways to execute system commands directly in Python, such as os.system, commands(removed from Python3), subprocess, etc.

The return value of os.system is only the return status code of the command, but does not get the output of the command. Commands do not work well on Windows; Subprocesses are more commonly used in everyday life.

Previously, the subprocess was simply used to encapsulate a method as follows:

# utf-8
"""Python 2.7 Execution environment"""
from subprocess import Popen, PIPE


def getoutput(cmd, cwd=None, env=None):
    p = Popen(cmd, stdout=PIPE, stdin=PIPE, 
              stderr=PIPE, cwd=cwd, env=env, 
              shell=True)
    out, err = p.communicate()
    if err:
        print err
    return p.returncode, out

status, output = getoutput('whoami')
Copy the code

Most of the time this method works fine, but a few days ago, we had a problem.

When you run the VGS and VGS commands on a Redhat 6.8 vm, a large number of exception information is displayed

File descriptor 4 (pipe:[526520]) leaked on pvremove invocation. Parent PID 46499:python
File descriptor 5 (pipe:[526520]) leaked on pvremove invocation. Parent PID 46499:python
File descriptor 6 (pipe:[526520]) leaked on pvremove invocation. Parent PID 46499:python
...
Copy the code

It is normal to execute LVS directly on the virtual machine

[root@localhost test]# lvs
LV          VG        Attr        LSize        Pool        Origin       Data%       Meta%        Move        log         Cpy%Sync  Convert
lvoradata   appvg    -wi-ao----   270.00g
lvu01       appvg    -wi-ao----   30.00g
root        vg00     -wi-ao----   19.53g
[root@localhost test]# echo $?
0
Copy the code

The output of the command is fine, and the return code is uniformly good. Why are there exceptions in Python execution?

Thanks again to the mighty (Google)[www.google.com] for finding a similar problem in Stackoverflow. The second answer is the desired result.

So for lvm commands like pvremove, you can avoid the leak by setting close_fds=True in your subprocess.Popen call.

Originally, LVM must have the stdin stdout stderr of the standard descriptor to be available on system calls, otherwise it will be turned off and the above prompt will be thrown. So, this problem is solved by specifying CLOSE_fds =True to inherit the file descriptor.

from subprocess import Popen, PIPE


def getoutput(cmd, cwd=None, env=None):
    p = Popen(cmd, stdout=PIPE, stdin=PIPE, 
              stderr=PIPE, cwd=cwd, env=env, 
              shell=True, close_fds=True)
    out, err = p.communicate()
    if err:
        print err
    return p.returncode, out
Copy the code

When you use the Subprocess.popen class, you don’t really know anything about the internal parameters, only that you do it, but not why. In the process of solving the above problems, it can be regarded as the source code of the subprocess, which has a simple understanding of the parameters.

As expected, still want to see more source code.

reference

Leak when run subprocess in python

Python subprocess