In this article, I’ll take a quick look at how Python probes work. To verify this principle, we will also implement a simple probe program that counts the execution time of specified functions.

The realization of the probe mainly involves the following points:

  • sys.meta_path
  • sitecustomize.py

sys.meta_path

Sys. meta_path is a simple way to implement import hook. When executing import-related operations, objects defined in sys.meta_path will be triggered. For more information about sys.meta_path, refer to the Python documentation for sys.meta_path and PEP 0302.

The object in sys.meta_path needs to implement a find_module method that returns None or an object that implements the load_module method (part1_ can be downloaded from Github) :

import sys

class MetaPathFinder:

    def find_module(self, fullname, path=None):
        print('find_module {}'.format(fullname))
        return MetaPathLoader()

class MetaPathLoader:

    def load_module(self, fullname):
        print('load_module {}'.format(fullname))
        sys.modules[fullname] = sys
        return sys

sys.meta_path.insert(0, MetaPathFinder())

if __name__ == '__main__':
    import http
    print(http)
    print(http.version_info)
Copy the code

The load_module method returns a module object, which is the import module object. For example, I replaced HTTP with sys module as I did above.

$ python meta_path1.py
find_module http
load_module http
<module 'sys' (built-in)>
sys.version_info(major=3, minor=5, micro=1, releaselevel='final', serial=0)
Copy the code

With sys.meta_path, we can implement the function of import hook: when importing a predefined module, we can change the objects in this module to obtain the execution time of functions or methods and other detection information.

Above said the leopard cat for prince, then how to an object for leopard cat for prince operation? For function objects, we can replace them with decorators (download part2 from github) :

import functools
import time

def func_wrapper(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print('start func')
        start = time.time()
        result = func(*args, **kwargs)
        end = time.time()
        print('spent {}s'.format(end - start))
        return result
    return wrapper

def sleep(n):
    time.sleep(n)
    return n

if __name__ == '__main__':
    func = func_wrapper(sleep)
    print(func(3))
Copy the code

Execution Result:

$ python func_wrapper.py
start func
spent 3.004966974258423s
3
Copy the code

Let’s implement a function that calculates the execution time of a given function for a given module (download part3 from github).

Suppose our module file is hello.py:

import time

def sleep(n):
    time.sleep(n)
    return n
Copy the code

Our import hook is hook.py:

import functools import importlib import sys import time _hook_modules = {'hello'} class MetaPathFinder: def find_module(self, fullname, path=None): print('find_module {}'.format(fullname)) if fullname in _hook_modules: return MetaPathLoader() class MetaPathLoader: def load_module(self, fullname): Print ('load_module {}'. Format (fullname)) # ' 'sys.modules'' if fullname in sys.modules: Return sys.modules[fullname] # Delete the custom finder from sys.meta_path # prevent recursion calls when import_module is triggered again Finder = sys.meta_path.pop(0) # Import module module = importlib.import_module(fullname) module_hook(fullname, module) sys.meta_path.insert(0, finder) return module sys.meta_path.insert(0, MetaPathFinder()) def module_hook(fullname, module): if fullname == 'hello': module.sleep = func_wrapper(module.sleep) def func_wrapper(func): @functools.wraps(func) def wrapper(*args, **kwargs): print('start func') start = time.time() result = func(*args, **kwargs) end = time.time() print('spent {}s'.format(end - start)) return result return wrapperCopy the code

Test code:

>>> import hook >>> import hello find_module hello load_module hello >>> >>> hello.sleep(3) start func spent 3.0029919147491455 s 3 > > >Copy the code

In fact, the above code already implements the basic functions of the probe. One problem is that the above code needs to perform an import hook operation to register our defined hooks.

Is there a way to automatically perform import hook operations when starting the Python interpreter? The answer is that you can do this by defining sitecustomize.py.

sitecustomize.py

To put it simply, the Python interpreter automatically imports sitecustomize and Usercustomize modules that exist under PYTHONPATH when initialized:

The directory structure of the lab project is as follows (the code can be downloaded from github in part4) :

$tree. ├─ sitecustomiz.py └─ usercustomize.pyCopy the code

sitecustomize.py:

$ cat sitecustomize.py
print('this is sitecustomize')
Copy the code

usercustomize.py:

$ cat usercustomize.py
print('this is usercustomize')
Copy the code

Add the current directory to PYTHONPATH and see what happens:

$export PYTHONPATH=. $python this is sitecustomize <---- this is usercustomize <---- Python 3.5.1 (default, Dec 24 2015, 17:20:27) [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on Darwin Type "help", "copyright", "credits" or "license" for more information. >>>Copy the code

You can see that it does import automatically. So we can change the previous probe program to support automatic import hook execution (the code can be downloaded from Github).

Directory structure:

$tree.├ ── Hello.py ├─ hook.py ├─ sitecustomize.pyCopy the code

sitecustomize.py:

$ cat sitecustomize.py
import hook
Copy the code

Results:

$export PYTHONPATH=. $python find_module usercustomize Python 3.5.1 (default, Dec 24, 2015, 17:20:27) [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on Darwin Type "help", "copyright", "credits" or "license" for more information. find_module readline find_module atexit find_module rlcompleter >>> >>> Import hello find_module hello load_module hello >>> >>> hello.sleep(3) start func spent 3.005002021789551s 3Copy the code

One problem with the above probe is that PYTHONPATH needs to be changed manually. Those of you who have used the probe program will remember that using a probe like NewRelic requires only one command: Newrelic -admin run-program python hello.py actually modifs PYTHONPATH in newrelic-admin.

Let’s implement a similar command line program called agent.py.

agent

Or on the basis of the last program to modify. Adjust a directory structure to place hook operations in a separate directory so that PYTHONPATH is set without interference (the code can be downloaded from github in part6).

Py $touch Bootstrap /__init__. Py $touch Agent.py $tree.├ ── Bootstrap $mv hook │ ├ ─ ─ just set py │ ├ ─ ─ _hook. Py │ └ ─ ─ sitecustomize. Py ├ ─ ─ hello. Py ├ ─ ─ test. Py ├ ─ ─ agent. PyCopy the code

The content of the bootstrap/sitecustomize. Py is amended as:

$ cat bootstrap/sitecustomize.py
import _hook
Copy the code

The contents of agent.py are as follows:

import os import sys current_dir = os.path.dirname(os.path.realpath(__file__)) boot_dir = os.path.join(current_dir, 'bootstrap') def main(): Args = sys.argv[1:] os.environ['PYTHONPATH'] = boot_dir # execute the following Python program command # sys.executable is the absolute path ' 'which' of the Python interpreter program Python ` ` # > > > sys. The executable # '/ usr/local/var/pyenv/versions / 3.5.1 track of/bin/python3.5' OS. The execl (sys. The executable, sys.executable, *args) if __name__ == '__main__': main()Copy the code

The contents of test.py are:

$ cat test.py
import sys
import hello

print(sys.argv)
print(hello.sleep(3))
Copy the code

Usage:

$ python agent.py test.py arg1 arg2 find_module usercustomize find_module hello load_module hello ['test.py', 'arg1', 'arg2'] start func return 3.005035161972046s 3Copy the code

At this point, we have implemented a simple Python probe program. Of course, this is a far cry from the actual probe program, so this article will focus on the implementation principles behind the probe.