In a Python project, how do you manage all the dependent libraries you use? The most common approach is to maintain a “requirements.txt” that records the name of the dependent library and its version number.

So, how do you generate this file? In the previous article, From Shallow to deep: How to automatically import missing libraries in Python? In, I mentioned a general approach:

pip freeze > requirements.txt
Copy the code

This method is convenient to use, but has several disadvantages:

  • The scope of its search for dependencies is global, so it adds libraries outside the project, creating redundancy (typically used in a virtual environment, but may still contain unrelated dependencies)
  • It only logs libraries installed as PIP Install
  • It makes no distinction between dependencies between dependent libraries
  • It can’t judge version differences and loop dependencies
  • …………………

There are a number of tools available for project dependency management. This article focuses on four tripartite libraries related to the requirements.txt file, providing a brief introduction to their use and listing some salient features. Which is the best management solution? I don’t know, read on…

pipreqs

This is a popular tool for managing dependency libraries in a project and can be installed with the “PIP install pipreqs” command. Its main features are:

  • Searching the scope of dependent libraries is a very targeted, directory-based approach
  • The search is based on what is imported in the script
  • Dependency files can be generated on environments where dependency libraries are not installed
  • When searching for software package information, you can specify the search method (only locally, in PyPi, or in a customized PyPi service).

The basic command options are as follows:

Usage:
    pipreqs [options] <path>

Options:
    --use-local           Use ONLY local package info instead of querying PyPI
    --pypi-server <url>   Use custom PyPi server
    --proxy <url>         Use Proxy, parameter will be passed to requests library. You can also just set the
                          environments parameter in your terminal:
                          $ export HTTP_PROXY="http://10.10.1.10:3128"
                          $ export HTTPS_PROXY="https://10.10.1.10:1080"
    --debug               Print debug information
    --ignore <dirs>...    Ignore extra directories
    --encoding <charset>  Use encoding parameter for file open
    --savepath <file>     Save the list of requirements in the given file
    --print               Output the list of requirements in the standard output
    --force               Overwrite existing requirements.txt
    --diff <file>         Compare modules in requirements.txt to project imports.
    --clean <file>        Clean up requirements.txt by removing modules that are not imported in project.
Copy the code

UnicodeDecodeError: ‘GBK’ codec can’t decode byte 0xae in UnicodeDecodeError: ‘GBK’ codec can’t decode byte 0xae in The encoding format is –encoding= UTf8.

In cases where the dependency file “requirements.txt” has been generated, it can force overrides, compare differences, and clean up dependencies that are no longer used.

pigar

Pigar also generates dependency files based on the project path and lists where dependency libraries are used in the file. This feature makes full use of comments in the requirements.txt file and can provide a wealth of information.

Pigar is useful for querying real import sources, such as the BS4 module from the Beautifulsoup4 library and the MySQLdb from the MySQL_Python library. You can use the “-s” parameter to find the actual dependent library.

$ pigar -s bs4 MySQLdb
Copy the code

It parses the AST rather than regular expressions, making it easy to extract dependent libraries from exec/eval parameters, docstrings, and documentation tests.

In addition, it supports differences between Python versions very well. For example, concurrent.futures is the standard library for Python 3.2+, whereas in earlier versions you needed to install the tripartite library Futures to use it. Pigar did this effectively. (PS: Pipreqs also supports this recognition, see this entry: github.com/bndr/pipreq…)

pip-tools

Pip-tools contains a set of tools for managing project dependencies, pip-compile and pip-sync, which can be installed using the PIP install pip-tools command. Its biggest advantage is the precise control of the project’s dependency library.

The usage and relationship diagram of the two tools are as follows:

The pip-compile command is used to generate dependency files and upgrade dependency libraries. In addition, it supports PIP’s “hash-checking Mode” and supports nesting of other dependency files within a dependency file (for example, in requirement-in files, You can import a dependency file with “-c requirements. TXT”).

It can generate requirements.txt from the setup.py file, so if a Flask project has “install_requires=[‘Flask’]” written in the setup.py file, all of its dependencies can be generated using commands:

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --output-file requirements.txt setup.py
#
click==6.7                # via flask
flask==0.122.
itsdangerous==0.24        # via flask
jinja2==2.96.             # via flask
markupsafe==1.0           # via jinja2
werkzeug==0.122.          # via flask
Copy the code

Without using the setup.py file, you can create “requirements.in”, write “Flask” and execute “pip-compile requirements.in” to achieve the same effect as before.

The pip-sync command is used to install, upgrade, or uninstall dependent libraries (except setuptools, PIP, and pip-tools) in a virtual environment based on the requirements. TXT file. This allows for targeted and on-demand lean management of dependent libraries in a virtual environment.

In addition, this command can merge multiple “*.txt” dependent files into one:

$ pip-sync dev-requirements.txt requirements.txt
Copy the code

pipdeptree

Its main purpose is to show the dependency tree of a Python project, showing their dependencies in a hierarchical indentation format, unlike the previous tools that generated flat juxtapositions.

Among other things, it can:

  • Generate the requirements.txt file for general use
  • How is reverse lookup of a dependent library introduced
  • Conflicting dependency libraries are displayed
  • You can discover cyclic dependencies and generate alarms
  • Generate dependency tree files in multiple formats (JSON, Graph, PDF, PNG, and so on)

It also has disadvantages, such as the inability to penetrate virtual environments. If you want to work in a virtual environment, you must install PipdepTree in that virtual environment. Because duplication or conflicts can occur across virtual environments, the virtual environment needs to be qualified. However, installing a PipdepTree for every virtual environment is still a bit frustrating.

All right, there are four libraries. Their core function is to analyze dependent libraries and generate requirements.txt files. At the same time, they have some differences that complement some of the shortcomings of traditional PIP.

This article does not make a comprehensive evaluation of them, just select some of the main features introduced, but they are easy to install (PIP install XXX), easy to use, interested students may wish to try.

For more rich details, please refer to the official documentation:

Github.com/bndr/pipreq…

Github.com/damnever/pi…

Github.com/jazzband/pi…

Github.com/naiquevin/p…

The public account “Python Cat”, this serial quality articles, cat philosophy series, Python advanced series, good books recommended series, technical writing, quality English recommended and translation, etc., welcome to pay attention to oh.