1. Why do I need to distribute and package the project?

We are used to using PIP to install some third-party modules. The installation process is easy because the module developer quietly does all the tedious work for us, and the process is packaging.

Packaging is to further encapsulate your source code and pre-arrange all project deployment so that users have it ready to use and don’t have to worry about deployment (if you don’t want to do it manually against a stack of deployment documents).

Whether you’re on the job or planning to write your own project to upload to PyPI, learn how to package your project.

Python has evolved over the years, and the project packaging tools have matured. What do they have?

You may have heard disutils, Distutils, Distutils2, setuptools, and so on. They seem familiar but strange. What are they?

2. The ancestor of packet distribution: Distutils

Distutils is a Python standard library, which is easily named as a distribution tool (UTLIS). It is an official distribution packaging tool developed by Python, and all subsequent packaging tools are based on it.

The essence of Distutils is to write setup.py, which is a guide for module distribution and installation.

So how do you write setup.py? Here the content is very much, I will be in the back of a detailed analysis, please be patient to look down.

You probably haven’t written setup.py, but you’ve definitely used setup.py to do things like the following command, which we often use to install modules.

$ python setup.py install
Copy the code

This is done from source code, as opposed to binary packages, which I’ll describe later.

3. Upgrade the distribution tool setupTools

Setuptools is an enhanced version of Distutils and is not included in the standard library. It has extended many features to help developers create and distribute Python packages better. Most Python users will use the more advanced SetupTools module.

Distribute, maybe you’ve seen it in other places, and I’ll mention it here.

Distribute is that SetupTools has a branch version, which may be because some developers think SetupTools is too slow to develop. But now, distribute is merged back into SetupTools. So we can think of them as the same thing.

Another large package distribution tool is Distutils2, which attempts to take full advantage of Distutils, DetupTools, and Distribute and is standard in the Python standard library. But the plan did not achieve the desired purpose, and has been an abandoned project.

Therefore, SetupTools is an excellent, reliable tool for installing and distributing Python packages.

So how do you install SetupTools in a clean environment?

There are two main methods:

  • Source code installation: at pypi.org/project/set… Run the python setup.py install command to decompress the zip package

  • Install by bootstrap: Download the bootstrap, which can be used to download or update the latest version of SetupTools

    $wget peak.telecommunity.com/dist/ez_set…

    The installation

    $ python ez_setup.py

    Update, either of the following

    Pythonezsetup. Py – Usetuptoolspython ez_setup. Py – U setuptoolspythonezsetup. Py – Usetuptools PIP install -u setuptools

Easy_install User Guide

When you install SetupTools, you have a third-party management tool called easy_install, which differentiates setupTools from Distutils.

Here’s a brief description of how it’s used, although it’s used very rarely.

First the package installation

Find the latest version from PyPI by package name Automatically download, compile, install $easy_install pkg_name # by the package name from the specified download page for links to install or upgrade packages $easy_install - http://pythonpaste.org/package_index.html # f Specify line installation package address $easy_install http://example.com/path/to/MyPackage-1.2.3.tgz # from local. Egg files installed $easy_install XXX. Egg # At installation time you can add additional parameters to specify the installation directory: --install-dir= dir, -d dir specifies the user to install: --userCopy the code

Then there is the upgrade of packages

$easy_install "SomePackage==2.0"Copy the code

Finally, the package is deleted

$ easy_install -m pkg_name
Copy the code

It is important to note that this deletion is only done in the easy-install.pth file, making it unusable in Python, but the actual package is still on your computer. To remove it completely, you need to manually delete the associated.egg and other files.

By default, easy_install will only download packages from pypi. Since this source is in a foreign country, the download speed is not ideal. If you have used PIP before, you may wonder if easy_install can specify the source for the installation.

The answer is yes.

Edit the configuration file /root/.pydistutils.cfg

[easy_install]
index-url=http://mirrors.aliyun.com/pypi/simple/
find-links=http://mirrors.aliyun.com/pypi/simple/
Copy the code

More than just introduces some commonly used methods of easy_install, want to learn more, you can see the official document: setuptools. Readthedocs. IO/en/latest/e…

Conclusion: SetupTools is an official professional package distribution tool, which is really simple from an installation point of view. More importantly, it is useful for package distribution, and the customization process is very high, and we still use it for release package distribution.

5. What is the difference between source code packages and binary packages?

Distribution of Python packages can be divided into two types:

  1. Distribute it as a source code package

Source package installation process, is first decompressed, then compiled, finally installed, so it is cross-platform, because each installation has to be compiled, the installation is slower than binary package installation.

The source package is essentially a compressed package, and its common formats are:

  1. Published in binary package format

Binary package installation process eliminates the compilation process, directly decompressed installation, so the installation speed is faster than the source package.

Since packages compiled for different platforms are not universal, multiple platforms need to be compiled before release.

Common formats for binary packages are:

What is the difference between eggs and wheels?

The Egg format was introduced by setuptools in 2004 and the Wheel format was defined by PEP427 in 2012. Wheel was developed as a replacement for Egg, which is essentially a ZIP package that is now considered the standard format for Python binary packages.

Here are the main differences between a Wheel and an Egg:

  • Wheel has an official PEP427 definition, while Egg has no PEP definition

  • Wheel is a distribution format, or package format. An Egg is both a distribution format and a runtime installation format, and can be imported directly

  • The Wheel file does not contain.pyc files

  • Wheel uses the PEP376-compatible.dist-info directory, while Egg uses the.egg-info directory

  • Wheel has a richer naming convention.

  • There are versions of wheels. Each Wheel file contains a version of the Wheel specification and a packaged implementation

  • Wheel is internally managed by sysconfig Path Type, making it easier to switch to other formats

Wheel packages can be installed using PIP, but the wheel module needs to be installed before using PIP commands.

$ pip install wheel
$ pip wheel --wheel-dir=/local/wheels pkg
Copy the code

7. How to write setup.py?

The most critical step in packaging distribution is to write the setup.py file.

Here is a simple use example of setup.py

From setuptools import setup, find_packages setup(name="mytest", version="1.0", author="wangbm", Author_email ="[email protected]", description="Learn to Pack Python Module --> author_email="[email protected]", description="Learn to Pack Python Module --> Python Programming time ", # project home URL ="http://iswbm.com/", # packages you want to install, setuptools.find_packages =find_packages())Copy the code

Next, I’ll slowly expand the setup function, adding more parameters so that you can understand what the setup function can do.

Program classification information

Classifiers Parameter description Package classification information. Example:

from setuptools import setup, Find_packages setup(# 3 - Alpha # 4 - Beta # 5 - Production/Stable 'Development Status: 3 - Alpha', # Intended Audience :: Developers', # What type of ' Build Tools', # License information 'License :: OSI Approved :: MIT License', # Target Python version 'Programming Language :: Python :: 2', 'Programming Language: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.3', 'Programming Language: Python :: 3.4', 'Programming Language :: Python :: 3.5',])Copy the code

Regarding the distribution of documents

From setuptools import setup, find_packages setup(name="mytest", version="1.0", author="wangbm", author_email="[email protected]", description="Learn to Pack Python Module", url="http://iswbm.com/", Packages =find_packages(), # static files to be installed, Such as configuration file, service files, pictures data_files = [(' ', [' the conf / *. Conf ']), ('/usr/lib/systemd/system/', [' bin / *. Service ']),]. # I hope the packaging file package_data = {' : [' *.txt], 'bandwidth_reporter: [' *.txt]}. Exclude_package_data ={'bandwidth_reporter':['*.txt']})Copy the code

In addition to the above parameter configuration, you can use a file called manifest.in to control file distribution.

Here is a sample manifest.in:

include *.txt recursive-include examples *.txt *.py prune examples/sample? /buildCopy the code

These configurations specify the following points

  • All files with the suffix TXT in the root directory will be distributed

  • Examples directories and TXT and py files in the root directory are distributed

  • Match the paths to examples/sample? /build will not be distributed

Manifest.in needs to be placed in the same top-level directory as setup.py, and setupTools automatically reads this file.

About dependency package download and installation

from setuptools import setup, find_packages setup( ... Pypi requires install_requires=['docutils>=0.3'], # setup. This is usually a configuration prepared for some setupTools plug-ins # the packages listed here do not install automatically. Setup_requires =[' PBR '], # Dependencies that are only needed for testing and are not used in normally published code. These three libraries can be installed automatically when python setup.py test is executed to ensure that the tests run properly. Tests_require = [' pytest > = 3.3.1 ', 'pytest - cov > = 2.5.1',]. $dependency_links=[$setup_requires / $tests_require $dependency_links=[$setup_requires / $tests_require "Http://example2.com/p/foobar-1.0.tar.gz",], # install_requires installation module automatically install # extras_require doesn't rely on package, Extras_require ={'PDF': ["ReportLab>=1.2", "RXP"], 'reST': ["ReportLab>=1.2", "RXP"] "], [" docutils > = 0.3})Copy the code

There are five common ways to express install_requires:

  1. ‘argparse’, which contains only the package name. This form checks only the existence of the package, not the version. Convenient, but not conducive to risk control.

  2. ‘setupTools ==38.2.4’, specify the version. This minimizes risk and ensures that development, testing and deployment versions are consistent and there are no surprises. The downside is that it is not conducive to updating, requiring code changes with each update.

  3. ‘docutils >= 0.3’, which is the more common form. This form automatically keeps the version up to date when a library is trusted.

  4. ‘Django > = 1.11,! = 1.11.1, <= 2’, this is a more complicated form. In this example, it is guaranteed that large versions of Django are between 1.11 and 2, i.e. 1.11.x; Also, version 1.11.1 with known problems was excluded (for example only). For large, complex libraries, this format is most appropriate.

  5. ‘requests[Security, socks] >= 2.18.4’, which is in the form of additional optional dependencies. A normal installation of Requests automatically installs the dependencies specified in its Install_requires, but not the set of dependencies security and SOCKS. These two sets of dependencies are defined in its EXTRas_REQUIRE. This form is used when using some libraries in depth.

Restrictions on the installation environment

Some libraries do not work in all versions of Python. If a library is installed in an incompatible Python environment, it should in theory not report an error at the time of use, but should fail during installation, prompting the installation to be prohibited.

Such functionality can be implemented using PYTHon_requires.

setup( ... Python_requires = '> = 2.7, the < = 3',)Copy the code

Generate distribution of executable files

From setuptools import setup, find_packages setup(name="mytest", version="1.0", author="wangbm", author_email="[email protected]", description="Learn to Pack Python Module", url="http://iswbm.com/", Packages =find_packages(), # to support automatic scripting, # this entry points to the main function of foo/main.py: entry_points={'console_scripts': ['foo = foo.main:main']}, # add the bin/foo.sh and bar.py scripts, /usr/bin/foo.sh and /usr/bin/bar.py scripts=['bin/foo.sh', 'bar.py'] )Copy the code

Setuptools will be moved to /usr/bin and given executable permissions after installation if the scripts in the scripts above have sh and py suffixes.

If you want to make further changes to these files, such as removing redundant suffixes, you can do so

from setuptools.command.install_scripts import install_scripts

class InstallScripts(install_scripts):

    def run(self):
        setuptools.command.install_scripts.install_scripts.run(self)

        # Rename some script files
        for script in self.get_outputs():
            if basename.endswith(".py") or basename.endswith(".sh"):
                dest = script[:-3]
            else:
                continue
            print("moving %s to %s" % (script, dest))
            shutil.move(script, dest)

setup(
    ...
    scripts=['bin/foo.sh', 'bar.py'],

    cmdclass={
        "install_scripts": InstallScripts
    }
)
Copy the code

ext_modules

The ext_modules argument is used to build C and C++ extension packages. It is a list of Extension instances. Each Extension instance describes an independent Extension module that can set Extension package names, header files, source files, link libraries and their paths, macro definitions, and editing parameters. Such as:

setup(
    # other arguments here...
    ext_modules=[
        Extension('foo',
                  glob(path.join(here, 'src', '*.c')),
                  libraries = [ 'rt' ],
                  include_dirs=[numpy.get_include()])
    ]
)
Copy the code

The specified release

Setup.py can only specify version, not release. If you need to change the version number, you can specify it using the –release parameter

python setup.py bdist_rpm --release=20200617
Copy the code

Setup.py has so many arguments that writing a setup.py without documentation is not easy. As a reminder, I’ve compiled some common parameters for the setup function:

8. What is PBR?

PBR is an auxiliary tool for SetupTools, originally developed for OpenStack and based on D2to1.

The PBR reads and filters the data in setup.cfg, and then feeds the parsed data to setup.py as a parameter. Contains the following functions:

  1. Get Version, AUTHORS, and ChangeLog information from Git

  2. Sphinx Autodoc. The PBR scans the project, finds all modules, and generates stub files

  3. Requirements. PBR will read requirements. TXT and generate install_requires/tests_require/dependency_links for the setup function

Note that in the header of the requirements. TXT file you can use: – the index, https://pypi.python.org/simple/, This line transforms an abstract dependency declaration like Requests ==1.2.0 into a concrete dependency declaration like Requests 1.2.0 from pypi.python.org/simple/

  1. Long_description. Generated from readme. RST, readme. TXT or README filelong_descriptionparameter

Using PBR is simple:

from setuptools import setup

setup(
    setup_requires=['pbr'],
    pbr=True,
)
Copy the code

There are some configurations in setup.cfg when using PBR. In [files], there are three keys: packages: specifies the packages to be included. The behavior is similar to setupTools. find_packages Namespace_packages: specifies namespace packages data_files: Specify destination directory and source file path, as an example:

[files]
data_files =
    etc/pbr = etc/pbr/*
    etc/neutron =
        etc/api-paste.ini
        etc/dhcp-agent.ini
    etc/init.d = neutron.init
Copy the code

The [entry_points] segment works in the same way as SetupTools.

So far, I’ve covered three ways to write using setup.py

  • Pass the arguments in one by one using command line arguments (highly recommended)

  • Specified in the setup function in setup.py (recommended)

  • Use PBR, specified in setup.cfg (easy to manage, preferred)

9. How do I build packages using setup.py

1. Build the source distribution package.

Use to publish a Python module or project that packages the source code as tar.gz (for Linux) or zip (for Windows)

$ python setup.py sdist
Copy the code

How do you install the package?

The answer is to use the easy_install tool available in SetupTools, which we’ll cover in the next section.

$ easy_install xxx.tar.gz
Copy the code

Using SDIST creates an archive in the default format based on the current platform. On Unix-like platforms, a gzip compressed tar file distribution with the suffix.tar.gz is created, while on Windows it is a ZIP file.

Of course, you can also break this default behavior by specifying the format of your distribution package

$ python setup.py sdist --formats=gztar,zip
Copy the code

What formats can you specify?

Create a compressed tarball and a zip file. Available formats are:

There are a few things to note about the above format:

  • Support for the Xztar format was only added in version 3.5

  • The zip format requires that you have installed the corresponding module: the zip program or zipfile module (which has become a Python standard library).

  • The Ztar format is being deprecated. Please try not to use it

Alternatively, if you want all files in the archive to be owned by root, you can specify this

python setup.py sdist --owner=root --group=root
Copy the code

2. Build a binary distribution.

In Windows, we are used to double-clicking exe to install software, and Python modules can also be packaged as binary packages such as exe.

$ python setup.py bdist_wininst
Copy the code

In Linux, RPM is used to install packages. You can use this command to build RPM packages

$ python setup.py bdist_rpm
Copy the code

If you prefer to use easy_install or PIP to install offline packages. You can pack it as an egg

$ python setup.py bdist_egg
Copy the code

If your project needs to install multiple platforms, both Windows and Linux, we need to execute multiple commands for multiple formats according to the above method. For convenience, you can execute the following command in one step to generate multiple formats of the base package

$ python setup.py bdist
Copy the code

10. How to use setup.py to install the package

Normally, we install modules from the source package or binary package built above.

However, when writing setup.py, you may need to debug it several times, so how do you test that your setup.py file works?

You can use this command to install your module into the global system environment

$ python setup.py install
Copy the code

If your project is still in development, installing modules frequently can be a hassle.

Instead of actually installing the package, you create a soft link in the system environment pointing to the directory where the package is actually located. This side can take effect without installation after modifying the package, which is convenient for debugging.

$ python setup.py develop
Copy the code

11. How do I distribute packages to PyPi?

If you think your module is good enough and you want to share it with others, you can upload it to PyPi (Python Package Index), which is a third-party repository of packages maintained by Python. Used to centrally store and manage Python packages published by developers.

To publish your own packages, you need to register with Pypi first. Then create the ~/. Pypirc file. In this file, configure the PyPI access address and account. For example, modify the. Pypirc file based on your own account.

A typical.pypirc file

[distutils]
index-servers = pypi

[pypi]
username:xxx
password:xxx
Copy the code

Then use this command to register the information, and when you’re done, you can see the project information on PyPi.

$ python setup.py register
Copy the code

After registration, you have to upload the source code package, others can use the download installation

$ python setup.py upload
Copy the code

Or you can also use twine tool register to upload, it is a special tool for interaction with pypi, details you can refer to website: www.ctolib.com/twine.html, not described here.