Pay attention to the “water drop and silver bullet” public account, the first time to obtain high-quality technical dry goods. 7 years of senior back-end development, with a simple way to explain the technology clearly.

It takes about 12 minutes to read this article.

In Python development, we often use the with syntax block to ensure that file descriptors are closed properly when reading or writing files, for example, to avoid resource leaks.

Have you ever thought about what’s behind with? What is the context manager we often hear about?

In this article we’ll take a look at the Python context manager and how with works.

Block with grammar

Before we get into the with syntax, let’s look at how to write code that doesn’t use with.

When we operate on a file, we can write code like this:

# Open file
f = open('file.txt')
for line in f:
    Read the contents of the file and perform other operations
    # do_something...
# close file
f.close()
Copy the code

This example is very simple, just open a file, then read the contents of the file, and finally close the file to free resources.

However, there is a problem with this code: after opening the file, if an exception occurs during the operation to do something else with the read content, the file handle cannot be released, resulting in resource leakage.

How to solve this problem?

Also very simple, we use try… Finally to optimize the code:

# Open file
f = open('file.txt')
try:
    for line in f:
        Read the contents of the file and perform other operations
        # do_something...
finally:
    Make sure the file is closed
    f.close()
Copy the code

The advantage of writing this way is that the file resources are guaranteed to be released at the end of the reading and operation, regardless of exceptions.

However, this optimization will make the structure of the code cumbersome, adding a try to the code logic every time… Finally is ok. Readability becomes poor.

In this case, we can use the with syntax block to solve the problem:

with open('file.txt') as f:
    for line in f:
        # do_something...
Copy the code

Using the with block allows you to do the same thing, but the benefit is that the structure of your code is very clear and readable.

Now that you know what with does, how does it work?

Context manager

First, let’s look at the syntax of with:

with context_expression [as target(s)]:
    with-body
Copy the code

The with syntax is very simple, we just need an expression with, and then we can execute our custom business logic.

But can the expression after “with” be written arbitrarily?

The answer is no. To use the with block, the object after with needs to implement the context manager protocol.

What is the context Manager protocol?

A class in Python implements the context manager protocol by implementing the following methods:

__enter__: in thewithBefore the syntax block is called, the return value is assigned towith 的 target
__exit__: on the exitwithSyntax block, commonly used for exception handling

Let’s look at an example that implements both methods:

class TestContext:

    def __enter__(self) :
        print('__enter__')
        return 1

    def __exit__(self, exc_type, exc_value, exc_tb) :
        print('exc_type: %s' % exc_type)
        print('exc_value: %s' % exc_value)
        print('exc_tb: %s' % exc_tb)

with TestContext() as t:
    print('t: %s' % t)
    
# Output:
# __enter__
# t: 1
# exc_type: None
# exc_value: None
# exc_tb: None
Copy the code

In this example, we define the TestContext class, which implements __enter__ and __exit__ methods, respectively.

In this way, TestContext can be used as a “context manager,” executed with TestContext() as t.

From the output results, we can see that the specific execution process is as follows:

__enter__When enteringwithThe return value of this method is assigned towithAfter thetvariable
__exit__The execution of thewithThe statement block is then called

If an exception occurs within the with block, the __exit__ method gets details about the exception:

exc_type: Exception type
exc_value: Exception object
exc_tb: Indicates abnormal stack information

Let’s look at an example where an exception occurs and observe how the __exit__ method gets the exception message:

with TestContext() as t:
    An exception will occur here
    a = 1 / 0 
    print('t: %s' % t)

# Output:
# __enter__
# exc_type: <type 'exceptions.ZeroDivisionError'>
# exc_value: integer division or modulo by zero
# exc_tb: <traceback object at 0x10d66dd88>
# Traceback (most recent call last):
# File "base.py", line 16, in 
      
# a = 1 / 0
# ZeroDivisionError: integer division or modulo by zero
Copy the code

From the output, we can see that when an exception occurs in the with block, __exit__ prints the details of the exception, including the exception type, exception object, and exception stack.

If we need to do special handling for exceptions, we can implement custom logic in this method.

Let’s go back to our original example of reading a file with. With can automatically close a file resource because the built-in file object implements the context manager protocol. The __enter__ method of the file object returns the file handle and closes the file resource in __exit__. In addition, when an exception occurs in the with block, An exception is thrown to the caller.

The pseudocode could be written like this:

class File:

    def __enter__(self) :
        return file_obj

    def __exit__(self, exc_type, exc_value, exc_tb) :
        # with Releases file resources on exit
        file_obj.close()
        Throw an exception if there is an exception in with
        if exc_type is not None:
            raise exception
Copy the code

In summary, we learned with that it is very suitable for scenarios requiring context processing, such as operating files and sockets. These scenarios require the release of resources after the execution of business logic.

Contextlib module

For scenarios that require context management, is there an easier way to do this than to implement __enter__ and __exit__ yourself?

The answer is yes. We can use the Contextlib module provided by the Python standard library to simplify our code.

Using the Contextlib module, we can use the context manager as a “decorator.”

The ContextLib module provides the ContextManager decorator and closing methods.

Let’s see how they are used by example.

Contextmanager decorator

Let’s look at the use of the ContextManager decorator:

from contextlib import contextmanager

@contextmanager
def test() :
    print('before')
    yield 'hello'
    print('after')

with test() as t:
    print(t)

# Output:
# before
# hello
# after
Copy the code

In this example, we use the contextmanager decorator in conjunction with yield to achieve the same functionality as the previous contextmanager, which is executed as follows:

performtest()Method, first print outbefore
performyield 'hello'.testMethod returns,helloThe return value is assigned towithThe blocktvariable
performwithThe logic inside the statement block, printed outtThe value of thehello
Back totestMethod, executeyieldThe logic behind it is printed outafter

This way, when we use the ContextManager decorator, instead of writing a class to implement the contextManager protocol, we simply decorate the corresponding method with a method to achieve the same functionality.

It is important to note, however, that when using the ContextManager decorator, if an exception occurs within the decorated method, we need to do exception handling in our own method, otherwise the logic after yield will not be executed.

@contextmanager
def test() :
    print('before')
    try:
        yield 'hello'
        If an exception occurs here, you must handle the exception logic yourself, otherwise the execution will not proceed
        a = 1 / 0 
    finally:
        print('after')

with test() as t:
    print(t)
Copy the code

Method of closing

Let’s look again at how the closing method provided by contextlib is used.

Closing is used primarily on resource objects that already implement the close method:

from contextlib import closing

class Test() :

    The closing decorator can only be used if the close method is defined
    def close(self) :
        print('closed')

The close method is automatically executed after the # with block completes execution
with closing(Test()):
    print('do something')
    
# Output:
# do something
# closed
Copy the code

As you can see from the execution result, the close method of the Test instance is automatically called after the with block is executed.

So, for scenarios that require custom closing resources, we can use this method in conjunction with with.

The realization of the contextlib

Contextlib: contextlib: contextLib: contextLib: contextLib: contextLib: contextLib

The contextlib module is contextlib.

class _GeneratorContextManagerBase:

    def __init__(self, func, args, kwds) :
        # receive a generator object (a method that contains a yield is a generator)
        self.gen = func(*args, **kwds)
        self.func, self.args, self.kwds = func, args, kwds
        doc = getattr(func, "__doc__".None)
        if doc is None:
            doc = type(self).__doc__
        self.__doc__ = doc

class _GeneratorContextManager(_GeneratorContextManagerBase, AbstractContextManager, ContextDecorator) :

    def __enter__(self) :
        try:
            The generator code will run the yield of the generator method
            return next(self.gen)
        except StopIteration:
            raise RuntimeError("generator didn't yield") from None

    def __exit__(self, type, value, traceback) :
        No exception occurred in # with
        if type is None:
            try:
                # continue with the generator
                next(self.gen)
            except StopIteration:
                return False
            else:
                raise RuntimeError("generator didn't stop")
        An exception occurred in # with
        else:
            if value is None:
                value = type(a)try:
                Throw an exception
                self.gen.throw(type, value, traceback)
            except StopIteration as exc:
                return exc is not value
            except RuntimeError as exc:
                if exc is value:
                    return False
                if type is StopIteration and exc.__cause__ is value:
                    return False
                raise
            except:
                if sys.exc_info()[1] is value:
                    return False
                raise
            raise RuntimeError("generator didn't stop after throw()")

def contextmanager(func) :
    @wraps(func)
    def helper(*args, **kwds) :
        return _GeneratorContextManager(func, args, kwds)
    return helper

class closing(AbstractContextManager) :
    def __init__(self, thing) :
        self.thing = thing
    def __enter__(self) :
        return self.thing
    def __exit__(self, *exc_info) :
        self.thing.close()
Copy the code

Source code I have added a good comment, you can look at it in detail.

The contextLib decorator implements the following logic:

Initialize one_GeneratorContextManagerClass, the constructor accepts a generatorgen
This class implements the context manager protocol__enter__ 和 __exit__
performwithWill enter__enter__Method, and then executes the generator, which runs at execution towithInside the syntax blockyield 处
__enter__returnyieldThe results of the
ifwithSyntax block is not abnormal,withWhen the execution is complete, the__exit__Method to execute the generator again, which runsyieldThen the code logic
ifwithSyntax block exception,__exit__It will pass this exception through the generator towithInside the syntax block, that is, throwing the exception to the caller

The closing method calls the close of the custom object in the __exit__ method, so that when the with method ends, the closing method is executed.

Usage scenarios

Now that you’ve learned about context managers, what exactly are they used for?

Here are some common examples that you can use in your own scenarios.

Redis distributed lock

from contextlib import contextmanager

@contextmanager
def lock(redis, lock_key, expire) :
    try:
        locked = redis.set(lock_key, 'locked', expire)
        yield locked
    finally:
        redis.delete(lock_key)

The lock resource is automatically released after the execution of the with code block
with lock(redis, 'locked'.3) as locked:
    if not locked:
        return
    # do something ...
Copy the code

In this example, we implement the Lock method for applying a distributed lock on Redis, and then decorate the method with the ContextManager decorator.

Our business can then use the with block when calling the Lock method.

The first step of the with syntax block is to determine whether a distributed lock has been applied for. If the application fails, the business logic returns directly. If the application is successful, the specific service logic will be executed. When the service logic is completed, the distributed lock will be automatically released when with exits, so there is no need to manually release the lock every time.

Redis things and pipes

from contextlib import contextmanager

@contextmanager
def pipeline(redis) :
    pipe = redis.pipeline()
    try:
        yield pipe
        pipe.execute()
    except Exception as exc:
        pipe.reset()
            
The execute method is automatically executed after the execution of the with code block
with pipeline(redis) as pipe:
    pipe.set('key1'.'a'.30)
    pipe.zadd('key2'.'a'.1)
    pipe.sadd('key3'.'a')
Copy the code

In this example, we define the pipeline method and use the decorator contextmanager to make it a contextmanager.

With Pipeline (redis) as pipe; / / Pipeline (redis) as pipe; / / Pipeline (redis) as pipe; / / Pipeline (redis) as pipe Send these commands in batches to the Redis server.

If an exception occurs during command execution, pipeline’s reset method is automatically called, abandoning execution of the transaction.

conclusion

To conclude, this article focuses on the use and implementation of the Python context manager.

First, we looked at the differences between manipulating files without and with, and then saw that using with makes our code structure cleaner. We then explored the implementation of with, which can be used in conjunction with the with syntax block as long as instances of the __enter__ and __exit__ methods are implemented.

We then introduced the Python Standard library’s Contextlib module, which provides a better way to implement context management by using the ContextManager decorator and closing methods to manipulate our resources.

Finally, I gave two examples to illustrate the specific use of the context manager, such as the use of distributed locks and transaction pipelines in Redis, the use of the context manager to help us manage resources, and the implementation of pre and post logic.

Therefore, if we use the context manager to implement the pre and post logic of operation resources in development, then our code structure and maintainability will also be improved, which is recommended.

My advanced Python series:

Python Advanced – How to implement a decorator?
Python Advanced – How to use magic methods correctly? (on)
Python Advanced – How to use magic methods correctly? (below)
Python Advanced — What is a metaclass?
Python Advanced – What is a Context manager?
Python Advancements — What is an iterator?
Python Advancements — How to use yield correctly?
Python Advanced – What is a descriptor?
Python Advancements – Why does GIL make multithreading so useless?

Crawler series:

How to build a crawler proxy service?
How to build a universal vertical crawler platform?
Scrapy source code analysis (a) architecture overview
Scrapy source code analysis (two) how to run Scrapy?
Scrapy source code analysis (three) what are the core components of Scrapy?
Scrapy source code analysis (four) how to complete the scraping task?

Want to read more hardcore technology articles? Focus on”Water drops and silver bullets”Public number, the first time to obtain high-quality technical dry goods. 7 years of senior back-end development, with a simple way to explain the technology clearly.

Python Advanced – What is a Context manager?