I’m sure you’re familiar with the with statement in Python, especially when it comes to reading and writing files, but I think most people are probably used to it without knowing the “secret” behind it.
So, how does the with statement work, what is the context manager associated with it, and how do they relate? This article will bring you the decryption
What is a context handler?
File input/output, database disconnection, and so on are common resource management operations in any programming language. But the resources are limited, when writing programs, we must ensure that these resources are released after use, otherwise it is easy to cause resource leakage, light makes the system processing slow, heavy will make the system crash.
You might not get it just by talking about these concepts, but here’s an example:
for x in range(10000000):
f = open('test.txt', 'w')
f.write('hello')
Copy the codeHere we have 10 million files open, but we don’t close them when we’re done. If you run this code, an error will be reported:
OSError: [Errno 23] Too many open files in system: 'test.txt'
Copy the codeThis is a classic example of a resource leak. Because the program opened too many files at the same time, occupy too many resources, causing the system crash.
To solve this problem, different programming languages have introduced different mechanisms. In Python, the solution is the Context Manager. Context managers, which help you automatically allocate and release resources, are typically used in the with statement. So, the correct way to write the above code should look like this:
for x in range(10000000):
with open('test.txt', 'w') as f:
f.write('hello')
Copy the codeIn this way, every time we open the file “test.txt” and write “hello”, the file will be closed automatically and the corresponding resources can be released to prevent resource leakage. Of course, the code for the with statement can also be expressed as follows:
f = open('test.txt', 'w')
try:
f.write('hello')
finally:
f.close()
Copy the codeNote that the last finally block is especially important because it guarantees that the file will eventually be closed even if an error exception occurs while writing to it. However, such code is redundant and prone to omissions compared to the with statement, so we generally prefer to use the with statement.
Another classic example is the threading. Lock class in Python. For example, if I wanted to acquire a lock, perform the corresponding operation, and release it after completion, the code would look like this:
some_lock = threading.Lock()
some_lock.acquire()
try:
...
finally:
some_lock.release()
Copy the codeThe with statement is also very succinct:
some_lock = threading.Lock()
with somelock:
...
Copy the codeAs we can see from these two examples, the use of the with statement can simplify the code and effectively prevent the occurrence of resource leakage.
Implementation of the context manager
Class-based context manager
Now that you understand the concepts and benefits of context management, let’s take a look at how the context manager works and understand its internal implementation through concrete examples. Here, I’ve created a custom context management class, FileManager, that emulates Python’s open and close file operations:
class FileManager: def __init__(self, name, mode): print('calling __init__ method') self.name = name self.mode = mode self.file = None def __enter__(self): print('calling __enter__ method') self.file = open(self.name, self.mode) return self.file def __exit__(self, exc_type, exc_val, exc_tb): print('calling __exit__ method') if self.file: self.file.close() with FileManager('test.txt', 'w') as f: Print ('ready to write to file') f.write(' Hello world') ## print calling __init__ method calling __enter__ method ready to write to file calling __exit__ methodCopy the codeNote that when we create a context manager with a class, we must ensure that the class includes both the method “__enter__()” and the method “__exit__()”. The method “__enter__()” returns the resource that needs to be managed, and the method “__exit__()” usually has operations to free or clean up the resource, such as closing the file in this example.
When we execute the context manager with the with statement:
with FileManager('test.txt', 'w') as f:
f.write('hello world')
Copy the codeThe following four steps will take place in sequence:
1. The method “__init__()” is called and the program initializes the object FileManager such that the file name (name) is “test.txt” and the file mode (mode) is ‘w’;
2. The method “__enter__()” is called, the file “test.txt” is opened in write mode, and the FileManager object is returned with the variable f;
3. The string Hello world is written to the test. TXT file.
4. The method “__exit__()” is called to close the previously opened file stream.
Therefore, the output of this program is:
calling __init__ method
calling __enter__ method
ready to write to file
calling __exit__ meth
Copy the codeIt is also worth mentioning that the arguments “exc_type, exc_val, exc_tb” in the method “__exit__()” represent Exception_type, exception_value, and traceback, respectively. When we execute the with statement with the context manager, if an exception is thrown, the exception information is contained in these three variables, passing in the method “__exit__()”.
Therefore, if you need to handle possible exceptions, you can add code to “__exit__()”, such as the following:
class Foo: def __init__(self): print('__init__ called') def __enter__(self): print('__enter__ called') return self def __exit__(self, exc_type, exc_value, exc_tb): print('__exit__ called') if exc_type: print(f'exc_type: {exc_type}') print(f'exc_value: {exc_value}') print(f'exc_traceback: {exc_tb}') print('exception handled') return True with Foo() as obj: Raise Exception(' Exception raised').with_traceback(None) # output __init__ called __enter__ called __exit__ called exc_type: <class 'Exception'> exc_value: exception raised exc_traceback: <traceback object at 0x1046036c8> exception handledCopy the codeHere, we manually raise the exception “Exception raised” in the with statement, and as you can see, the exception in the “__exit__()” method is caught and handled smoothly. Note, however, that if the method “__exit__()” does not return True, the exception will still be thrown. Therefore, if you are sure that the exception has been handled, add the “return True” statement at the end of “__exit__()”.
Generator-based context manager
Of course, class-based context managers are widely used in Python and are often seen in this form, but context managers in Python are not limited to this. In addition to being class-based, it can also be implemented based on generators. Let’s look at an example.
. For example, you can use a decorator contextlib contextmanager, to define themselves based on the context of the generator needed for the manager, to support with statement. Or take the previous class context manager FileManager for example, we can also use the following form:
from contextlib import contextmanager
@contextmanager
def file_manager(name, mode):
try:
f = open(name, mode)
yield f
finally:
f.close()
with file_manager('test.txt', 'w') as f:
f.write('hello world')
Copy the codeIn this code, the function file_manager() is a generator. When we execute the with statement, we open the file and return the file object f; When the with statement completes, the file closure operation in finally block is executed.
As you can see, with the generator-based contextmanager, we no longer define the “__enter__()” and “__exit__()” methods, but be sure to add the decorator @contextmanager, which is easy for beginners to miss.
Having covered these two different principles of context managers, it is important to note that the class-based context manager and the generator-based context manager are functionally identical. only
- The class-based context manager is more
flexible, suitable for large-scale system development; - The generator-based context manager is more convenient and concise, suitable for small and medium sized programs.