preface
Most of the main content of this paper comes from the translation of PEP 343’s original text, and the rest is my understanding of the original text. In the process of sorting out, I did not deliberately distinguish between the translation part and my personal understanding part, and these two parts were mixed together to form this paper. So don’t read with the idea that everything in this article is 100 percent correct. If something in the article leaves you wondering, you can leave a comment to discuss it with me or compare PEP 343 to confirm it.
Abstract
This PEP adds a new syntax to Python: with, which is an easier alternative to the standard try/finally syntax.
In this PEP, the context manager provides __enter__() and __exit__() methods, which are called when entering and exiting the with statement, respectively.
Historical background
This PEP was originally written in the first person by Guido, followed by Nick Coghlan to update those later discussed on Python-Dev.
After much discussion of PEP 340 and its alternatives, Guido decided to withdraw PEP 340 and propose a new version based on PEP 310. This version adds a throw() method to throw an exception in the paused generator and a close() method to throw the GeneratorExit exception and changes the syntax key to with (block is defined in PEP 340).
After PEP 343 was officially adopted, the following PEPs were rejected or withdrawn due to overlap:
- PEP 310: Original proposal for the WITH syntax
- PEP 319: The scenario it describes can be implemented by providing the appropriate with syntax
- PEP 340
- PEP 346
There has been some discussion of an earlier version of this PEP on the Python Wiki.
Proposal motivation
PEP 340 illustrates a number of good ideas, such as using generators as templates for block syntax, adding exception handling and finalization (which I can interpret as destructor methods) to generators, and more. In addition to endorsing it, it is also opposed by many people because of the fact that the underlying structure is circular. The use of a loop structure means that breaks and continues in the block syntax can disrupt normal code logic and create uncertainty, even if it is used as a management tool for non-circular resources.
An article by Raymond Chen arguing against process control macros led Guido to make the final decision to drop PEP 340. In Raymond’s article, Guido made the point that hiding flow control in macros makes code harder to read and harder to maintain. Guido found this to be true of both Python and C, and realized that Templates, as introduced by PEP 340, hides almost all control flow. For example, the auto_retry function in PEP 340’s Examples 4 can only catch exceptions and repeat blocks up to three times (which is hidden inside the syntax).
By contrast, PEP 310’s with syntax does not hide process control, and although finally temporarily aborts the control flow, at the end (after everything in finally has been executed) the control flow continues to execute as if finally did not exist.
PEP 310 provides a rough syntax like this (where VAR is optional) :
with VAR = EXPR:
BLOCK
Copy the code
The syntax above roughly translates like this:
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
Copy the code
But when implementing the following code based on PEP 310’s syntax:
with f = open("/ect/passwd"):
BLOCK1
BLOCK2
Copy the code
Following the example given above, this code can be translated roughly like this:
f = open("/etc/passwd")
f.__enter__()
try:
BLOCK1
finally:
f.__exit__()
BLOCK2
Copy the code
The above code seems to assume that the BLOCK1 part will execute without exception, and the BLOCK2 part will be called immediately afterwards. However, BLOCK2 cannot be successfully executed if an exception is thrown or a non-local GOto (e.g., break, continue, return) is executed in BOLCK1. The magic added to the end of the with syntax (__exit__()) does not solve this problem.
You may ask: What happens if an exception is thrown in the __exit__() method? If that happens, it’s all over, but it’s no worse than the exception that would otherwise be raised.
The nature of an exception is that it can be thrown anywhere in the code, and you just have to live with that. Even if you write code that doesn’t throw any exceptions, a KeyboardInterrupt exception will still cause it to exit between any two virtual machine opcodes. Even if there is nothing wrong with your program, a program that is executing normally may exit because of your forced exit behavior, such as the common Ctrl + C).
The discussion above and the features shown in PEP 310 have led Guido to prefer the SYNTAX of PEP 310, but he still wants to implement what PEP 340 proposes: using generators as “templates” for abstractions such as obtaining and releasing locks or opening and closing files. This is a useful feature that can be seen in the PEP 340 example.
Inspired by one of Phillip Eby’s counter-suggestions to PEP 340, Guido tried to create a decorator that would turn a suitable generator into an object with the necessary __enter__() and __exit__() methods. However, he ran into a snag in the implementation process: handling the locking example was not difficult, but handling the opening example was nearly impossible. Here’s how he defines syntax templates:
@contextmanager
def opening(filename) :
f = open(filename)
try:
yield f
finally:
f.close()
Copy the code
And can be used like this:
withf = opening(filename): ... read datafrom f...
Copy the code
The problem is that in PEP 310, the results of EXPR expressions are assigned directly to VAR, and then VAR’s __exit__() method is called upon exiting BLOCK1. But in this case, VAR obviously needs to get the open file object, which means __exit__() must be a method on the file object.
While this problem can be solved with proxy classes, this solution is not elegant. After thinking about it, Guido realized that he could easily write the decorators he needed with a few changes to the syntax template. The change is to set VAR to the result of the __enter__() method call and save the value of EXPR for later calls to its __exit__() method. The decorator can then return an instance of the Wrapper class, whose __enter__() method calls the generator’s next() method to return what next() returns. The wrapper class instance’s __exit__() method calls the generator’s next() method again and (hopefully) throws a Stoplteration exception. See the generator decorator section below for details.
So now the final hurdle is the syntax of PEP 310:
with VAR = EXPR:
BLOCK1
Copy the code
Instead of using an assignment, you should use an implicit operation, since EXPR is not assigned to VAR per se. Refer to the implementation of PEP 340, which can be modified to:
with EXPR as VAR:
BLOCK1
Copy the code
It was also evident during the discussion of the proposal that developers generally want to be able to catch exceptions in generators, even if only for logging. But generators do not allow yield because the with syntax should not be designed as a loop (it is slightly acceptable to throw a different exception).
To implement this requirement, a new generator method, throw(), is proposed that takes one to three optional arguments (type, value, traceback) representing exceptions and throws them when the generator pauses.
The throw() method is followed by another generator method, close(). The close() method can call the throw() method with a special exception called GeneratorExit. This behavior is equivalent to telling the generator to prepare to quit, with a minor improvement in that the close() method is automatically triggered when the generator is collected by the garbage collection mechanism.
From there, we can add yield to the try-finally syntax, because we can now guarantee that the finally statement will be executed at the end. But there are also some common warnings about finalization Apply: processes can end abruptly without destructing any objects, and objects can persist forever because of loops or memory leaks in the application (as opposed to loops or memory leaks in Python controlled by GC).
Note that there is no guarantee that the finally part will be executed immediately after the generator object has no reference, although this is implemented in CPython. This is similar to auto-closing files: while an object in CPython is deleted as soon as the last reference to it disappears, other GC algorithms don’t necessarily do the same.
See PEP 342 for details of how generators have been modified.
Grammar specification
A new declaration is proposed with the syntax:
with EXPR as VAR:
BLOCK
Copy the code
With and as are the new keywords. Expression EXPR is an arbitrary (but not be an expression list), the distribution of the VAR is a single goal, it can’t be a comma separated variable sequence, but can be brackets parcel comma-separated variable sequence (this limitation makes the grammar of the future is likely to expand into multiple comma-separated resources, each resource has a corresponding as statement).
The AS VAR section is optional, and the above statement is translated as:
mgr = (EXPR)
exit = type(mgr).__exit__ # Not calling it yet
value = type(mgr).__enter__(mgr)
exc = True
try:
try:
VAR = value # Only if "as VAR" is present
BLOCK
except:
# The exceptional case is handled here
# If the BLCOK part is abnormal
exc = False
if not exit(mgr, *sys.exc_info()):
raise
# The exception is swallowed if exit() returns true
# If exit() returns true, the exception is ignored
finally:
# The normal and non-local goto case are handled here
# if the BLOCK has no exception or executes a non-local goto
if exc:
exit(mgr, None.None.None)
Copy the code
The lower-case variables (MGR, exit, value, exc) are internal variables that are inaccessible to users and are most likely implemented as special registers or stack positions. For more information about sys.exc_info, see the sys.exc_info section
The above code translation details are intended to specify exact semantics. If the method is not found as expected, the interpreter throws AttributeError. Also, if any of the calls throws an exception, the effect is the same as the code above. Finally, if the BLOCK contains a non-local goto statement, the __exit__() method is called as if the BLOCK had been executed normally, with three None arguments. That is, these “pseudo-exceptions” are not treated as exceptions by __exit__().
If the syntax of the as VAR part is omitted, the VAR = part of the translation is also omitted, but Mgr.__enter () is still called.
The call convention for Mgr.__exit__ () is as follows:
- If the content in the BLOCK completes normally or a non-local goto is triggered, the finally syntax BLOCK is called Mgr.__exit__ () with three None arguments.
- If the contents of the BLOCK raise an exception, when executing the finally syntax BLOCK, Mgr.__exit__ () is called, passing three parameters representing the exception type, value, and traceback respectively.
Important: If Mgr.__exit__ () returns True, the exception is ignored. That is, if Mgr.__exit__ () returns True, the next statement after the with statement will still be executed, even if an exception occurs inside the with syntax.
However, if the with syntax is interrupted by a non-local goto, when Mgr.__exit__ () returns, the non-local return is restored without considering the return value. The purpose of this is to make it possible for mgr.__exit__() to ignore exception throwing without being triggered frequently by mistake (since the default return value of Mgr.__exit__ () is None equals Flase, which enables the exception to be rethrown).
The main purpose of ignoring exception throws is to make it possible to write the @Contextmanger decorator. The @Contextmanger decorator, implemented with the ability to ignore exceptions thrown, causes the try/except block inside a decorated generator to behave as if the generator’s body had been extended online in place of the with syntax.
The motivation to pass exception details to __exit__() is consistent with the no-argument __exit__() in PEP 310, as described in the example section Transaction: Database Transaction Management. In this example, the generator function must decide whether to roll back a thing based on whether an exception has occurred. In this case, passing complete exception information is more extensible than using a Boolean value to mark the occurrence of an exception, such as providing an interface for an exception logging tool.
Relying on sys.exc_info() for exception information is rejected because the semantics of sys.exc_info() are so complex that it is entirely possible to return an exception that was caught long ago. There is also a proposal to add a Boolean value to distinguish between a BLOCK that executes successfully and a BLOCK that is interrupted by a non-local goto. This proposal was also rejected as too complicated and unnecessary. A non-local GOTO should be considered an exception for database transaction rollback decisions.
To make it easy to directly manipulate the context manager’s Python code inline, __exit__() methods should not rethrow exceptions passed to them. It should always be the responsibility of the caller of the __exit__() method to decide when to rethrow an exception.
Therefore, the caller can tell whether __exit__() failed by throwing an exception or not:
- If __exit__() does not throw an exception, the method itself executes successfully, regardless of whether the exception passed is ignored;
- If __exit__() throws an exception, the method itself failed.
Therefore, when implementing the __exit__() method, you should avoid throwing exceptions unless there is an exception (there is no need to avoid throwing those that are passed in).
Generator decorator
After PEP 342 passes, we can write a decorator that makes it possible for a generator decorated by this decorator to be used by the with syntax, and that generator yeild just once. Here is the code to complete such a decorator:
class GeneratorContextManager(object) :
def __init__(self, gen) :
self.gen = gen
def __enter__(self) :
try:
return self.gen.next(a)except StopIteration:
raise RuntimeError("generator didn't yield")
def __exit__(self, type, value, traceback) :
if type is None:
try:
self.gen.next(a)except StopIteration:
return
else:
raise RuntimeError("generator didn't stop")
else:
try:
self.gen.throw(type, value, traceback)
raise RuntimeError("generator didn't stop after throw()")
except StopIteration:
return True
except:
# only re-raise if it's not the exception that was
# passed to throw(), because __exit__() must not raise
# an exception unless __exit__() itself failed. But
# throw() has to raise the exception to signal
# propagation, so thi fixes the impedance mismatch
# between the throw() protocol and the __exit__()
# protocol .
It is rethrown only if it is not an exception passed to throw(),
__exit__() can only throw an exception if the method itself fails
But throw() must throw an exception to pass a signal
# So thi fixes the mismatch between throw() and __exit__() protocols
if sys.exc_info()[1] is not value:
raise
def contextmanager(func) :
def helper(*args, **kwds) :
return GeneratorContextManager(func(*args, **kwds))
return helper
Copy the code
This decorator can be used like this:
@contextmanager
def opening(filename) :
f = open(filename) # IOError is untouched by GeneratorContext
try:
yield f
finally:
f.close() # Ditto for errors here(howere unlikely)
Copy the code
A more complete implementation of this decorator has become part of the standard library.
A context manager that exists in the standard library
We can add __enter__() and __exit__() methods to files, sockets, locks, and so on, so that we can manipulate these objects as follows:
with locking(myLock):
BLOCK
Copy the code
Or you could write it as a shorthand
with myLock:
BLOCK
Copy the code
Caution should be exercised, however, as it may lead to similar errors:
f = open(filename)
with f:
BLOCK1
with f:
BLOCK2
Copy the code
The above code does not execute as smoothly as one might expect, because F is turned off before entering BLCOK2.
There are many other errors similar to those in the example above, such as:
- Generators decorated with the @ContextManager decorator mentioned above raise a RuntimeError when called f.__enter__() the second time by the with syntax;
- A similar error is raised if __enter__ is called on a closed file object.
For Python 2.5, the following types have been identified as context managers:
- file
- thread.LockType
- threading.Lock
- threading.RLock
- threading.Condition
- threading.Semaphore
- threading.BoundedSemaphore
Copy the code
A context manager will also be added to the Decimal module to support the use of the local Decimal context in the with statement, automatically restoring the original context when the with statement exits.
Standard terminology
This PEP proposes to call the protocol consisting of __enter__() and __exit__() methods a “context management protocol,” and the objects that implement it are called “context managers.”
The expression immediately following the with keyword in a statement is a “context expression.”
There is no special terminology for the code in the body of the with statement and for variable names (or names) after the AS keyword. The generic terms “Statement body” and “target list” can be used, and if these terms are not clear, they can be prefixed with statement or with statement.
The existence of objects such as the arithmetic context of decimal modules makes the use of the term “context” alone potentially ambiguous. If necessary, the description can be made more concrete by using “context manager” to represent concrete objects created by context expressions, and “runtime context” or (preferably) “runtime environment” to represent actual state changes made by the context manager.
When we briefly discuss the use of the with statement, this ambiguity should not be important because the context expression fully describes the changes made to the runtime environment. But the distinction between these nouns is especially important when discussing the mechanics of the with statement itself and how context managers are actually implemented.
How do I cache the context manager
Many context managers, such as files and generator-based contexts, are disposable objects. Once the __exit__() method is called, the context manager is not reusable (for example, the file has been closed, or the underlying generator has finished executing).
Creating a new context manager object for each WITH statement is the simplest solution to avoid the problem of multi-threaded code or nested WITH statements trying to use the same context manager. All of the reusable context managers in the library come from the threading module and have been designed to address the problems associated with threading and nested use.
This means that in order to reuse a context manager with specific initialization parameters in multiple WITH statements, it is often necessary to store it in a callable object with zero parameters and then call it in the context expression of each statement, rather than directly caching the context manager.
When this restriction does not apply, the documentation for the affected context manager should make this clear.
Rejected option
For months, PEP prohibited ignoring exception throws in order to avoid hiding the control flow, but it turned out to be a hassle to avoid during implementation, so Guido turned it back on.
Another core of this PEP is a __context__() method, similar to iterable’s __iter__() method. This led to endless discussion of questions and terminology. Guido eventually dropped the concept entirely as new problems arose in the process of explaining the __content__() method.
Defining the with syntax directly using PEP 342’s generator API was also briefly discussed, but quickly dismissed, because doing so would make writing context manager programs that are not generators extremely difficult.
The sample
The generator-based example relies on PEP 342. In addition, some examples are not necessary in practice because existing objects, such as threading.rlock, can be used directly in the with statement.
The tenses used for context names in the example are not arbitrary:
- The past tense (Ed) represents an action completed in the __enter__ method and undone in the __eixt__ method;
- The progressive tense (ing) represents an action completed in an __exit__ method.
Locked: locks are managed
Acquire the lock at the start of a statement and release it on departure:
@contextmanager
def locked(lock) :
lock.acquire()
try:
yield
finally:
lock.release()
Copy the code
It can be used like this:
with locked(myLock):
# Code here executes with myLock held. The lock is
# guaranteed to be released when the block is left (even
# if via return or by an uncaught exception).
# this code is executed with myLock in hand.
# This lock is guaranteed to be released after leaving the with statement, even if there are interrupts or exceptions during execution
Copy the code
Opened: File management
Open a file in a specific mode at the beginning of a statement and close it when leaving:
@contextmanager
def opened(filename, mode="r") :
f = open(filename, mode)
try:
yield f
finally:
f.close()
Copy the code
It can be used like this:
with opened("/etc/passwd") as f:
for line in f:
print line.rstrip()
Copy the code
Transaction: Database transaction management
Commit or roll back a database transaction:
@contextmanager
def transaction(db) :
db.begin()
try:
yield None
except:
db.rollback()
raise
else:
db.commit()
Copy the code
Class locked: Lock management without generators
Rewrite Example 1 without generators:
class locked:
def __init__(self, lock) :
self.lock = lock
def __enter__(self) :
self.lock.acquire()
def __exit__(self, type, value, tb) :
self.lock.release()
Copy the code
This example can easily be modified to other relatively stateless examples, showing that it is easy to avoid dependency on generators if you do not need to preserve special states.
Stdout_redirected: Output redirection management
Temporarily redirect stdout:
@contextmanager
def stdout_redirected(new_stdout) :
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout = save_stdout
Copy the code
It can be used like this:
with opened(filename, "w") as f:
with stdout_redirected(f):
print "Hello world"
Copy the code
This implementation is not thread-safe, but it is not thread-safe if you implement the same action manually. This is a popular processing scheme in single-threaded programs (for example, in scripts).
Opened_w_error: file (__enter__ returns both values)
A variant of Opened () that returns both a file handle and the exception contents:
@contextmanager
def opened_w_error(filename, mode="r") :
try:
f = open(filename, mode)
except IOError, err:
# Use tuples to produce two return values at the same time
yield None, err
else:
try:
# Use tuples to produce two return values at the same time
yield f, None
finally:
f.close()
Copy the code
It can be used like this:
Use a tuple to receive the return value after as
with opened_w_error("/etc/passwd"."a") as (f, err):
if err:
print "IOError:", err
else:
f.write("guido::0:0::/:/bin/sh\n")
Copy the code
Signal: signal interception
Another useful example is the signal blocking operation, which is implemented as follows:
import signal
with signal.blocked():
# code executed without worrying about signals
Execute code without worrying about signals
Copy the code
You can pass in a list of signals to block, and by default, all signals are blocked.
decimal
For a decimal context, here is a simple example:
import decimal
@contextmanager
def extra_precision(places=2) :
c = decimal.getcontext()
saved_prec = c.prec
c.prec += places
try:
yield None
finally:
c.prec = saved_prec
Copy the code
Use examples (adapted from Python Library Reference) :
def sin(x) :
# Return the sine of x as measured in radians.
# return the sine of x in radians.
with extra_precision():
i, lasts, s, fact, num, sign = 1.0, x, 1, x, 1
whiles ! = lasts: lasts = s i +=2
fact *= i * (i-1)
num *= x * x
sign *= -1
s += num / fact * sign
# The "+s" rounds back to the original precision,
# so this must be outside the with-statement:
return +s
Copy the code
another decimal
Context manager for a simple Decimal module:
@contextmanager
def localcontext(ctx=None) :
"""Set a new local decimal context for the block"""
# Default to using the current context
if ctx is None:
ctx = getcontext()
# We set the thread context to a copy of this context
# to ensure that changes within the block are kept
# local to the block.
newctx = ctx.copy()
oldctx = decimal.getcontext()
decimal.setcontext(newctx)
try:
yield newctx
finally:
# Always restore the original context
decimal.setcontext(oldctx)
Copy the code
Use cases:
from decimal import localcontext, ExtendedContext
def sin(x) :
with localcontext() as ctx:
ctx.prec += 2
# Rest of sin calculation algorithm
# uses a precision 2 greater than normal
return +s # Convert result to normal precision
def sin(x) :
with localcontext(ExtendedContext):
# Rest of sin calculation algorithm
# uses the Extended Context from the
# General Decimal Arithmetic Specification
return +s # Convert result to normal context
Copy the code
Closing: Object-closing context manager
Generic object-Closing context manager:
class closing(object) :
def __init__(self, obj) :
self.obj = obj
def __enter__(self) :
return self.obj
def __exit__(self, *exc_info) :
try:
close_it = self.obj.close
except AttributeError:
pass
else:
close_it()
Copy the code
Can be used to exactly close any object that has a close method, be it a file, generator, or anything else. It can also be used when it is uncertain whether the object needs to be closed (for example, a function that takes an arbitrary iterable) :
# emulate opening():
with closing(open("argument.txt")) as contradiction:
for line in contradiction:
print line
# deterministically finalize an iterator:
with closing(iter(data_source)) as data:
for datum in data:
process(datum)
Copy the code
Python 2.5’s Contextlib module contains the implementation of this context manager (version 3.x also remains).
Released: Lock management
PEP 319 gives a use case for using a release context to temporarily release a previously acquired lock.
This can be achieved by swapping the order of acquisition() and release() calls.
The implementation code is as follows:
class released:
def __init__(self, lock) :
self.lock = lock
def __enter__(self) :
self.lock.release()
def __exit__(self, type, value, tb) :
self.lock.acquire()
Copy the code
Use cases:
with my_lock:
# Operations with the lock held
with released(my_lock):
# Operations without the lock
# e.g. blocking I/O
# Lock is held again here
Copy the code
Nested: nested use
A “nested” context manager that automatically nested the provided context from left to right to avoid excessive indentation:
@contextmanager
def nested(*contexts) :
exits = []
vars = []
try:
try:
for context in contexts:
exit = context.__exit__
enter = context.__enter__
vars.append(enter())
exits.append(exit)
yield vars
except:
exc = sys.exc_info()
else:
exc = (None.None.None)
finally:
while exits:
exit = exits.pop()
try:
exit(*exc)
except:
exc = sys.exc_info()
else:
exc = (None.None.None)
ifexc ! = (None.None.None) :# sys.exc_info() may have been
# changed by one of the exit methods
# so provide explicit exception info
raise exc[0], exc[1], exc[2]
Copy the code
Use case
with nested(a, b, c) as (x, y, z):
# Perform operation
Copy the code
Is equal to:
with a as x:
with b as y:
with c as z:
# Perform operation
Copy the code
Python 2.5’s Contextlib module contains the implementation of this context manager (the method with the same name is not found in the 3.x documentation).
The implementation process
This PEP was originally accepted by Guido during the EuroPython keynote on June 27, 2005. It was later accepted again with the __context__ method added.
This PEP was implemented in Subversion in Python 2.5 A1, with the __context__() method removed in Python 2.5 B1.
The appendix
Generator state
Can inspect. Getgeneratorstate generator (generator) checked to see:
- GEN_CREATED: Waiting to start execution.
- GEN_RUNNING: Currently being executed by the interpreter.
- GEN_SUSPENDED: Currently suspended at a yield expression.
- GEN_CLOSED: Execution has completed.
sys.exc_info
The tuple returned by this function contains three values that give information about the exception currently being handled. The information returned is limited to the current thread and current stack frame. If the current stack frame does not have an exception being handled, the information is retrieved from the lower rung called stack frame or the higher rung caller, and so on, until the stack frame that is handling the exception is found. Here “handling exceptions” means “executing except clauses”. Any stack frame can only access information about the exception currently being handled.
If there is no exception being handled throughout the stack, a tuple of three None values is returned. Otherwise the return value is (type, value, traceback). What they mean is:
- Type: The type of exception being handled (which is a subclass of BaseException);
- Value: exception instance (instance of the exception type);
- Traceback: A traceback object that encapsulates the call stack when the exception originally occurred.
reference
- PEP 343 original
- PEP 310 original
- PEP 340 original
- Raymond: A rant against flow control macros
- Brett Cannon: Unravelling the `with` statement
- Python standard Library – Sys library
- Python official documentation – Data Model – Traceback objects