Python functions beyond basics and a deep look at closure in Python.

All examples in this article use Python 3.3+.

1 Python functions beyond basics

Let’s introduce several important features of Python functions first.

1.1 Python function as a variable

Python function, essentially is also object, like a common Python variable, can be assigned to a variable.

Let’s see a simple demo. First of all, define a random string. Then call print and len function and pass the defined string.

>>> slogan = "Life is short, I use Python"

>>> print
>>> <built-in function print>
>>> print(slogan)
>>> Life is short, I use Python

>>> len
>>> <built-in function len>
>>> len(slogan)
>>> 27Copy the code

These work fine, nothing special.

Next step:

  • assign print to a new variable original_print
  • then assign len to print.
>>> original_print = print    # assign print to original_print
>>> original_print    # original_print now becomes print
>>> <built-in function print>

>>> original_print(slogan)
>>> Life is short, I use Python

>>> print = len    # assign len to print
>>> print    # print now becomes len
>>> <built-in function len>

>>> print(slogan)
>>> 27Copy the code

The conclusion here is straightforward: Python function can be assigned to a variable.

1.2 Python function as function argument

A Python function can be passed as a argument to a function.

1.2.1 the Custom example

Let’s give a smart_add function as an example. It takes three arguments and the third argument is a function.

def smart_add(x, y, f):
    return f(x) + f(y)Copy the code

Try to call smart_add and pass abs function as the third argument to it.

>>> smart_add(-3, 7, abs)    # abs(-3) + abs(7)
>>> 10Copy the code

Again, try to pass another one: math.sqrt.

>>> import math >>> Smart_add (4, 9, math.sqrt) # math.sqrt(4) + math.sqrt(9) >>> 5.0Copy the code
1.2.2 Built – in example:map
>>> help(map)
# ...
map(func, *iterables) --> map object
# ...Copy the code

According to the document: map function will make an iterator that computes the function using arguments from each of the iterables. Stops when the shortest iterable is exhausted.

In short, map will take each item x in iterables and map it to func(x).

x in iterables |--map to--> func(x)Copy the code

The first parameter of map is a function, and the second one is an iterable collection. For example, pass len function as the first argument and map string to its length.

>>> names = ["Tom", "Jerry", "Bugs Bunny"]
>>> mapped_obj = map(len, names)

>>> mapped_obj
>>> <map object at 0x102629320>

>>> print(list(mapped_obj))
[3, 5, 10]Copy the code

Many other functions in Python are similar to map, which takes a function as argument, such as reduce, filter.

You may have noticed that map function can be replaced with list comprehensions.

# same effect as the map function
[func(item) for item in iterables]Copy the code

In fact, the reduce function was demoted from built-in in Python 2.x to the functools module in Python 3 on that account. But the map and filter functions are still built-ins in Python 3.

Anyway, what we learned from this part is that Python function can be passed as an argument to a function.

1.3 Return a function in a Python function

Let’s define an inner function within an outer function and then return this inner function from outer.

def outer(): print('call outer() ... ') # define an inner function within the outer function def inner(): print('call inner() ... ') # return the inner function return innerCopy the code

Call the outer function and notice that the returned result is a function.

>>> r = outer()    # call outer()
call outer() ...
>>> r    # Returned result by calling outer() is a function
>>> <function outer.<locals>.inner at 0x1089c6d08>
>>> r()    # Call the returned function
call inner() ...Copy the code

One important thing to remember is not to confuse “return a function” with “return a data value”.

import math

def demo_one():
    return math.sqrt    # return a function

def demo_two(x):
    return math.sqrt(x)    # return a data valueCopy the code

Look at another example below.

# pow_later.py def pow_later(x): y = 2 def lazy_pow(): print('calculate pow({}, {})... '.format(x, y)) return pow(x, y) # Use Python built-in function: pow return lazy_powCopy the code

Try it in Python shell.

>>> from pow_later import pow_later

>>> my_pow = pow_later(3)
>>> my_pow
>>> <function pow_later.<locals>.lazy_pow at 0x10a043d08>Copy the code

pow_later returns a function that will actually calculate the result of pow(3, 2) in the future.

So call it when you need, and you will get the real calculated result:

>>> my_pow()
calculate pow(3, 2)...
9Copy the code

1.4 Bonus: higher-order function and first-class function

A function that meet at least one of the following criteria is called a higher-order function.

  • takes one or more functions as arguments
  • returns a function as its result

In fact, A Python function is not only a higher-order function, but also a first-class function, which satisfies following four criteria:

  1. can be created at runtime
  2. can be assigned to a variable
  3. can be passed as a argument to a function
  4. can be returned as the result of a function

2 Python closure

Now take a deeper look at the latest example mentioned above.

def pow_later(x): y = 2 def lazy_pow(): print('calculate pow({}, {})... '.format(x, y)) return pow(x, y) return lazy_powCopy the code

We called pow_later(3) and it returned a function object.

>>> my_pow = pow_later(3)
>>> my_pow
>>> <function pow_later.<locals>.lazy_pow at 0x10a043d08>Copy the code

then we invoked the returned function object.

>>> my_pow()
calculate pow(3, 2)...
9Copy the code

Obviously, the variable y and the parameter x are local variables of pow_later function. So when my_pow() was called, the pow_later function had already returned, and its local variables also had gone. But in fact my_pow() still remembered the vaules of x and y even the outer scope pow_later was long gone. How did this happen?

2.1 Free variable

If a variable in a function is neither a local variable nor a parameter of that function, this variable is called a free variable of that function.

In short, free variables are variables that are used locally, but defined in an enclosing scope.

In our case, x is a parameter of pow_later and y is a local variable of pow_later. But within lazy_pow, x and y are free variables.

2.2 a Closure

2.2.1 What is a closure

Specifically speaking, my_pow, actually the function object returned by calling pow_later(x), is a closure.

Note that the closure for lazy_pow extends the scope of lazy_pow function to include the binding for the free variables: x and y.

Generally speaking, a closure is a structure (code blocks, function object, callable object, etc.) storing a function together with an environment. The environment here means information about free variables that function bounded, especially values or storage locations of free variables.

For example, a closure is created, returned and assigned to my_pow after following function call.

>>> my_pow = pow_later(3)Copy the code

Essentially, this closure is the codes of function lazy_pow together with free variables x and y.

2.2.2 Inspect closure

You can see that the closure keeps names of free variables by inspecting __code__ attribute of my_pow function which represents the compiled body of the function.

>>> my_pow.__code__.co_freevars
>>> ('x', 'y')Copy the code

Meanwhile, pow_later will also keep names of local variables that are referenced by its nested functions in co_cellvars attribute of its code object.

>>> pow_later.__code__.co_cellvars
>>> ('x', 'y')Copy the code

However, where is the values of free variables?

>>> dir(my_pow)

>>> my_pow.__closure__
>>> (<cell at 0x10a428348: int object at 0x109e06b60>, <cell at 0x10a428378: int object at 0x109e06b40>)Copy the code

Note that my_pow has an attribute named __closure__ and it’s a tuple with two elements.

>>> dir(my_pow.__closure__[0])
>>> ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'cell_contents']

>>> my_pow.__closure__[0].cell_contents
>>> 3

>>> my_pow.__closure__[1].cell_contents
>>> 2Copy the code

So __closure__ is a tuple of cells that contain bounded values of free variables.

If your Python version is 3.3+, you can use inspect module to inspect. The nonlocals dictionary in inspecting result is exactly the bounded free variables and their values.

>>> import inspect
>>> inspect.getclosurevars(my_pow)
ClosureVars(nonlocals={'x': 3, 'y': 2}, globals={}, builtins={'print': <built-in function print>, 'pow': <built-in function pow>, 'format': <built-in function format>}, unbound=set())Copy the code
2.2.3 __closure__

Functions without free variables are not closures.

def f(x):
    def g():
        pass
    return gCopy the code

Note that returned function g has no free variable. And its __closure__ is None.

>>> h=f(1)
>>> h
>>> <function f.<locals>.g at 0x10f650158>
>>> h.__code__.co_freevars
>>> ()

>>> print(h.__closure__)
>>> NoneCopy the code

Global variables are not free variables in Python. So global functions are not closures.

>>> data=200 # global >>> def d(): # global >>> print(data) ... . >>> d() >>> 200 >>> d.__code__.co_freevars >>> () >>> print(d.__closure__) >>> NoneCopy the code

__closure__ attribute of global functions is None.

2.2.4 nonlocal declaration

Let’s review our pow_later(x) function.

  • pass a number x to function pow_later;
  • pow_later will return a function object;
  • the returned function object my_pow will calculate x**2 (y=2) each time it is called.

Now I’d like to change above behavior, let y increase 1 automatically each time my_pow is called. That is:

  • the firt time call, calculate x**2;
  • the second time call, calculate x**3;
  • the third time call, calculate x**4;
  • .

The updated source codes are as follows.

# pow_later.py def pow_later(x): y = 2 def lazy_pow(): print('calculate pow({}, {})... '.format(x, y)) result = pow(x, y) y = y + 1 # increase y return result return lazy_powCopy the code

Try it in Python shell.

>>> from pow_later import pow_later
>>> my_pow = pow_later(3)
>>> my_pow 
>>> <function pow_later.<locals>.lazy_pow at 0x108e020d0>Copy the code

So far so good, let’s call my_pow to see result.

>>> my_pow()
>>> Traceback (most recent call last):
...
UnboundLocalError: local variable 'y' referenced before assignmentCopy the code

The error message is clear enough.

  • It’s a UnboundLocalError
  • y is a local variable
  • local variable y referenced before assignment

The problem happens in this line: y = y + 1.

We are actually assigning to y in lazy_pow scope, and that makes y becomes local to lazy_pow scope. So Python considers y a local variable of lazy_pow. Before assigning to that local variable, Python will first read the local variable y. But y is a free variable as mentioned eariler and there is no local variable named y in lazy_pow scope at all.

You may think, OK, we don’t assign! How about use y += 1 instead of y = y + 1? The += operation is performed in-place, meaning that rather than creating and assigning a new value to the variable, the old variable is modified instead.

The answer is: no change here. Because y is a number, which is an immutable type. += will also create a new number object with new value behind the scene and assign the reference of the new object to y.

To deal with this situation, a nonlocal declaration was introduced in Python 3. It marks a variable as a free variable even though it is assigned a new value within the function.

# pow_later.py def pow_later(x): y = 2 def lazy_pow(): nonlocal y # nonlocal declaration print('calculate pow({}, {})... '.format(x, y)) result = pow(x, y) y = y + 1 return result return lazy_powCopy the code

Now the closure works well.

>>> from pow_later import pow_later
>>> my_pow = pow_later(3)
>>> my_pow()
>>> calculate pow(3, 2)...
9
>>> my_pow()
>>> calculate pow(3, 3)...
27
>>> my_pow()
>>> calculate pow(3, 4)...
81Copy the code

3 Summary

Two topics were discussed in this article.

First, Python functions are first-class functions.

Second, what is closure and how it works in Python.