I try to do all of my interactive Python development using Jupyter notebooks or IPython sessions. One of the main reasons I love these environments is the % autoReload magic. What’s so special about % autoReload, and why does it often make development faster and easier?

Why IPython and Jupyter?

Before going any further, if you haven’t already used IPython and Jupyter, check out the IPython interactive tutorial. It goes a long way towards explaining why using IPython is better than the default Python interpreter. It has many useful features, but in this article I will only discuss one feature (magic), and one magic in particular (%autoreload). Jupyter Notebook, like IPython, supports most of the same magic, so much of this tutorial can be used in an interactive IPython session or a Jupyter notebook session. One thing to note here is that I am talking about Python, not any of the other languages running in Jupyter notebooks.

What is magic?

Magic is a special function that you can call in an IPython or Jupyter session. They come in two forms: rows and cells. A line spell is prefixed with one %, and a unit spell is prefixed with two %%. A line mana consumes one line, while a unit mana consumes the line below the mana, allowing for more input. In this article, we will only look at one of the casters, the %autoreload magic.

Why auto-load?

The %autoreload magic alters the Python session so that the module automatically reloads in that session before entering the code entered at the IPython prompt (or the Jupyter notebook unit). This means that modules loaded into your session can be modified (outside of your session), and these changes will be detected and reloaded without you having to restart your session.

This can be very useful. Let me describe a typical scenario. Suppose you have a Jupyter notebook that you have created and are enhancing, and you need to get data from several sources. You get the data by executing functions in modules that you imported at the beginning of the meeting, which are Python code that you control. For many users, this will be a very typical use case. Also, suppose that on your laptop, you load all your data into memory, which takes a full five minutes. Then you start working with the data and quickly realize that you need slightly different data from a function in one of the modules you control, so you need to add another parameter to query the different data. How do you

  1. Make this change
  2. Test the change
  3. Go on with your work

In most cases, you open the underlying code in an editor or IDE, make changes to it, test it in another session (or use unit tests), and then have the option of installing the changes locally. But what about a laptop that already has some data loaded? One way to continue is to restart your Jupyter kernel to receive the changes you just made, reload all the data into memory (at least 5 minutes), and continue your work.

But there’s a better way, use Autoreload. In your Jupyter session, you first load the Autoreload extension, using the %load_ext magic.

%load_ext autoreload
Copy the code

The %autoreload magic is now available in your session. It can accept a single parameter specifying how the autoreloading of the module will proceed. The extension also provides another magic, % AIMPort, which allows fine control of which modules are affected by autoloading. If %autoreload is not given, it immediately reloads all modules (except those excluded by % AIMPort, as shown below). You can run it once and use your updated code.

The optional parameter autoreload has three valid values.

  • 0 – Disables automatic reloading
  • 1 – Reloads the input Python code each time before executing it%aimportAll imported modules
  • 2 – Reload all modules (except those that were) each time before executing the Python code you have entered%aimportExcluded modules)

To adjust modules affected by AutoReload, use the % AIMport talisman. Here’s how it works.

  • No arguments – Lists modules that will or will not be imported
  • There is one argument — the supplied module will be imported.%autoreload 1
  • Separate arguments with commas – all modules in the list will be imported.%autoreload 1
  • Parameters before-– The module will not be automatically loaded.

For me, the most common way I use %autoreload is in my initial development work, when IT is possible for me to change Python modules and notebook code to just include everything (i.e., run % AutoReload 2) and in other cases, not use it at all. However, it’s useful to have control, especially if you’re loading a lot of modules.

example

For a concrete example, you can use it to learn by making two Python files, auto.py and auto2.py, and saving them in the Jupyter notebook and entering them below. There should be a simple function in every Python file, as shown below.

# in auto.py
def my_api(model, year):
    # dummy result
    return { 'model': model, 'year': year, }

# in auto2.py
def my_api2(model, year):
    # dummy result
    return { 'model': model, 'year': year, }
Copy the code

Now, let’s import these two modules and use IPython/Jupyter to help check the API methods and attach one to the function? . You should see that the imported module matches your code in the Python file.

import auto
import auto2

auto.my_api?
Copy the code
Signature: auto.my_api(model, year)
Docstring: <no docstring>
File:      ~/projects/python_blogposts/tools/auto.py
Type:      function
Copy the code

Now, in a separate editor, add a third argument to the auto. My_api function (and perhaps make it accept a third color argument). Save the file. Do we see it? Refresh the help unit to find out.

No, not yet. Let’s turn on autoload.

%autoreload 2
Copy the code

Now, when I check auto. My_api, I see the new parameter. It worked!

Now I can change the Settings so that I reload only the AUTO2 module, not Auto. But first, let’s look at the modules to reload and skip. By default, it includes all modules and skips none (because I use 2 as the initial argument).

%aimport
Modules to reload:


Modules to skip:
Copy the code

Let’s close Auto.

%aimport -auto
%aimport
Modules to reload:


Modules to skip:
auto
Copy the code

Now, if I change the code in Auto, I should not see those changes in this session. With % AIMPort, you can limit which code is reloaded.

Matters needing attention

It is worth noting that module overloading is not perfect. You should not use this feature in production code, it will slow things down. Also, if you are editing your code in real time and leaving it in a broken state, the last code that successfully loaded will be the code that ran in your session, so it can be confusing for you. This may not be the way you want to modify a lot of code, but it works fine when making incremental changes.

To see what the broken code looks like, open the module that is being loaded automatically (auto2.py), add a syntax error (for example, maybe put a mismatched Parese somewhere) and save the file, then execute the module’s functions in the notebook unit. You should see the autoReload report tracking syntax errors in the cell. You will only see this error once, and if you re-execute the cell, it will not display the same error, but will use the version of the code that was last loaded.

Also, be aware that some things don’t always work, such as removing functions from modules, changing the @ attribute in a class to a normal method, or reloading C extensions. In these cases, you will need to restart your session. You can see more details in the documentation.

conclusion

If you haven’t used % autoReload before, try it when you have an IPython or Jupyter session with a lot of data and want to make a small change to the local module. I hope it saves you some time.

You may want to check out this article to see how you can use other magic to view variables in a Jupyter or IPython session.

The postUsing autoreload to speed up IPython and Jupyter workappeared first onwrighters.io.