Dynamically importing modules PREMIUM

Let's talk about importing modules dynamically in Python.

This is a big topic that could involve a few different things, and all of them are a little bit weird, so we're going to take a look at a few examples.

Dynamically importing a module using its name

Let's say we have a string that represents a module name, and we don't have this module name when we write our Python code, but we do have this name at runtime:

>>> module_name = "csv"

We'd like to import this module.

To do this, we could use the import_module function from Python's importlib module:

>>> import importlib
>>> module_object = importlib.import_module(module_name)
>>> module_object
<module 'csv' from '/usr/lib/python3.11/csv.py'>

The import_module function basically works like an import statement, except it allows us to dynamically specify the module_name as a string.

Dynamically importing a module using its path

What if we our Python module isn't in our Python path (it's directory isn't in sys.path)?

Let's say know the path to a .py path that represents our module:

>>> module_name = "silly_module"
>>> module_path = "/home/trey/silly_module.py"

The importlib module can help in this case too, though our solution will need to be a little bit more complex.

I've written a function here called import_from_path:

import importlib.util
import sys

def import_from_path(module_name, file_path):
    """Import a module given its name and file path."""
    spec = importlib.util.spec_from_file_location(module_name, file_path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    return module

This function accepts a module_name and a file_path, and it makes a module specification object (spec).

    spec = importlib.util.spec_from_file_location(module_name, file_path)

After that, it makes a module object (module) from that module specification, and then caches that module object into Python's sys.modules dictionary.

    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module

Python normally does all of this for us, but we have to do it manually because we're not using an import statement here!

Finally, we execute the code in our module object (remember: importing a module runs code):

    spec.loader.exec_module(module)
    return module

That's really where the magic happens. Executing our module will populate the attributes on that module object (just like an import statement would).

>>> module_name = "silly_module"
>>> module_path = "/home/trey/silly_module.py"
>>> module_object = import_from_path(module_name, module_path)
>>> module_object
<module 'silly_module' from '/home/trey/silly_module.py'>

This code works, but it's a little bit more complex than we probably need.

Inserting a directory into the Python path

Typically, the easiest way to import a .py file is to make sure the directory that stores our module is available in our Python path.

For example, here, we're taking a vendored_utils subdirectory of our directory (using pathlib) and we're inserting it into our sys.path list:

from pathlib import Path
import sys


our_directory = Path(__file__).parent
sys.path.insert(0, our_directory / "vendored_utils")

The sys.path list contains directories that Python checks when looking for modules and packages to import.

Now that we've added our module's directory to our Python path, we can dynamically import our module:

import importlib
module_name = "requests"
importlib.import_module(module_name)

But if we know the name of our module when we write our code, it would be better to just use a regular import statement at this point:

from pathlib import Path
import sys


our_directory = Path(__file__).parent
sys.path.insert(0, our_directory / "vendored_utils")

import requests

Adding a module's parent directory to the sys.path list is the easiest and most typical way to import a module dynamically in Python. If you can, I would definitely recommend this approach over all others.

Dynamically importing a module from source code

What if we don't even have a file to import? What if we just have a string that represents some source code that we'd like to treat as a module:

>>> source_code = """
... from dataclasses import dataclass
...
... @dataclass
... class Point:
...     x: float
...     y: float
... """

It's also possible to import a module from a string in Python! Here's an import_from_string function that can do just that:

import importlib.util
import sys

def import_from_string(module_name, source_code):
    spec = importlib.util.spec_from_loader(module_name, loader=None)
    module = importlib.util.module_from_spec(spec)
    exec(source_code, module.__dict__)
    sys.modules[spec.name] = module
    return module

This import_from_string function accepts a module name (module_name) and the source that represents that module (source_code). Then we make a module specification (spec) with a module name, but without a Python path this time:

    spec = importlib.util.spec_from_loader(module_name, loader=None)

Then we make a module object (module) from that specification:

    module = importlib.util.module_from_spec(spec)

But the magic here is a little bit different. We tell Python to execute that source code using module.__dict__ as the global variables dictionary for our code:

    exec(source_code, module.__dict__)

That __dict__ dictionary is where modules store their attributes (it's actually where all objects typically store their attributes). That exec call populates our module object with the attributes we'd expect it to have.

After that, just as before, we cache our module object:

    sys.modules[spec.name] = module

So given a string of source code and the name of our module, we can make a module object that has the attributes we'd expect:

>>> source_code = """
... from dataclasses import dataclass
...
... @dataclass
... class Point:
...     x: float
...     y: float
... """
>>> point = import_from_string("point", source_code)
>>> point
<module 'point'>
>>> point.Point
<class 'point.Point'>

Inserting a directory into sys.path is usually the easiest way

If you'd like to import a module dynamically in Python, first ask yourself: "do I need to do this and what do I really need to do here?" Typically, it's easiest to insert a directory into sys.path.

But if you really need to, you can use the importlib module from the Python Standard Library to dynamically import modules.