Python Imports

status:Work In Progress

An overview of the history and functionality of Python’s import machinery.

This page is an outgrowth of a talk proposal I made for PyCon 2012. Hopefully it’s helpful as a reference and adds a little perspective on imports in Python. If you have any suggestions or corrections just let me know (see the project issue tracker).

A High-level Overview of Python’s import

Python’s import statement and the concept of self-contained namespaces have been a feature of the language since the very beginning. It’s one of the simple yet powerful ways that Python enables you to write code you’ll still be able to read in 6 months.

Chances are that you take imports for granted: the simplicity of the syntax; the whirring and intermeshing going on behing the scenes. Imports are, to a large measure, what make Python tick. You use them in every piece of Python code you write.

Why does all this matter? It’s because understanding the lower layers empowers you to get the most out of the higher ones [1]. With that tool in your belt you can fix your problems more quickly. To top it all off, Python provides a number of import tools that actually make sense once you wrap your brain around the behind-the-scenes stuff.

So, what makes import tick? Glad you asked...

Pulling It Apart

As implied, Python’s import machinery is made up of many pieces at many different levels. Why the complexity? Because imports cover a lot of territory and are called on to do some pretty hairy stuff. By the time you’re done here, you’ll understand.

Since it’s the point of this document, we’ll be looking at each cog and sprocket, with a high-level view coming immediately. The principal piece is the module object, which we’ll discuss next. Later sections will go more in-depth, while the appendices go all the way.

Imports work at several levels. Recognizing them is key to putting together the big picture. While we’ll look at each later, here’s a quick run-down of those layers, from top to bottom:

  1. the import statement
  2. builtins.__import__
  3. PEP 302 finders
  4. PEP 302 path importers

(and woven throughout is the interpreter’s import state).

What is a Module?

The central piece to imports is the module. A module is the object that the import machinery spits out. While we may call our files “modules” sometimes, the file is not the module. Instead, the module object gets created during import and the file gets executed in the module’s namespace. It’s a subtle difference, but a crucial one. That’s because imports are about much more than just files.

Let’s look at how modules fit in.

Note: make sure you understand the difference between running a .py file as a script and importing the corresponding module (see Modules vs. Scripts).

What Happens During Import?

With that concept of modules in mind, let’s step down, layer by layer, through import process.

At the highest level the compiler maps the various forms of the import statement to a handful of instructions for the interpreter. The end result is that one or more module has been imported and one or more names has been bound (in the local namespace).

At the point that the interpreter actually goes to import the module, it calls the builtins.__import__() function, which does the bulk of the work. The wasn’t always the case, but thankfully it is now, because we can take advantage of it to customize the behavior of imports.

When you import a module, you’re actually importing the chain of modules defined by the dots in the module name. Each of the names is imported, from left to right, with each imported relative to the previous one (the parent).

If one of these modules has not already been imported, then the process described in PEP 302 is used to find it and load it. This is implemented within builtins.__import__().

First it tries using custom import hooks to find the module. Then the system falls back to looking in special internal modules. Finally, it looks across a variety of filesystem paths. Through the API defined by PEP 302, this part is the last (and deepest) opportunity for customization. Realistically, it’s also the last chance for a module to be located during import.

If the module is never located, an ImportError gets raised. Otherwise the module gets loaded and the process continues.

Python’s Data and Execution Models

Namespaces

...

In the context of imports, there are two important namespaces: modules and packages. We’ve already talked about modules and how they are the namespace in which your files are executed. A package is simply a module associated with a directory rather than a file.

(packages should never be put directly on sys.path (even by the sys.path[0] behavior of the __main__ module) )

See the “modules” section of the data model documentation.

Execution Blocks

See the execution model documentation.

Modules vs. Scripts

The Import Syntax

http://docs.python.org/dev/reference/simple_stmts.html

For more detail see the appendix.

The import statement

  • usage
  • effect

The as clause

  • usage
  • effect
  • benefits

The from statement

  • usage
  • effect
  • from ... import *
  • dangers

Parentheses

  • usage
  • effect

Other Semantics

  • implicit relative imports
  • files (__init__.py, .py, .pyc, .pyo, etc.)
  • builtins.__import__()

Note

As noted at the beginning of this section, the appendix provides a more thorough under-the-hood look at the import syntax.

sys.modules

sys.path

  • .pth files

(Also see Appendix B) (Also see I.1 for more on why ‘’, a.k.a. CWD, is added to sys.path)

sys.meta_path

sys.path_hooks

sys.path_importer_cache

site-packages

http://docs.python.org/dev/library/site.html

site-packages user site-packages

(Also see Appendix B)

The site Module

http://docs.python.org/dev/library/site.html

site.py sitecustomize.py usercustomize.py

(Also see Appendix B)

Customizing Import Behavior

We’ll take a look at each layer.

The Import Syntax

Overriding builtins.__import__()

Using PEP 302 Finders (and Loaders)

Manipulating the Path Importer Cache

Using PEP 302 Path Importers

Directly Modifying the Import State

Consenting Adults

.pth Files

Other Resources


Appendices

  • annotated step-by-step through the CPython source for the import process
orphan:

Appendix: Import Syntax Under the Hood

Standard Library

Lib/importlib/*.py Lib/pkgutil.py

Tests

Lib/Test/...

CPython

Python/import.c Python/importdl.c Python/importdl.h Include/import.h Python/sysmodule.c Python/pythonrun.c

PyPy

Jython

IronPython

Appendix: An Extended Timeline of Importing in Python

orphan:

Some Context

(origins) http://www.python.org/community/sigs/retired/import-sig/ http://www.python.org/dev/peps/pep-3121/#id11 (1.5) http://www.python.org/doc/essays/packages.html #http://www.python.org/doc/essays/packages/ Modula-3 influence: http://python-history.blogspot.com/2009/02/adding-support-for-user-defined-classes.html http://python-history.blogspot.com/2009/01/brief-timeline-of-python.html http://python-history.blogspot.com/2009/03/dynamically-loaded-modules.html http://docs.python.org/dev/whatsnew/index.html http://python.org/download/releases/src/ http://hg.python.org/cpython-fullhistory/tags http://hg.python.org/cpython-fullhistory/graph/3cd033e6b530?revcount=800 http://hg.python.org/cpython-fullhistory/log/62bdb1cbe0f5/Python/import.c?revcount=120 initial: http://hg.python.org/cpython-fullhistory/file/fc6fcd7df4f7/Python/import.c 0.9.8: http://hg.python.org/cpython-fullhistory/file/17eff686be30/Python/import.c builtin___import__(), importdl.c: http://hg.python.org/cpython-fullhistory/rev/d7e91437f0a2 PyImport_Import: http://hg.python.org/cpython-fullhistory/rev/292193170da1 highlights of “What’s New”: http://nedbatchelder.com/blog/201109/whats_in_which_python.html code_swarm: http://vimeo.com/1093745

(ni) introduced (1.3): http://hg.python.org/cpython-fullhistory/rev/ec0b42889243 deprecated (1.5): http://docs.python.org/release/1.5/lib/node40.html still lives: http://docs.python.org/library/imputil.html#examples

(ihooks) introduced (1.3): http://hg.python.org/cpython-fullhistory/rev/ec0b42889243 removed (3.0): http://docs.python.org/release/2.6.2/library/undoc.html#miscellaneous-useful-utilities http://pydoc.org/2.4.1/ihooks.html

—–

The versions and dates are derived from a post on Guido’s “History of Python” blog. I’ve correlated the entries in section B.1 to versions by either explicit reference or by matching their commits to a version. Section B.2 also maps commits to versions. In both cases, I did my best to determine that mapping, but some may be off by a version.

The Extended Timeline

Initial Checkin (1990)
  • Checks sys.modules
  • Loads modules from sys.path or current dir (if sys.path is empty)
  • Supports IMPORT_NAME and IMPORT_FROM opcodes
  • No support for .pyc files
  • No support for packages
  • No support for C extension modules?
  • No ImportError
Python 0.9.1 (Feb. 1991)
Python 1.0 (1994)
  • Support for extension modules
  • Support for .pyc files
Python 1.2 (1995)
  • (Python/bltinmodule.c) __import__() builtin introduced
  • (Python/import.c) dynamic module support factored out into importdl.c
Python 1.3 (1995)
  • “ni” module introduced
Python 1.4 (1996)
http://docs.python.org/release/1.4/ref/ *
Python 1.5 (1998)
  • Support for packages
  • “site-packages” and “site-python” directories introduced
  • “__all__” introduced
  • “ni” module deprecated
  • (Python/import.c) PyImport_Import() introduced
Python 2.0 (2000)
Python 2.1 (2001)
  • PEP 235 – Import on Case-Insensitive Platforms
Python 2.2 (2001)
Python 2.3 (2003)
  • PEP 273 – Import Modules from Zip Archives
  • PEP 302 – New Import Hooks
Python 2.4 (2004)
  • PEP 328 – Imports: Multi-Line and Absolute/Relative (multi-line portion)
Python 2.5 (2006)
  • PEP 328 (relative imports portion)
  • PEP 338 – Executing modules as scripts
Python 2.6/3.0 (2008)
  • PEP 366 – Main module explicit relative imports
  • PEP 370 – Per user site-packages directory
Python 3.0 (2008)
  • reload removed from builtins
  • ihooks module removed from stdlib
  • imputil module removed from stdlib
Python 3.1 (2009)
Python 3.2 (2011)
  • PEP 3147 – PYC Repository Directories
Python 3.3 (2012)
  • see appendix D

Appendix: Ongoing Core Efforts to Improve Importing

orphan:

PEPs

  • PEP 369 – Post import hooks
  • PEP 382 – Namespace Packages
  • PEP 395 – Module Aliasing
  • PEP 402 – Simplified Package Layout and Partitioning
  • PEP ??? – import engine

Rejected PEPs:

  • PEP 299 – Special __main__() function in modules
  • PEP 3122 – Delineation of the main module

Projects

  • importlib.__import__ as the default builtins.__import__

Currently in Python, “builtin___import__()” in Python/bltinmodule.c makes a call to PyImport_ImportModuleLevelObject. Brett Cannon is working on making importlib.__import__ the default import call[1].

  • the __experimental__ module

http://mail.python.org/pipermail/python-ideas/2010-June/007357.html http://mail.python.org/pipermail/python-ideas/2011-August/011278.html

like the __future__ module, but for less-stable APIs that are likely to go in focus on stdlib (room for experimental syntax too?) (higher exposure testing)

Appendix: Imports in Alternate Python Implementations

IronPython

http://ironpython.codeplex.com/wiki/search?tab=Home&SearchText=import

Appendix: Easter Eggs

The Python devs are a playful lot.

from __future__ import braces

import __hello__

Appendix: Import Examples

orphan:

How It Works

  • Example: Plain Syntax Handler
  • Example: From Name Syntax Handler
  • Example: From Star Syntax Handler
  • Example: builtins.__import__

Causes of ImportError

  • turn into ImportError subclasses, __cause__

Other Exceptions During Import

  • SyntaxError
  • IOError?

block imports on the current working directory

By default Python will look for a module in your current working directory before trying the stdlib. The explicit relative import syntax of 2.7 help with this, but only to an extent.

To completely keep Python from trying the CWD, simply run “sys.path.remove(‘’)” and optionally follow that with “sys.path.append(‘’)”.

So the question remains, when did the empty string get added to (the front of) sys.path, and why?

...

Online References

orphan:

Dr. Brett Cannon gave a talk at PyCon 2010 <http://python.mirocommunity.org/video/1491> and PyCon 2008 (can’t find video).

importlib extensions: http://packages.python.org/importers/

flowchart: http://svn.python.org/view/sandbox/trunk/import_in_py/docs/

http://mail.python.org/mailman/listinfo/import-sig http://docs.python.org/dev/reference/simple_stmts.html#the-import-statement http://docs.python.org/dev/reference/simple_stmts.html#future-statements http://docs.python.org/dev/reference/toplevel_components.html http://docs.python.org/dev/reference/executionmodel.html#naming-and-binding http://docs.python.org/dev/reference/datamodel.html#the-standard-type-hierarchy (modules)

http://docs.python.org/dev/tutorial/modules.html http://docs.python.org/release/1.5.1p1/tut/modules.html http://docs.python.org/release/1.4/tut/node41.html#SECTION00700000000000000000 http://docs.python.org/dev/library/modules.html http://docs.python.org/dev/faq/programming.html#what-are-the-best-practices-for-using-import-in-a-module http://docs.python.org/dev/faq/programming.html#how-can-i-have-modules-that-mutually-import-each-other http://docs.python.org/dev/faq/programming.html#import-x-y-z-returns-module-x-how-do-i-get-z http://docs.python.org/dev/faq/programming.html#when-i-edit-an-imported-module-and-reimport-it-the-changes-don-t-show-up-why-does-this-happen

http://docs.python.org/dev/library/py_compile.html http://docs.python.org/dev/library/compileall.html

http://www.doughellmann.com/PyMOTW/sys/imports.html

http://lucumr.pocoo.org/2011/9/21/python-import-blackbox/

http://code.google.com/p/backport/ http://mirnazim.org/writings/python-ecosystem-introduction/ http://nedbatchelder.com/blog/201112/duplicitous_django_settings.html

http://lucumr.pocoo.org/2011/12/7/thoughts-on-python3/ http://pythonic.pocoo.org/2009/3/4/imports-in-functions-you-sure-about-that

Open bugs: http://bugs.python.org/issue?%40search_text=import&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40columns=creation&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=1&%40columns=status&%40group=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search

Not closed: http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40columns=creation&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=-1%2C1%2C3&%40columns=status&%40group=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search

Closed: http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40columns=creation&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=2&%40columns=status&%40group=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search

<stack overflow>

http://stackoverflow.com/questions/279237/python-import-a-module-from-a-folder

<cookbook recipes>

<others>

https://github.com/zacharyvoase/metaspace

http://www.youtube.com/watch?v=DkW5CSZ_VII http://aroberge.blogspot.com/2006/02/python-wish-new-meaning-for-import-as.html

Import Who’s Who

orphan:

People who have been involved with Python’s imports (incomplete):

“experts”: Brett Cannon, Nick Coghlan

Brett Cannon (importlib) Just van Rossum (PEP 302) Paul Moore (PEP 302) Aahz (PEP 328) Martin v. Loewis (PEP 382) P.J. Eby (PEP 402) James C. Ahlstrom (PEP 273) Nick Coghlan (PEP 338, PEP 366, PEP 395) Christian Heimes (PEP 370) Thomas Wouters (PEP 221) Barry Warsaw (PEP 3147) Tim Peters (PEP 225) Guido van Rossum (pretty much everything else <wink>)

Glossary

orphan:

The terminology surrounding imports can get confusing. This glossary should help.

import hook
...
finder
An object with a find() method that conforms to PEP 302. May also refer to the class of such an object.
loader
An object with a load() method that conforms to PEP 302. May also refer to the class of such an object.
importer
Mostly synonymous with path importer.
path importer
An object, class, or other code that may be plugged into the PEP 302 import machinery. Often this term refers specifically to those that are used with sys.path_hooks.
module
The object generated at the highest level of the import process. In the normal import statement, it is the object bound to the name.
module name
The value bound to the __name__ attribute of the corresponding module object. This will be the full qualified name relative to the sys.path value at import time.
package
A module corresponding to a directory. The module is populated with the results of evaluating the __init__.py file in the directory. Other .py files and directories in the directory may be imported as submodules of the package.
namespace package
A package, possibly without its own module execution, into which subpackages are aggregated according to a single namespace. The “zope” package is a good example.
package portion
...

The import Statement

The import statement is the syntactic mechanism you use to invoke Python’s powerful import machinery. It has two forms: the regular import and from-import. In a moment we’ll walk through the ins and outs of both forms.

When you use the import statement in either form, you identify a module and its parent modules together as a module name. By default, each successive parent module is imported from the outside in, followed by the actual module you wanted. After that the appropriate name is bound in the current local namespace. Most imports are going to happen at the module level where the name will be bound in that module’s [global] namespace.

A module object is the result of importing. We use the term “module” to refer to this object as well as to the thing that Python used to create the object, usually a file. A package is a special kind of module. Where a normal module corresponds to a file, a package corresponds to a directory.

Changing the Import Behavior

You can override the full import machinery by overriding builtins.__import__().

Prior to Python 2.3 the only way to override the import behavior was by replacing builtins.__import__() with some other function that did what you wanted. This changed with PEP PEP 302.

Now you can add special “loader” objects to a couple of different places in the sys module to take control of imports in more targeted ways. A loader translates a module name into a “finder” object, if it can. The finder, in turn, converts the module name into the corresponding module object, which it sticks into sys.modules.

This entire process is explained much more in-depth in a later section and in the appendix.

Import State

All the Python variables related to the default import behavior is stored in the sys module. This includes sys.path, sys.modules, sys.meta_path, and sys.path_hooks.

ImportError

What It Means

When It Happens and When Not

PEP 302

With the release of Python 3.2, a powerful way of customizing imports became available with PEP 302.

Finders

Loaders

The Default Import Process

See the appendix.

Implicit Finders

Builtin Modules

Frozen Modules

Zipped Modules

The Python Path Finder

Implementations

The imp Module

  • iterative
  • Python/import.c

PyImport_GetModuleDict() used to get sys.modules (see J.3.9).

  • Python/bltinmodule.c - _builtin___import__()

This is the default handler for the import statement. In 2.7 it is __builtin__.__import__().

The importlib Module

  • recursive
  • Lib/importlib/

sys.modules used to get sys.modules.

  • importlib.__import__()
  • importlib.import_module()

.pth Files

See the site module documentation.

More in Appendix B.

A History of Python’s import Statement

A Brief History of Python

The Origins of Python’s import

http://python-history.blogspot.com/2009/02/adding-support-for-user-defined-classes.html http://www.python.org/doc/essays/foreword/ http://python-history.blogspot.com/2009/01/brief-timeline-of-python.html

The import statement has been a part of Python since the very beginning, though with more limited behavior.

Like many things in Python, the syntax for the import statement has its roots in Modula-3.

Early Changes

  • builtins.__import__()
  • ni.py
  • ihooks.py

The Intervening Years

Recent Changes