status: Work In Progress
An overview of the history and functionality of Python’s import machinery.
This page is an outgrowth of a talk proposal I made for PyCon 2012. Hopefully it’s helpful as a reference and adds a little perspective on imports in Python. If you have any suggestions or corrections just let me know (see the project issue tracker).
A High-level Overview of Python’s import¶
Python’s import statement and the concept of self-contained namespaces have been a feature of the language since the very beginning. It’s one of the simple yet powerful ways that Python enables you to write code you’ll still be able to read in 6 months.
Chances are that you take imports for granted: the simplicity of the syntax; the whirring and intermeshing going on behing the scenes. Imports are, to a large measure, what make Python tick. You use them in every piece of Python code you write.
Why does all this matter? It’s because understanding the lower layers empowers you to get the most out of the higher ones . With that tool in your belt you can fix your problems more quickly. To top it all off, Python provides a number of import tools that actually make sense once you wrap your brain around the behind-the-scenes stuff.
So, what makes import tick? Glad you asked...
Pulling It Apart¶
As implied, Python’s import machinery is made up of many pieces at many different levels. Why the complexity? Because imports cover a lot of territory and are called on to do some pretty hairy stuff. By the time you’re done here, you’ll understand.
Since it’s the point of this document, we’ll be looking at each cog and sprocket, with a high-level view coming immediately. The principal piece is the module object, which we’ll discuss next. Later sections will go more in-depth, while the appendices go all the way.
Imports work at several levels. Recognizing them is key to putting together the big picture. While we’ll look at each later, here’s a quick run-down of those layers, from top to bottom:
- the import statement
- PEP 302 finders
- PEP 302 path importers
(and woven throughout is the interpreter’s import state).
What is a Module?¶
The central piece to imports is the module. A module is the object that the import machinery spits out. While we may call our files “modules” sometimes, the file is not the module. Instead, the module object gets created during import and the file gets executed in the module’s namespace. It’s a subtle difference, but a crucial one. That’s because imports are about much more than just files.
Let’s look at how modules fit in.
Note: make sure you understand the difference between running a .py file as a script and importing the corresponding module (see Modules vs. Scripts).
What Happens During Import?¶
With that concept of modules in mind, let’s step down, layer by layer, through import process.
At the highest level the compiler maps the various forms of the import statement to a handful of instructions for the interpreter. The end result is that one or more module has been imported and one or more names has been bound (in the local namespace).
At the point that the interpreter actually goes to import the module, it calls the builtins.__import__() function, which does the bulk of the work. The wasn’t always the case, but thankfully it is now, because we can take advantage of it to customize the behavior of imports.
When you import a module, you’re actually importing the chain of modules defined by the dots in the module name. Each of the names is imported, from left to right, with each imported relative to the previous one (the parent).
If one of these modules has not already been imported, then the process described in PEP 302 is used to find it and load it. This is implemented within builtins.__import__().
First it tries using custom import hooks to find the module. Then the system falls back to looking in special internal modules. Finally, it looks across a variety of filesystem paths. Through the API defined by PEP 302, this part is the last (and deepest) opportunity for customization. Realistically, it’s also the last chance for a module to be located during import.
If the module is never located, an ImportError gets raised. Otherwise the module gets loaded and the process continues.
Python’s Data and Execution Models¶
In the context of imports, there are two important namespaces: modules and packages. We’ve already talked about modules and how they are the namespace in which your files are executed. A package is simply a module associated with a directory rather than a file.
(packages should never be put directly on sys.path (even by the sys.path behavior of the __main__ module) )
See the “modules” section of the data model documentation.
Modules vs. Scripts¶
The Import Syntax¶
For more detail see the appendix.
The import statement¶
The as clause¶
The from statement¶
- from ... import *
- implicit relative imports
- files (__init__.py, .py, .pyc, .pyo, etc.)
As noted at the beginning of this section, the appendix provides a more thorough under-the-hood look at the import syntax.
- .pth files
(Also see Appendix B) (Also see I.1 for more on why ‘’, a.k.a. CWD, is added to sys.path)
site-packages user site-packages
(Also see Appendix B)
The site Module¶
site.py sitecustomize.py usercustomize.py
(Also see Appendix B)
Customizing Import Behavior¶
We’ll take a look at each layer.
The Import Syntax¶
Using PEP 302 Finders (and Loaders)¶
Manipulating the Path Importer Cache¶
Using PEP 302 Path Importers¶
Directly Modifying the Import State¶
- annotated step-by-step through the CPython source for the import process
Appendix: Import Syntax Under the Hood¶
Python/import.c Python/importdl.c Python/importdl.h Include/import.h Python/sysmodule.c Python/pythonrun.c
(origins) http://www.python.org/community/sigs/retired/import-sig/ http://www.python.org/dev/peps/pep-3121/#id11 (1.5) http://www.python.org/doc/essays/packages.html #http://www.python.org/doc/essays/packages/ Modula-3 influence: http://python-history.blogspot.com/2009/02/adding-support-for-user-defined-classes.html http://python-history.blogspot.com/2009/01/brief-timeline-of-python.html http://python-history.blogspot.com/2009/03/dynamically-loaded-modules.html http://docs.python.org/dev/whatsnew/index.html http://python.org/download/releases/src/ http://hg.python.org/cpython-fullhistory/tags http://hg.python.org/cpython-fullhistory/graph/3cd033e6b530?revcount=800 http://hg.python.org/cpython-fullhistory/log/62bdb1cbe0f5/Python/import.c?revcount=120 initial: http://hg.python.org/cpython-fullhistory/file/fc6fcd7df4f7/Python/import.c 0.9.8: http://hg.python.org/cpython-fullhistory/file/17eff686be30/Python/import.c builtin___import__(), importdl.c: http://hg.python.org/cpython-fullhistory/rev/d7e91437f0a2 PyImport_Import: http://hg.python.org/cpython-fullhistory/rev/292193170da1 highlights of “What’s New”: http://nedbatchelder.com/blog/201109/whats_in_which_python.html code_swarm: http://vimeo.com/1093745
(ni) introduced (1.3): http://hg.python.org/cpython-fullhistory/rev/ec0b42889243 deprecated (1.5): http://docs.python.org/release/1.5/lib/node40.html still lives: http://docs.python.org/library/imputil.html#examples
(ihooks) introduced (1.3): http://hg.python.org/cpython-fullhistory/rev/ec0b42889243 removed (3.0): http://docs.python.org/release/2.6.2/library/undoc.html#miscellaneous-useful-utilities http://pydoc.org/2.4.1/ihooks.html
The versions and dates are derived from a post on Guido’s “History of Python” blog. I’ve correlated the entries in section B.1 to versions by either explicit reference or by matching their commits to a version. Section B.2 also maps commits to versions. In both cases, I did my best to determine that mapping, but some may be off by a version.
The Extended Timeline¶
- Initial Checkin (1990)
- Checks sys.modules
- Loads modules from sys.path or current dir (if sys.path is empty)
- Supports IMPORT_NAME and IMPORT_FROM opcodes
- No support for .pyc files
- No support for packages
- No support for C extension modules?
- No ImportError
- Python 0.9.1 (Feb. 1991)
- builtin module support (C extention modules)
- Python 1.0 (1994)
- Support for extension modules
- Support for .pyc files
- Python 1.2 (1995)
- (Python/bltinmodule.c) __import__() builtin introduced
- (Python/import.c) dynamic module support factored out into importdl.c
- Python 1.3 (1995)
- “ni” module introduced
- Python 1.4 (1996)
- http://docs.python.org/release/1.4/ref/ *
- Python 1.5 (1998)
- Support for packages
- “site-packages” and “site-python” directories introduced
- “__all__” introduced
- “ni” module deprecated
- (Python/import.c) PyImport_Import() introduced
- Python 2.0 (2000)
- PEP 221 – Import As
- Python 2.1 (2001)
- PEP 235 – Import on Case-Insensitive Platforms
- Python 2.2 (2001)
- Python 2.3 (2003)
- Python 2.4 (2004)
- PEP 328 – Imports: Multi-Line and Absolute/Relative (multi-line portion)
- Python 2.5 (2006)
- Python 2.6/3.0 (2008)
- Python 3.0 (2008)
- Python 3.1 (2009)
- importlib module added
- Python 3.2 (2011)
- PEP 3147 – PYC Repository Directories
- Python 3.3 (2012)
- see appendix D
Appendix: Ongoing Core Efforts to Improve Importing¶
- PEP 369 – Post import hooks
- PEP 382 – Namespace Packages
- PEP 395 – Module Aliasing
- PEP 402 – Simplified Package Layout and Partitioning
- PEP ??? – import engine
- importlib.__import__ as the default builtins.__import__
Currently in Python, “builtin___import__()” in Python/bltinmodule.c makes a call to PyImport_ImportModuleLevelObject. Brett Cannon is working on making importlib.__import__ the default import call.
- the __experimental__ module
like the __future__ module, but for less-stable APIs that are likely to go in focus on stdlib (room for experimental syntax too?) (higher exposure testing)
Appendix: Imports in Alternate Python Implementations¶
http://readthedocs.org/search/project/?q=import&selected_facets=project%3Apypy http://codespeak.net/pypy/dist/pypy/doc/clr-module.html http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html https://bugs.pypy.org/issue367
Appendix: Easter Eggs¶
The Python devs are a playful lot.
from __future__ import flufl¶
from __future__ import braces¶
How It Works¶
- Example: Plain Syntax Handler
- Example: From Name Syntax Handler
- Example: From Star Syntax Handler
- Example: builtins.__import__
- Example: Naively Track Imports
- Example: Import Tracking, Take 2
- Example: Statement Local Namespaces
- Example: Protecting a High-Latency Filesystem
- Example: Customizing Access to a Specific Module Path
- Example: PEPS 382 and 402 as Import Hooks
- Example: Import Engine as an Import Hook
- Example: lazy imports
- Example: “importing” straight from a file
Appendix: Imports in the Python Community¶
Appendix: Troubleshooting Imports¶
Causes of ImportError¶
- turn into ImportError subclasses, __cause__
Other Exceptions During Import¶
block imports on the current working directory¶
By default Python will look for a module in your current working directory before trying the stdlib. The explicit relative import syntax of 2.7 help with this, but only to an extent.
To completely keep Python from trying the CWD, simply run “sys.path.remove(‘’)” and optionally follow that with “sys.path.append(‘’)”.
So the question remains, when did the empty string get added to (the front of) sys.path, and why?
Dr. Brett Cannon gave a talk at PyCon 2010 <http://python.mirocommunity.org/video/1491> and PyCon 2008 (can’t find video).
importlib extensions: http://packages.python.org/importers/
http://mail.python.org/mailman/listinfo/import-sig http://docs.python.org/dev/reference/simple_stmts.html#the-import-statement http://docs.python.org/dev/reference/simple_stmts.html#future-statements http://docs.python.org/dev/reference/toplevel_components.html http://docs.python.org/dev/reference/executionmodel.html#naming-and-binding http://docs.python.org/dev/reference/datamodel.html#the-standard-type-hierarchy (modules)
http://docs.python.org/dev/tutorial/modules.html http://docs.python.org/release/1.5.1p1/tut/modules.html http://docs.python.org/release/1.4/tut/node41.html#SECTION00700000000000000000 http://docs.python.org/dev/library/modules.html http://docs.python.org/dev/faq/programming.html#what-are-the-best-practices-for-using-import-in-a-module http://docs.python.org/dev/faq/programming.html#how-can-i-have-modules-that-mutually-import-each-other http://docs.python.org/dev/faq/programming.html#import-x-y-z-returns-module-x-how-do-i-get-z http://docs.python.org/dev/faq/programming.html#when-i-edit-an-imported-module-and-reimport-it-the-changes-don-t-show-up-why-does-this-happen
Open bugs: http://bugs.python.org/issue?%40search_text=import&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40columns=creation&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=1&%40columns=status&%40group=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search
Not closed: http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40columns=creation&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=-1%2C1%2C3&%40columns=status&%40group=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search
Import Who’s Who¶
People who have been involved with Python’s imports (incomplete):
“experts”: Brett Cannon, Nick Coghlan
Brett Cannon (importlib) Just van Rossum (PEP 302) Paul Moore (PEP 302) Aahz (PEP 328) Martin v. Loewis (PEP 382) P.J. Eby (PEP 402) James C. Ahlstrom (PEP 273) Nick Coghlan (PEP 338, PEP 366, PEP 395) Christian Heimes (PEP 370) Thomas Wouters (PEP 221) Barry Warsaw (PEP 3147) Tim Peters (PEP 225) Guido van Rossum (pretty much everything else <wink>)
- import hook
- An object with a find() method that conforms to PEP 302. May also refer to the class of such an object.
- An object with a load() method that conforms to PEP 302. May also refer to the class of such an object.
- Mostly synonymous with path importer.
- path importer
- An object, class, or other code that may be plugged into the PEP 302 import machinery. Often this term refers specifically to those that are used with sys.path_hooks.
- The object generated at the highest level of the import process. In the normal import statement, it is the object bound to the name.
- module name
- The value bound to the __name__ attribute of the corresponding module object. This will be the full qualified name relative to the sys.path value at import time.
- A module corresponding to a directory. The module is populated with the results of evaluating the __init__.py file in the directory. Other .py files and directories in the directory may be imported as submodules of the package.
- namespace package
- A package, possibly without its own module execution, into which subpackages are aggregated according to a single namespace. The “zope” package is a good example.
- package portion
The import Statement¶
The import statement is the syntactic mechanism you use to invoke Python’s powerful import machinery. It has two forms: the regular import and from-import. In a moment we’ll walk through the ins and outs of both forms.
When you use the import statement in either form, you identify a module and its parent modules together as a module name. By default, each successive parent module is imported from the outside in, followed by the actual module you wanted. After that the appropriate name is bound in the current local namespace. Most imports are going to happen at the module level where the name will be bound in that module’s [global] namespace.
A module object is the result of importing. We use the term “module” to refer to this object as well as to the thing that Python used to create the object, usually a file. A package is a special kind of module. Where a normal module corresponds to a file, a package corresponds to a directory.
Changing the Import Behavior¶
You can override the full import machinery by overriding builtins.__import__().
Now you can add special “loader” objects to a couple of different places in the sys module to take control of imports in more targeted ways. A loader translates a module name into a “finder” object, if it can. The finder, in turn, converts the module name into the corresponding module object, which it sticks into sys.modules.
This entire process is explained much more in-depth in a later section and in the appendix.
All the Python variables related to the default import behavior is stored in the sys module. This includes sys.path, sys.modules, sys.meta_path, and sys.path_hooks.
What It Means¶
When It Happens and When Not¶
The imp Module¶
PyImport_GetModuleDict() used to get sys.modules (see J.3.9).
- Python/bltinmodule.c - _builtin___import__()
This is the default handler for the import statement. In 2.7 it is __builtin__.__import__().
The importlib Module¶
sys.modules used to get sys.modules.
See the site module documentation.
More in Appendix B.
A History of Python’s import Statement¶
A Brief History of Python¶
The Origins of Python’s import¶
http://python-history.blogspot.com/2009/02/adding-support-for-user-defined-classes.html http://www.python.org/doc/essays/foreword/ http://python-history.blogspot.com/2009/01/brief-timeline-of-python.html
The import statement has been a part of Python since the very beginning, though with more limited behavior.
Like many things in Python, the syntax for the import statement has its roots in Modula-3.