orphan:

Appendix: Import Syntax Under the Hood

Appendix: Import-related Files

orphan:

sys.path related

http://docs.python.org/library/sys.html#sys.path http://docs.python.org/c-api/init.html#Py_GetPath http://hg.python.org/cpython/file/default/Modules/getpath.c#l21

http://docs.python.org/dev/library/site.html

1. calculate the 4 site-packages dirs
2. add them to sys.path
3. check for and execute .pth files in those site-packages dirs
4. calculate user site-packages

site.getsitepackages()

Unix/Mac:

<sys.prefix>/lib/python#.#/site-packages <sys.exec_prefix>/lib/python#.#/site-packages <sys.prefix>/lib/site-python <sys.exec_prefix>/lib/site-python

Windows:

<sys.prefix>/ <sys.exec_prefix>/ <sys.prefix>/lib/site-packages <sys.exec_prefix>/lib/site-packages

site related

http://docs.python.org/dev/library/site.html

• module: site
• module: sitecustomize
• module: usercustomize

site.getusersitepackages()

Standard Library

Lib/importlib/*.py Lib/pkgutil.py

Tests

Lib/Test/...

CPython

Python/import.c Python/importdl.c Python/importdl.h Include/import.h Python/sysmodule.c Python/pythonrun.c

PyPy

Jython

IronPython

Appendix: An Extended Timeline of Importing in Python

orphan:

Some Context

(origins) http://www.python.org/community/sigs/retired/import-sig/ http://www.python.org/dev/peps/pep-3121/#id11 (1.5) http://www.python.org/doc/essays/packages.html #http://www.python.org/doc/essays/packages/ Modula-3 influence: http://python-history.blogspot.com/2009/02/adding-support-for-user-defined-classes.html http://python-history.blogspot.com/2009/01/brief-timeline-of-python.html http://python-history.blogspot.com/2009/03/dynamically-loaded-modules.html http://docs.python.org/dev/whatsnew/index.html http://python.org/download/releases/src/ http://hg.python.org/cpython-fullhistory/tags http://hg.python.org/cpython-fullhistory/graph/3cd033e6b530?revcount=800 http://hg.python.org/cpython-fullhistory/log/62bdb1cbe0f5/Python/import.c?revcount=120 initial: http://hg.python.org/cpython-fullhistory/file/fc6fcd7df4f7/Python/import.c 0.9.8: http://hg.python.org/cpython-fullhistory/file/17eff686be30/Python/import.c builtin___import__(), importdl.c: http://hg.python.org/cpython-fullhistory/rev/d7e91437f0a2 PyImport_Import: http://hg.python.org/cpython-fullhistory/rev/292193170da1 highlights of “What’s New”: http://nedbatchelder.com/blog/201109/whats_in_which_python.html code_swarm: http://vimeo.com/1093745

(ni) introduced (1.3): http://hg.python.org/cpython-fullhistory/rev/ec0b42889243 deprecated (1.5): http://docs.python.org/release/1.5/lib/node40.html still lives: http://docs.python.org/library/imputil.html#examples

(ihooks) introduced (1.3): http://hg.python.org/cpython-fullhistory/rev/ec0b42889243 removed (3.0): http://docs.python.org/release/2.6.2/library/undoc.html#miscellaneous-useful-utilities http://pydoc.org/2.4.1/ihooks.html

—–

The versions and dates are derived from a post on Guido’s “History of Python” blog. I’ve correlated the entries in section B.1 to versions by either explicit reference or by matching their commits to a version. Section B.2 also maps commits to versions. In both cases, I did my best to determine that mapping, but some may be off by a version.

The Extended Timeline

Initial Checkin (1990)

• Checks sys.modules
• Loads modules from sys.path or current dir (if sys.path is empty)
• Supports IMPORT_NAME and IMPORT_FROM opcodes
• No support for .pyc files
• No support for packages
• No support for C extension modules?
• No ImportError

Python 0.9.1 (Feb. 1991)

• builtin module support (C extention modules)

Python 1.0 (1994)

• Support for extension modules
• Support for .pyc files

Python 1.2 (1995)

• (Python/bltinmodule.c) __import__() builtin introduced
• (Python/import.c) dynamic module support factored out into importdl.c

Python 1.3 (1995)

• “ni” module introduced

Python 1.4 (1996)

http://docs.python.org/release/1.4/ref/ *

Python 1.5 (1998)

• Support for packages
• “site-packages” and “site-python” directories introduced
• “__all__” introduced
• “ni” module deprecated
• (Python/import.c) PyImport_Import() introduced

Python 2.0 (2000)

• PEP 221 – Import As

Python 2.1 (2001)

• PEP 235 – Import on Case-Insensitive Platforms

Python 2.2 (2001)

Python 2.3 (2003)

• PEP 273 – Import Modules from Zip Archives
• PEP 302 – New Import Hooks

Python 2.4 (2004)

• PEP 328 – Imports: Multi-Line and Absolute/Relative (multi-line portion)

Python 2.5 (2006)

• PEP 328 (relative imports portion)
• PEP 338 – Executing modules as scripts

Python 2.6/3.0 (2008)

• PEP 366 – Main module explicit relative imports
• PEP 370 – Per user site-packages directory

Python 3.0 (2008)

• reload removed from builtins
• ihooks module removed from stdlib
• imputil module removed from stdlib

Python 3.1 (2009)

• importlib module added

Python 3.2 (2011)

• PEP 3147 – PYC Repository Directories

Python 3.3 (2012)

• see appendix D

Appendix: A Timeline of Import-related Commits

orphan:

http://hg.python.org/cpython-fullhistory/log/62bdb1cbe0f5/Python/import.c?revcount=120

Python 0.9.0 (1991)

Python 0.9.1 (1991)

Python 0.9.2 (1991)

Python 0.9.4 (1991)

Python 0.9.5 (1992)

Python 0.9.6 (1992)

Python 0.9.7 (1992)

Python 0.9.8 (1993)

Python 0.9.9 (1993)

Python 1.0.0 (1994)

Python 1.0.2 (1994)

Python 1.0.3 (1994)

Python 1.0.4 (1994)

Python 1.1 (1994)

Python 1.1.1 (1994)

Python 1.2 (1995)

Python 1.3 (1995)

Python 1.4 (1996)

Python 1.5 (1998)

Python 1.5.1 (1998)

Python 1.5.2 (1999)

Python 1.6 (2000)

Python 2.0 (2000)

Python 2.1 (2001)

Python 2.2 (2001)

Python 2.3 (2003)

Python 2.4 (2004)

Python 2.5 (2006)

Python 2.6 (2008)

Python 3.0 (2008)

Python 2.7 (2010)

Python 3.1 (2010)

Python 3.2 (2011)

Python 3.3 (2012)

Appendix: Ongoing Core Efforts to Improve Importing

orphan:

PEPs

• PEP 369 – Post import hooks
• PEP 382 – Namespace Packages
• PEP 395 – Module Aliasing
• PEP 402 – Simplified Package Layout and Partitioning
• PEP ??? – import engine

Rejected PEPs:

• PEP 299 – Special __main__() function in modules
• PEP 3122 – Delineation of the main module

Projects

• importlib.__import__ as the default builtins.__import__

Currently in Python, “builtin___import__()” in Python/bltinmodule.c makes a call to PyImport_ImportModuleLevelObject. Brett Cannon is working on making importlib.__import__ the default import call[1].

• the __experimental__ module

http://mail.python.org/pipermail/python-ideas/2010-June/007357.html http://mail.python.org/pipermail/python-ideas/2011-August/011278.html

like the __future__ module, but for less-stable APIs that are likely to go in focus on stdlib (room for experimental syntax too?) (higher exposure testing)

Appendix: Imports in Alternate Python Implementations

PyPy

http://readthedocs.org/search/project/?q=import&selected_facets=project%3Apypy http://codespeak.net/pypy/dist/pypy/doc/clr-module.html http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html https://bugs.pypy.org/issue367

Jython

http://www.google.com/search?sitesearch=www.jython.org&q=import&Search=Search

IronPython

http://ironpython.codeplex.com/wiki/search?tab=Home&SearchText=import

Appendix: Easter Eggs

The Python devs are a playful lot.

import this

http://www.wefearchange.org/2010/06/import-this-and-zen-of-python.html

import antigravity

http://xkcd.com/353/ http://python-history.blogspot.com/2010/06/import-antigravity.html

from __future__ import flufl

http://www.python.org/dev/peps/pep-0401/ http://sayspy.blogspot.com/2009/03/guido-has-retired-as-bdfl.html

from __future__ import braces

import __hello__

Appendix: Import Examples

orphan:

How It Works

• Example: Plain Syntax Handler
• Example: From Name Syntax Handler
• Example: From Star Syntax Handler
• Example: builtins.__import__

Uncategorized

• Example: Naively Track Imports
• Example: Import Tracking, Take 2
• Example: Statement Local Namespaces
• Example: Protecting a High-Latency Filesystem
• Example: Customizing Access to a Specific Module Path
• Example: PEPS 382 and 402 as Import Hooks
• Example: Import Engine as an Import Hook
• Example: lazy imports
• Example: “importing” straight from a file

Appendix: Imports in the Python Community

Appendix: Troubleshooting Imports

orphan:

Causes of ImportError

• turn into ImportError subclasses, __cause__

Other Exceptions During Import

• SyntaxError
• IOError?

Common Import-related Problems

circular imports

module behaves differently when run as script

• don’t run non-scripts as scripts; import in a test script

imports in scripts and at REPL behave differently than expected

• minimize the amount of code in scripts
• relative imports behave a little differently in scripts

reloading modules

no .pyc created

• .pyc for <name> is created only for “import <name>”
• caching was turned off
• file is actually in __pycache__ directory
• python run with -O flag (optimized) so .pyc files created

undesired import conflicts with files in CWD

By default the current working directory is first on sys.path. If this is causing trouble, you can move it to the back of the line:

try: sys.path.remove('')
except ValueError: pass
else: sys.path.append('')

Alternatively, you could remove it entirely (don’t append it back on).

stale code

http://bugs.python.org/issue8087

orphaned pyc file getting used

If a directory has a pyc file but no matching py file, the module will be loaded from the pyc file directly. Starting with PEP 3147 (Python 3.2), orphaned pyc files in the __pycache__ directory are NOT loaded. The behavior use of pyc files otherwise stays the same. Either way, if you don’t want the module to be loaded from an orphaned pyc file, delete that file. Also see this tracker ticket <orphaned_pyc_files>.

alternate sys.modules ignored

Sometimes it can be helpful to replace sys.modules with a custom dictionary. However, in CPython, this does not affect the underlying dictionary that was originally bound to sys.modules. That is a separate part of the interpreter state. The behavior of the default _builtin___import__(), implemented in Python/import.c, actually uses this underlying dictionary through PyImport_GetModuleDict(), rather than explicitly pulling sys.modules. So your fancy-pantsy sys.modules is never used.

Luckily, importlib does explicitly use sys.modules, so if you switch over to that it should work just fine. This will be an even smaller issue once importlib’s __import__ because the default builtin.

(see http://bugs.python.org/issue12633).

import loads some other mysterious module

If you have a module file in your sys.path, and you try to import it, sometimes the import will succeed but the module will be the wrong one. This can be both mysterious and perplexing.

The first thing to do is to see if you have a package (directory with a __init__.py) by the same name in the same place as that module file. If so, Python will import from the package instead of the module. To verify this, import the module: “import <name>” and then check the module in sys.modules: “import sys; print(sys.modules[‘<name>’])”. You should see it pointing to the __init__.py of the package instead of the module file you were expecting.

Appendix: Import Tips and Tricks

orphan:

block imports on the current working directory

By default Python will look for a module in your current working directory before trying the stdlib. The explicit relative import syntax of 2.7 help with this, but only to an extent.

To completely keep Python from trying the CWD, simply run “sys.path.remove(‘’)” and optionally follow that with “sys.path.append(‘’)”.

So the question remains, when did the empty string get added to (the front of) sys.path, and why?

...

Appendix: Other Import-related Resources

Online References

orphan:

Dr. Brett Cannon gave a talk at PyCon 2010 <http://python.mirocommunity.org/video/1491> and PyCon 2008 (can’t find video).

importlib extensions: http://packages.python.org/importers/

flowchart: http://svn.python.org/view/sandbox/trunk/import_in_py/docs/

http://mail.python.org/mailman/listinfo/import-sig http://docs.python.org/dev/reference/simple_stmts.html#the-import-statement http://docs.python.org/dev/reference/simple_stmts.html#future-statements http://docs.python.org/dev/reference/toplevel_components.html http://docs.python.org/dev/reference/executionmodel.html#naming-and-binding http://docs.python.org/dev/reference/datamodel.html#the-standard-type-hierarchy (modules)

http://docs.python.org/dev/tutorial/modules.html http://docs.python.org/release/1.5.1p1/tut/modules.html http://docs.python.org/release/1.4/tut/node41.html#SECTION00700000000000000000 http://docs.python.org/dev/library/modules.html http://docs.python.org/dev/faq/programming.html#what-are-the-best-practices-for-using-import-in-a-module http://docs.python.org/dev/faq/programming.html#how-can-i-have-modules-that-mutually-import-each-other http://docs.python.org/dev/faq/programming.html#import-x-y-z-returns-module-x-how-do-i-get-z http://docs.python.org/dev/faq/programming.html#when-i-edit-an-imported-module-and-reimport-it-the-changes-don-t-show-up-why-does-this-happen

http://docs.python.org/dev/library/py_compile.html http://docs.python.org/dev/library/compileall.html

http://www.doughellmann.com/PyMOTW/sys/imports.html

http://lucumr.pocoo.org/2011/9/21/python-import-blackbox/

http://code.google.com/p/backport/ http://mirnazim.org/writings/python-ecosystem-introduction/ http://nedbatchelder.com/blog/201112/duplicitous_django_settings.html

http://lucumr.pocoo.org/2011/12/7/thoughts-on-python3/ http://pythonic.pocoo.org/2009/3/4/imports-in-functions-you-sure-about-that

Open bugs: http://bugs.python.org/issue?%40search_text=import&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40columns=creation&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=1&%40columns=status&%40group=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search

Not closed: http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40columns=creation&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=-1%2C1%2C3&%40columns=status&%40group=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search

Closed: http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40columns=creation&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=2&%40columns=status&%40group=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search

<stack overflow>

http://stackoverflow.com/questions/279237/python-import-a-module-from-a-folder

<cookbook recipes>

<others>

https://github.com/zacharyvoase/metaspace

http://www.youtube.com/watch?v=DkW5CSZ_VII http://aroberge.blogspot.com/2006/02/python-wish-new-meaning-for-import-as.html

Import Who’s Who

orphan:

People who have been involved with Python’s imports (incomplete):

“experts”: Brett Cannon, Nick Coghlan

Brett Cannon (importlib) Just van Rossum (PEP 302) Paul Moore (PEP 302) Aahz (PEP 328) Martin v. Loewis (PEP 382) P.J. Eby (PEP 402) James C. Ahlstrom (PEP 273) Nick Coghlan (PEP 338, PEP 366, PEP 395) Christian Heimes (PEP 370) Thomas Wouters (PEP 221) Barry Warsaw (PEP 3147) Tim Peters (PEP 225) Guido van Rossum (pretty much everything else <wink>)

Glossary

orphan:

The terminology surrounding imports can get confusing. This glossary should help.

import hook

...

finder

An object with a find() method that conforms to PEP 302. May also refer to the class of such an object.

loader

An object with a load() method that conforms to PEP 302. May also refer to the class of such an object.

importer

Mostly synonymous with path importer.

path importer

An object, class, or other code that may be plugged into the PEP 302 import machinery. Often this term refers specifically to those that are used with sys.path_hooks.

module

The object generated at the highest level of the import process. In the normal import statement, it is the object bound to the name.

module name

The value bound to the __name__ attribute of the corresponding module object. This will be the full qualified name relative to the sys.path value at import time.

package

A module corresponding to a directory. The module is populated with the results of evaluating the __init__.py file in the directory. Other .py files and directories in the directory may be imported as submodules of the package.

namespace package

A package, possibly without its own module execution, into which subpackages are aggregated according to a single namespace. The “zope” package is a good example.

package portion

...

• Index
• Search Page