## 問題
You would like to customize Python’s import statement so that it can transparently load modules from a remote machine.
## 解決方案
First, a serious disclaimer about security. The idea discussed in this recipe would be wholly bad without some kind of extra security and authentication layer. That said, the main goal is actually to take a deep dive into the inner workings of Python’s import statement. If you get this recipe to work and understand the inner workings, you’ll have a solid foundation of customizing import for almost any other purpose. With that out of the way, let’s carry on.
At the core of this recipe is a desire to extend the functionality of the import statement. There are several approaches for doing this, but for the purposes of illustration, start by making the following directory of Python code:
~~~
testcode/
spam.py
fib.py
grok/
__init__.py
blah.py
~~~
The content of these files doesn’t matter, but put a few simple statements and functions in each file so you can test them and see output when they’re imported. For example:
~~~
# spam.py
print("I'm spam")
def hello(name):
print('Hello %s' % name)
# fib.py
print("I'm fib")
def fib(n):
if n < 2:
return 1
else:
return fib(n-1) + fib(n-2)
# grok/__init__.py
print("I'm grok.__init__")
# grok/blah.py
print("I'm grok.blah")
~~~
The goal here is to allow remote access to these files as modules. Perhaps the easiest way to do this is to publish them on a web server. Simply go to the testcode directory and run Python like this:
~~~
bash % cd testcode
bash % python3 -m http.server 15000
Serving HTTP on 0.0.0.0 port 15000 ...
~~~
Leave that server running and start up a separate Python interpreter. Make sure you can access the remote files using urllib. For example:
~~~
>>> from urllib.request import urlopen
>>> u = urlopen('http://localhost:15000/fib.py')
>>> data = u.read().decode('utf-8')
>>> print(data)
# fib.py
print("I'm fib")
def fib(n):
if n < 2:
return 1
else:
return fib(n-1) + fib(n-2)
>>>
~~~
Loading source code from this server is going to form the basis for the remainder of this recipe. Specifically, instead of manually grabbing a file of source code using urlop en(), the import statement will be customized to do it transparently behind the scenes.
The first approach to loading a remote module is to create an explicit loading function for doing it. For example:
~~~
import imp
import urllib.request
import sys
def load_module(url):
u = urllib.request.urlopen(url)
source = u.read().decode('utf-8')
mod = sys.modules.setdefault(url, imp.new_module(url))
code = compile(source, url, 'exec')
mod.__file__ = url
mod.__package__ = ''
exec(code, mod.__dict__)
return mod
~~~
This function merely downloads the source code, compiles it into a code object using compile(), and executes it in the dictionary of a newly created module object. Here’s how you would use the function:
~~~
>>> fib = load_module('http://localhost:15000/fib.py')
I'm fib
>>> fib.fib(10)
89
>>> spam = load_module('http://localhost:15000/spam.py')
I'm spam
>>> spam.hello('Guido')
Hello Guido
>>> fib
<module 'http://localhost:15000/fib.py' from 'http://localhost:15000/fib.py'>
>>> spam
<module 'http://localhost:15000/spam.py' from 'http://localhost:15000/spam.py'>
>>>
~~~
As you can see, it “works” for simple modules. However, it’s not plugged into the usual import statement, and extending the code to support more advanced constructs, such as packages, would require additional work.
A much slicker approach is to create a custom importer. The first way to do this is to create what’s known as a meta path importer. Here is an example:
~~~
# urlimport.py
import sys
import importlib.abc
import imp
from urllib.request import urlopen
from urllib.error import HTTPError, URLError
from html.parser import HTMLParser
# Debugging
import logging
log = logging.getLogger(__name__)
# Get links from a given URL
def _get_links(url):
class LinkParser(HTMLParser):
def handle_starttag(self, tag, attrs):
if tag == 'a':
attrs = dict(attrs)
links.add(attrs.get('href').rstrip('/'))
links = set()
try:
log.debug('Getting links from %s' % url)
u = urlopen(url)
parser = LinkParser()
parser.feed(u.read().decode('utf-8'))
except Exception as e:
log.debug('Could not get links. %s', e)
log.debug('links: %r', links)
return links
class UrlMetaFinder(importlib.abc.MetaPathFinder):
def __init__(self, baseurl):
self._baseurl = baseurl
self._links = { }
self._loaders = { baseurl : UrlModuleLoader(baseurl) }
def find_module(self, fullname, path=None):
log.debug('find_module: fullname=%r, path=%r', fullname, path)
if path is None:
baseurl = self._baseurl
else:
if not path[0].startswith(self._baseurl):
return None
baseurl = path[0]
parts = fullname.split('.')
basename = parts[-1]
log.debug('find_module: baseurl=%r, basename=%r', baseurl, basename)
# Check link cache
if basename not in self._links:
self._links[baseurl] = _get_links(baseurl)
# Check if it's a package
if basename in self._links[baseurl]:
log.debug('find_module: trying package %r', fullname)
fullurl = self._baseurl + '/' + basename
# Attempt to load the package (which accesses __init__.py)
loader = UrlPackageLoader(fullurl)
try:
loader.load_module(fullname)
self._links[fullurl] = _get_links(fullurl)
self._loaders[fullurl] = UrlModuleLoader(fullurl)
log.debug('find_module: package %r loaded', fullname)
except ImportError as e:
log.debug('find_module: package failed. %s', e)
loader = None
return loader
# A normal module
filename = basename + '.py'
if filename in self._links[baseurl]:
log.debug('find_module: module %r found', fullname)
return self._loaders[baseurl]
else:
log.debug('find_module: module %r not found', fullname)
return None
def invalidate_caches(self):
log.debug('invalidating link cache')
self._links.clear()
# Module Loader for a URL
class UrlModuleLoader(importlib.abc.SourceLoader):
def __init__(self, baseurl):
self._baseurl = baseurl
self._source_cache = {}
def module_repr(self, module):
return '<urlmodule %r from %r>' % (module.__name__, module.__file__)
# Required method
def load_module(self, fullname):
code = self.get_code(fullname)
mod = sys.modules.setdefault(fullname, imp.new_module(fullname))
mod.__file__ = self.get_filename(fullname)
mod.__loader__ = self
mod.__package__ = fullname.rpartition('.')[0]
exec(code, mod.__dict__)
return mod
# Optional extensions
def get_code(self, fullname):
src = self.get_source(fullname)
return compile(src, self.get_filename(fullname), 'exec')
def get_data(self, path):
pass
def get_filename(self, fullname):
return self._baseurl + '/' + fullname.split('.')[-1] + '.py'
def get_source(self, fullname):
filename = self.get_filename(fullname)
log.debug('loader: reading %r', filename)
if filename in self._source_cache:
log.debug('loader: cached %r', filename)
return self._source_cache[filename]
try:
u = urlopen(filename)
source = u.read().decode('utf-8')
log.debug('loader: %r loaded', filename)
self._source_cache[filename] = source
return source
except (HTTPError, URLError) as e:
log.debug('loader: %r failed. %s', filename, e)
raise ImportError("Can't load %s" % filename)
def is_package(self, fullname):
return False
# Package loader for a URL
class UrlPackageLoader(UrlModuleLoader):
def load_module(self, fullname):
mod = super().load_module(fullname)
mod.__path__ = [ self._baseurl ]
mod.__package__ = fullname
def get_filename(self, fullname):
return self._baseurl + '/' + '__init__.py'
def is_package(self, fullname):
return True
# Utility functions for installing/uninstalling the loader
_installed_meta_cache = { }
def install_meta(address):
if address not in _installed_meta_cache:
finder = UrlMetaFinder(address)
_installed_meta_cache[address] = finder
sys.meta_path.append(finder)
log.debug('%r installed on sys.meta_path', finder)
def remove_meta(address):
if address in _installed_meta_cache:
finder = _installed_meta_cache.pop(address)
sys.meta_path.remove(finder)
log.debug('%r removed from sys.meta_path', finder)
~~~
Here is an interactive session showing how to use the preceding code:
~~~
>>> # importing currently fails
>>> import fib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'fib'
>>> # Load the importer and retry (it works)
>>> import urlimport
>>> urlimport.install_meta('http://localhost:15000')
>>> import fib
I'm fib
>>> import spam
I'm spam
>>> import grok.blah
I'm grok.__init__
I'm grok.blah
>>> grok.blah.__file__
'http://localhost:15000/grok/blah.py'
>>>
~~~
This particular solution involves installing an instance of a special finder object UrlMe taFinder as the last entry in sys.meta_path. Whenever modules are imported, the finders in sys.meta_path are consulted in order to locate the module. In this example, the UrlMetaFinder instance becomes a finder of last resort that’s triggered when a module can’t be found in any of the normal locations.
As for the general implementation approach, the UrlMetaFinder class wraps around a user-specified URL. Internally, the finder builds sets of valid links by scraping them from the given URL. When imports are made, the module name is compared against this set of known links. If a match can be found, a separate UrlModuleLoader class is used to load source code from the remote machine and create the resulting module object. One reason for caching the links is to avoid unnecessary HTTP requests on repeated imports.
The second approach to customizing import is to write a hook that plugs directly into the sys.path variable, recognizing certain directory naming patterns. Add the following class and support functions to urlimport.py:
~~~
# urlimport.py
# ... include previous code above ...
# Path finder class for a URL
class UrlPathFinder(importlib.abc.PathEntryFinder):
def __init__(self, baseurl):
self._links = None
self._loader = UrlModuleLoader(baseurl)
self._baseurl = baseurl
def find_loader(self, fullname):
log.debug('find_loader: %r', fullname)
parts = fullname.split('.')
basename = parts[-1]
# Check link cache
if self._links is None:
self._links = [] # See discussion
self._links = _get_links(self._baseurl)
# Check if it's a package
if basename in self._links:
log.debug('find_loader: trying package %r', fullname)
fullurl = self._baseurl + '/' + basename
# Attempt to load the package (which accesses __init__.py)
loader = UrlPackageLoader(fullurl)
try:
loader.load_module(fullname)
log.debug('find_loader: package %r loaded', fullname)
except ImportError as e:
log.debug('find_loader: %r is a namespace package', fullname)
loader = None
return (loader, [fullurl])
# A normal module
filename = basename + '.py'
if filename in self._links:
log.debug('find_loader: module %r found', fullname)
return (self._loader, [])
else:
log.debug('find_loader: module %r not found', fullname)
return (None, [])
def invalidate_caches(self):
log.debug('invalidating link cache')
self._links = None
# Check path to see if it looks like a URL
_url_path_cache = {}
def handle_url(path):
if path.startswith(('http://', 'https://')):
log.debug('Handle path? %s. [Yes]', path)
if path in _url_path_cache:
finder = _url_path_cache[path]
else:
finder = UrlPathFinder(path)
_url_path_cache[path] = finder
return finder
else:
log.debug('Handle path? %s. [No]', path)
def install_path_hook():
sys.path_hooks.append(handle_url)
sys.path_importer_cache.clear()
log.debug('Installing handle_url')
def remove_path_hook():
sys.path_hooks.remove(handle_url)
sys.path_importer_cache.clear()
log.debug('Removing handle_url')
~~~
To use this path-based finder, you simply add URLs to sys.path. For example:
~~~
>>> # Initial import fails
>>> import fib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'fib'
>>> # Install the path hook
>>> import urlimport
>>> urlimport.install_path_hook()
>>> # Imports still fail (not on path)
>>> import fib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'fib'
>>> # Add an entry to sys.path and watch it work
>>> import sys
>>> sys.path.append('http://localhost:15000')
>>> import fib
I'm fib
>>> import grok.blah
I'm grok.__init__
I'm grok.blah
>>> grok.blah.__file__
'http://localhost:15000/grok/blah.py'
>>>
~~~
The key to this last example is the handle_url() function, which is added to the sys.path_hooks variable. When the entries on sys.path are being processed, the functions in sys.path_hooks are invoked. If any of those functions return a finder object, that finder is used to try to load modules for that entry on sys.path.
It should be noted that the remotely imported modules work exactly like any other module. For instance:
~~~
>>> fib
<urlmodule 'fib' from 'http://localhost:15000/fib.py'>
>>> fib.__name__
'fib'
>>> fib.__file__
'http://localhost:15000/fib.py'
>>> import inspect
>>> print(inspect.getsource(fib))
# fib.py
print("I'm fib")
def fib(n):
if n < 2:
return 1
else:
return fib(n-1) + fib(n-2)
>>>
~~~
## 討論
Before discussing this recipe in further detail, it should be emphasized that Python’s module, package, and import mechanism is one of the most complicated parts of the entire language—often poorly understood by even the most seasoned Python programmers unless they’ve devoted effort to peeling back the covers. There are several critical documents that are worth reading, including the documentation for the?[importlib module](https://docs.python.org/3/library/importlib.html)?and?[PEP 302](http://www.python.org/dev/peps/pep-0302). That documentation won’t be repeated here, but some essential highlights will be discussed.
First, if you want to create a new module object, you use the imp.new_module() function. For example:
~~~
>>> import imp
>>> m = imp.new_module('spam')
>>> m
<module 'spam'>
>>> m.__name__
'spam'
>>>
~~~
Module objects usually have a few expected attributes, including __file__ (the name of the file that the module was loaded from) and __package__ (the name of the enclosing package, if any).
Second, modules are cached by the interpreter. The module cache can be found in the dictionary sys.modules. Because of this caching, it’s common to combine caching and module creation together into a single step. For example:
~~~
>>> import sys
>>> import imp
>>> m = sys.modules.setdefault('spam', imp.new_module('spam'))
>>> m
<module 'spam'>
>>>
~~~
The main reason for doing this is that if a module with the given name already exists, you’ll get the already created module instead. For example:
~~~
>>> import math
>>> m = sys.modules.setdefault('math', imp.new_module('math'))
>>> m
<module 'math' from '/usr/local/lib/python3.3/lib-dynload/math.so'>
>>> m.sin(2)
0.9092974268256817
>>> m.cos(2)
-0.4161468365471424
>>>
~~~
Since creating modules is easy, it is straightforward to write simple functions, such as the load_module() function in the first part of this recipe. A downside of this approach is that it is actually rather tricky to handle more complicated cases, such as package imports. In order to handle a package, you would have to reimplement much of the underlying logic that’s already part of the normal import statement (e.g., checking for directories, looking for __init__.py files, executing those files, setting up paths, etc.). This complexity is one of the reasons why it’s often better to extend the import statement directly rather than defining a custom function.
Extending the import statement is straightforward, but involves a number of moving parts. At the highest level, import operations are processed by a list of “meta-path” finders that you can find in the list sys.meta_path. If you output its value, you’ll see the following:
~~~
>>> from pprint import pprint
>>> pprint(sys.meta_path)
[<class '_frozen_importlib.BuiltinImporter'>,
<class '_frozen_importlib.FrozenImporter'>,
<class '_frozen_importlib.PathFinder'>]
>>>
~~~
When executing a statement such as import fib, the interpreter walks through the finder objects on sys.meta_path and invokes their find_module() method in order to locate an appropriate module loader. It helps to see this by experimentation, so define the following class and try the following:
~~~
>>> class Finder:
... def find_module(self, fullname, path):
... print('Looking for', fullname, path)
... return None
...
>>> import sys
>>> sys.meta_path.insert(0, Finder()) # Insert as first entry
>>> import math
Looking for math None
>>> import types
Looking for types None
>>> import threading
Looking for threading None
Looking for time None
Looking for traceback None
Looking for linecache None
Looking for tokenize None
Looking for token None
>>>
~~~
Notice how the find_module() method is being triggered on every import. The role of the path argument in this method is to handle packages. When packages are imported, it is a list of the directories that are found in the package’s __path__ attribute. These are the paths that need to be checked to find package subcomponents. For example, notice the path setting for xml.etree and xml.etree.ElementTree:
~~~
>>> import xml.etree.ElementTree
Looking for xml None
Looking for xml.etree ['/usr/local/lib/python3.3/xml']
Looking for xml.etree.ElementTree ['/usr/local/lib/python3.3/xml/etree']
Looking for warnings None
Looking for contextlib None
Looking for xml.etree.ElementPath ['/usr/local/lib/python3.3/xml/etree']
Looking for _elementtree None
Looking for copy None
Looking for org None
Looking for pyexpat None
Looking for ElementC14N None
>>>
~~~
The placement of the finder on sys.meta_path is critical. Remove it from the front of the list to the end of the list and try more imports:
~~~
>>> del sys.meta_path[0]
>>> sys.meta_path.append(Finder())
>>> import urllib.request
>>> import datetime
~~~
Now you don’t see any output because the imports are being handled by other entries in sys.meta_path. In this case, you would only see it trigger when nonexistent modules are imported:
~~~
>>> import fib
Looking for fib None
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'fib'
>>> import xml.superfast
Looking for xml.superfast ['/usr/local/lib/python3.3/xml']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'xml.superfast'
>>>
~~~
The fact that you can install a finder to catch unknown modules is the key to the UrlMetaFinder class in this recipe. An instance of UrlMetaFinder is added to the end of sys.meta_path, where it serves as a kind of importer of last resort. If the requested module name can’t be located by any of the other import mechanisms, it gets handled by this finder. Some care needs to be taken when handling packages. Specifically, the value presented in the path argument needs to be checked to see if it starts with the URL registered in the finder. If not, the submodule must belong to some other finder and should be ignored.
Additional handling of packages is found in the UrlPackageLoader class. This class, rather than importing the package name, tries to load the underlying __init__.py file. It also sets the module __path__ attribute. This last part is critical, as the value set will be passed to subsequent find_module() calls when loading package submodules. The path-based import hook is an extension of these ideas, but based on a somewhat different mechanism. As you know, sys.path is a list of directories where Python looks for modules. For example:
~~~
>>> from pprint import pprint
>>> import sys
>>> pprint(sys.path)
['',
'/usr/local/lib/python33.zip',
'/usr/local/lib/python3.3',
'/usr/local/lib/python3.3/plat-darwin',
'/usr/local/lib/python3.3/lib-dynload',
'/usr/local/lib/...3.3/site-packages']
>>>
~~~
Each entry in sys.path is additionally attached to a finder object. You can view these finders by looking at sys.path_importer_cache:
~~~
>>> pprint(sys.path_importer_cache)
{'.': FileFinder('.'),
'/usr/local/lib/python3.3': FileFinder('/usr/local/lib/python3.3'),
'/usr/local/lib/python3.3/': FileFinder('/usr/local/lib/python3.3/'),
'/usr/local/lib/python3.3/collections': FileFinder('...python3.3/collections'),
'/usr/local/lib/python3.3/encodings': FileFinder('...python3.3/encodings'),
'/usr/local/lib/python3.3/lib-dynload': FileFinder('...python3.3/lib-dynload'),
'/usr/local/lib/python3.3/plat-darwin': FileFinder('...python3.3/plat-darwin'),
'/usr/local/lib/python3.3/site-packages': FileFinder('...python3.3/site-packages'),
'/usr/local/lib/python33.zip': None}
>>>
~~~
sys.path_importer_cache tends to be much larger than sys.path because it records finders for all known directories where code is being loaded. This includes subdirectories of packages which usually aren’t included on sys.path.
To execute import fib, the directories on sys.path are checked in order. For each directory, the name fib is presented to the associated finder found in sys.path_im porter_cache. This is also something that you can investigate by making your own finder and putting an entry in the cache. Try this experiment:
~~~
>>> class Finder:
... def find_loader(self, name):
... print('Looking for', name)
... return (None, [])
...
>>> import sys
>>> # Add a "debug" entry to the importer cache
>>> sys.path_importer_cache['debug'] = Finder()
>>> # Add a "debug" directory to sys.path
>>> sys.path.insert(0, 'debug')
>>> import threading
Looking for threading
Looking for time
Looking for traceback
Looking for linecache
Looking for tokenize
Looking for token
>>>
~~~
Here, you’ve installed a new cache entry for the name debug and installed the name debug as the first entry on sys.path. On all subsequent imports, you see your finder being triggered. However, since it returns (None, []), processing simply continues to the next entry.
The population of sys.path_importer_cache is controlled by a list of functions stored in sys.path_hooks. Try this experiment, which clears the cache and adds a new path checking function to sys.path_hooks:
~~~
>>> sys.path_importer_cache.clear()
>>> def check_path(path):
... print('Checking', path)
... raise ImportError()
...
>>> sys.path_hooks.insert(0, check_path)
>>> import fib
Checked debug
Checking .
Checking /usr/local/lib/python33.zip
Checking /usr/local/lib/python3.3
Checking /usr/local/lib/python3.3/plat-darwin
Checking /usr/local/lib/python3.3/lib-dynload
Checking /Users/beazley/.local/lib/python3.3/site-packages
Checking /usr/local/lib/python3.3/site-packages
Looking for fib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'fib'
>>>
~~~
As you can see, the check_path() function is being invoked for every entry on sys.path. However, since an ImportError exception is raised, nothing else happens (checking just moves to the next function on sys.path_hooks).
Using this knowledge of how sys.path is processed, you can install a custom path checking function that looks for filename patterns, such as URLs. For instance:
~~~
>>> def check_url(path):
... if path.startswith('http://'):
... return Finder()
... else:
... raise ImportError()
...
>>> sys.path.append('http://localhost:15000')
>>> sys.path_hooks[0] = check_url
>>> import fib
Looking for fib # Finder output!
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'fib'
>>> # Notice installation of Finder in sys.path_importer_cache
>>> sys.path_importer_cache['http://localhost:15000']
<__main__.Finder object at 0x10064c850>
>>>
~~~
This is the key mechanism at work in the last part of this recipe. Essentially, a custom path checking function has been installed that looks for URLs in sys.path. When they are encountered, a new UrlPathFinder instance is created and installed into sys.path_importer_cache. From that point forward, all import statements that pass through that part of sys.path will try to use your custom finder.
Package handling with a path-based importer is somewhat tricky, and relates to the return value of the find_loader() method. For simple modules, find_loader() returns a tuple (loader, None) where loader is an instance of a loader that will import the module.
For a normal package, find_loader() returns a tuple (loader, path) where loader is the loader instance that will import the package (and execute __init__.py) and path is a list of the directories that will make up the initial setting of the package’s __path__ attribute. For example, if the base URL was[http://localhost:15000](http://localhost:15000/)?and a user executed import grok, the path returned by find_loader() would be [ ‘[http://local](http://local/)?host:15000/grok’ ].
The find_loader() must additionally account for the possibility of a namespace package. A namespace package is a package where a valid package directory name exists, but no __init__.py file can be found. For this case, find_loader() must return a tuple (None, path) where path is a list of directories that would have made up the package’s __path__ attribute had it defined an __init__.py file. For this case, the import mechanism moves on to check further directories on sys.path. If more namespace packages are found, all of the resulting paths are joined together to make a final namespace package. See Recipe 10.5 for more information on namespace packages.
There is a recursive element to package handling that is not immediately obvious in the solution, but also at work. All packages contain an internal path setting, which can be found in __path__ attribute. For example:
~~~
>>> import xml.etree.ElementTree
>>> xml.__path__
['/usr/local/lib/python3.3/xml']
>>> xml.etree.__path__
['/usr/local/lib/python3.3/xml/etree']
>>>
~~~
As mentioned, the setting of __path__ is controlled by the return value of the find_load er() method. However, the subsequent processing of __path__ is also handled by the functions in sys.path_hooks. Thus, when package subcomponents are loaded, the entries in __path__ are checked by the handle_url() function. This causes new instances of UrlPathFinder to be created and added to sys.path_importer_cache.
One remaining tricky part of the implementation concerns the behavior of the han dle_url() function and its interaction with the _get_links() function used internally. If your implementation of a finder involves the use of other modules (e.g., urllib.re quest), there is a possibility that those modules will attempt to make further imports in the middle of the finder’s operation. This can actually cause handle_url() and other parts of the finder to get executed in a kind of recursive loop. To account for this possibility, the implementation maintains a cache of created finders (one per URL). This avoids the problem of creating duplicate finders. In addition, the following fragment of code ensures that the finder doesn’t respond to any import requests while it’s in the processs of getting the initial set of links:
~~~
# Check link cache
if self._links is None:
self._links = [] # See discussion
self._links = _get_links(self._baseurl)
~~~
You may not need this checking in other implementations, but for this example involving URLs, it was required.
Finally, the invalidate_caches() method of both finders is a utility method that is supposed to clear internal caches should the source code change. This method is triggered when a user invokes importlib.invalidate_caches(). You might use it if you want the URL importers to reread the list of links, possibly for the purpose of being able to access newly added files.
In comparing the two approaches (modifying sys.meta_path or using a path hook), it helps to take a high-level view. Importers installed using sys.meta_path are free to handle modules in any manner that they wish. For instance, they could load modules out of a database or import them in a manner that is radically different than normal module/package handling. This freedom also means that such importers need to do more bookkeeping and internal management. This explains, for instance, why the implementation of UrlMetaFinder needs to do its own caching of links, loaders, and other details. On the other hand, path-based hooks are more narrowly tied to the processing of sys.path. Because of the connection to sys.path, modules loaded with such extensions will tend to have the same features as normal modules and packages that programmers are used to.
Assuming that your head hasn’t completely exploded at this point, a key to understanding and experimenting with this recipe may be the added logging calls. You can enable logging and try experiments such as this:
~~~
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> import urlimport
>>> urlimport.install_path_hook()
DEBUG:urlimport:Installing handle_url
>>> import fib
DEBUG:urlimport:Handle path? /usr/local/lib/python33.zip. [No]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'fib'
>>> import sys
>>> sys.path.append('http://localhost:15000')
>>> import fib
DEBUG:urlimport:Handle path? http://localhost:15000\. [Yes]
DEBUG:urlimport:Getting links from http://localhost:15000
DEBUG:urlimport:links: {'spam.py', 'fib.py', 'grok'}
DEBUG:urlimport:find_loader: 'fib'
DEBUG:urlimport:find_loader: module 'fib' found
DEBUG:urlimport:loader: reading 'http://localhost:15000/fib.py'
DEBUG:urlimport:loader: 'http://localhost:15000/fib.py' loaded
I'm fib
>>>
~~~
Last, but not least, spending some time sleeping with?[PEP 302](http://www.python.org/dev/peps/pep-0302)?and the documentation for importlib under your pillow may be advisable.
- Copyright
- 前言
- 第一章:數據結構和算法
- 1.1 解壓序列賦值給多個變量
- 1.2 解壓可迭代對象賦值給多個變量
- 1.3 保留最后N個元素
- 1.4 查找最大或最小的N個元素
- 1.5 實現一個優先級隊列
- 1.6 字典中的鍵映射多個值
- 1.7 字典排序
- 1.8 字典的運算
- 1.9 查找兩字典的相同點
- 1.10 刪除序列相同元素并保持順序
- 1.11 命名切片
- 1.12 序列中出現次數最多的元素
- 1.13 通過某個關鍵字排序一個字典列表
- 1.14 排序不支持原生比較的對象
- 1.15 通過某個字段將記錄分組
- 1.16 過濾序列元素
- 1.17 從字典中提取子集
- 1.18 映射名稱到序列元素
- 1.19 轉換并同時計算數據
- 1.20 合并多個字典或映射
- 第二章:字符串和文本
- 2.1 使用多個界定符分割字符串
- 2.2 字符串開頭或結尾匹配
- 2.3 用Shell通配符匹配字符串
- 2.4 字符串匹配和搜索
- 2.5 字符串搜索和替換
- 2.6 字符串忽略大小寫的搜索替換
- 2.7 最短匹配模式
- 2.8 多行匹配模式
- 2.9 將Unicode文本標準化
- 2.10 在正則式中使用Unicode
- 2.11 刪除字符串中不需要的字符
- 2.12 審查清理文本字符串
- 2.13 字符串對齊
- 2.14 合并拼接字符串
- 2.15 字符串中插入變量
- 2.16 以指定列寬格式化字符串
- 2.17 在字符串中處理html和xml
- 2.18 字符串令牌解析
- 2.19 實現一個簡單的遞歸下降分析器
- 2.20 字節字符串上的字符串操作
- 第三章:數字日期和時間
- 3.1 數字的四舍五入
- 3.2 執行精確的浮點數運算
- 3.3 數字的格式化輸出
- 3.4 二八十六進制整數
- 3.5 字節到大整數的打包與解包
- 3.6 復數的數學運算
- 3.7 無窮大與NaN
- 3.8 分數運算
- 3.9 大型數組運算
- 3.10 矩陣與線性代數運算
- 3.11 隨機選擇
- 3.12 基本的日期與時間轉換
- 3.13 計算最后一個周五的日期
- 3.14 計算當前月份的日期范圍
- 3.15 字符串轉換為日期
- 3.16 結合時區的日期操作
- 第四章:迭代器與生成器
- 4.1 手動遍歷迭代器
- 4.2 代理迭代
- 4.3 使用生成器創建新的迭代模式
- 4.4 實現迭代器協議
- 4.5 反向迭代
- 4.6 帶有外部狀態的生成器函數
- 4.7 迭代器切片
- 4.8 跳過可迭代對象的開始部分
- 4.9 排列組合的迭代
- 4.10 序列上索引值迭代
- 4.11 同時迭代多個序列
- 4.12 不同集合上元素的迭代
- 4.13 創建數據處理管道
- 4.14 展開嵌套的序列
- 4.15 順序迭代合并后的排序迭代對象
- 4.16 迭代器代替while無限循環
- 第五章:文件與IO
- 5.1 讀寫文本數據
- 5.2 打印輸出至文件中
- 5.3 使用其他分隔符或行終止符打印
- 5.4 讀寫字節數據
- 5.5 文件不存在才能寫入
- 5.6 字符串的I/O操作
- 5.7 讀寫壓縮文件
- 5.8 固定大小記錄的文件迭代
- 5.9 讀取二進制數據到可變緩沖區中
- 5.10 內存映射的二進制文件
- 5.11 文件路徑名的操作
- 5.12 測試文件是否存在
- 5.13 獲取文件夾中的文件列表
- 5.14 忽略文件名編碼
- 5.15 打印不合法的文件名
- 5.16 增加或改變已打開文件的編碼
- 5.17 將字節寫入文本文件
- 5.18 將文件描述符包裝成文件對象
- 5.19 創建臨時文件和文件夾
- 5.20 與串行端口的數據通信
- 5.21 序列化Python對象
- 第六章:數據編碼和處理
- 6.1 讀寫CSV數據
- 6.2 讀寫JSON數據
- 6.3 解析簡單的XML數據
- 6.4 增量式解析大型XML文件
- 6.5 將字典轉換為XML
- 6.6 解析和修改XML
- 6.7 利用命名空間解析XML文檔
- 6.8 與關系型數據庫的交互
- 6.9 編碼和解碼十六進制數
- 6.10 編碼解碼Base64數據
- 6.11 讀寫二進制數組數據
- 6.12 讀取嵌套和可變長二進制數據
- 6.13 數據的累加與統計操作
- 第七章:函數
- 7.1 可接受任意數量參數的函數
- 7.2 只接受關鍵字參數的函數
- 7.3 給函數參數增加元信息
- 7.4 返回多個值的函數
- 7.5 定義有默認參數的函數
- 7.6 定義匿名或內聯函數
- 7.7 匿名函數捕獲變量值
- 7.8 減少可調用對象的參數個數
- 7.9 將單方法的類轉換為函數
- 7.10 帶額外狀態信息的回調函數
- 7.11 內聯回調函數
- 7.12 訪問閉包中定義的變量
- 第八章:類與對象
- 8.1 改變對象的字符串顯示
- 8.2 自定義字符串的格式化
- 8.3 讓對象支持上下文管理協議
- 8.4 創建大量對象時節省內存方法
- 8.5 在類中封裝屬性名
- 8.6 創建可管理的屬性
- 8.7 調用父類方法
- 8.8 子類中擴展property
- 8.9 創建新的類或實例屬性
- 8.10 使用延遲計算屬性
- 8.11 簡化數據結構的初始化
- 8.12 定義接口或者抽象基類
- 8.13 實現數據模型的類型約束
- 8.14 實現自定義容器
- 8.15 屬性的代理訪問
- 8.16 在類中定義多個構造器
- 8.17 創建不調用init方法的實例
- 8.18 利用Mixins擴展類功能
- 8.19 實現狀態對象或者狀態機
- 8.20 通過字符串調用對象方法
- 8.21 實現訪問者模式
- 8.22 不用遞歸實現訪問者模式
- 8.23 循環引用數據結構的內存管理
- 8.24 讓類支持比較操作
- 8.25 創建緩存實例
- 第九章:元編程
- 9.1 在函數上添加包裝器
- 9.2 創建裝飾器時保留函數元信息
- 9.3 解除一個裝飾器
- 9.4 定義一個帶參數的裝飾器
- 9.5 可自定義屬性的裝飾器
- 9.6 帶可選參數的裝飾器
- 9.7 利用裝飾器強制函數上的類型檢查
- 9.8 將裝飾器定義為類的一部分
- 9.9 將裝飾器定義為類
- 9.10 為類和靜態方法提供裝飾器
- 9.11 裝飾器為被包裝函數增加參數
- 9.12 使用裝飾器擴充類的功能
- 9.13 使用元類控制實例的創建
- 9.14 捕獲類的屬性定義順序
- 9.15 定義有可選參數的元類
- 9.16 *args和**kwargs的強制參數簽名
- 9.17 在類上強制使用編程規約
- 9.18 以編程方式定義類
- 9.19 在定義的時候初始化類的成員
- 9.20 利用函數注解實現方法重載
- 9.21 避免重復的屬性方法
- 9.22 定義上下文管理器的簡單方法
- 9.23 在局部變量域中執行代碼
- 9.24 解析與分析Python源碼
- 9.25 拆解Python字節碼
- 第十章:模塊與包
- 10.1 構建一個模塊的層級包
- 10.2 控制模塊被全部導入的內容
- 10.3 使用相對路徑名導入包中子模塊
- 10.4 將模塊分割成多個文件
- 10.5 利用命名空間導入目錄分散的代碼
- 10.6 重新加載模塊
- 10.7 運行目錄或壓縮文件
- 10.8 讀取位于包中的數據文件
- 10.9 將文件夾加入到sys.path
- 10.10 通過字符串名導入模塊
- 10.11 通過導入鉤子遠程加載模塊
- 10.12 導入模塊的同時修改模塊
- 10.13 安裝私有的包
- 10.14 創建新的Python環境
- 10.15 分發包
- 第十一章:網絡與Web編程
- 11.1 作為客戶端與HTTP服務交互
- 11.2 創建TCP服務器
- 11.3 創建UDP服務器
- 11.4 通過CIDR地址生成對應的IP地址集
- 11.5 生成一個簡單的REST接口
- 11.6 通過XML-RPC實現簡單的遠程調用
- 11.7 在不同的Python解釋器之間交互
- 11.8 實現遠程方法調用
- 11.9 簡單的客戶端認證
- 11.10 在網絡服務中加入SSL
- 11.11 進程間傳遞Socket文件描述符
- 11.12 理解事件驅動的IO
- 11.13 發送與接收大型數組
- 第十二章:并發編程
- 12.1 啟動與停止線程
- 12.2 判斷線程是否已經啟動
- 12.3 線程間的通信
- 12.4 給關鍵部分加鎖
- 12.5 防止死鎖的加鎖機制
- 12.6 保存線程的狀態信息
- 12.7 創建一個線程池
- 12.8 簡單的并行編程
- 12.9 Python的全局鎖問題
- 12.10 定義一個Actor任務
- 12.11 實現消息發布/訂閱模型
- 12.12 使用生成器代替線程
- 12.13 多個線程隊列輪詢
- 12.14 在Unix系統上面啟動守護進程
- 第十三章:腳本編程與系統管理
- 13.1 通過重定向/管道/文件接受輸入
- 13.2 終止程序并給出錯誤信息
- 13.3 解析命令行選項
- 13.4 運行時彈出密碼輸入提示
- 13.5 獲取終端的大小
- 13.6 執行外部命令并獲取它的輸出
- 13.7 復制或者移動文件和目錄
- 13.8 創建和解壓壓縮文件
- 13.9 通過文件名查找文件
- 13.10 讀取配置文件
- 13.11 給簡單腳本增加日志功能
- 13.12 給內庫增加日志功能
- 13.13 記錄程序執行的時間
- 13.14 限制內存和CPU的使用量
- 13.15 啟動一個WEB瀏覽器
- 第十四章:測試調試和異常
- 14.1 測試輸出到標準輸出上
- 14.2 在單元測試中給對象打補丁
- 14.3 在單元測試中測試異常情況
- 14.4 將測試輸出用日志記錄到文件中
- 14.5 忽略或者期望測試失敗
- 14.6 處理多個異常
- 14.7 捕獲所有異常
- 14.8 創建自定義異常
- 14.9 捕獲異常后拋出另外的異常
- 14.10 重新拋出最后的異常
- 14.11 輸出警告信息
- 14.12 調試基本的程序崩潰錯誤
- 14.13 給你的程序做基準測試
- 14.14 讓你的程序跑的更快
- 第十五章:C語言擴展
- 15.1 使用ctypes訪問C代碼
- 15.2 簡單的C擴展模塊
- 15.3 一個操作數組的擴展函數
- 15.4 在C擴展模塊中操作隱形指針
- 15.5 從擴張模塊中定義和導出C的API
- 15.6 從C語言中調用Python代碼
- 15.7 從C擴展中釋放全局鎖
- 15.8 C和Python中的線程混用
- 15.9 用WSIG包裝C代碼
- 15.10 用Cython包裝C代碼
- 15.11 用Cython寫高性能的數組操作
- 15.12 將函數指針轉換為可調用對象
- 15.13 傳遞NULL結尾的字符串給C函數庫
- 15.14 傳遞Unicode字符串給C函數庫
- 15.15 C字符串轉換為Python字符串
- 15.16 不確定編碼格式的C字符串
- 15.17 傳遞文件名給C擴展
- 15.18 傳遞已打開的文件給C擴展
- 15.19 從C語言中讀取類文件對象
- 15.20 處理C語言中的可迭代對象
- 15.21 診斷分析代碼錯誤
- 附錄A
- 關于譯者
- Roadmap