Python 3.5+：如何在给定完整文件路径的情况下动态导入模块（在存在隐式兄弟导入的情况下）？

Question

The standard library clearly documents how to import source files directly (given the absolute file path to the source file), but this approach does not work if that source file uses implicit sibling imports as described in the example below.

标准库清楚地记录了如何直接导入源文件(给定源文件的绝对文件路径),但如果源文件使用隐式同级导入,则此方法不起作用,如下例所示。

How could that example be adapted to work in the presence of implicit sibling imports?

如果这个例子适用于隐式兄弟导入的存在?

I already checked out this and this other * questions on the topic, but they do not address implicit sibling imports within the file being imported by hand.

我已经检查了这个以及关于该主题的其他*问题,但它们没有解决手动导入的文件中的隐式兄弟导入问题。

Setup/Example

Here's an illustrative example

这是一个说明性的例子

Directory structure:

root/
  - directory/
    - app.py
  - folder/
    - implicit_sibling_import.py
    - lib.py

---

app.py:

import os
import importlib.util

# construct absolute paths
root = os.path.abspath(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
isi_path = os.path.join(root, 'folder', 'implicit_sibling_import.py')

def path_import(absolute_path):
   '''implementation taken from https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly'''
   spec = importlib.util.spec_from_file_location(absolute_path, absolute_path)
   module = importlib.util.module_from_spec(spec)
   spec.loader.exec_module(module)
   return module

isi = path_import(isi_path)
print(isi.hello_wrapper())

---

lib.py:

def hello():
    return 'world'

---

implicit_sibling_import.py:

import lib # this is the implicit sibling import. grabs root/folder/lib.py

def hello_wrapper():
    return "ISI says: " + lib.hello()

#if __name__ == '__main__':
#    print(hello_wrapper())

---

Running python folder/implicit_sibling_import.py with the if __name__ == '__main__': block commented out yields ISI says: world in Python 3.6.

运行python文件夹/ implicit_sibling_import.py与if __name__ =='__ main__':块注释掉产生ISI说:Python 3.6中的世界。

But running python directory/app.py yields:

但是运行python目录/ app.py会产生:

Traceback (most recent call last):
  File "directory/app.py", line 10, in <module>
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/Users/pedro/test/folder/implicit_sibling_import.py", line 1, in <module>
    import lib
ModuleNotFoundError: No module named 'lib'

Workaround

If I add import sys; sys.path.insert(0, os.path.dirname(isi_path)) to app.py, python app.py yields world as intended, but I would like to avoid munging the sys.path if possible.

如果我添加import sys; sys.path.insert(0,os.path.dirname(isi_path))到app.py,python app.py产生了预期的世界,但我想避免在可能的情况下修改sys.path。

Answer requirements

I'd like python app.py to print ISI says: world and I'd like to accomplish this by modifying the path_import function.

我想要打印ISI的python app.py说:世界,我想通过修改path_import函数来实现这一点。

I'm not sure of the implications of mangling sys.path. Eg. if there was directory/requests.py and I added the path to directory to the sys.path, I wouldn't want import requests to start importing directory/requests.py instead of importing the requests library that I installed with pip install requests.

我不确定修改sys.path的含义。例如。如果有目录/ requests.py并且我将目录路径添加到sys.path,我不希望导入请求开始导入目录/ requests.py,而不是导入我使用pip安装请求安装的请求库。

The solution MUST be implemented as a python function that accepts the absolute file path to the desired module and returns the module object.

解决方案必须实现为python函数,它接受所需模块的绝对文件路径并返回模块对象。

Ideally, the solution should not introduce side-effects (eg. if it does modify sys.path, it should return sys.path to its original state). If the solution does introduce side-effects, it should explain why a solution cannot be achieved without introducing side-effects.

理想情况下,解决方案不应引入副作用(例如,如果它确实修改了sys.path,它应该将sys.path返回到其原始状态)。如果解决方案确实引入了副作用,那么它应该解释为什么在不引入副作用的情况下无法实现解决方案。

---

PYTHONPATH

If I have multiple projects doing this, I don't want to have to remember to set PYTHONPATH every time I switch between them. The user should just be able to pip install my project and run it without any additional setup.

如果我有多个项目这样做,我不想记得每次在它们之间切换时设置PYTHONPATH。用户应该只需能够安装我的项目并运行它而无需任何额外的设置。

-m

The -m flag is the recommended/pythonic approach, but the standard library also clearly documents How to import source files directly. I'd like to know how I can adapt that approach to cope with implicit relative imports. Clearly, Python's internals must do this, so how do the internals differ from the "import source files directly" documentation?

-m标志是推荐/ pythonic方法,但标准库也清楚地记录了如何直接导入源文件。我想知道如何调整这种方法来应对隐含的相对进口。显然,Python的内部必须这样做,那么内部结构与“直接导入源文件”文档有何不同?

5 个解决方案

#1

The easiest solution I could come up with is to temporarily modify sys.path in the function doing the import:

我能想到的最简单的解决方案是在执行导入的函数中临时修改sys.path:

from contextlib import contextmanager

@contextmanager
def add_to_path(p):
    import sys
    old_path = sys.path
    sys.path = sys.path[:]
    sys.path.insert(0, p)
    try:
        yield
    finally:
        sys.path = old_path

def path_import(absolute_path):
   '''implementation taken from https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly'''
   with add_to_path(os.path.dirname(absolute_path)):
       spec = importlib.util.spec_from_file_location(absolute_path, absolute_path)
       module = importlib.util.module_from_spec(spec)
       spec.loader.exec_module(module)
       return module

This should not cause any problems unless you do imports in another thread concurrently. Otherwise, since sys.path is restored to its previous state, there should be no unwanted side effects.

除非您同时在另一个线程中导入,否则这不会导致任何问题。否则,由于sys.path恢复到以前的状态,因此不应存在不必要的副作用。

Edit:

I realize that my answer is somewhat unsatisfactory but, digging into the code reveals that, the line spec.loader.exec_module(module) basically results in exec(spec.loader.get_code(module.__name__),module.__dict__) getting called. Here spec.loader.get_code(module.__name__) is simply the code contained in lib.py.

我意识到我的答案有些不尽如人意,但是,深入研究代码会发现,行spec.loader.exec_module(模块)基本上会导致exec(spec.loader.get_code(module .__ name __),module .__ dict__)被调用。这里spec.loader.get_code(module .__ name__)就是lib.py中包含的代码。

Thus a better answer to the question would have to find a way to make the import statement behave differently by simply injecting one or more global variables through the second argument of the exec-statement. However, "whatever you do to make the import machinery look in that file's folder, it'll have to linger beyond the duration of the initial import, since functions from that file might perform further imports when you call them", as stated by @user2357112 in the question comments.

因此,对问题的更好回答必须找到一种方法,通过简单地通过exec语句的第二个参数注入一个或多个全局变量,使import语句表现不同。但是,“无论你做什么使导入机器看起来都在该文件的文件夹中,它必须在初始导入的时间内停留,因为当你调用它们时,该文件中的函数可能会执行进一步的导入”,如@所述。 user2357112在问题评论中。

Unfortunately the only way to change the behavior of the import statement seems to be to change sys.path or in a package __path__. module.__dict__ already contains __path__ so that doesn't seem to work which leaves sys.path (Or trying to figure out why exec does not treat the code as a package even though it has __path__ and __package__ ... - But I don't know where to start - Maybe it has something to do with having no __init__.py file).

不幸的是,改变import语句行为的唯一方法似乎是更改sys.path或包__path__。模块.__ dict__已经包含了__path__,所以似乎没有工作离开sys.path(或试图找出为什么exec不将代码视为包,即使它有__path__和__package__ ... - 但我不知道知道从哪里开始 - 也许它与没有__init__.py文件有关。

Furthermore this issue does not seem to be specific to importlib but rather a general problem with sibling imports.

此外,这个问题似乎并不特定于importlib,而是兄弟导入的一般问题。

Edit2: If you don't want the module to end up in sys.modules the following should work (Note that any modules added to sys.modules during the import are removed):

Edit2:如果您不希望模块在sys.modules中结束,则以下内容应该有效(请注意,导入期间添加到sys.modules的所有模块都将被删除):

from contextlib import contextmanager

@contextmanager
def add_to_path(p):
    import sys
    old_path = sys.path
    old_modules = sys.modules
    sys.modules = old_modules.copy()
    sys.path = sys.path[:]
    sys.path.insert(0, p)
    try:
        yield
    finally:
        sys.path = old_path
        sys.modules = old_modules

#2

add to the PYTHONPATH environment variable the path your application is on

将应用程序所在的路径添加到PYTHONPATH环境变量中

Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored.

增加模块文件的默认搜索路径。格式与shell的PATH相同:一个或多个目录路径名由os.pathsep分隔(例如Unix上的冒号或Windows上的分号)。默认忽略不存在的目录。

on bash its like this:

在bash上它是这样的:

export PYTHONPATH="./folder/:${PYTHONPATH}"

or run directly:

或直接运行:

PYTHONPATH="./folder/:${PYTHONPATH}" python directory/app.py

#3

1. Make sure your root is in a folder that is explicitly searched in the PYTHONPATH

确保您的根位于PYTHONPATH中显式搜索的文件夹中

2. Use an absolute import:

   使用绝对导入:

   from root.folder import implicit_sibling_import #called from app.py

   来自root.folder导入implicit_sibling_import #called来自app.py

#4

The OP's idea is great, this work only for this example by adding sibling modules with proper name to the sys.modules, I would say it is the SAME as adding PYTHONPATH. tested and working with version 3.5.1.

OP的想法很棒,这个工作只针对这个例子,通过添加具有适当名称的兄弟模块到sys.modules,我会说它是添加PYTHONPATH的SAME。测试并使用3.5.1版。

import os
import sys
import importlib.util


class PathImport(object):

    def get_module_name(self, absolute_path):
        module_name = os.path.basename(absolute_path)
        module_name = module_name.replace('.py', '')
        return module_name

    def add_sibling_modules(self, sibling_dirname):
        for current, subdir, files in os.walk(sibling_dirname):
            for file_py in files:
                if not file_py.endswith('.py'):
                    continue
                if file_py == '__init__.py':
                    continue
                python_file = os.path.join(current, file_py)
                (module, spec) = self.path_import(python_file)
                sys.modules[spec.name] = module

    def path_import(self, absolute_path):
        module_name = self.get_module_name(absolute_path)
        spec = importlib.util.spec_from_file_location(module_name, absolute_path)
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
        return (module, spec)

def main():
    pathImport = PathImport()
    root = os.path.abspath(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
    isi_path = os.path.join(root, 'folder', 'implicit_sibling_import.py')
    sibling_dirname = os.path.dirname(isi_path)
    pathImport.add_sibling_modules(sibling_dirname)
    (lib, spec) = pathImport.path_import(isi_path)
    print (lib.hello())

if __name__ == '__main__':
    main()

#5

Try:

export PYTHONPATH="./folder/:${PYTHONPATH}"

or run directly:

或直接运行:

PYTHONPATH="./folder/:${PYTHONPATH}" python directory/app.py

Make sure your root is in a folder that is explicitly searched in the PYTHONPATH. Use an absolute import:

确保您的根位于PYTHONPATH中显式搜索的文件夹中。使用绝对导入:

from root.folder import implicit_sibling_import #called from app.py