管道代码跨越Apache Beam / Dataflow中的多个文件

After a lengthy search, I haven't found an example of a Dataflow / Beam pipeline that spans several files. Beam docs do suggest a file structure (under the section "Multiple File Dependencies"), but the Juliaset example they give has in effect a single code/source file (and the main file that calls it). Based on the Juliaset example, I need a similar file structure:

经过漫长的搜索，我还没有找到跨越多个文件的数据流/光束管道的示例。 Beam docs确实建议了一个文件结构（在“Multiple File Dependencies”部分下），但是它们提供的Juliaset示例实际上是一个代码/源文件（以及调用它的主文件）。基于Juliaset示例，我需要一个类似的文件结构：

juliaset/__init__.py
juliaset/juliaset.py # actual code
juliaset/some_conf.py
__init__.py
juliaset_main.py
setup.py

Now I want to import .some_conf from juliaset/juliaset.py, which works when run locally but gives me an error when run on Dataflow

现在我想从juliaset / juliaset.py导入.some_conf，它在本地运行时有效，但在Dataflow上运行时给出错误

INFO:root:2017-12-15T17:34:09.333Z: JOB_MESSAGE_ERROR: (8cdf3e226105b90a): Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 706, in run
    self._load_main_session(self.local_staging_directory)
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 446, in _load_main_session
    pickler.load_session(session_file)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 247, in load_session
    return dill.load_session(file_path)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 363, in load_session
    module = unpickler.load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 767, in _import_module
    return getattr(__import__(module, None, None, [obj]), obj)
ImportError: No module named package_name.juliaset.some_conf

A full working example would be very much appreciated!

一个完整的工作示例将非常感谢！

1 个解决方案

#1

Can you verify your setup.py containing a structure like:

你可以验证你的setup.py包含如下结构：

import setuptools

setuptools.setup(
    name='My Project',
    version='1.0',
    install_requires=[],
    packages=setuptools.find_packages(),
)

Import your modules like from juliaset.juliaset import SomeClass

从juliaset.juliaset导入SomeClass导入模块

And when you call the Python script, use python -m juliaset_main (without the .py)

当你调用Python脚本时，使用python -m juliaset_main（不带.py）

Not sure if you already tried this, but just to be sure.

不确定你是否已经尝试过这个，但只是为了确定。

#1