解析配置文件，环境和命令行参数，以获取单个选项集合

Python's standard library has modules for configuration file parsing (configparser), environment variable reading (os.environ), and command-line argument parsing (argparse). I want to write a program that does all those, and also:

Python的标准库包含用于配置文件解析（configparser），环境变量读取（os.environ）和命令行参数解析（argparse）的模块。我想写一个程序来完成所有这些，还有：

• Has a cascade of option values:

  有一系列选项值：

  • default option values, overridden by
  • 默认选项值，被覆盖
  • config file options, overridden by
  • 配置文件选项，被覆盖
  • environment variables, overridden by
  • 环境变量，被覆盖
  • command-line options.
  • 命令行选项。

• Allows one or more configuration file locations specified on the command line with e.g. --config-file foo.conf, and reads that (either instead of, or additional to, the usual configuration file). This must still obey the above cascade.

  允许在命令行上指定的一个或多个配置文件位置，例如： --config-file foo.conf，并读取（通常配置文件的替代或补充）。这仍然必须遵守上述级联。

• Allows option definitions in a single place to determine the parsing behaviour for configuration files and the command line.

  允许在一个位置使用选项定义来确定配置文件和命令行的解析行为。

• Unifies the parsed options into a single collection of option values for the rest of the program to access without caring where they came from.

  将解析后的选项统一到一个选项值集合中，以便程序的其余部分可以访问，而无需关心它们来自何处。

Everything I need is apparently in the Python standard library, but they don't work together smoothly.

我需要的一切显然都在Python标准库中，但它们并不能顺利地协同工作。

How can I achieve this with minimum deviation from the Python standard library?

如何以最小的Python标准库偏差实现这一目标？

9 个解决方案

#1

The argparse module makes this not nuts, as long as you're happy with a config file that looks like command line. (I think this is an advantage, because users will only have to learn one syntax.) Setting fromfile_prefix_chars to, for example, @, makes it so that,

只要您对配置文件看起来像命令行感到满意，argparse模块就不会这么做。 （我认为这是一个优势，因为用户只需要学习一种语法。）将fromfile_prefix_chars设置为例如@，使得它成为，

my_prog --foo=bar

is equivalent to

相当于

my_prog @baz.conf

if @baz.conf is,

if @ baz.conf是，

--foo
bar

You can even have your code look for foo.conf automatically by modifying argv

您甚至可以通过修改argv让代码自动查找foo.conf

if os.path.exists('foo.conf'):
    argv = ['@foo.conf'] + argv
args = argparser.parse_args(argv)

The format of these configuration files is modifiable by making a subclass of ArgumentParser and adding a convert_arg_line_to_args method.

通过创建ArgumentParser的子类并添加convert_arg_line_to_args方法，可以修改这些配置文件的格式。

#2

Here's a little something that I hacked together. Feel free suggest improvements/bug-reports in the comments:

这是我一起入侵的一些东西。在评论中随意建议改进/错误报告：

import argparse
import ConfigParser
import os

def _identity(x):
    return x

_SENTINEL = object()


class AddConfigFile(argparse.Action):
    def __call__(self,parser,namespace,values,option_string=None):
        # I can never remember if `values` is a list all the time or if it
        # can be a scalar string; this takes care of both.
        if isinstance(values,basestring):
            parser.config_files.append(values)
        else:
            parser.config_files.extend(values)


class ArgumentConfigEnvParser(argparse.ArgumentParser):
    def __init__(self,*args,**kwargs):
        """
        Added 2 new keyword arguments to the ArgumentParser constructor:

           config --> List of filenames to parse for config goodness
           default_section --> name of the default section in the config file
        """
        self.config_files = kwargs.pop('config',[])  #Must be a list
        self.default_section = kwargs.pop('default_section','MAIN')
        self._action_defaults = {}
        argparse.ArgumentParser.__init__(self,*args,**kwargs)


    def add_argument(self,*args,**kwargs):
        """
        Works like `ArgumentParser.add_argument`, except that we've added an action:

           config: add a config file to the parser

        This also adds the ability to specify which section of the config file to pull the 
        data from, via the `section` keyword.  This relies on the (undocumented) fact that
        `ArgumentParser.add_argument` actually returns the `Action` object that it creates.
        We need this to reliably get `dest` (although we could probably write a simple
        function to do this for us).
        """

        if 'action' in kwargs and kwargs['action'] == 'config':
            kwargs['action'] = AddConfigFile
            kwargs['default'] = argparse.SUPPRESS

        # argparse won't know what to do with the section, so 
        # we'll pop it out and add it back in later.
        #
        # We also have to prevent argparse from doing any type conversion,
        # which is done explicitly in parse_known_args.  
        #
        # This way, we can reliably check whether argparse has replaced the default.
        #
        section = kwargs.pop('section', self.default_section)
        type = kwargs.pop('type', _identity)
        default = kwargs.pop('default', _SENTINEL)

        if default is not argparse.SUPPRESS:
            kwargs.update(default=_SENTINEL)
        else:  
            kwargs.update(default=argparse.SUPPRESS)

        action = argparse.ArgumentParser.add_argument(self,*args,**kwargs)
        kwargs.update(section=section, type=type, default=default)
        self._action_defaults[action.dest] = (args,kwargs)
        return action

    def parse_known_args(self,args=None, namespace=None):
        # `parse_args` calls `parse_known_args`, so we should be okay with this...
        ns, argv = argparse.ArgumentParser.parse_known_args(self, args=args, namespace=namespace)
        config_parser = ConfigParser.SafeConfigParser()
        config_files = [os.path.expanduser(os.path.expandvars(x)) for x in self.config_files]
        config_parser.read(config_files)

        for dest,(args,init_dict) in self._action_defaults.items():
            type_converter = init_dict['type']
            default = init_dict['default']
            obj = default

            if getattr(ns,dest,_SENTINEL) is not _SENTINEL: # found on command line
                obj = getattr(ns,dest)
            else: # not found on commandline
                try:  # get from config file
                    obj = config_parser.get(init_dict['section'],dest)
                except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): # Nope, not in config file
                    try: # get from environment
                        obj = os.environ[dest.upper()]
                    except KeyError:
                        pass

            if obj is _SENTINEL:
                setattr(ns,dest,None)
            elif obj is argparse.SUPPRESS:
                pass
            else:
                setattr(ns,dest,type_converter(obj))

        return ns, argv


if __name__ == '__main__':
    fake_config = """
[MAIN]
foo:bar
bar:1
"""
    with open('_config.file','w') as fout:
        fout.write(fake_config)

    parser = ArgumentConfigEnvParser()
    parser.add_argument('--config-file', action='config', help="location of config file")
    parser.add_argument('--foo', type=str, action='store', default="grape", help="don't know what foo does ...")
    parser.add_argument('--bar', type=int, default=7, action='store', help="This is an integer (I hope)")
    parser.add_argument('--baz', type=float, action='store', help="This is an float(I hope)")
    parser.add_argument('--qux', type=int, default='6', action='store', help="this is another int")
    ns = parser.parse_args([])

    parser_defaults = {'foo':"grape",'bar':7,'baz':None,'qux':6}
    config_defaults = {'foo':'bar','bar':1}
    env_defaults = {"baz":3.14159}

    # This should be the defaults we gave the parser
    print ns
    assert ns.__dict__ == parser_defaults

    # This should be the defaults we gave the parser + config defaults
    d = parser_defaults.copy()
    d.update(config_defaults)
    ns = parser.parse_args(['--config-file','_config.file'])
    print ns
    assert ns.__dict__ == d

    os.environ['BAZ'] = "3.14159"

    # This should be the parser defaults + config defaults + env_defaults
    d = parser_defaults.copy()
    d.update(config_defaults)
    d.update(env_defaults)
    ns = parser.parse_args(['--config-file','_config.file'])
    print ns
    assert ns.__dict__ == d

    # This should be the parser defaults + config defaults + env_defaults + commandline
    commandline = {'foo':'3','qux':4} 
    d = parser_defaults.copy()
    d.update(config_defaults)
    d.update(env_defaults)
    d.update(commandline)
    ns = parser.parse_args(['--config-file','_config.file','--foo=3','--qux=4'])
    print ns
    assert ns.__dict__ == d

    os.remove('_config.file')

TODO

This implementation is still incomplete. Here's a partial TODO list:

此实施仍然不完整。这是部分TODO列表：

• (easy) Interaction with parser defaults
• （简单）与解析器默认值交互
• (easy) If type conversion doesn't work, check against how argparse handles error messages
• （简单）如果类型转换不起作用，请检查argparse如何处理错误消息

Conform to documented behavior

• (easy) Write a function that figures out dest from args in add_argument, instead of relying on the Action object
• （简单）编写一个函数，从add_argument中的args中找出dest，而不是依赖于Action对象
• (trivial) Write a parse_args function which uses parse_known_args. (e.g. copy parse_args from the cpython implementation to guarantee it calls parse_known_args.)
• （平凡）编写一个使用parse_known_args的parse_args函数。 （例如，从cpython实现复制parse_args以保证它调用parse_known_args。）

Less Easy Stuff…

I haven't tried any of this yet. It's unlikely—but still possible!—that it could just work…

我还没有尝试过这个。它不太可能 - 但仍然可能！ - 它可以正常工作......

• (hard?) Mutual Exclusion
• （难吗？）相互排斥
• (hard?) Argument Groups (If implemented, these groups should get a section in the config file.)
• （硬？）参数组（如果实现，这些组应该在配置文件中得到一个部分。）
• (hard?) Sub Commands (Sub-commands should also get a section in the config file.)
• （硬？）子命令（子命令也应该在配置文件中得到一个部分。）

#3

There's library that does exactly this called configglue.

有一个名为configglue的库。

configglue is a library that glues together python's optparse.OptionParser and ConfigParser.ConfigParser, so that you don't have to repeat yourself when you want to export the same options to a configuration file and a commandline interface.

configglue是一个将python的optparse.OptionParser和ConfigParser.ConfigParser粘合在一起的库，因此当您想要将相同的选项导出到配置文件和命令行界面时，您不必重复自己。

It also supports environment variables.

它还支持环境变量。

There's also another library called ConfigArgParse which is

还有另一个名为ConfigArgParse的库

A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.

argparse的替代品，允许通过配置文件和/或环境变量设置选项。

You might be interested in PyCon talk about configuration by Łukasz Langa - Let Them Configure!

你可能对PyCon谈论ŁukaszLanga的配置感兴趣 - 让他们配置！

#4

While I haven't tried it by my own, there is ConfigArgParse library which states that it does most of things that you want:

虽然我没有亲自尝试过，但有一个ConfigArgParse库，它表明它可以完成你想要的大部分事情：

A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.

argparse的替代品，允许通过配置文件和/或环境变量设置选项。

#5

It seems the standard library doesn't address this, leaving each programmer to cobble configparser and argparse and os.environ all together in clunky ways.

似乎标准库没有解决这个问题，让每个程序员以笨重的方式将configparser和argparse以及os.environ拼凑在一起。

#6

The Python standard library does not provide this, as far as I know. I solved this for myself by writing code to use optparse and ConfigParser to parse the command line and config files, and provide an abstraction layer on top of them. However, you would need this as a separate dependency, which from your earlier comment seems to be unpalatable.

据我所知，Python标准库不提供此功能。我通过编写代码来使用optparse和ConfigParser解析命令行和配置文件，并在它们之上提供一个抽象层，为自己解决了这个问题。但是，你需要将它作为一个单独的依赖项，从你之前的评论中看起来似乎是不合适的。

If you want to look at the code I wrote, it's at http://liw.fi/cliapp/. It's integrated into my "command line application framework" library, since that's a large part of what the framework needs to do.

如果你想查看我写的代码，请访问http://liw.fi/cliapp/。它已集成到我的“命令行应用程序框架”库中，因为这是框架需要做的很大一部分。

#7

To hit all those requirements, I would recommend writing your own library that uses both [opt|arg]parse and configparser for the underlying functionality.

为了满足所有这些要求，我建议编写自己的库，使用[opt | arg] parse和configparser作为底层功能。

Given the first two and the last requirement, I'd say you want:

考虑到前两个和最后一个要求，我想说你想要：

Step one: Do a command line parser pass that only looks for the --config-file option.

第一步：执行命令行解析器传递，只查找--config-file选项。

Step two: Parse the config file.

第二步：解析配置文件。

Step three: set up a second command line parser pass using the output of the config file pass as the defaults.

第三步：使用配置文件传递的输出作为默认值设置第二个命令行解析器传递。

The third requirement likely means you have to design your own option definition system to expose all the functionality of optparse and configparser that you care about, and write some plumbing to do conversions in between.

第三个要求可能意味着您必须设计自己的选项定义系统以公开您关心的optparse和configparser的所有功能，并编写一些管道以在其间进行转换。

#8

I was tried something like this recently, using "optparse".

我最近尝试使用“optparse”这样的东西。

I set it up as a sub-class of OptonParser, with a '--Store' and a '--Check' command.

我把它设置为OptonParser的子类，带有'--Store'和'--Check'命令。

The code below should pretty much have you covered. You just need to define your own 'load' and 'store' methods which accept/return dictionaries and you're prey much set.

下面的代码几乎应该涵盖你。您只需要定义自己的“加载”和“存储”方法，这些方法接受/返回字典，并且您可以设置很多东西。

class SmartParse(optparse.OptionParser):
    def __init__(self,defaults,*args,**kwargs):
        self.smartDefaults=defaults
        optparse.OptionParser.__init__(self,*args,**kwargs)
        fileGroup = optparse.OptionGroup(self,'handle stored defaults')
        fileGroup.add_option(
            '-S','--Store',
            dest='Action',
            action='store_const',const='Store',
            help='store command line settings'
        )
        fileGroup.add_option(
            '-C','--Check',
            dest='Action',
            action='store_const',const='Check',
            help ='check stored settings'
        )
        self.add_option_group(fileGroup)
    def parse_args(self,*args,**kwargs):
        (options,arguments) = optparse.OptionParser.parse_args(self,*args,**kwargs)
        action = options.__dict__.pop('Action')
        if action == 'Check':
            assert all(
                value is None 
                for (key,value) in options.__dict__.iteritems() 
            )
            print 'defaults:',self.smartDefaults
            print 'config:',self.load()
            sys.exit()
        elif action == 'Store':
            self.store(options.__dict__)
            sys.exit()
        else:
            config=self.load()
            commandline=dict(
                [key,val] 
                for (key,val) in options.__dict__.iteritems() 
                if val is not None
            )
            result = {}
            result.update(self.defaults)
            result.update(config)
            result.update(commandline)
            return result,arguments
    def load(self):
        return {}
    def store(self,optionDict):
        print 'Storing:',optionDict

#9

Here's a module I hacked together that reads command-line arguments, environment settings, ini files, and keyring values as well. It's also available in a gist.

这是我一起攻击的模块，它同时读取命令行参数，环境设置，ini文件和密钥环值。它也有一个要点。

"""
Configuration Parser

Configurable parser that will parse config files, environment variables,
keyring, and command-line arguments.



Example test.ini file:

    [defaults]
    gini=10

    [app]
    xini = 50

Example test.arg file:

    --xfarg=30

Example test.py file:

    import os
    import sys

    import config


    def main(argv):
        '''Test.'''
        options = [
            config.Option("xpos",
                          help="positional argument",
                          nargs='?',
                          default="all",
                          env="APP_XPOS"),
            config.Option("--xarg",
                          help="optional argument",
                          default=1,
                          type=int,
                          env="APP_XARG"),
            config.Option("--xenv",
                          help="environment argument",
                          default=1,
                          type=int,
                          env="APP_XENV"),
            config.Option("--xfarg",
                          help="@file argument",
                          default=1,
                          type=int,
                          env="APP_XFARG"),
            config.Option("--xini",
                          help="ini argument",
                          default=1,
                          type=int,
                          ini_section="app",
                          env="APP_XINI"),
            config.Option("--gini",
                          help="global ini argument",
                          default=1,
                          type=int,
                          env="APP_GINI"),
            config.Option("--karg",
                          help="secret keyring arg",
                          default=-1,
                          type=int),
        ]
        ini_file_paths = [
            '/etc/default/app.ini',
            os.path.join(os.path.dirname(os.path.abspath(__file__)),
                         'test.ini')
        ]

        # default usage
        conf = config.Config(prog='app', options=options,
                             ini_paths=ini_file_paths)
        conf.parse()
        print conf

        # advanced usage
        cli_args = conf.parse_cli(argv=argv)
        env = conf.parse_env()
        secrets = conf.parse_keyring(namespace="app")
        ini = conf.parse_ini(ini_file_paths)
        sources = {}
        if ini:
            for key, value in ini.iteritems():
                conf[key] = value
                sources[key] = "ini-file"
        if secrets:
            for key, value in secrets.iteritems():
                conf[key] = value
                sources[key] = "keyring"
        if env:
            for key, value in env.iteritems():
                conf[key] = value
                sources[key] = "environment"
        if cli_args:
            for key, value in cli_args.iteritems():
                conf[key] = value
                sources[key] = "command-line"
        print '\n'.join(['%s:\t%s' % (k, v) for k, v in sources.items()])


    if __name__ == "__main__":
        if config.keyring:
            config.keyring.set_password("app", "karg", "13")
        main(sys.argv)

Example results:

    $APP_XENV=10 python test.py api --xarg=2 @test.arg
    <Config xpos=api, gini=1, xenv=10, xini=50, karg=13, xarg=2, xfarg=30>
    xpos:   command-line
    xenv:   environment
    xini:   ini-file
    karg:   keyring
    xarg:   command-line
    xfarg:  command-line


"""
import argparse
import ConfigParser
import copy
import os
import sys

try:
    import keyring
except ImportError:
    keyring = None


class Option(object):
    """Holds a configuration option and the names and locations for it.

    Instantiate options using the same arguments as you would for an
    add_arguments call in argparse. However, you have two additional kwargs
    available:

        env: the name of the environment variable to use for this option
        ini_section: the ini file section to look this value up from
    """

    def __init__(self, *args, **kwargs):
        self.args = args or []
        self.kwargs = kwargs or {}

    def add_argument(self, parser, **override_kwargs):
        """Add an option to a an argparse parser."""
        kwargs = {}
        if self.kwargs:
            kwargs = copy.copy(self.kwargs)
            try:
                del kwargs['env']
            except KeyError:
                pass
            try:
                del kwargs['ini_section']
            except KeyError:
                pass
        kwargs.update(override_kwargs)
        parser.add_argument(*self.args, **kwargs)

    @property
    def type(self):
        """The type of the option.

        Should be a callable to parse options.
        """
        return self.kwargs.get("type", str)

    @property
    def name(self):
        """The name of the option as determined from the args."""
        for arg in self.args:
            if arg.startswith("--"):
                return arg[2:].replace("-", "_")
            elif arg.startswith("-"):
                continue
            else:
                return arg.replace("-", "_")

    @property
    def default(self):
        """The default for the option."""
        return self.kwargs.get("default")


class Config(object):
    """Parses configuration sources."""

    def __init__(self, options=None, ini_paths=None, **parser_kwargs):
        """Initialize with list of options.

        :param ini_paths: optional paths to ini files to look up values from
        :param parser_kwargs: kwargs used to init argparse parsers.
        """
        self._parser_kwargs = parser_kwargs or {}
        self._ini_paths = ini_paths or []
        self._options = copy.copy(options) or []
        self._values = {option.name: option.default
                        for option in self._options}
        self._parser = argparse.ArgumentParser(**parser_kwargs)
        self.pass_thru_args = []

    @property
    def prog(self):
        """Program name."""
        return self._parser.prog

    def __getitem__(self, key):
        return self._values[key]

    def __setitem__(self, key, value):
        self._values[key] = value

    def __delitem__(self, key):
        del self._values[key]

    def __contains__(self, key):
        return key in self._values

    def __iter__(self):
        return iter(self._values)

    def __len__(self):
        return len(self._values)

    def get(self, key, *args):
        """
        Return the value for key if it exists otherwise the default.
        """
        return self._values.get(key, *args)

    def __getattr__(self, attr):
        if attr in self._values:
            return self._values[attr]
        else:
            raise AttributeError("'config' object has no attribute '%s'"
                                 % attr)

    def build_parser(self, options, **override_kwargs):
        """."""
        kwargs = copy.copy(self._parser_kwargs)
        kwargs.update(override_kwargs)
        if 'fromfile_prefix_chars' not in kwargs:
            kwargs['fromfile_prefix_chars'] = '@'
        parser = argparse.ArgumentParser(**kwargs)
        if options:
            for option in options:
                option.add_argument(parser)
        return parser

    def parse_cli(self, argv=None):
        """Parse command-line arguments into values."""
        if not argv:
            argv = sys.argv
        options = []
        for option in self._options:
            temp = Option(*option.args, **option.kwargs)
            temp.kwargs['default'] = argparse.SUPPRESS
            options.append(temp)
        parser = self.build_parser(options=options)
        parsed, extras = parser.parse_known_args(argv[1:])
        if extras:
            valid, pass_thru = self.parse_passthru_args(argv[1:])
            parsed, extras = parser.parse_known_args(valid)
            if extras:
                raise AttributeError("Unrecognized arguments: %s" %
                                     ' ,'.join(extras))
            self.pass_thru_args = pass_thru + extras
        return vars(parsed)

    def parse_env(self):
        results = {}
        for option in self._options:
            env_var = option.kwargs.get('env')
            if env_var and env_var in os.environ:
                value = os.environ[env_var]
                results[option.name] = option.type(value)
        return results

    def get_defaults(self):
        """Use argparse to determine and return dict of defaults."""
        parser = self.build_parser(options=self._options)
        parsed, _ = parser.parse_known_args([])
        return vars(parsed)

    def parse_ini(self, paths=None):
        """Parse config files and return configuration options.

        Expects array of files that are in ini format.
        :param paths: list of paths to files to parse (uses ConfigParse logic).
                      If not supplied, uses the ini_paths value supplied on
                      initialization.
        """
        results = {}
        config = ConfigParser.SafeConfigParser()
        config.read(paths or self._ini_paths)
        for option in self._options:
            ini_section = option.kwargs.get('ini_section')
            if ini_section:
                try:
                    value = config.get(ini_section, option.name)
                    results[option.name] = option.type(value)
                except ConfigParser.NoSectionError:
                    pass
        return results

    def parse_keyring(self, namespace=None):
        """."""
        results = {}
        if not keyring:
            return results
        if not namespace:
            namespace = self.prog
        for option in self._options:
            secret = keyring.get_password(namespace, option.name)
            if secret:
                results[option.name] = option.type(secret)
        return results

    def parse(self, argv=None):
        """."""
        defaults = self.get_defaults()
        args = self.parse_cli(argv=argv)
        env = self.parse_env()
        secrets = self.parse_keyring()
        ini = self.parse_ini()

        results = defaults
        results.update(ini)
        results.update(secrets)
        results.update(env)
        results.update(args)

        self._values = results
        return self

    @staticmethod
    def parse_passthru_args(argv):
        """Handles arguments to be passed thru to a subprocess using '--'.

        :returns: tuple of two lists; args and pass-thru-args
        """
        if '--' in argv:
            dashdash = argv.index("--")
            if dashdash == 0:
                return argv[1:], []
            elif dashdash > 0:
                return argv[0:dashdash], argv[dashdash + 1:]
        return argv, []

    def __repr__(self):
        return "<Config %s>" % ', '.join([
            '%s=%s' % (k, v) for k, v in self._values.iteritems()])


def comma_separated_strings(value):
    """Handles comma-separated arguments passed in command-line."""
    return map(str, value.split(","))


def comma_separated_pairs(value):
    """Handles comma-separated key/values passed in command-line."""
    pairs = value.split(",")
    results = {}
    for pair in pairs:
        key, pair_value = pair.split('=')
        results[key] = pair_value
    return results