ConfigArgParse-0.13.0/0000755€$£B€_ 40000000000013235650327014343 5ustar weisburd00000000000000ConfigArgParse-0.13.0/ConfigArgParse.egg-info/0000755€$£B€_ 40000000000013235650327020667 5ustar weisburd00000000000000ConfigArgParse-0.13.0/ConfigArgParse.egg-info/dependency_links.txt0000644€$£B€_ 40000000000113235650327024735 0ustar weisburd00000000000000 ConfigArgParse-0.13.0/ConfigArgParse.egg-info/PKG-INFO0000644€$£B€_ 40000004362113235650327021772 0ustar weisburd00000000000000Metadata-Version: 1.1 Name: ConfigArgParse Version: 0.13.0 Summary: A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables. Home-page: https://github.com/bw2/ConfigArgParse Author: Zorro Author-email: zorro3.github@gmail.com License: MIT Description-Content-Type: UNKNOWN Description: Overview ~~~~~~~~ Applications with more than a handful of user-settable options are best configured through a combination of command line args, config files, hard-coded defaults, and in some cases, environment variables. Python's command line parsing modules such as argparse have very limited support for config files and environment variables, so this module extends argparse to add these features. Available on PyPI: http://pypi.python.org/pypi/ConfigArgParse .. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master :target: https://travis-ci.org/bw2/ConfigArgParse Features ~~~~~~~~ - command-line, config file, env var, and default settings can now be defined, documented, and parsed in one go using a single API (if a value is specified in more than one way then: command line > environment variables > config file values > defaults) - config files can have .ini or .yaml style syntax (eg. key=value or key: value) - user can provide a config file via a normal-looking command line arg (eg. -c path/to/config.txt) rather than the argparse-style @config.txt - one or more default config file paths can be specified (eg. ['/etc/bla.conf', '~/.my_config'] ) - all argparse functionality is fully supported, so this module can serve as a drop-in replacement (verified by argparse unittests). - env vars and config file keys & syntax are automatically documented in the -h help message - new method :code:`print_values()` can report keys & values and where they were set (eg. command line, env var, config file, or default). - lite-weight (no 3rd-party library dependencies except (optionally) PyYAML) - extensible (:code:`ConfigFileParser` can be subclassed to define a new config file format) - unittested by running the unittests that came with argparse but on configargparse, and using tox to test with python2.7+ and python3+ Example ~~~~~~~ *my_script.py*: Script that defines 4 options and a positional arg and then parses and prints the values. Also, it prints out the help message as well as the string produced by :code:`format_values()` to show what they look like. .. code:: py import configargparse p = configargparse.ArgParser(default_config_files=['/etc/app/conf.d/*.conf', '~/.my_settings']) p.add('-c', '--my-config', required=True, is_config_file=True, help='config file path') p.add('--genome', required=True, help='path to genome file') # this option can be set in a config file because it starts with '--' p.add('-v', help='verbose', action='store_true') p.add('-d', '--dbsnp', help='known variants .vcf', env_var='DBSNP_PATH') # this option can be set in a config file because it starts with '--' p.add('vcf', nargs='+', help='variant file(s)') options = p.parse_args() print(options) print("----------") print(p.format_help()) print("----------") print(p.format_values()) # useful for logging where different settings came from *config.txt:* Since the script above set the config file as required=True, lets create a config file to give it: .. code:: py # settings for my_script.py genome = HCMV # cytomegalovirus genome dbsnp = /data/dbsnp/variants.vcf *command line:* Now run the script and pass it the config file: .. code:: bash python my_script.py --genome hg19 --my-config config.txt f1.vcf f2.vcf *output:* Here is the result: .. code:: bash Namespace(dbsnp='/data/dbsnp/variants.vcf', genome='hg19', my_config='config.txt', vcf=['f1.vcf', 'f2.vcf'], verbose=False) ---------- usage: my_script.py [-h] --genome GENOME [-v] -c MY_CONFIG [-d DBSNP] vcf [vcf ...] Args that start with '--' (eg. --genome) can also be set in a config file (/etc/settings.ini or /home/jeff/.my_settings or provided via -c) by using .ini or .yaml-style syntax (eg. genome=value). Command-line values override environment variables which override config file values which override defaults. positional arguments: vcf variant file optional arguments: -h, --help show this help message and exit --genome GENOME path to genome file -v verbose -c MY_CONFIG, --my-config MY_CONFIG config file path -d DBSNP, --dbsnp DBSNP known variants .vcf [env var: DBSNP_PATH] ---------- Command Line Args: --genome hg19 --my-config config.txt f1.vcf f2.vcf Config File (config.txt): dbsnp: /data/dbsnp/variants.vcf Special Values ~~~~~~~~~~~~~~ Under the hood, configargparse handles environment variables and config file values by converting them to their corresponding command line arg. For example, "key = value" will be processed as if "--key value" was specified on the command line. Also, the following special values (whether in a config file or an environment variable) are handled in a special way to support booleans and lists: - :code:`key = true` is handled as if "--key" was specified on the command line. In your python code this key must be defined as a boolean flag (eg. action="store_true" or similar). - :code:`key = [value1, value2, ...]` is handled as if "--key value1 --key value2" etc. was specified on the command line. In your python code this key must be defined as a list (eg. action="append"). Config File Syntax ~~~~~~~~~~~~~~~~~~ Only command line args that have a long version (eg. one that starts with '--') can be set in a config file. For example, "--color" can be set by putting "color=green" in a config file. The config file syntax depends on the constuctor arg: :code:`config_file_parser_class` which can be set to one of the provided classes: :code:`DefaultConfigFileParser` or :code:`YAMLConfigFileParser`, or to your own subclass of the :code:`ConfigFileParser` abstract class. *DefaultConfigFileParser* - the full range of valid syntax is: .. code:: yaml # this is a comment ; this is also a comment (.ini style) --- # lines that start with --- are ignored (yaml style) ------------------- [section] # .ini-style section names are treated as comments # how to specify a key-value pair (all of these are equivalent): name value # key is case sensitive: "Name" isn't "name" name = value # (.ini style) (white space is ignored, so name = value same as name=value) name: value # (yaml style) --name value # (argparse style) # how to set a flag arg (eg. arg which has action="store_true") --name name name = True # "True" and "true" are the same # how to specify a list arg (eg. arg which has action="append") fruit = [apple, orange, lemon] indexes = [1, 12, 35 , 40] *YAMLConfigFileParser* - allows a subset of YAML syntax (http://goo.gl/VgT2DU) .. code:: yaml # a comment name1: value name2: true # "True" and "true" are the same fruit: [apple, orange, lemon] indexes: [1, 12, 35, 40] ArgParser Singletons ~~~~~~~~~~~~~~~~~~~~~~~~~ To make it easier to configure different modules in an application, configargparse provides globally-available ArgumentParser instances via configargparse.get_argument_parser('name') (similar to logging.getLogger('name')). Here is an example of an application with a utils module that also defines and retrieves its own command-line args. *main.py* .. code:: py import configargparse import utils p = configargparse.get_argument_parser() p.add_argument("-x", help="Main module setting") p.add_argument("--m-setting", help="Main module setting") options = p.parse_known_args() # using p.parse_args() here may raise errors. *utils.py* .. code:: py import configargparse p = configargparse.get_argument_parser() p.add_argument("--utils-setting", help="Config-file-settable option for utils") if __name__ == "__main__": options = p.parse_known_args() Help Formatters ~~~~~~~~~~~~~~~ :code:`ArgumentDefaultsRawHelpFormatter` is a new HelpFormatter that both adds default values AND disables line-wrapping. It can be passed to the constructor: :code:`ArgParser(.., formatter_class=ArgumentDefaultsRawHelpFormatter)` Aliases ~~~~~~~ The configargparse.ArgumentParser API inherits its class and method names from argparse and also provides the following shorter names for convenience: - p = configargparse.get_arg_parser() # get global singleton instance - p = configargparse.get_parser() - p = configargparse.ArgParser() # create a new instance - p = configargparse.Parser() - p.add_arg(..) - p.add(..) - options = p.parse(..) HelpFormatters: - RawFormatter = RawDescriptionHelpFormatter - DefaultsFormatter = ArgumentDefaultsHelpFormatter - DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter Design Notes ~~~~~~~~~~~~ Unit tests: tests/test_configargparse.py contains custom unittests for features specific to this module (such as config file and env-var support), as well as a hook to load and run argparse unittests (see the built-in test.test_argparse module) but on configargparse in place of argparse. This ensures that configargparse will work as a drop in replacement for argparse in all usecases. Previously existing modules (PyPI search keywords: config argparse): - argparse (built-in module python v2.7+ ) - Good: - fully featured command line parsing - can read args from files using an easy to understand mechanism - Bad: - syntax for specifying config file path is unusual (eg. @file.txt)and not described in the user help message. - default config file syntax doesn't support comments and is unintuitive (eg. --namevalue) - no support for environment variables - ConfArgParse v1.0.15 (https://pypi.python.org/pypi/ConfArgParse) - Good: - extends argparse with support for config files parsed by ConfigParser - clear documentation in README - Bad: - config file values are processed using ArgumentParser.set_defaults(..) which means "required" and "choices" are not handled as expected. For example, if you specify a required value in a config file, you still have to specify it again on the command line. - doesn't work with python 3 yet - no unit tests, code not well documented - appsettings v0.5 (https://pypi.python.org/pypi/appsettings) - Good: - supports config file (yaml format) and env_var parsing - supports config-file-only setting for specifying lists and dicts - Bad: - passes in config file and env settings via parse_args namespace param - tests not finished and don't work with python3 (import StringIO) - argparse_config v0.5.1 (https://pypi.python.org/pypi/argparse_config) - Good: - similar features to ConfArgParse v1.0.15 - Bad: - doesn't work with python3 (error during pip install) - yconf v0.3.2 - (https://pypi.python.org/pypi/yconf) - features and interface not that great - hieropt v0.3 - (https://pypi.python.org/pypi/hieropt) - doesn't appear to be maintained, couldn't find documentation - configurati v0.2.3 - (https://pypi.python.org/pypi/configurati) - Good: - JSON, YAML, or Python configuration files - handles rich data structures such as dictionaries - can group configuration names into sections (like .ini files) - Bad: - doesn't work with python3 - 2+ years since last release to PyPI - apparently unmaintained Design choices: 1. all options must be settable via command line. Having options that can only be set using config files or env. vars adds complexity to the API, and is not a useful enough feature since the developer can split up options into sections and call a section "config file keys", with command line args that are just "--" plus the config key. 2. config file and env. var settings should be processed by appending them to the command line (another benefit of #1). This is an easy-to-implement solution and implicitly takes care of checking that all "required" args are provied, etc., plus the behavior should be easy for users to understand. 3. configargparse shouldn't override argparse's convert_arg_line_to_args method so that all argparse unit tests can be run on configargparse. 4. in terms of what to allow for config file keys, the "dest" value of an option can't serve as a valid config key because many options can have the same dest. Instead, since multiple options can't use the same long arg (eg. "--long-arg-x"), let the config key be either "--long-arg-x" or "long-arg-x". This means the developer can allow only a subset of the command-line args to be specified via config file (eg. short args like -x would be excluded). Also, that way config keys are automatically documented whenever the command line args are documented in the help message. 5. don't force users to put config file settings in the right .ini [sections]. This doesn't have a clear benefit since all options are command-line settable, and so have a globally unique key anyway. Enforcing sections just makes things harder for the user and adds complexity to the implementation. 6. if necessary, config-file-only args can be added later by implementing a separate add method and using the namespace arg as in appsettings_v0.5 Relevant sites: - http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll - http://tricksntweaks.blogspot.com/2013_05_01_archive.html - http://www.youtube.com/watch?v=vvCwqHgZJc8#t=35 .. |Travis CI Status for bw2/ConfigArgParse| image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master Keywords: options,argparse,ConfigArgParse,config,environment variables,envvars,ENV,environment,optparse,YAML,INI Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: MIT License Classifier: Natural Language :: English Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy ConfigArgParse-0.13.0/ConfigArgParse.egg-info/requires.txt0000644€$£B€_ 40000000001713235650327023265 0ustar weisburd00000000000000 [yaml] PyYAML ConfigArgParse-0.13.0/ConfigArgParse.egg-info/SOURCES.txt0000644€$£B€_ 40000000044513235650327022556 0ustar weisburd00000000000000LICENSE MANIFEST.in README.rst configargparse.py setup.py ConfigArgParse.egg-info/PKG-INFO ConfigArgParse.egg-info/SOURCES.txt ConfigArgParse.egg-info/dependency_links.txt ConfigArgParse.egg-info/requires.txt ConfigArgParse.egg-info/top_level.txt tests/__init__.py tests/test_configargparse.pyConfigArgParse-0.13.0/ConfigArgParse.egg-info/top_level.txt0000644€$£B€_ 40000000001713235650327023417 0ustar weisburd00000000000000configargparse ConfigArgParse-0.13.0/configargparse.py0000644€$£B€_ 40000012074613235644610017717 0ustar weisburd00000000000000import argparse import glob import os import re import sys import types if sys.version_info >= (3, 0): from io import StringIO else: from StringIO import StringIO if sys.version_info < (2, 7): from ordereddict import OrderedDict else: from collections import OrderedDict ACTION_TYPES_THAT_DONT_NEED_A_VALUE = (argparse._StoreTrueAction, argparse._StoreFalseAction, argparse._CountAction, argparse._StoreConstAction, argparse._AppendConstAction) # global ArgumentParser instances _parsers = {} def init_argument_parser(name=None, **kwargs): """Creates a global ArgumentParser instance with the given name, passing any args other than "name" to the ArgumentParser constructor. This instance can then be retrieved using get_argument_parser(..) """ if name is None: name = "default" if name in _parsers: raise ValueError(("kwargs besides 'name' can only be passed in the" " first time. '%s' ArgumentParser already exists: %s") % ( name, _parsers[name])) kwargs.setdefault('formatter_class', argparse.ArgumentDefaultsHelpFormatter) kwargs.setdefault('conflict_handler', 'resolve') _parsers[name] = ArgumentParser(**kwargs) def get_argument_parser(name=None, **kwargs): """Returns the global ArgumentParser instance with the given name. The 1st time this function is called, a new ArgumentParser instance will be created for the given name, and any args other than "name" will be passed on to the ArgumentParser constructor. """ if name is None: name = "default" if len(kwargs) > 0 or name not in _parsers: init_argument_parser(name, **kwargs) return _parsers[name] class ArgumentDefaultsRawHelpFormatter( argparse.ArgumentDefaultsHelpFormatter, argparse.RawTextHelpFormatter, argparse.RawDescriptionHelpFormatter): """HelpFormatter that adds default values AND doesn't do line-wrapping""" pass class ConfigFileParser(object): """This abstract class can be extended to add support for new config file formats""" def get_syntax_description(self): """Returns a string describing the config file syntax.""" raise NotImplementedError("get_syntax_description(..) not implemented") def parse(self, stream): """Parses the keys and values from a config file. NOTE: For keys that were specified to configargparse as action="store_true" or "store_false", the config file value must be one of: "yes", "no", "true", "false". Otherwise an error will be raised. Args: stream: A config file input stream (such as an open file object). Returns: OrderedDict of items where the keys have type string and the values have type either string or list (eg. to support config file formats like YAML which allow lists). """ raise NotImplementedError("parse(..) not implemented") def serialize(self, items): """Does the inverse of config parsing by taking parsed values and converting them back to a string representing config file contents. Args: items: an OrderedDict of items to be converted to the config file format. Keys should be strings, and values should be either strings or lists. Returns: Contents of config file as a string """ raise NotImplementedError("serialize(..) not implemented") class ConfigFileParserException(Exception): """Raised when config file parsing failed.""" class DefaultConfigFileParser(ConfigFileParser): """Based on a simplified subset of INI and YAML formats. Here is the supported syntax: # this is a comment ; this is also a comment (.ini style) --- # lines that start with --- are ignored (yaml style) ------------------- [section] # .ini-style section names are treated as comments # how to specify a key-value pair (all of these are equivalent): name value # key is case sensitive: "Name" isn't "name" name = value # (.ini style) (white space is ignored, so name = value same as name=value) name: value # (yaml style) --name value # (argparse style) # how to set a flag arg (eg. arg which has action="store_true") --name name name = True # "True" and "true" are the same # how to specify a list arg (eg. arg which has action="append") fruit = [apple, orange, lemon] indexes = [1, 12, 35 , 40] """ def get_syntax_description(self): msg = ("Config file syntax allows: key=value, flag=true, stuff=[a,b,c] " "(for details, see syntax at https://goo.gl/R74nmi).") return msg def parse(self, stream): """Parses the keys + values from a config file.""" items = OrderedDict() for i, line in enumerate(stream): line = line.strip() if not line or line[0] in ["#", ";", "["] or line.startswith("---"): continue white_space = "\\s*" key = "(?P[^:=;#\s]+?)" value = white_space+"[:=\s]"+white_space+"(?P.+?)" comment = white_space+"(?P\\s[;#].*)?" key_only_match = re.match("^" + key + comment + "$", line) if key_only_match: key = key_only_match.group("key") items[key] = "true" continue key_value_match = re.match("^"+key+value+comment+"$", line) if key_value_match: key = key_value_match.group("key") value = key_value_match.group("value") if value.startswith("[") and value.endswith("]"): # handle special case of lists value = [elem.strip() for elem in value[1:-1].split(",")] items[key] = value continue raise ConfigFileParserException("Unexpected line %s in %s: %s" % (i, getattr(stream, 'name', 'stream'), line)) return items def serialize(self, items): """Does the inverse of config parsing by taking parsed values and converting them back to a string representing config file contents. """ r = StringIO() for key, value in items.items(): if isinstance(value, list): # handle special case of lists value = "["+", ".join(map(str, value))+"]" r.write("%s = %s\n" % (key, value)) return r.getvalue() class YAMLConfigFileParser(ConfigFileParser): """Parses YAML config files. Depends on the PyYAML module. https://pypi.python.org/pypi/PyYAML """ def get_syntax_description(self): msg = ("The config file uses YAML syntax and must represent a YAML " "'mapping' (for details, see http://learn.getgrav.org/advanced/yaml).") return msg def _load_yaml(self): """lazy-import PyYAML so that configargparse doesn't have to dependend on it unless this parser is used.""" try: import yaml except ImportError: raise ConfigFileParserException("Could not import yaml. " "It can be installed by running 'pip install PyYAML'") return yaml def parse(self, stream): """Parses the keys and values from a config file.""" yaml = self._load_yaml() try: parsed_obj = yaml.safe_load(stream) except Exception as e: raise ConfigFileParserException("Couldn't parse config file: %s" % e) if not isinstance(parsed_obj, dict): raise ConfigFileParserException("The config file doesn't appear to " "contain 'key: value' pairs (aka. a YAML mapping). " "yaml.load('%s') returned type '%s' instead of 'dict'." % ( getattr(stream, 'name', 'stream'), type(parsed_obj).__name__)) result = OrderedDict() for key, value in parsed_obj.items(): if isinstance(value, list): result[key] = value else: result[key] = str(value) return result def serialize(self, items, default_flow_style=False): """Does the inverse of config parsing by taking parsed values and converting them back to a string representing config file contents. Args: default_flow_style: defines serialization format (see PyYAML docs) """ # lazy-import so there's no dependency on yaml unless this class is used yaml = self._load_yaml() # it looks like ordering can't be preserved: http://pyyaml.org/ticket/29 items = dict(items) return yaml.dump(items, default_flow_style=default_flow_style) # used while parsing args to keep track of where they came from _COMMAND_LINE_SOURCE_KEY = "command_line" _ENV_VAR_SOURCE_KEY = "environment_variables" _CONFIG_FILE_SOURCE_KEY = "config_file" _DEFAULTS_SOURCE_KEY = "defaults" class ArgumentParser(argparse.ArgumentParser): """Drop-in replacement for argparse.ArgumentParser that adds support for environment variables and .ini or .yaml-style config files. """ def __init__(self, add_config_file_help=True, add_env_var_help=True, auto_env_var_prefix=None, default_config_files=[], ignore_unknown_config_file_keys=False, config_file_parser_class=DefaultConfigFileParser, args_for_setting_config_path=[], config_arg_is_required=False, config_arg_help_message="config file path", args_for_writing_out_config_file=[], write_out_config_file_arg_help_message="takes the current command line " "args and writes them out to a config file at the given path, then " "exits", **kwargs ): """Supports args of the argparse.ArgumentParser constructor as **kwargs, as well as the following additional args. Additional Args: add_config_file_help: Whether to add a description of config file syntax to the help message. add_env_var_help: Whether to add something to the help message for args that can be set through environment variables. auto_env_var_prefix: If set to a string instead of None, all config- file-settable options will become also settable via environment variables whose names are this prefix followed by the config file key, all in upper case. (eg. setting this to "foo_" will allow an arg like "--my-arg" to also be set via the FOO_MY_ARG environment variable) default_config_files: When specified, this list of config files will be parsed in order, with the values from each config file taking precedence over pervious ones. This allows an application to look for config files in multiple standard locations such as the install directory, home directory, and current directory. Also, shell * syntax can be used to specify all conf files in a directory. For exmaple: ["/etc/conf/app_config.ini", "/etc/conf/conf-enabled/*.ini", "~/.my_app_config.ini", "./app_config.txt"] ignore_unknown_config_file_keys: If true, settings that are found in a config file but don't correspond to any defined configargparse args will be ignored. If false, they will be processed and appended to the commandline like other args, and can be retrieved using parse_known_args() instead of parse_args() config_file_parser_class: configargparse.ConfigFileParser subclass which determines the config file format. configargparse comes with DefaultConfigFileParser and YAMLConfigFileParser. args_for_setting_config_path: A list of one or more command line args to be used for specifying the config file path (eg. ["-c", "--config-file"]). Default: [] config_arg_is_required: When args_for_setting_config_path is set, set this to True to always require users to provide a config path. config_arg_help_message: the help message to use for the args listed in args_for_setting_config_path. args_for_writing_out_config_file: A list of one or more command line args to use for specifying a config file output path. If provided, these args cause configargparse to write out a config file with settings based on the other provided commandline args, environment variants and defaults, and then to exit. (eg. ["-w", "--write-out-config-file"]). Default: [] write_out_config_file_arg_help_message: The help message to use for the args in args_for_writing_out_config_file. """ self._add_config_file_help = add_config_file_help self._add_env_var_help = add_env_var_help self._auto_env_var_prefix = auto_env_var_prefix argparse.ArgumentParser.__init__(self, **kwargs) # parse the additional args if config_file_parser_class is None: self._config_file_parser = DefaultConfigFileParser() else: self._config_file_parser = config_file_parser_class() self._default_config_files = default_config_files self._ignore_unknown_config_file_keys = ignore_unknown_config_file_keys if args_for_setting_config_path: self.add_argument(*args_for_setting_config_path, dest="config_file", required=config_arg_is_required, help=config_arg_help_message, is_config_file_arg=True) if args_for_writing_out_config_file: self.add_argument(*args_for_writing_out_config_file, dest="write_out_config_file_to_this_path", metavar="CONFIG_OUTPUT_PATH", help=write_out_config_file_arg_help_message, is_write_out_config_file_arg=True) def parse_args(self, args = None, namespace = None, config_file_contents = None, env_vars = os.environ): """Supports all the same args as the ArgumentParser.parse_args(..), as well as the following additional args. Additional Args: args: a list of args as in argparse, or a string (eg. "-x -y bla") config_file_contents: String. Used for testing. env_vars: Dictionary. Used for testing. """ args, argv = self.parse_known_args(args = args, namespace = namespace, config_file_contents = config_file_contents, env_vars = env_vars) if argv: self.error('unrecognized arguments: %s' % ' '.join(argv)) return args def parse_known_args(self, args = None, namespace = None, config_file_contents = None, env_vars = os.environ): """Supports all the same args as the ArgumentParser.parse_args(..), as well as the following additional args. Additional Args: args: a list of args as in argparse, or a string (eg. "-x -y bla") config_file_contents: String. Used for testing. env_vars: Dictionary. Used for testing. """ if args is None: args = sys.argv[1:] elif isinstance(args, str): args = args.split() else: args = list(args) # normalize args by converting args like --key=value to --key value normalized_args = list() for arg in args: if arg and arg[0] in self.prefix_chars and '=' in arg: key, value = arg.split('=', 1) normalized_args.append(key) normalized_args.append(value) else: normalized_args.append(arg) args = normalized_args for a in self._actions: a.is_positional_arg = not a.option_strings # maps a string describing the source (eg. env var) to a settings dict # to keep track of where values came from (used by print_values()). # The settings dicts for env vars and config files will then map # the config key to an (argparse Action obj, string value) 2-tuple. self._source_to_settings = OrderedDict() if args: a_v_pair = (None, list(args)) # copy args list to isolate changes self._source_to_settings[_COMMAND_LINE_SOURCE_KEY] = {'': a_v_pair} # handle auto_env_var_prefix __init__ arg by setting a.env_var as needed if self._auto_env_var_prefix is not None: for a in self._actions: config_file_keys = self.get_possible_config_keys(a) if config_file_keys and not (a.env_var or a.is_positional_arg or a.is_config_file_arg or a.is_write_out_config_file_arg or isinstance(a, argparse._HelpAction)): stripped_config_file_key = config_file_keys[0].strip( self.prefix_chars) a.env_var = (self._auto_env_var_prefix + stripped_config_file_key).replace('-', '_').upper() # add env var settings to the commandline that aren't there already env_var_args = [] actions_with_env_var_values = [a for a in self._actions if not a.is_positional_arg and a.env_var and a.env_var in env_vars and not already_on_command_line(args, a.option_strings)] for action in actions_with_env_var_values: key = action.env_var value = env_vars[key] # Make list-string into list. if action.nargs or isinstance(action, argparse._AppendAction): element_capture = re.match('\[(.*)\]', value) if element_capture: value = [val.strip() for val in element_capture.group(1).split(',') if val.strip()] env_var_args += self.convert_item_to_command_line_arg( action, key, value) args = args + env_var_args if env_var_args: self._source_to_settings[_ENV_VAR_SOURCE_KEY] = OrderedDict( [(a.env_var, (a, env_vars[a.env_var])) for a in actions_with_env_var_values]) # before parsing any config files, check if -h was specified. supports_help_arg = any( a for a in self._actions if isinstance(a, argparse._HelpAction)) skip_config_file_parsing = supports_help_arg and ( "-h" in args or "--help" in args) # prepare for reading config file(s) known_config_keys = dict((config_key, action) for action in self._actions for config_key in self.get_possible_config_keys(action)) # open the config file(s) config_streams = [] if config_file_contents is not None: stream = StringIO(config_file_contents) stream.name = "method arg" config_streams = [stream] elif not skip_config_file_parsing: config_streams = self._open_config_files(args) # parse each config file for stream in reversed(config_streams): try: config_items = self._config_file_parser.parse(stream) except ConfigFileParserException as e: self.error(e) finally: if hasattr(stream, "close"): stream.close() # add each config item to the commandline unless it's there already config_args = [] for key, value in config_items.items(): if key in known_config_keys: action = known_config_keys[key] discard_this_key = already_on_command_line( args, action.option_strings) else: action = None discard_this_key = self._ignore_unknown_config_file_keys or \ already_on_command_line( args, [self.get_command_line_key_for_unknown_config_file_setting(key)]) if not discard_this_key: config_args += self.convert_item_to_command_line_arg( action, key, value) source_key = "%s|%s" %(_CONFIG_FILE_SOURCE_KEY, stream.name) if source_key not in self._source_to_settings: self._source_to_settings[source_key] = OrderedDict() self._source_to_settings[source_key][key] = (action, value) args = args + config_args # save default settings for use by print_values() default_settings = OrderedDict() for action in self._actions: cares_about_default_value = (not action.is_positional_arg or action.nargs in [OPTIONAL, ZERO_OR_MORE]) if (already_on_command_line(args, action.option_strings) or not cares_about_default_value or action.default is None or action.default == SUPPRESS or isinstance(action, ACTION_TYPES_THAT_DONT_NEED_A_VALUE)): continue else: if action.option_strings: key = action.option_strings[-1] else: key = action.dest default_settings[key] = (action, str(action.default)) if default_settings: self._source_to_settings[_DEFAULTS_SOURCE_KEY] = default_settings # parse all args (including commandline, config file, and env var) namespace, unknown_args = argparse.ArgumentParser.parse_known_args( self, args=args, namespace=namespace) # handle any args that have is_write_out_config_file_arg set to true # check if the user specified this arg on the commandline output_file_paths = [getattr(namespace, a.dest, None) for a in self._actions if getattr(a, "is_write_out_config_file_arg", False)] output_file_paths = [a for a in output_file_paths if a is not None] self.write_config_file(namespace, output_file_paths, exit_after=True) return namespace, unknown_args def write_config_file(self, parsed_namespace, output_file_paths, exit_after=False): """Write the given settings to output files. Args: parsed_namespace: namespace object created within parse_known_args() output_file_paths: any number of file paths to write the config to exit_after: whether to exit the program after writing the config files """ for output_file_path in output_file_paths: # validate the output file path try: with open(output_file_path, "w") as output_file: pass except IOError as e: raise ValueError("Couldn't open %s for writing: %s" % ( output_file_path, e)) if output_file_paths: # generate the config file contents config_items = self.get_items_for_config_file_output( self._source_to_settings, parsed_namespace) file_contents = self._config_file_parser.serialize(config_items) for output_file_path in output_file_paths: with open(output_file_path, "w") as output_file: output_file.write(file_contents) message = "Wrote config file to " + ", ".join(output_file_paths) if exit_after: self.exit(0, message) else: print(message) def get_command_line_key_for_unknown_config_file_setting(self, key): """Compute a commandline arg key to be used for a config file setting that doesn't correspond to any defined configargparse arg (and so doesn't have a user-specified commandline arg key). Args: key: The config file key that was being set. """ key_without_prefix_chars = key.strip(self.prefix_chars) command_line_key = self.prefix_chars[0]*2 + key_without_prefix_chars return command_line_key def get_items_for_config_file_output(self, source_to_settings, parsed_namespace): """Converts the given settings back to a dictionary that can be passed to ConfigFormatParser.serialize(..). Args: source_to_settings: the dictionary described in parse_known_args() parsed_namespace: namespace object created within parse_known_args() Returns: an OrderedDict where keys are strings and values are either strings or lists """ config_file_items = OrderedDict() for source, settings in source_to_settings.items(): if source == _COMMAND_LINE_SOURCE_KEY: _, existing_command_line_args = settings[''] for action in self._actions: config_file_keys = self.get_possible_config_keys(action) if config_file_keys and not action.is_positional_arg and \ already_on_command_line(existing_command_line_args, action.option_strings): value = getattr(parsed_namespace, action.dest, None) if value is not None: if isinstance(value, bool): value = str(value).lower() elif callable(action.type): found = [i for i in range(0, len(existing_command_line_args)-1) if existing_command_line_args[i] in config_file_keys] if found: value = existing_command_line_args[found[-1] + 1] config_file_items[config_file_keys[0]] = value elif source == _ENV_VAR_SOURCE_KEY: for key, (action, value) in settings.items(): config_file_keys = self.get_possible_config_keys(action) if config_file_keys: value = getattr(parsed_namespace, action.dest, None) if value is not None: config_file_items[config_file_keys[0]] = value elif source.startswith(_CONFIG_FILE_SOURCE_KEY): for key, (action, value) in settings.items(): config_file_items[key] = value elif source == _DEFAULTS_SOURCE_KEY: for key, (action, value) in settings.items(): config_file_keys = self.get_possible_config_keys(action) if config_file_keys: value = getattr(parsed_namespace, action.dest, None) if value is not None: config_file_items[config_file_keys[0]] = value return config_file_items def convert_item_to_command_line_arg(self, action, key, value): """Converts a config file or env var key + value to a list of commandline args to append to the commandline. Args: action: The argparse Action object for this setting, or None if this config file setting doesn't correspond to any defined configargparse arg. key: string (config file key or env var name) value: parsed value of type string or list """ args = [] if action is None: command_line_key = \ self.get_command_line_key_for_unknown_config_file_setting(key) else: command_line_key = action.option_strings[-1] # handle boolean value if action is not None and isinstance(action, ACTION_TYPES_THAT_DONT_NEED_A_VALUE): if value.lower() in ("true", "yes", "1"): args.append( command_line_key ) elif value.lower() in ("false", "no", "0"): # don't append when set to "false" / "no" pass else: self.error("Unexpected value for %s: '%s'. Expecting 'true', " "'false', 'yes', 'no', '1' or '0'" % (key, value)) elif isinstance(value, list): if action is None or isinstance(action, argparse._AppendAction): for list_elem in value: args.append( command_line_key ) args.append( str(list_elem) ) elif (isinstance(action, argparse._StoreAction) and action.nargs in ('+', '*')) or ( isinstance(action.nargs, int) and action.nargs > 1): args.append( command_line_key ) for list_elem in value: args.append( str(list_elem) ) else: self.error(("%s can't be set to a list '%s' unless its action type is changed " "to 'append' or nargs is set to '*', '+', or > 1") % (key, value)) elif isinstance(value, str): args.append( command_line_key ) args.append( value ) else: raise ValueError("Unexpected value type %s for value: %s" % ( type(value), value)) return args def get_possible_config_keys(self, action): """This method decides which actions can be set in a config file and what their keys will be. It returns a list of 0 or more config keys that can be used to set the given action's value in a config file. """ keys = [] # Do not write out the config options for writing out a config file if getattr(action, 'is_write_out_config_file_arg', None): return keys for arg in action.option_strings: if any([arg.startswith(2*c) for c in self.prefix_chars]): keys += [arg[2:], arg] # eg. for '--bla' return ['bla', '--bla'] return keys def _open_config_files(self, command_line_args): """Tries to parse config file path(s) from within command_line_args. Returns a list of opened config files, including files specified on the commandline as well as any default_config_files specified in the constructor that are present on disk. Args: command_line_args: List of all args (already split on spaces) """ # open any default config files config_files = [open(f) for files in map(glob.glob, map(os.path.expanduser, self._default_config_files)) for f in files] # list actions with is_config_file_arg=True. Its possible there is more # than one such arg. user_config_file_arg_actions = [ a for a in self._actions if getattr(a, "is_config_file_arg", False)] if not user_config_file_arg_actions: return config_files for action in user_config_file_arg_actions: # try to parse out the config file path by using a clean new # ArgumentParser that only knows this one arg/action. arg_parser = argparse.ArgumentParser( prefix_chars=self.prefix_chars, add_help=False) arg_parser._add_action(action) # make parser not exit on error by replacing its error method. # Otherwise it sys.exits(..) if, for example, config file # is_required=True and user doesn't provide it. def error_method(self, message): pass arg_parser.error = types.MethodType(error_method, arg_parser) # check whether the user provided a value parsed_arg = arg_parser.parse_known_args(args=command_line_args) if not parsed_arg: continue namespace, _ = parsed_arg user_config_file = getattr(namespace, action.dest, None) if not user_config_file: continue # validate the user-provided config file path user_config_file = os.path.expanduser(user_config_file) if not os.path.isfile(user_config_file): self.error('File not found: %s' % user_config_file) config_files += [open(user_config_file)] return config_files def format_values(self): """Returns a string with all args and settings and where they came from (eg. commandline, config file, enviroment variable or default) """ source_key_to_display_value_map = { _COMMAND_LINE_SOURCE_KEY: "Command Line Args: ", _ENV_VAR_SOURCE_KEY: "Environment Variables:\n", _CONFIG_FILE_SOURCE_KEY: "Config File (%s):\n", _DEFAULTS_SOURCE_KEY: "Defaults:\n" } r = StringIO() for source, settings in self._source_to_settings.items(): source = source.split("|") source = source_key_to_display_value_map[source[0]] % tuple(source[1:]) r.write(source) for key, (action, value) in settings.items(): if key: r.write(" %-19s%s\n" % (key+":", value)) else: if isinstance(value, str): r.write(" %s\n" % value) elif isinstance(value, list): r.write(" %s\n" % ' '.join(value)) return r.getvalue() def print_values(self, file = sys.stdout): """Prints the format_values() string (to sys.stdout or another file).""" file.write(self.format_values()) def format_help(self): msg = "" added_config_file_help = False added_env_var_help = False if self._add_config_file_help: default_config_files = self._default_config_files cc = 2*self.prefix_chars[0] # eg. -- config_settable_args = [(arg, a) for a in self._actions for arg in a.option_strings if self.get_possible_config_keys(a) and not (a.dest == "help" or a.is_config_file_arg or a.is_write_out_config_file_arg)] config_path_actions = [a for a in self._actions if getattr(a, "is_config_file_arg", False)] if config_settable_args and (default_config_files or config_path_actions): self._add_config_file_help = False # prevent duplication added_config_file_help = True msg += ("Args that start with '%s' (eg. %s) can also be set in " "a config file") % (cc, config_settable_args[0][0]) config_arg_string = " or ".join(a.option_strings[0] for a in config_path_actions if a.option_strings) if config_arg_string: config_arg_string = "specified via " + config_arg_string if default_config_files or config_arg_string: msg += " (%s)." % " or ".join(tuple(default_config_files) + tuple(filter(None, [config_arg_string]))) msg += " " + self._config_file_parser.get_syntax_description() if self._add_env_var_help: env_var_actions = [(a.env_var, a) for a in self._actions if getattr(a, "env_var", None)] for env_var, a in env_var_actions: env_var_help_string = " [env var: %s]" % env_var if not a.help: a.help = "" if env_var_help_string not in a.help: a.help += env_var_help_string added_env_var_help = True self._add_env_var_help = False # prevent duplication if added_env_var_help or added_config_file_help: value_sources = ["defaults"] if added_config_file_help: value_sources = ["config file values"] + value_sources if added_env_var_help: value_sources = ["environment variables"] + value_sources msg += (" If an arg is specified in more than one place, then " "commandline values override %s.") % ( " which override ".join(value_sources)) if msg: self.description = (self.description or "") + " " + msg return argparse.ArgumentParser.format_help(self) def add_argument(self, *args, **kwargs): """ This method supports the same args as ArgumentParser.add_argument(..) as well as the additional args below. Additional Args: env_var: If set, the value of this environment variable will override any config file or default values for this arg (but can itself be overriden on the commandline). Also, if auto_env_var_prefix is set in the constructor, this env var name will be used instead of the automatic name. is_config_file_arg: If True, this arg is treated as a config file path This provides an alternative way to specify config files in place of the ArgumentParser(fromfile_prefix_chars=..) mechanism. Default: False is_write_out_config_file_arg: If True, this arg will be treated as a config file path, and, when it is specified, will cause configargparse to write all current commandline args to this file as config options and then exit. Default: False """ env_var = kwargs.pop("env_var", None) is_config_file_arg = kwargs.pop( "is_config_file_arg", None) or kwargs.pop( "is_config_file", None) # for backward compat. is_write_out_config_file_arg = kwargs.pop( "is_write_out_config_file_arg", None) action = self.original_add_argument_method(*args, **kwargs) action.is_positional_arg = not action.option_strings action.env_var = env_var action.is_config_file_arg = is_config_file_arg action.is_write_out_config_file_arg = is_write_out_config_file_arg if action.is_positional_arg and env_var: raise ValueError("env_var can't be set for a positional arg.") if action.is_config_file_arg and not isinstance(action, argparse._StoreAction): raise ValueError("arg with is_config_file_arg=True must have " "action='store'") if action.is_write_out_config_file_arg: error_prefix = "arg with is_write_out_config_file_arg=True " if not isinstance(action, argparse._StoreAction): raise ValueError(error_prefix + "must have action='store'") if is_config_file_arg: raise ValueError(error_prefix + "can't also have " "is_config_file_arg=True") return action def already_on_command_line(existing_args_list, potential_command_line_args): """Utility method for checking if any of the potential_command_line_args is already present in existing_args. """ return any(potential_arg in existing_args_list for potential_arg in potential_command_line_args) # wrap ArgumentParser's add_argument(..) method with the one above argparse._ActionsContainer.original_add_argument_method = argparse._ActionsContainer.add_argument argparse._ActionsContainer.add_argument = add_argument # add all public classes and constants from argparse module's namespace to this # module's namespace so that the 2 modules are truly interchangeable HelpFormatter = argparse.HelpFormatter RawDescriptionHelpFormatter = argparse.RawDescriptionHelpFormatter RawTextHelpFormatter = argparse.RawTextHelpFormatter ArgumentDefaultsHelpFormatter = argparse.ArgumentDefaultsHelpFormatter ArgumentError = argparse.ArgumentError ArgumentTypeError = argparse.ArgumentTypeError Action = argparse.Action FileType = argparse.FileType Namespace = argparse.Namespace ONE_OR_MORE = argparse.ONE_OR_MORE OPTIONAL = argparse.OPTIONAL REMAINDER = argparse.REMAINDER SUPPRESS = argparse.SUPPRESS ZERO_OR_MORE = argparse.ZERO_OR_MORE # deprecated PEP-8 incompatible API names. initArgumentParser = init_argument_parser getArgumentParser = get_argument_parser getArgParser = get_argument_parser getParser = get_argument_parser # create shorter aliases for the key methods and class names get_arg_parser = get_argument_parser get_parser = get_argument_parser ArgParser = ArgumentParser Parser = ArgumentParser argparse._ActionsContainer.add_arg = argparse._ActionsContainer.add_argument argparse._ActionsContainer.add = argparse._ActionsContainer.add_argument ArgumentParser.parse = ArgumentParser.parse_args ArgumentParser.parse_known = ArgumentParser.parse_known_args RawFormatter = RawDescriptionHelpFormatter DefaultsFormatter = ArgumentDefaultsHelpFormatter DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter ConfigArgParse-0.13.0/LICENSE0000644€$£B€_ 40000000205613235644610015351 0ustar weisburd00000000000000The MIT License (MIT) Copyright (c) 2015 bw2 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ConfigArgParse-0.13.0/MANIFEST.in0000644€$£B€_ 40000000006313235644610016076 0ustar weisburd00000000000000include LICENSE include README.rst include tests/* ConfigArgParse-0.13.0/PKG-INFO0000644€$£B€_ 40000004362113235650327015446 0ustar weisburd00000000000000Metadata-Version: 1.1 Name: ConfigArgParse Version: 0.13.0 Summary: A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables. Home-page: https://github.com/bw2/ConfigArgParse Author: Zorro Author-email: zorro3.github@gmail.com License: MIT Description-Content-Type: UNKNOWN Description: Overview ~~~~~~~~ Applications with more than a handful of user-settable options are best configured through a combination of command line args, config files, hard-coded defaults, and in some cases, environment variables. Python's command line parsing modules such as argparse have very limited support for config files and environment variables, so this module extends argparse to add these features. Available on PyPI: http://pypi.python.org/pypi/ConfigArgParse .. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master :target: https://travis-ci.org/bw2/ConfigArgParse Features ~~~~~~~~ - command-line, config file, env var, and default settings can now be defined, documented, and parsed in one go using a single API (if a value is specified in more than one way then: command line > environment variables > config file values > defaults) - config files can have .ini or .yaml style syntax (eg. key=value or key: value) - user can provide a config file via a normal-looking command line arg (eg. -c path/to/config.txt) rather than the argparse-style @config.txt - one or more default config file paths can be specified (eg. ['/etc/bla.conf', '~/.my_config'] ) - all argparse functionality is fully supported, so this module can serve as a drop-in replacement (verified by argparse unittests). - env vars and config file keys & syntax are automatically documented in the -h help message - new method :code:`print_values()` can report keys & values and where they were set (eg. command line, env var, config file, or default). - lite-weight (no 3rd-party library dependencies except (optionally) PyYAML) - extensible (:code:`ConfigFileParser` can be subclassed to define a new config file format) - unittested by running the unittests that came with argparse but on configargparse, and using tox to test with python2.7+ and python3+ Example ~~~~~~~ *my_script.py*: Script that defines 4 options and a positional arg and then parses and prints the values. Also, it prints out the help message as well as the string produced by :code:`format_values()` to show what they look like. .. code:: py import configargparse p = configargparse.ArgParser(default_config_files=['/etc/app/conf.d/*.conf', '~/.my_settings']) p.add('-c', '--my-config', required=True, is_config_file=True, help='config file path') p.add('--genome', required=True, help='path to genome file') # this option can be set in a config file because it starts with '--' p.add('-v', help='verbose', action='store_true') p.add('-d', '--dbsnp', help='known variants .vcf', env_var='DBSNP_PATH') # this option can be set in a config file because it starts with '--' p.add('vcf', nargs='+', help='variant file(s)') options = p.parse_args() print(options) print("----------") print(p.format_help()) print("----------") print(p.format_values()) # useful for logging where different settings came from *config.txt:* Since the script above set the config file as required=True, lets create a config file to give it: .. code:: py # settings for my_script.py genome = HCMV # cytomegalovirus genome dbsnp = /data/dbsnp/variants.vcf *command line:* Now run the script and pass it the config file: .. code:: bash python my_script.py --genome hg19 --my-config config.txt f1.vcf f2.vcf *output:* Here is the result: .. code:: bash Namespace(dbsnp='/data/dbsnp/variants.vcf', genome='hg19', my_config='config.txt', vcf=['f1.vcf', 'f2.vcf'], verbose=False) ---------- usage: my_script.py [-h] --genome GENOME [-v] -c MY_CONFIG [-d DBSNP] vcf [vcf ...] Args that start with '--' (eg. --genome) can also be set in a config file (/etc/settings.ini or /home/jeff/.my_settings or provided via -c) by using .ini or .yaml-style syntax (eg. genome=value). Command-line values override environment variables which override config file values which override defaults. positional arguments: vcf variant file optional arguments: -h, --help show this help message and exit --genome GENOME path to genome file -v verbose -c MY_CONFIG, --my-config MY_CONFIG config file path -d DBSNP, --dbsnp DBSNP known variants .vcf [env var: DBSNP_PATH] ---------- Command Line Args: --genome hg19 --my-config config.txt f1.vcf f2.vcf Config File (config.txt): dbsnp: /data/dbsnp/variants.vcf Special Values ~~~~~~~~~~~~~~ Under the hood, configargparse handles environment variables and config file values by converting them to their corresponding command line arg. For example, "key = value" will be processed as if "--key value" was specified on the command line. Also, the following special values (whether in a config file or an environment variable) are handled in a special way to support booleans and lists: - :code:`key = true` is handled as if "--key" was specified on the command line. In your python code this key must be defined as a boolean flag (eg. action="store_true" or similar). - :code:`key = [value1, value2, ...]` is handled as if "--key value1 --key value2" etc. was specified on the command line. In your python code this key must be defined as a list (eg. action="append"). Config File Syntax ~~~~~~~~~~~~~~~~~~ Only command line args that have a long version (eg. one that starts with '--') can be set in a config file. For example, "--color" can be set by putting "color=green" in a config file. The config file syntax depends on the constuctor arg: :code:`config_file_parser_class` which can be set to one of the provided classes: :code:`DefaultConfigFileParser` or :code:`YAMLConfigFileParser`, or to your own subclass of the :code:`ConfigFileParser` abstract class. *DefaultConfigFileParser* - the full range of valid syntax is: .. code:: yaml # this is a comment ; this is also a comment (.ini style) --- # lines that start with --- are ignored (yaml style) ------------------- [section] # .ini-style section names are treated as comments # how to specify a key-value pair (all of these are equivalent): name value # key is case sensitive: "Name" isn't "name" name = value # (.ini style) (white space is ignored, so name = value same as name=value) name: value # (yaml style) --name value # (argparse style) # how to set a flag arg (eg. arg which has action="store_true") --name name name = True # "True" and "true" are the same # how to specify a list arg (eg. arg which has action="append") fruit = [apple, orange, lemon] indexes = [1, 12, 35 , 40] *YAMLConfigFileParser* - allows a subset of YAML syntax (http://goo.gl/VgT2DU) .. code:: yaml # a comment name1: value name2: true # "True" and "true" are the same fruit: [apple, orange, lemon] indexes: [1, 12, 35, 40] ArgParser Singletons ~~~~~~~~~~~~~~~~~~~~~~~~~ To make it easier to configure different modules in an application, configargparse provides globally-available ArgumentParser instances via configargparse.get_argument_parser('name') (similar to logging.getLogger('name')). Here is an example of an application with a utils module that also defines and retrieves its own command-line args. *main.py* .. code:: py import configargparse import utils p = configargparse.get_argument_parser() p.add_argument("-x", help="Main module setting") p.add_argument("--m-setting", help="Main module setting") options = p.parse_known_args() # using p.parse_args() here may raise errors. *utils.py* .. code:: py import configargparse p = configargparse.get_argument_parser() p.add_argument("--utils-setting", help="Config-file-settable option for utils") if __name__ == "__main__": options = p.parse_known_args() Help Formatters ~~~~~~~~~~~~~~~ :code:`ArgumentDefaultsRawHelpFormatter` is a new HelpFormatter that both adds default values AND disables line-wrapping. It can be passed to the constructor: :code:`ArgParser(.., formatter_class=ArgumentDefaultsRawHelpFormatter)` Aliases ~~~~~~~ The configargparse.ArgumentParser API inherits its class and method names from argparse and also provides the following shorter names for convenience: - p = configargparse.get_arg_parser() # get global singleton instance - p = configargparse.get_parser() - p = configargparse.ArgParser() # create a new instance - p = configargparse.Parser() - p.add_arg(..) - p.add(..) - options = p.parse(..) HelpFormatters: - RawFormatter = RawDescriptionHelpFormatter - DefaultsFormatter = ArgumentDefaultsHelpFormatter - DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter Design Notes ~~~~~~~~~~~~ Unit tests: tests/test_configargparse.py contains custom unittests for features specific to this module (such as config file and env-var support), as well as a hook to load and run argparse unittests (see the built-in test.test_argparse module) but on configargparse in place of argparse. This ensures that configargparse will work as a drop in replacement for argparse in all usecases. Previously existing modules (PyPI search keywords: config argparse): - argparse (built-in module python v2.7+ ) - Good: - fully featured command line parsing - can read args from files using an easy to understand mechanism - Bad: - syntax for specifying config file path is unusual (eg. @file.txt)and not described in the user help message. - default config file syntax doesn't support comments and is unintuitive (eg. --namevalue) - no support for environment variables - ConfArgParse v1.0.15 (https://pypi.python.org/pypi/ConfArgParse) - Good: - extends argparse with support for config files parsed by ConfigParser - clear documentation in README - Bad: - config file values are processed using ArgumentParser.set_defaults(..) which means "required" and "choices" are not handled as expected. For example, if you specify a required value in a config file, you still have to specify it again on the command line. - doesn't work with python 3 yet - no unit tests, code not well documented - appsettings v0.5 (https://pypi.python.org/pypi/appsettings) - Good: - supports config file (yaml format) and env_var parsing - supports config-file-only setting for specifying lists and dicts - Bad: - passes in config file and env settings via parse_args namespace param - tests not finished and don't work with python3 (import StringIO) - argparse_config v0.5.1 (https://pypi.python.org/pypi/argparse_config) - Good: - similar features to ConfArgParse v1.0.15 - Bad: - doesn't work with python3 (error during pip install) - yconf v0.3.2 - (https://pypi.python.org/pypi/yconf) - features and interface not that great - hieropt v0.3 - (https://pypi.python.org/pypi/hieropt) - doesn't appear to be maintained, couldn't find documentation - configurati v0.2.3 - (https://pypi.python.org/pypi/configurati) - Good: - JSON, YAML, or Python configuration files - handles rich data structures such as dictionaries - can group configuration names into sections (like .ini files) - Bad: - doesn't work with python3 - 2+ years since last release to PyPI - apparently unmaintained Design choices: 1. all options must be settable via command line. Having options that can only be set using config files or env. vars adds complexity to the API, and is not a useful enough feature since the developer can split up options into sections and call a section "config file keys", with command line args that are just "--" plus the config key. 2. config file and env. var settings should be processed by appending them to the command line (another benefit of #1). This is an easy-to-implement solution and implicitly takes care of checking that all "required" args are provied, etc., plus the behavior should be easy for users to understand. 3. configargparse shouldn't override argparse's convert_arg_line_to_args method so that all argparse unit tests can be run on configargparse. 4. in terms of what to allow for config file keys, the "dest" value of an option can't serve as a valid config key because many options can have the same dest. Instead, since multiple options can't use the same long arg (eg. "--long-arg-x"), let the config key be either "--long-arg-x" or "long-arg-x". This means the developer can allow only a subset of the command-line args to be specified via config file (eg. short args like -x would be excluded). Also, that way config keys are automatically documented whenever the command line args are documented in the help message. 5. don't force users to put config file settings in the right .ini [sections]. This doesn't have a clear benefit since all options are command-line settable, and so have a globally unique key anyway. Enforcing sections just makes things harder for the user and adds complexity to the implementation. 6. if necessary, config-file-only args can be added later by implementing a separate add method and using the namespace arg as in appsettings_v0.5 Relevant sites: - http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll - http://tricksntweaks.blogspot.com/2013_05_01_archive.html - http://www.youtube.com/watch?v=vvCwqHgZJc8#t=35 .. |Travis CI Status for bw2/ConfigArgParse| image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master Keywords: options,argparse,ConfigArgParse,config,environment variables,envvars,ENV,environment,optparse,YAML,INI Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: MIT License Classifier: Natural Language :: English Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy ConfigArgParse-0.13.0/README.rst0000644€$£B€_ 40000003333013235644610016032 0ustar weisburd00000000000000Overview ~~~~~~~~ Applications with more than a handful of user-settable options are best configured through a combination of command line args, config files, hard-coded defaults, and in some cases, environment variables. Python's command line parsing modules such as argparse have very limited support for config files and environment variables, so this module extends argparse to add these features. Available on PyPI: http://pypi.python.org/pypi/ConfigArgParse .. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master :target: https://travis-ci.org/bw2/ConfigArgParse Features ~~~~~~~~ - command-line, config file, env var, and default settings can now be defined, documented, and parsed in one go using a single API (if a value is specified in more than one way then: command line > environment variables > config file values > defaults) - config files can have .ini or .yaml style syntax (eg. key=value or key: value) - user can provide a config file via a normal-looking command line arg (eg. -c path/to/config.txt) rather than the argparse-style @config.txt - one or more default config file paths can be specified (eg. ['/etc/bla.conf', '~/.my_config'] ) - all argparse functionality is fully supported, so this module can serve as a drop-in replacement (verified by argparse unittests). - env vars and config file keys & syntax are automatically documented in the -h help message - new method :code:`print_values()` can report keys & values and where they were set (eg. command line, env var, config file, or default). - lite-weight (no 3rd-party library dependencies except (optionally) PyYAML) - extensible (:code:`ConfigFileParser` can be subclassed to define a new config file format) - unittested by running the unittests that came with argparse but on configargparse, and using tox to test with python2.7+ and python3+ Example ~~~~~~~ *my_script.py*: Script that defines 4 options and a positional arg and then parses and prints the values. Also, it prints out the help message as well as the string produced by :code:`format_values()` to show what they look like. .. code:: py import configargparse p = configargparse.ArgParser(default_config_files=['/etc/app/conf.d/*.conf', '~/.my_settings']) p.add('-c', '--my-config', required=True, is_config_file=True, help='config file path') p.add('--genome', required=True, help='path to genome file') # this option can be set in a config file because it starts with '--' p.add('-v', help='verbose', action='store_true') p.add('-d', '--dbsnp', help='known variants .vcf', env_var='DBSNP_PATH') # this option can be set in a config file because it starts with '--' p.add('vcf', nargs='+', help='variant file(s)') options = p.parse_args() print(options) print("----------") print(p.format_help()) print("----------") print(p.format_values()) # useful for logging where different settings came from *config.txt:* Since the script above set the config file as required=True, lets create a config file to give it: .. code:: py # settings for my_script.py genome = HCMV # cytomegalovirus genome dbsnp = /data/dbsnp/variants.vcf *command line:* Now run the script and pass it the config file: .. code:: bash python my_script.py --genome hg19 --my-config config.txt f1.vcf f2.vcf *output:* Here is the result: .. code:: bash Namespace(dbsnp='/data/dbsnp/variants.vcf', genome='hg19', my_config='config.txt', vcf=['f1.vcf', 'f2.vcf'], verbose=False) ---------- usage: my_script.py [-h] --genome GENOME [-v] -c MY_CONFIG [-d DBSNP] vcf [vcf ...] Args that start with '--' (eg. --genome) can also be set in a config file (/etc/settings.ini or /home/jeff/.my_settings or provided via -c) by using .ini or .yaml-style syntax (eg. genome=value). Command-line values override environment variables which override config file values which override defaults. positional arguments: vcf variant file optional arguments: -h, --help show this help message and exit --genome GENOME path to genome file -v verbose -c MY_CONFIG, --my-config MY_CONFIG config file path -d DBSNP, --dbsnp DBSNP known variants .vcf [env var: DBSNP_PATH] ---------- Command Line Args: --genome hg19 --my-config config.txt f1.vcf f2.vcf Config File (config.txt): dbsnp: /data/dbsnp/variants.vcf Special Values ~~~~~~~~~~~~~~ Under the hood, configargparse handles environment variables and config file values by converting them to their corresponding command line arg. For example, "key = value" will be processed as if "--key value" was specified on the command line. Also, the following special values (whether in a config file or an environment variable) are handled in a special way to support booleans and lists: - :code:`key = true` is handled as if "--key" was specified on the command line. In your python code this key must be defined as a boolean flag (eg. action="store_true" or similar). - :code:`key = [value1, value2, ...]` is handled as if "--key value1 --key value2" etc. was specified on the command line. In your python code this key must be defined as a list (eg. action="append"). Config File Syntax ~~~~~~~~~~~~~~~~~~ Only command line args that have a long version (eg. one that starts with '--') can be set in a config file. For example, "--color" can be set by putting "color=green" in a config file. The config file syntax depends on the constuctor arg: :code:`config_file_parser_class` which can be set to one of the provided classes: :code:`DefaultConfigFileParser` or :code:`YAMLConfigFileParser`, or to your own subclass of the :code:`ConfigFileParser` abstract class. *DefaultConfigFileParser* - the full range of valid syntax is: .. code:: yaml # this is a comment ; this is also a comment (.ini style) --- # lines that start with --- are ignored (yaml style) ------------------- [section] # .ini-style section names are treated as comments # how to specify a key-value pair (all of these are equivalent): name value # key is case sensitive: "Name" isn't "name" name = value # (.ini style) (white space is ignored, so name = value same as name=value) name: value # (yaml style) --name value # (argparse style) # how to set a flag arg (eg. arg which has action="store_true") --name name name = True # "True" and "true" are the same # how to specify a list arg (eg. arg which has action="append") fruit = [apple, orange, lemon] indexes = [1, 12, 35 , 40] *YAMLConfigFileParser* - allows a subset of YAML syntax (http://goo.gl/VgT2DU) .. code:: yaml # a comment name1: value name2: true # "True" and "true" are the same fruit: [apple, orange, lemon] indexes: [1, 12, 35, 40] ArgParser Singletons ~~~~~~~~~~~~~~~~~~~~~~~~~ To make it easier to configure different modules in an application, configargparse provides globally-available ArgumentParser instances via configargparse.get_argument_parser('name') (similar to logging.getLogger('name')). Here is an example of an application with a utils module that also defines and retrieves its own command-line args. *main.py* .. code:: py import configargparse import utils p = configargparse.get_argument_parser() p.add_argument("-x", help="Main module setting") p.add_argument("--m-setting", help="Main module setting") options = p.parse_known_args() # using p.parse_args() here may raise errors. *utils.py* .. code:: py import configargparse p = configargparse.get_argument_parser() p.add_argument("--utils-setting", help="Config-file-settable option for utils") if __name__ == "__main__": options = p.parse_known_args() Help Formatters ~~~~~~~~~~~~~~~ :code:`ArgumentDefaultsRawHelpFormatter` is a new HelpFormatter that both adds default values AND disables line-wrapping. It can be passed to the constructor: :code:`ArgParser(.., formatter_class=ArgumentDefaultsRawHelpFormatter)` Aliases ~~~~~~~ The configargparse.ArgumentParser API inherits its class and method names from argparse and also provides the following shorter names for convenience: - p = configargparse.get_arg_parser() # get global singleton instance - p = configargparse.get_parser() - p = configargparse.ArgParser() # create a new instance - p = configargparse.Parser() - p.add_arg(..) - p.add(..) - options = p.parse(..) HelpFormatters: - RawFormatter = RawDescriptionHelpFormatter - DefaultsFormatter = ArgumentDefaultsHelpFormatter - DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter Design Notes ~~~~~~~~~~~~ Unit tests: tests/test_configargparse.py contains custom unittests for features specific to this module (such as config file and env-var support), as well as a hook to load and run argparse unittests (see the built-in test.test_argparse module) but on configargparse in place of argparse. This ensures that configargparse will work as a drop in replacement for argparse in all usecases. Previously existing modules (PyPI search keywords: config argparse): - argparse (built-in module python v2.7+ ) - Good: - fully featured command line parsing - can read args from files using an easy to understand mechanism - Bad: - syntax for specifying config file path is unusual (eg. @file.txt)and not described in the user help message. - default config file syntax doesn't support comments and is unintuitive (eg. --namevalue) - no support for environment variables - ConfArgParse v1.0.15 (https://pypi.python.org/pypi/ConfArgParse) - Good: - extends argparse with support for config files parsed by ConfigParser - clear documentation in README - Bad: - config file values are processed using ArgumentParser.set_defaults(..) which means "required" and "choices" are not handled as expected. For example, if you specify a required value in a config file, you still have to specify it again on the command line. - doesn't work with python 3 yet - no unit tests, code not well documented - appsettings v0.5 (https://pypi.python.org/pypi/appsettings) - Good: - supports config file (yaml format) and env_var parsing - supports config-file-only setting for specifying lists and dicts - Bad: - passes in config file and env settings via parse_args namespace param - tests not finished and don't work with python3 (import StringIO) - argparse_config v0.5.1 (https://pypi.python.org/pypi/argparse_config) - Good: - similar features to ConfArgParse v1.0.15 - Bad: - doesn't work with python3 (error during pip install) - yconf v0.3.2 - (https://pypi.python.org/pypi/yconf) - features and interface not that great - hieropt v0.3 - (https://pypi.python.org/pypi/hieropt) - doesn't appear to be maintained, couldn't find documentation - configurati v0.2.3 - (https://pypi.python.org/pypi/configurati) - Good: - JSON, YAML, or Python configuration files - handles rich data structures such as dictionaries - can group configuration names into sections (like .ini files) - Bad: - doesn't work with python3 - 2+ years since last release to PyPI - apparently unmaintained Design choices: 1. all options must be settable via command line. Having options that can only be set using config files or env. vars adds complexity to the API, and is not a useful enough feature since the developer can split up options into sections and call a section "config file keys", with command line args that are just "--" plus the config key. 2. config file and env. var settings should be processed by appending them to the command line (another benefit of #1). This is an easy-to-implement solution and implicitly takes care of checking that all "required" args are provied, etc., plus the behavior should be easy for users to understand. 3. configargparse shouldn't override argparse's convert_arg_line_to_args method so that all argparse unit tests can be run on configargparse. 4. in terms of what to allow for config file keys, the "dest" value of an option can't serve as a valid config key because many options can have the same dest. Instead, since multiple options can't use the same long arg (eg. "--long-arg-x"), let the config key be either "--long-arg-x" or "long-arg-x". This means the developer can allow only a subset of the command-line args to be specified via config file (eg. short args like -x would be excluded). Also, that way config keys are automatically documented whenever the command line args are documented in the help message. 5. don't force users to put config file settings in the right .ini [sections]. This doesn't have a clear benefit since all options are command-line settable, and so have a globally unique key anyway. Enforcing sections just makes things harder for the user and adds complexity to the implementation. 6. if necessary, config-file-only args can be added later by implementing a separate add method and using the namespace arg as in appsettings_v0.5 Relevant sites: - http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll - http://tricksntweaks.blogspot.com/2013_05_01_archive.html - http://www.youtube.com/watch?v=vvCwqHgZJc8#t=35 .. |Travis CI Status for bw2/ConfigArgParse| image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master ConfigArgParse-0.13.0/setup.cfg0000644€$£B€_ 40000000004613235650327016164 0ustar weisburd00000000000000[egg_info] tag_build = tag_date = 0 ConfigArgParse-0.13.0/setup.py0000644€$£B€_ 40000000735313235644774016076 0ustar weisburd00000000000000import logging import os import sys try: from setuptools import setup except ImportError: print("WARNING: setuptools not installed. Will try using distutils instead..") from distutils.core import setup def launch_http_server(directory): assert os.path.isdir(directory) try: try: from SimpleHTTPServer import SimpleHTTPRequestHandler except ImportError: from http.server import SimpleHTTPRequestHandler try: import SocketServer except ImportError: import socketserver as SocketServer import socket for port in [80] + list(range(8000, 8100)): try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('localhost', port)) s.close() except socket.error as e: logging.debug("Can't use port %d: %s" % (port, e.strerror)) continue print("HTML coverage report now available at http://%s%s" % ( socket.gethostname(), (":%s" % port) if port != 80 else "")) os.chdir(directory) SocketServer.TCPServer(("", port), SimpleHTTPRequestHandler).serve_forever() else: logging.debug("All network port. ") except Exception as e: logging.error("ERROR: while starting an HTTP server to serve " "the coverage report: %s" % e) command = sys.argv[-1] if command == 'publish': os.system('python setup.py sdist upload') sys.exit() elif command == "coverage": try: import coverage except: sys.exit("coverage.py not installed (pip install --user coverage)") setup_py_path = os.path.abspath(__file__) os.system('coverage run --source=configargparse ' + setup_py_path +' test') os.system('coverage report') os.system('coverage html') print("Done computing coverage") launch_http_server(directory="htmlcov") sys.exit() long_description = '' if command not in ['test', 'coverage']: long_description = open('README.rst').read() install_requires = [] tests_require = [ 'PyYAML', ] if sys.version_info < (2, 7): install_requires.extend([ 'argparse', 'ordereddict', ]) tests_require.extend([ 'unittest2', ]) setup( name='ConfigArgParse', version="0.13.0", description='A drop-in replacement for argparse that allows options to ' 'also be set via config files and/or environment variables.', long_description=long_description, author='Zorro', author_email='zorro3.github@gmail.com', url='https://github.com/bw2/ConfigArgParse', py_modules=['configargparse'], include_package_data=True, license="MIT", keywords='options, argparse, ConfigArgParse, config, environment variables, ' 'envvars, ENV, environment, optparse, YAML, INI', classifiers=[ 'Development Status :: 4 - Beta', 'Intended Audience :: Developers', 'License :: OSI Approved :: MIT License', 'Natural Language :: English', "Programming Language :: Python :: 2", 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', ], test_suite='tests', install_requires=install_requires, tests_require=tests_require, extras_require = { 'yaml': ["PyYAML"], } ) ConfigArgParse-0.13.0/tests/0000755€$£B€_ 40000000000013235650327015505 5ustar weisburd00000000000000ConfigArgParse-0.13.0/tests/__init__.py0000644€$£B€_ 40000000005613235644610017615 0ustar weisburd00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- ConfigArgParse-0.13.0/tests/test_configargparse.py0000644€$£B€_ 40000013052013235644610022107 0ustar weisburd00000000000000import argparse import configargparse import inspect import logging import sys import tempfile import types try: import mock except ImportError: from unittest import mock if sys.version_info < (2, 7): import unittest2 as unittest else: import unittest if sys.version_info >= (3, 0): from io import StringIO else: from StringIO import StringIO # enable logging to simplify debugging logger = logging.getLogger() logger.level = logging.DEBUG stream_handler = logging.StreamHandler(sys.stdout) logger.addHandler(stream_handler) def replace_error_method(arg_parser): """Swap out arg_parser's error(..) method so that instead of calling sys.exit(..) it just raises an error. """ def error_method(self, message): raise argparse.ArgumentError(None, message) def exit_method(self, status, message): self._exit_method_called = True arg_parser._exit_method_called = False arg_parser.error = types.MethodType(error_method, arg_parser) arg_parser.exit = types.MethodType(exit_method, arg_parser) return arg_parser class TestCase(unittest.TestCase): def initParser(self, *args, **kwargs): p = configargparse.ArgParser(*args, **kwargs) self.parser = replace_error_method(p) self.add_arg = self.parser.add_argument self.parse = self.parser.parse_args self.parse_known = self.parser.parse_known_args self.format_values = self.parser.format_values self.format_help = self.parser.format_help if not hasattr(self, "assertRegex"): self.assertRegex = self.assertRegexpMatches if not hasattr(self, "assertRaisesRegex"): self.assertRaisesRegex = self.assertRaisesRegexp return self.parser def assertParseArgsRaises(self, regex, *args, **kwargs): self.assertRaisesRegex(argparse.ArgumentError, regex, self.parse, *args, **kwargs) class TestBasicUseCases(TestCase): def setUp(self): self.initParser(args_for_setting_config_path=[]) def testBasicCase1(self): ## Test command line and config file values self.add_arg("filenames", nargs="+", help="positional arg") self.add_arg("-x", "--arg-x", action="store_true") self.add_arg("-y", "--arg-y", dest="y1", type=int, required=True) self.add_arg("--arg-z", action="append", type=float, required=True) # make sure required args are enforced self.assertParseArgsRaises("too few arg" if sys.version_info < (3,3) else "the following arguments are required", args="") self.assertParseArgsRaises("argument -y/--arg-y is required" if sys.version_info < (3,3) else "the following arguments are required: -y/--arg-y", args="-x --arg-z 11 file1.txt") self.assertParseArgsRaises("argument --arg-z is required" if sys.version_info < (3,3) else "the following arguments are required: --arg-z", args="file1.txt file2.txt file3.txt -x -y 1") # check values after setting args on command line ns = self.parse(args="file1.txt --arg-x -y 3 --arg-z 10", config_file_contents="") self.assertListEqual(ns.filenames, ["file1.txt"]) self.assertEqual(ns.arg_x, True) self.assertEqual(ns.y1, 3) self.assertEqual(ns.arg_z, [10]) self.assertRegex(self.format_values(), 'Command Line Args: file1.txt --arg-x -y 3 --arg-z 10') # check values after setting args in config file ns = self.parse(args="file1.txt file2.txt", config_file_contents=""" # set all required args in config file arg-x = True arg-y = 10 arg-z = 30 arg-z = 40 """) self.assertListEqual(ns.filenames, ["file1.txt", "file2.txt"]) self.assertEqual(ns.arg_x, True) self.assertEqual(ns.y1, 10) self.assertEqual(ns.arg_z, [40]) self.assertRegex(self.format_values(), 'Command Line Args: \s+ file1.txt file2.txt\n' 'Config File \(method arg\):\n' ' arg-x: \s+ True\n' ' arg-y: \s+ 10\n' ' arg-z: \s+ 40\n') # check values after setting args in both command line and config file ns = self.parse(args="file1.txt file2.txt --arg-x -y 3 --arg-z 100 ", config_file_contents="""arg-y = 31.5 arg-z = 30 """) self.format_help() self.format_values() self.assertListEqual(ns.filenames, ["file1.txt", "file2.txt"]) self.assertEqual(ns.arg_x, True) self.assertEqual(ns.y1, 3) self.assertEqual(ns.arg_z, [100]) self.assertRegex(self.format_values(), "Command Line Args: file1.txt file2.txt --arg-x -y 3 --arg-z 100") def testBasicCase2(self, use_groups=False): ## Test command line, config file and env var values default_config_file = tempfile.NamedTemporaryFile(mode="w", delete=True) default_config_file.flush() p = self.initParser(default_config_files=['/etc/settings.ini', '/home/jeff/.user_settings', default_config_file.name]) p.add_arg('vcf', nargs='+', help='Variant file(s)') if not use_groups: self.add_arg('--genome', help='Path to genome file', required=True) self.add_arg('-v', dest='verbose', action='store_true') self.add_arg('-g', '--my-cfg-file', required=True, is_config_file=True) self.add_arg('-d', '--dbsnp', env_var='DBSNP_PATH') self.add_arg('-f', '--format', choices=["BED", "MAF", "VCF", "WIG", "R"], dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT", default="BED") else: g = p.add_argument_group(title="g1") g.add_arg('--genome', help='Path to genome file', required=True) g.add_arg('-v', dest='verbose', action='store_true') g.add_arg('-g', '--my-cfg-file', required=True, is_config_file=True) g = p.add_argument_group(title="g2") g.add_arg('-d', '--dbsnp', env_var='DBSNP_PATH') g.add_arg('-f', '--format', choices=["BED", "MAF", "VCF", "WIG", "R"], dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT", default="BED") # make sure required args are enforced self.assertParseArgsRaises("too few arg" if sys.version_info < (3,3) else "the following arguments are required: vcf, -g/--my-cfg-file", args="--genome hg19") self.assertParseArgsRaises("not found: file.txt", args="-g file.txt") # check values after setting args on command line config_file2 = tempfile.NamedTemporaryFile(mode="w", delete=True) config_file2.flush() ns = self.parse(args="--genome hg19 -g %s bla.vcf " % config_file2.name) self.assertEqual(ns.genome, "hg19") self.assertEqual(ns.verbose, False) self.assertEqual(ns.dbsnp, None) self.assertEqual(ns.fmt, "BED") self.assertListEqual(ns.vcf, ["bla.vcf"]) self.assertRegex(self.format_values(), 'Command Line Args: --genome hg19 -g [^\s]+ bla.vcf\n' 'Defaults:\n' ' --format: \s+ BED\n') # check precedence: args > env > config > default using the --format arg default_config_file.write("--format MAF") default_config_file.flush() ns = self.parse(args="--genome hg19 -g %s f.vcf " % config_file2.name) self.assertEqual(ns.fmt, "MAF") self.assertRegex(self.format_values(), 'Command Line Args: --genome hg19 -g [^\s]+ f.vcf\n' 'Config File \([^\s]+\):\n' ' --format: \s+ MAF\n') config_file2.write("--format VCF") config_file2.flush() ns = self.parse(args="--genome hg19 -g %s f.vcf " % config_file2.name) self.assertEqual(ns.fmt, "VCF") self.assertRegex(self.format_values(), 'Command Line Args: --genome hg19 -g [^\s]+ f.vcf\n' 'Config File \([^\s]+\):\n' ' --format: \s+ VCF\n') ns = self.parse(env_vars={"OUTPUT_FORMAT":"R", "DBSNP_PATH":"/a/b.vcf"}, args="--genome hg19 -g %s f.vcf " % config_file2.name) self.assertEqual(ns.fmt, "R") self.assertEqual(ns.dbsnp, "/a/b.vcf") self.assertRegex(self.format_values(), 'Command Line Args: --genome hg19 -g [^\s]+ f.vcf\n' 'Environment Variables:\n' ' DBSNP_PATH: \s+ /a/b.vcf\n' ' OUTPUT_FORMAT: \s+ R\n') ns = self.parse(env_vars={"OUTPUT_FORMAT":"R", "DBSNP_PATH":"/a/b.vcf", "ANOTHER_VAR":"something"}, args="--genome hg19 -g %s --format WIG f.vcf" % config_file2.name) self.assertEqual(ns.fmt, "WIG") self.assertEqual(ns.dbsnp, "/a/b.vcf") self.assertRegex(self.format_values(), 'Command Line Args: --genome hg19 -g [^\s]+ --format WIG f.vcf\n' 'Environment Variables:\n' ' DBSNP_PATH: \s+ /a/b.vcf\n') if not use_groups: self.assertRegex(self.format_help(), 'usage: .* \[-h\] --genome GENOME \[-v\] -g MY_CFG_FILE' ' \[-d DBSNP\]\s+\[-f FRMT\]\s+vcf \[vcf ...\]\n\n' + 9*'(.+\s+)'+ # repeated 8 times because .+ matches atmost 1 line 'positional arguments:\n' ' vcf \s+ Variant file\(s\)\n\n' 'optional arguments:\n' ' -h, --help \s+ show this help message and exit\n' ' --genome GENOME \s+ Path to genome file\n' ' -v\n' ' -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n' ' -d DBSNP, --dbsnp DBSNP\s+\[env var: DBSNP_PATH\]\n' ' -f FRMT, --format FRMT\s+\[env var: OUTPUT_FORMAT\]\n') else: self.assertRegex(self.format_help(), 'usage: .* \[-h\] --genome GENOME \[-v\] -g MY_CFG_FILE' ' \[-d DBSNP\]\s+\[-f FRMT\]\s+vcf \[vcf ...\]\n\n'+ 9*'.+\s+'+ # repeated 8 times because .+ matches atmost 1 line 'positional arguments:\n' ' vcf \s+ Variant file\(s\)\n\n' 'optional arguments:\n' ' -h, --help \s+ show this help message and exit\n\n' 'g1:\n' ' --genome GENOME \s+ Path to genome file\n' ' -v\n' ' -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n\n' 'g2:\n' ' -d DBSNP, --dbsnp DBSNP\s+\[env var: DBSNP_PATH\]\n' ' -f FRMT, --format FRMT\s+\[env var: OUTPUT_FORMAT\]\n') self.assertParseArgsRaises("invalid choice: 'ZZZ'", args="--genome hg19 -g %s --format ZZZ f.vcf" % config_file2.name) self.assertParseArgsRaises("unrecognized arguments: --bla", args="--bla --genome hg19 -g %s f.vcf" % config_file2.name) default_config_file.close() config_file2.close() def testBasicCase2_WithGroups(self): self.testBasicCase2(use_groups=True) def testPositionalAndConfigVarLists(self): self.initParser() self.add_arg("a") self.add_arg("-x", "--arg", nargs="+") ns = self.parse("positional_value", config_file_contents="""arg = [Shell, someword, anotherword]""") self.assertEqual(ns.arg, ['Shell', 'someword', 'anotherword']) self.assertEqual(ns.a, "positional_value") def testMutuallyExclusiveArgs(self): config_file = tempfile.NamedTemporaryFile(mode="w", delete=True) p = self.parser g = p.add_argument_group(title="group1") g.add_arg('--genome', help='Path to genome file', required=True) g.add_arg('-v', dest='verbose', action='store_true') g = p.add_mutually_exclusive_group(required=True) g.add_arg('-f1', '--type1-cfg-file', is_config_file=True) g.add_arg('-f2', '--type2-cfg-file', is_config_file=True) g = p.add_mutually_exclusive_group(required=True) g.add_arg('-f', '--format', choices=["BED", "MAF", "VCF", "WIG", "R"], dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT", default="BED") g.add_arg('-b', '--bam', dest='fmt', action="store_const", const="BAM", env_var='BAM_FORMAT') ns = self.parse(args="--genome hg19 -f1 %s --bam" % config_file.name) self.assertEqual(ns.genome, "hg19") self.assertEqual(ns.verbose, False) self.assertEqual(ns.fmt, "BAM") ns = self.parse(env_vars={"BAM_FORMAT" : "true"}, args="--genome hg19 -f1 %s" % config_file.name) self.assertEqual(ns.genome, "hg19") self.assertEqual(ns.verbose, False) self.assertEqual(ns.fmt, "BAM") self.assertRegex(self.format_values(), 'Command Line Args: --genome hg19 -f1 [^\s]+\n' 'Environment Variables:\n' ' BAM_FORMAT: \s+ true\n' 'Defaults:\n' ' --format: \s+ BED\n') self.assertRegex(self.format_help(), 'usage: .* \[-h\] --genome GENOME \[-v\]\s+ \(-f1 TYPE1_CFG_FILE \|' ' \s*-f2 TYPE2_CFG_FILE\)\s+\(-f FRMT \| -b\)\n\n' + 7*'.+\s+'+ # repeated 7 times because .+ matches atmost 1 line 'optional arguments:\n' ' -h, --help show this help message and exit\n' ' -f1 TYPE1_CFG_FILE, --type1-cfg-file TYPE1_CFG_FILE\n' ' -f2 TYPE2_CFG_FILE, --type2-cfg-file TYPE2_CFG_FILE\n' ' -f FRMT, --format FRMT\s+\[env var: OUTPUT_FORMAT\]\n' ' -b, --bam\s+\[env var: BAM_FORMAT\]\n\n' 'group1:\n' ' --genome GENOME Path to genome file\n' ' -v\n') config_file.close() def testSubParsers(self): config_file1 = tempfile.NamedTemporaryFile(mode="w", delete=True) config_file1.write("--i = B") config_file1.flush() config_file2 = tempfile.NamedTemporaryFile(mode="w", delete=True) config_file2.write("p = 10") config_file2.flush() parser = configargparse.ArgumentParser(prog="myProg") subparsers = parser.add_subparsers(title="actions") parent_parser = configargparse.ArgumentParser(add_help=False) parent_parser.add_argument("-p", "--p", type=int, required=True, help="set db parameter") create_p = subparsers.add_parser("create", parents=[parent_parser], help="create the orbix environment") create_p.add_argument("--i", env_var="INIT", choices=["A","B"], default="A") create_p.add_argument("-config", is_config_file=True) update_p = subparsers.add_parser("update", parents=[parent_parser], help="update the orbix environment") update_p.add_argument("-config2", is_config_file=True, required=True) ns = parser.parse_args(args = "create -p 2 -config "+config_file1.name) self.assertEqual(ns.p, 2) self.assertEqual(ns.i, "B") ns = parser.parse_args(args = "update -config2 " + config_file2.name) self.assertEqual(ns.p, 10) config_file1.close() config_file2.close() def testAddArgsErrors(self): self.assertRaisesRegex(ValueError, "arg with " "is_write_out_config_file_arg=True can't also have " "is_config_file_arg=True", self.add_arg, "-x", "--X", is_config_file=True, is_write_out_config_file_arg=True) self.assertRaisesRegex(ValueError, "arg with " "is_write_out_config_file_arg=True must have action='store'", self.add_arg, "-y", "--Y", action="append", is_write_out_config_file_arg=True) def testConfigFileSyntax(self): self.add_arg('-x', required=True, type=int) self.add_arg('--y', required=True, type=float) self.add_arg('--z') self.add_arg('--c') self.add_arg('--b', action="store_true") self.add_arg('--a', action="append", type=int) ns = self.parse(args="-x 1", env_vars={}, config_file_contents=""" #inline comment 1 # inline comment 2 # inline comment 3 ;inline comment 4 ; inline comment 5 ;inline comment 6 --- # separator 1 ------------- # separator 2 y=1.1 y = 2.1 y= 3.1 # with comment y= 4.1 ; with comment --- y:5.1 y : 6.1 y: 7.1 # with comment y: 8.1 ; with comment --- y \t 9.1 y 10.1 y 11.1 # with comment y 12.1 ; with comment --- b b = True b: True ---- a = 33 --- z z 1 """) self.assertEqual(ns.x, 1) self.assertEqual(ns.y, 12.1) self.assertEqual(ns.z, 'z 1') self.assertEqual(ns.c, None) self.assertEqual(ns.b, True) self.assertEqual(ns.a, [33]) self.assertRegex(self.format_values(), 'Command Line Args: \s+ -x 1\n' 'Config File \(method arg\):\n' ' y: \s+ 12.1\n' ' b: \s+ True\n' ' a: \s+ 33\n' ' z: \s+ z 1\n') # -x is not a long arg so can't be set via config file self.assertParseArgsRaises("argument -x is required" if sys.version_info < (3,3) else "the following arguments are required: -x, --y", config_file_contents="-x 3") self.assertParseArgsRaises("invalid float value: 'abc'", args="-x 5", config_file_contents="y: abc") self.assertParseArgsRaises("argument --y is required" if sys.version_info < (3,3) else "the following arguments are required: --y", args="-x 5", config_file_contents="z: 1") # test unknown config file args self.assertParseArgsRaises("bla", args="-x 1 --y 2.3", config_file_contents="bla=3") ns, args = self.parse_known("-x 10 --y 3.8", config_file_contents="bla=3", env_vars={"bla": "2"}) self.assertListEqual(args, ["--bla", "3"]) self.initParser(ignore_unknown_config_file_keys=False) ns, args = self.parse_known(args="-x 1", config_file_contents="bla=3", env_vars={"bla": "2"}) self.assertEqual(set(args), set(["--bla", "3", "-x", "1"])) def testBooleanValuesCanBeExpressedAsNumbers(self): self.initParser() store_true_env_var_name = "STORE_TRUE" self.add_arg("--boolean_store_true", action="store_true", env_var=store_true_env_var_name) result_namespace = self.parse("", config_file_contents="""boolean_store_true = 1""") self.assertTrue(result_namespace.boolean_store_true) result_namespace = self.parse("", config_file_contents="""boolean_store_true = 0""") self.assertFalse(result_namespace.boolean_store_true) result_namespace = self.parse("", env_vars={store_true_env_var_name: "1"}) self.assertTrue(result_namespace.boolean_store_true) result_namespace = self.parse("", env_vars={store_true_env_var_name: "0"}) self.assertFalse(result_namespace.boolean_store_true) self.initParser() store_false_env_var_name = "STORE_FALSE" self.add_arg("--boolean_store_false", action="store_false", env_var=store_false_env_var_name) result_namespace = self.parse("", config_file_contents="""boolean_store_false = 1""") self.assertFalse(result_namespace.boolean_store_false) result_namespace = self.parse("", config_file_contents="""boolean_store_false = 0""") self.assertTrue(result_namespace.boolean_store_false) result_namespace = self.parse("", env_vars={store_false_env_var_name: "1"}) self.assertFalse(result_namespace.boolean_store_false) result_namespace = self.parse("", env_vars={store_false_env_var_name: "0"}) self.assertTrue(result_namespace.boolean_store_false) def testConfigOrEnvValueErrors(self): # error should occur when a flag arg is set to something other than "true" or "false" self.initParser() self.add_arg("--height", env_var = "HEIGHT", required=True) self.add_arg("--do-it", dest="x", env_var = "FLAG1", action="store_true") self.add_arg("--dont-do-it", dest="y", env_var = "FLAG2", action="store_false") ns = self.parse("", env_vars = {"HEIGHT": "tall", "FLAG1": "yes"}) self.assertEqual(ns.height, "tall") self.assertEqual(ns.x, True) ns = self.parse("", env_vars = {"HEIGHT": "tall", "FLAG2": "yes"}) self.assertEqual(ns.y, False) ns = self.parse("", env_vars = {"HEIGHT": "tall", "FLAG2": "no"}) self.assertEqual(ns.y, True) # error should occur when flag arg is given a value self.initParser() self.add_arg("-v", "--verbose", env_var="VERBOSE", action="store_true") self.assertParseArgsRaises("Unexpected value for VERBOSE: 'bla'. " "Expecting 'true', 'false', 'yes', 'no', '1' or '0'", env_vars={"VERBOSE" : "bla"}) ns = self.parse("", config_file_contents="verbose=true", env_vars={"HEIGHT": "true"}) self.assertEqual(ns.verbose, True) ns = self.parse("", config_file_contents="verbose", env_vars={"HEIGHT": "true"}) self.assertEqual(ns.verbose, True) ns = self.parse("", env_vars = {"HEIGHT": "true", "VERBOSE": "true"}) self.assertEqual(ns.verbose, True) ns = self.parse("", env_vars = {"HEIGHT": "true", "VERBOSE": "false"}) self.assertEqual(ns.verbose, False) ns = self.parse("", config_file_contents="--verbose", env_vars = {"HEIGHT": "true"}) self.assertEqual(ns.verbose, True) # error should occur is non-append arg is given a list value self.initParser() self.add_arg("-f", "--file", env_var="FILES", action="append", type=int) ns = self.parse("", env_vars = {"file": "[1,2,3]", "VERBOSE": "true"}) self.assertEqual(ns.file, None) def testAutoEnvVarPrefix(self): self.initParser(auto_env_var_prefix="TEST_") self.add_arg("-a", "--arg0", is_config_file_arg=True) self.add_arg("-b", "--arg1", is_write_out_config_file_arg=True) self.add_arg("-x", "--arg2", env_var="TEST2", type=int) self.add_arg("-y", "--arg3", action="append", type=int) self.add_arg("-z", "--arg4", required=True) self.add_arg("-w", "--arg4-more", required=True) ns = self.parse("", env_vars = { "TEST_ARG0": "0", "TEST_ARG1": "1", "TEST_ARG2": "2", "TEST2": "22", "TEST_ARG4": "arg4_value", "TEST_ARG4_MORE": "magic"}) self.assertEqual(ns.arg0, None) self.assertEqual(ns.arg1, None) self.assertEqual(ns.arg2, 22) self.assertEqual(ns.arg4, "arg4_value") self.assertEqual(ns.arg4_more, "magic") def testEnvVarLists(self): self.initParser() self.add_arg("-x", "--arg2", env_var="TEST2") self.add_arg("-y", "--arg3", env_var="TEST3", type=int) self.add_arg("-z", "--arg4", env_var="TEST4", nargs="+") self.add_arg("-u", "--arg5", env_var="TEST5", nargs="+", type=int) self.add_arg("--arg6", env_var="TEST6") self.add_arg("--arg7", env_var="TEST7", action="append") ns = self.parse("", env_vars={"TEST2": "22", "TEST3": "22", "TEST4": "[Shell, someword, anotherword]", "TEST5": "[22, 99, 33]", "TEST6": "[value6.1, value6.2, value6.3]", "TEST7": "[value7.1, value7.2, value7.3]", }) self.assertEqual(ns.arg2, "22") self.assertEqual(ns.arg3, 22) self.assertEqual(ns.arg4, ['Shell', 'someword', 'anotherword']) self.assertEqual(ns.arg5, [22, 99, 33]) self.assertEqual(ns.arg6, "[value6.1, value6.2, value6.3]") self.assertEqual(ns.arg7, ["value7.1", "value7.2", "value7.3"]) def testPositionalAndEnvVarLists(self): self.initParser() self.add_arg("a") self.add_arg("-x", "--arg", env_var="TEST", nargs="+") ns = self.parse("positional_value", env_vars={"TEST": "[Shell, someword, anotherword]"}) self.assertEqual(ns.arg, ['Shell', 'someword', 'anotherword']) self.assertEqual(ns.a, "positional_value") class TestMisc(TestCase): # TODO test different action types with config file, env var """Test edge cases""" def setUp(self): self.initParser(args_for_setting_config_path=[]) @mock.patch('argparse.ArgumentParser.__init__') def testKwrgsArePassedToArgParse(self, argparse_init): kwargs_for_argparse = {"allow_abbrev": False, "whatever_other_arg": "something"} parser = configargparse.ArgumentParser(add_config_file_help=False, **kwargs_for_argparse) argparse_init.assert_called_with(parser, **kwargs_for_argparse) def testGlobalInstances(self, name=None): p = configargparse.getArgumentParser(name, prog="prog", usage="test") self.assertEqual(p.usage, "test") self.assertEqual(p.prog, "prog") self.assertRaisesRegex(ValueError, "kwargs besides 'name' can only be " "passed in the first time", configargparse.getArgumentParser, name, prog="prog") p2 = configargparse.getArgumentParser(name) self.assertEqual(p, p2) def testGlobalInstances_WithName(self): self.testGlobalInstances("name1") self.testGlobalInstances("name2") def testAddArguments_ArgValidation(self): self.assertRaises(ValueError, self.add_arg, 'positional', env_var="bla") action = self.add_arg('positional') self.assertIsNotNone(action) self.assertEqual(action.dest, "positional") def testAddArguments_IsConfigFilePathArg(self): self.assertRaises(ValueError, self.add_arg, 'c', action="store_false", is_config_file=True) self.add_arg("-c", "--config", is_config_file=True) self.add_arg("--x", required=True) # verify parsing from config file config_file = tempfile.NamedTemporaryFile(mode="w", delete=True) config_file.write("x=bla") config_file.flush() ns = self.parse(args="-c %s" % config_file.name) self.assertEqual(ns.x, "bla") def testConstructor_ConfigFileArgs(self): # Test constructor args: # args_for_setting_config_path # config_arg_is_required # config_arg_help_message temp_cfg = tempfile.NamedTemporaryFile(mode="w", delete=True) temp_cfg.write("genome=hg19") temp_cfg.flush() self.initParser(args_for_setting_config_path=["-c", "--config"], config_arg_is_required = True, config_arg_help_message = "my config file", default_config_files=[temp_cfg.name]) self.add_arg('--genome', help='Path to genome file', required=True) self.assertParseArgsRaises("argument -c/--config is required" if sys.version_info < (3,3) else "arguments are required: -c/--config",) temp_cfg2 = tempfile.NamedTemporaryFile(mode="w", delete=True) ns = self.parse("-c " + temp_cfg2.name) self.assertEqual(ns.genome, "hg19") # temp_cfg2 config file should override default config file values temp_cfg2.write("genome=hg20") temp_cfg2.flush() ns = self.parse("-c " + temp_cfg2.name) self.assertEqual(ns.genome, "hg20") self.assertRegex(self.format_help(), 'usage: .* \[-h\] -c CONFIG_FILE --genome GENOME\n\n'+ 7*'.+\s+'+ # repeated 7 times because .+ matches atmost 1 line 'optional arguments:\n' ' -h, --help\s+ show this help message and exit\n' ' -c CONFIG_FILE, --config CONFIG_FILE\s+ my config file\n' ' --genome GENOME\s+ Path to genome file\n') # just run print_values() to make sure it completes and returns None self.assertEqual(self.parser.print_values(file=sys.stderr), None) # test ignore_unknown_config_file_keys=False self.initParser(ignore_unknown_config_file_keys=False) self.assertRaisesRegex(argparse.ArgumentError, "unrecognized arguments", self.parse, config_file_contents="arg1 = 3") ns, args = self.parse_known(config_file_contents="arg1 = 3") self.assertEqual(getattr(ns, "arg1", ""), "") # test ignore_unknown_config_file_keys=True self.initParser(ignore_unknown_config_file_keys=True) ns = self.parse(args="", config_file_contents="arg1 = 3") self.assertEqual(getattr(ns, "arg1", ""), "") ns, args = self.parse_known(config_file_contents="arg1 = 3") self.assertEqual(getattr(ns, "arg1", ""), "") def test_AbbrevConfigFileArgs(self): """Tests that abbreviated values don't get pulled from config file. """ temp_cfg = tempfile.NamedTemporaryFile(mode="w", delete=True) temp_cfg.write("a2a = 0.5\n") temp_cfg.write("a3a = 0.5\n") temp_cfg.flush() self.initParser() self.add_arg('-c', '--config_file', required=False, is_config_file=True, help='config file path') self.add_arg('--hello', type=int, required=False) command = '-c {0} --hello 2'.format(temp_cfg.name) known, unknown = self.parse_known(command) self.assertListEqual(unknown, ['--a2a', '0.5', '--a3a', '0.5']) def test_FormatHelp(self): self.initParser(args_for_setting_config_path=["-c", "--config"], config_arg_is_required = True, config_arg_help_message = "my config file", default_config_files=["~/.myconfig"], args_for_writing_out_config_file=["-w", "--write-config"], ) self.add_arg('--arg1', help='Arg1 help text', required=True) self.add_arg('--flag', help='Flag help text', action="store_true") self.assertRegex(self.format_help(), 'usage: .* \[-h\] -c CONFIG_FILE\s+' '\[-w CONFIG_OUTPUT_PATH\]\s* --arg1 ARG1\s*\[--flag\]\s*' 'Args that start with \'--\' \(eg. --arg1\) can also be set in a ' 'config file\s*\(~/.myconfig or specified via -c\).\s*' 'Config file syntax allows: key=value,\s*flag=true, stuff=\[a,b,c\] ' '\(for details, see syntax at https://goo.gl/R74nmi\).\s*' 'If an arg is specified in more than\s*one place, then ' 'commandline values\s*override config file values which override\s*' 'defaults.\s*' 'optional arguments:\s*' '-h, --help \s* show this help message and exit\n\s*' '-c CONFIG_FILE, --config CONFIG_FILE\s+my config file\s*' '-w CONFIG_OUTPUT_PATH, --write-config CONFIG_OUTPUT_PATH\s*takes\s*' 'the current command line args and writes them\s*' 'out to a config file at the given path, then exits\s*' '--arg1 ARG1\s*Arg1 help text\s*' '--flag \s*Flag help text' ) class CustomClass(object): def __init__(self, name): self.name = name @staticmethod def valid_custom(s): if s == "invalid": raise Exception("invalid name") return TestMisc.CustomClass(s) def testConstructor_WriteOutConfigFileArgs(self): # Test constructor args: # args_for_writing_out_config_file # write_out_config_file_arg_help_message cfg_f = tempfile.NamedTemporaryFile(mode="w+", delete=True) self.initParser(args_for_writing_out_config_file=["-w"], write_out_config_file_arg_help_message="write config") self.add_arg("-not-config-file-settable") self.add_arg("--config-file-settable-arg", type=int) self.add_arg("--config-file-settable-arg2", type=int, default=3) self.add_arg("--config-file-settable-flag", action="store_true") self.add_arg("--config-file-settable-custom", type=TestMisc.valid_custom) self.add_arg("-l", "--config-file-settable-list", action="append") # write out a config file command_line_args = "-w %s " % cfg_f.name command_line_args += "--config-file-settable-arg 1 " command_line_args += "--config-file-settable-flag " command_line_args += "--config-file-settable-custom custom_value " command_line_args += "-l a -l b -l c -l d " self.assertFalse(self.parser._exit_method_called) ns = self.parse(command_line_args) self.assertTrue(self.parser._exit_method_called) cfg_f.seek(0) expected_config_file_contents = "config-file-settable-arg = 1\n" expected_config_file_contents += "config-file-settable-flag = true\n" expected_config_file_contents += "config-file-settable-custom = custom_value\n" expected_config_file_contents += "config-file-settable-list = [a, b, c, d]\n" expected_config_file_contents += "config-file-settable-arg2 = 3\n" self.assertEqual(cfg_f.read().strip(), expected_config_file_contents.strip()) self.assertRaisesRegex(ValueError, "Couldn't open / for writing:", self.parse, args = command_line_args + " -w /") def testConstructor_WriteOutConfigFileArgs2(self): # Test constructor args: # args_for_writing_out_config_file # write_out_config_file_arg_help_message cfg_f = tempfile.NamedTemporaryFile(mode="w+", delete=True) self.initParser(args_for_writing_out_config_file=["-w"], write_out_config_file_arg_help_message="write config") self.add_arg("-not-config-file-settable") self.add_arg("-a", "--arg1", type=int, env_var="ARG1") self.add_arg("-b", "--arg2", type=int, default=3) self.add_arg("-c", "--arg3") self.add_arg("-d", "--arg4") self.add_arg("-e", "--arg5") self.add_arg("--config-file-settable-flag", action="store_true", env_var="FLAG_ARG") self.add_arg("-l", "--config-file-settable-list", action="append") # write out a config file command_line_args = "-w %s " % cfg_f.name command_line_args += "-l a -l b -l c -l d " self.assertFalse(self.parser._exit_method_called) ns = self.parse(command_line_args, env_vars={"ARG1": "10", "FLAG_ARG": "true", "SOME_OTHER_ENV_VAR": "2"}, config_file_contents="arg3 = bla3\narg4 = bla4") self.assertTrue(self.parser._exit_method_called) cfg_f.seek(0) expected_config_file_contents = "config-file-settable-list = [a, b, c, d]\n" expected_config_file_contents += "arg1 = 10\n" expected_config_file_contents += "config-file-settable-flag = True\n" expected_config_file_contents += "arg3 = bla3\n" expected_config_file_contents += "arg4 = bla4\n" expected_config_file_contents += "arg2 = 3\n" self.assertEqual(cfg_f.read().strip(), expected_config_file_contents.strip()) self.assertRaisesRegex(ValueError, "Couldn't open / for writing:", self.parse, args = command_line_args + " -w /") def testConstructor_WriteOutConfigFileArgsLong(self): """Test config writing with long version of arg There was a bug where the long version of the args_for_writing_out_config_file was being dumped into the resultant output config file """ # Test constructor args: # args_for_writing_out_config_file # write_out_config_file_arg_help_message cfg_f = tempfile.NamedTemporaryFile(mode="w+", delete=True) self.initParser(args_for_writing_out_config_file=["--write-config"], write_out_config_file_arg_help_message="write config") self.add_arg("-not-config-file-settable") self.add_arg("--config-file-settable-arg", type=int) self.add_arg("--config-file-settable-arg2", type=int, default=3) self.add_arg("--config-file-settable-flag", action="store_true") self.add_arg("-l", "--config-file-settable-list", action="append") # write out a config file command_line_args = "--write-config %s " % cfg_f.name command_line_args += "--config-file-settable-arg 1 " command_line_args += "--config-file-settable-flag " command_line_args += "-l a -l b -l c -l d " self.assertFalse(self.parser._exit_method_called) ns = self.parse(command_line_args) self.assertTrue(self.parser._exit_method_called) cfg_f.seek(0) expected_config_file_contents = "config-file-settable-arg = 1\n" expected_config_file_contents += "config-file-settable-flag = true\n" expected_config_file_contents += "config-file-settable-list = [a, b, c, d]\n" expected_config_file_contents += "config-file-settable-arg2 = 3\n" self.assertEqual(cfg_f.read().strip(), expected_config_file_contents.strip()) self.assertRaisesRegex(ValueError, "Couldn't open / for writing:", self.parse, args = command_line_args + " --write-config /") def testMethodAliases(self): p = self.parser p.add("-a", "--arg-a", default=3) p.add_arg("-b", "--arg-b", required=True) p.add_argument("-c") g1 = p.add_argument_group(title="group1", description="descr") g1.add("-d", "--arg-d", required=True) g1.add_arg("-e", "--arg-e", required=True) g1.add_argument("-f", "--arg-f", default=5) g2 = p.add_mutually_exclusive_group(required=True) g2.add("-x", "--arg-x") g2.add_arg("-y", "--arg-y") g2.add_argument("-z", "--arg-z", default=5) # verify that flags must be globally unique g2 = p.add_argument_group(title="group2", description="descr") self.assertRaises(argparse.ArgumentError, g1.add, "-c") self.assertRaises(argparse.ArgumentError, g2.add, "-f") self.initParser() p = self.parser options = p.parse(args=[]) self.assertDictEqual(vars(options), {}) class TestConfigFileParsers(TestCase): """Test ConfigFileParser subclasses in isolation""" def testDefaultConfigFileParser_Basic(self): p = configargparse.DefaultConfigFileParser() self.assertTrue(len(p.get_syntax_description()) > 0) # test the simplest case input_config_str = StringIO("""a: 3\n""") parsed_obj = p.parse(input_config_str) output_config_str = p.serialize(parsed_obj) self.assertEqual(input_config_str.getvalue().replace(": ", " = "), output_config_str) self.assertDictEqual(parsed_obj, dict([('a', '3')])) def testDefaultConfigFileParser_All(self): p = configargparse.DefaultConfigFileParser() # test the all syntax case config_lines = [ "# comment1 ", "[ some section ]", "----", "---------", "_a: 3", "; comment2 ", "_b = c", "_list_arg1 = [a, b, c]", "_str_arg = true", "_list_arg2 = [1, 2, 3]", ] # test parse input_config_str = StringIO("\n".join(config_lines)+"\n") parsed_obj = p.parse(input_config_str) # test serialize output_config_str = p.serialize(parsed_obj) self.assertEqual("\n".join( l.replace(': ', ' = ') for l in config_lines if l.startswith('_'))+"\n", output_config_str) self.assertDictEqual(parsed_obj, dict([ ('_a', '3'), ('_b', 'c'), ('_list_arg1', ['a', 'b', 'c']), ('_str_arg', 'true'), ('_list_arg2', ['1', '2', '3']), ])) self.assertListEqual(parsed_obj['_list_arg1'], ['a', 'b', 'c']) self.assertListEqual(parsed_obj['_list_arg2'], ['1', '2', '3']) def testYAMLConfigFileParser_Basic(self): try: import yaml except: logging.warning("WARNING: PyYAML not installed. " "Couldn't test YAMLConfigFileParser") return p = configargparse.YAMLConfigFileParser() self.assertTrue(len(p.get_syntax_description()) > 0) input_config_str = StringIO("""a: '3'\n""") parsed_obj = p.parse(input_config_str) output_config_str = p.serialize(dict(parsed_obj)) self.assertEqual(input_config_str.getvalue(), output_config_str) self.assertDictEqual(parsed_obj, dict([('a', '3')])) def testYAMLConfigFileParser_All(self): try: import yaml except: logging.warning("WARNING: PyYAML not installed. " "Couldn't test YAMLConfigFileParser") return p = configargparse.YAMLConfigFileParser() # test the all syntax case config_lines = [ "a: '3'", "list_arg:", "- 1", "- 2", "- 3", ] # test parse input_config_str = StringIO("\n".join(config_lines)+"\n") parsed_obj = p.parse(input_config_str) # test serialize output_config_str = p.serialize(parsed_obj) self.assertEqual(input_config_str.getvalue(), output_config_str) self.assertDictEqual(parsed_obj, dict([ ('a', '3'), ('list_arg', [1,2,3]), ])) ################################################################################ # since configargparse should work as a drop-in replacement for argparse # in all situations, run argparse unittests on configargparse by modifying # their source code to use configargparse.ArgumentParser try: import test.test_argparse #Sig = test.test_argparse.Sig #NS = test.test_argparse.NS except ImportError: if sys.version_info < (2, 7): logging.info("\n\n" + ("=" * 30) + "\nINFO: Skipping tests for argparse (Python < 2.7)\n" + ("=" * 30) + "\n") else: logging.error("\n\n" "============================\n" "ERROR: Many tests couldn't be run because 'import test.test_argparse' " "failed. Try building/installing python from source rather than through" " a package manager.\n" "============================\n") else: test_argparse_source_code = inspect.getsource(test.test_argparse) test_argparse_source_code = test_argparse_source_code.replace( 'argparse.ArgumentParser', 'configargparse.ArgumentParser') # run or debug a subset of the argparse tests #test_argparse_source_code = test_argparse_source_code.replace( # "(TestCase)", "").replace( # "(ParserTestCase)", "").replace( # "(HelpTestCase)", "").replace( # ", TestCase", "").replace( # ", ParserTestCase", "") #test_argparse_source_code = test_argparse_source_code.replace( # "class TestMessageContentError", "class TestMessageContentError(TestCase)") exec(test_argparse_source_code) # print argparse unittest source code def print_source_code(source_code, line_numbers, context_lines=10): for n in line_numbers: logging.debug("##### Code around line %s #####" % n) lines_to_print = set(range(n - context_lines, n + context_lines)) for n2, line in enumerate(source_code.split("\n"), 1): if n2 in lines_to_print: logging.debug("%s %5d: %s" % ( "**" if n2 == n else " ", n2, line)) #print_source_code(test_argparse_source_code, [4540, 4565])