ConfigArgParse-1.5.3/ 0000755 0000000 0000024 00000000000 14126173212 014435 5 ustar root staff 0000000 0000000 ConfigArgParse-1.5.3/PKG-INFO 0000644 0000000 0000024 00000053516 14126173212 015544 0 ustar root staff 0000000 0000000 Metadata-Version: 2.1
Name: ConfigArgParse
Version: 1.5.3
Summary: A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.
Home-page: https://github.com/bw2/ConfigArgParse
License: MIT
Description: ConfigArgParse
--------------
.. image:: https://img.shields.io/pypi/v/ConfigArgParse.svg?style=flat
:alt: PyPI version
:target: https://pypi.python.org/pypi/ConfigArgParse
.. image:: https://img.shields.io/pypi/pyversions/ConfigArgParse.svg
:alt: Supported Python versions
:target: https://pypi.python.org/pypi/ConfigArgParse
.. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
:alt: Travis CI build
:target: https://travis-ci.org/bw2/ConfigArgParse
.. image:: https://img.shields.io/badge/-API_Documentation-blue
:alt: API Documentation
:target: https://bw2.github.io/ConfigArgParse/
Overview
~~~~~~~~
Applications with more than a handful of user-settable options are best
configured through a combination of command line args, config files,
hard-coded defaults, and in some cases, environment variables.
Python's command line parsing modules such as argparse have very limited
support for config files and environment variables, so this module
extends argparse to add these features.
Available on PyPI: http://pypi.python.org/pypi/ConfigArgParse
.. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
:target: https://travis-ci.org/bw2/ConfigArgParse
Features
~~~~~~~~
- command-line, config file, env var, and default settings can now be
defined, documented, and parsed in one go using a single API (if a
value is specified in more than one way then: command line >
environment variables > config file values > defaults)
- config files can have .ini or .yaml style syntax (eg. key=value or
key: value)
- user can provide a config file via a normal-looking command line arg
(eg. -c path/to/config.txt) rather than the argparse-style @config.txt
- one or more default config file paths can be specified
(eg. ['/etc/bla.conf', '~/.my_config'] )
- all argparse functionality is fully supported, so this module can
serve as a drop-in replacement (verified by argparse unittests).
- env vars and config file keys & syntax are automatically documented
in the -h help message
- new method :code:`print_values()` can report keys & values and where
they were set (eg. command line, env var, config file, or default).
- lite-weight (no 3rd-party library dependencies except (optionally) PyYAML)
- extensible (:code:`ConfigFileParser` can be subclassed to define a new
config file format)
- unittested by running the unittests that came with argparse but on
configargparse, and using tox to test with Python 2.7 and Python 3+
Example
~~~~~~~
*config_test.py*:
Script that defines 4 options and a positional arg and then parses and prints the values. Also,
it prints out the help message as well as the string produced by :code:`format_values()` to show
what they look like.
.. code:: py
import configargparse
p = configargparse.ArgParser(default_config_files=['/etc/app/conf.d/*.conf', '~/.my_settings'])
p.add('-c', '--my-config', required=True, is_config_file=True, help='config file path')
p.add('--genome', required=True, help='path to genome file') # this option can be set in a config file because it starts with '--'
p.add('-v', help='verbose', action='store_true')
p.add('-d', '--dbsnp', help='known variants .vcf', env_var='DBSNP_PATH') # this option can be set in a config file because it starts with '--'
p.add('vcf', nargs='+', help='variant file(s)')
options = p.parse_args()
print(options)
print("----------")
print(p.format_help())
print("----------")
print(p.format_values()) # useful for logging where different settings came from
*config.txt:*
Since the script above set the config file as required=True, lets create a config file to give it:
.. code:: py
# settings for config_test.py
genome = HCMV # cytomegalovirus genome
dbsnp = /data/dbsnp/variants.vcf
*command line:*
Now run the script and pass it the config file:
.. code:: bash
DBSNP_PATH=/data/dbsnp/variants_v2.vcf python config_test.py --my-config config.txt f1.vcf f2.vcf
*output:*
Here is the result:
.. code:: bash
Namespace(dbsnp='/data/dbsnp/variants_v2.vcf', genome='HCMV', my_config='config.txt', v=False, vcf=['f1.vcf', 'f2.vcf'])
----------
usage: config_test.py [-h] -c MY_CONFIG --genome GENOME [-v] [-d DBSNP]
vcf [vcf ...]
Args that start with '--' (eg. --genome) can also be set in a config file
(/etc/app/conf.d/*.conf or ~/.my_settings or specified via -c). Config file
syntax allows: key=value, flag=true, stuff=[a,b,c] (for details, see syntax at
https://goo.gl/R74nmi). If an arg is specified in more than one place, then
commandline values override environment variables which override config file
values which override defaults.
positional arguments:
vcf variant file(s)
optional arguments:
-h, --help show this help message and exit
-c MY_CONFIG, --my-config MY_CONFIG
config file path
--genome GENOME path to genome file
-v verbose
-d DBSNP, --dbsnp DBSNP
known variants .vcf [env var: DBSNP_PATH]
----------
Command Line Args: --my-config config.txt f1.vcf f2.vcf
Environment Variables:
DBSNP_PATH: /data/dbsnp/variants_v2.vcf
Config File (config.txt):
genome: HCMV
Special Values
~~~~~~~~~~~~~~
Under the hood, configargparse handles environment variables and config file
values by converting them to their corresponding command line arg. For
example, "key = value" will be processed as if "--key value" was specified
on the command line.
Also, the following special values (whether in a config file or an environment
variable) are handled in a special way to support booleans and lists:
- :code:`key = true` is handled as if "--key" was specified on the command line.
In your python code this key must be defined as a boolean flag
(eg. action="store_true" or similar).
- :code:`key = [value1, value2, ...]` is handled as if "--key value1 --key value2"
etc. was specified on the command line. In your python code this key must
be defined as a list (eg. action="append").
Config File Syntax
~~~~~~~~~~~~~~~~~~
Only command line args that have a long version (eg. one that starts with '--')
can be set in a config file. For example, "--color" can be set by putting
"color=green" in a config file. The config file syntax depends on the constuctor
arg: :code:`config_file_parser_class` which can be set to one of the provided
classes: :code:`DefaultConfigFileParser`, :code:`YAMLConfigFileParser`,
:code:`ConfigparserConfigFileParser` or to your own subclass of the
:code:`ConfigFileParser` abstract class.
*DefaultConfigFileParser* - the full range of valid syntax is:
.. code:: yaml
# this is a comment
; this is also a comment (.ini style)
--- # lines that start with --- are ignored (yaml style)
-------------------
[section] # .ini-style section names are treated as comments
# how to specify a key-value pair (all of these are equivalent):
name value # key is case sensitive: "Name" isn't "name"
name = value # (.ini style) (white space is ignored, so name = value same as name=value)
name: value # (yaml style)
--name value # (argparse style)
# how to set a flag arg (eg. arg which has action="store_true")
--name
name
name = True # "True" and "true" are the same
# how to specify a list arg (eg. arg which has action="append")
fruit = [apple, orange, lemon]
indexes = [1, 12, 35 , 40]
*YAMLConfigFileParser* - allows a subset of YAML syntax (http://goo.gl/VgT2DU)
.. code:: yaml
# a comment
name1: value
name2: true # "True" and "true" are the same
fruit: [apple, orange, lemon]
indexes: [1, 12, 35, 40]
*ConfigparserConfigFileParser* - allows a subset of python's configparser
module syntax (https://docs.python.org/3.7/library/configparser.html). In
particular the following configparser options are set:
.. code:: py
config = configparser.ArgParser(
delimiters=("=",":"),
allow_no_value=False,
comment_prefixes=("#",";"),
inline_comment_prefixes=("#",";"),
strict=True,
empty_lines_in_values=False,
)
Once configparser parses the config file all section names are removed, thus all
keys must have unique names regardless of which INI section they are defined
under. Also, any keys which have python list syntax are converted to lists by
evaluating them as python code using ast.literal_eval
(https://docs.python.org/3/library/ast.html#ast.literal_eval). To facilitate
this all multi-line values are converted to single-line values. Thus multi-line
string values will have all new-lines converted to spaces. Note, since key-value
pairs that have python dictionary syntax are saved as single-line strings, even
if formatted across multiple lines in the config file, dictionaries can be read
in and converted to valid python dictionaries with PyYAML's safe_load. Example
given below:
.. code:: py
# inside your config file (e.g. config.ini)
[section1] # INI sections treated as comments
system1_settings: { # start of multi-line dictionary
'a':True,
'b':[2, 4, 8, 16],
'c':{'start':0, 'stop':1000},
'd':'experiment 32 testing simulation with parameter a on'
} # end of multi-line dictionary value
.......
# in your configargparse setup
import configargparse
import yaml
parser = configargparse.ArgParser(
config_file_parser_class=configargparse.ConfigparserConfigFileParser
)
parser.add_argument('--system1_settings', type=yaml.safe_load)
args = parser.parse_args() # now args.system1 is a valid python dict
ArgParser Singletons
~~~~~~~~~~~~~~~~~~~~~~~~~
To make it easier to configure different modules in an application,
configargparse provides globally-available ArgumentParser instances
via configargparse.get_argument_parser('name') (similar to
logging.getLogger('name')).
Here is an example of an application with a utils module that also
defines and retrieves its own command-line args.
*main.py*
.. code:: py
import configargparse
import utils
p = configargparse.get_argument_parser()
p.add_argument("-x", help="Main module setting")
p.add_argument("--m-setting", help="Main module setting")
options = p.parse_known_args() # using p.parse_args() here may raise errors.
*utils.py*
.. code:: py
import configargparse
p = configargparse.get_argument_parser()
p.add_argument("--utils-setting", help="Config-file-settable option for utils")
if __name__ == "__main__":
options = p.parse_known_args()
Help Formatters
~~~~~~~~~~~~~~~
:code:`ArgumentDefaultsRawHelpFormatter` is a new HelpFormatter that both adds
default values AND disables line-wrapping. It can be passed to the constructor:
:code:`ArgParser(.., formatter_class=ArgumentDefaultsRawHelpFormatter)`
Aliases
~~~~~~~
The configargparse.ArgumentParser API inherits its class and method
names from argparse and also provides the following shorter names for
convenience:
- p = configargparse.get_arg_parser() # get global singleton instance
- p = configargparse.get_parser()
- p = configargparse.ArgParser() # create a new instance
- p = configargparse.Parser()
- p.add_arg(..)
- p.add(..)
- options = p.parse(..)
HelpFormatters:
- RawFormatter = RawDescriptionHelpFormatter
- DefaultsFormatter = ArgumentDefaultsHelpFormatter
- DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter
API Documentation
~~~~~~~~~~~~~~~~~
You can review the generated API Documentation for the ``configargparse`` module: `HERE `_
Design Notes
~~~~~~~~~~~~
Unit tests:
tests/test_configargparse.py contains custom unittests for features
specific to this module (such as config file and env-var support), as
well as a hook to load and run argparse unittests (see the built-in
test.test_argparse module) but on configargparse in place of argparse.
This ensures that configargparse will work as a drop in replacement for
argparse in all usecases.
Previously existing modules (PyPI search keywords: config argparse):
- argparse (built-in module Python v2.7+)
- Good:
- fully featured command line parsing
- can read args from files using an easy to understand mechanism
- Bad:
- syntax for specifying config file path is unusual (eg.
@file.txt)and not described in the user help message.
- default config file syntax doesn't support comments and is
unintuitive (eg. --namevalue)
- no support for environment variables
- ConfArgParse v1.0.15
(https://pypi.python.org/pypi/ConfArgParse)
- Good:
- extends argparse with support for config files parsed by
ConfigParser
- clear documentation in README
- Bad:
- config file values are processed using
ArgumentParser.set_defaults(..) which means "required" and
"choices" are not handled as expected. For example, if you
specify a required value in a config file, you still have to
specify it again on the command line.
- doesn't work with Python 3 yet
- no unit tests, code not well documented
- appsettings v0.5 (https://pypi.python.org/pypi/appsettings)
- Good:
- supports config file (yaml format) and env_var parsing
- supports config-file-only setting for specifying lists and
dicts
- Bad:
- passes in config file and env settings via parse_args
namespace param
- tests not finished and don't work with Python 3 (import
StringIO)
- argparse_config v0.5.1
(https://pypi.python.org/pypi/argparse_config)
- Good:
- similar features to ConfArgParse v1.0.15
- Bad:
- doesn't work with Python 3 (error during pip install)
- yconf v0.3.2 - (https://pypi.python.org/pypi/yconf) - features
and interface not that great
- hieropt v0.3 - (https://pypi.python.org/pypi/hieropt) - doesn't
appear to be maintained, couldn't find documentation
- configurati v0.2.3 - (https://pypi.python.org/pypi/configurati)
- Good:
- JSON, YAML, or Python configuration files
- handles rich data structures such as dictionaries
- can group configuration names into sections (like .ini files)
- Bad:
- doesn't work with Python 3
- 2+ years since last release to PyPI
- apparently unmaintained
Design choices:
1. all options must be settable via command line. Having options that
can only be set using config files or env. vars adds complexity to
the API, and is not a useful enough feature since the developer can
split up options into sections and call a section "config file keys",
with command line args that are just "--" plus the config key.
2. config file and env. var settings should be processed by appending
them to the command line (another benefit of #1). This is an
easy-to-implement solution and implicitly takes care of checking that
all "required" args are provied, etc., plus the behavior should be
easy for users to understand.
3. configargparse shouldn't override argparse's
convert_arg_line_to_args method so that all argparse unit tests
can be run on configargparse.
4. in terms of what to allow for config file keys, the "dest" value of
an option can't serve as a valid config key because many options can
have the same dest. Instead, since multiple options can't use the
same long arg (eg. "--long-arg-x"), let the config key be either
"--long-arg-x" or "long-arg-x". This means the developer can allow
only a subset of the command-line args to be specified via config
file (eg. short args like -x would be excluded). Also, that way
config keys are automatically documented whenever the command line
args are documented in the help message.
5. don't force users to put config file settings in the right .ini
[sections]. This doesn't have a clear benefit since all options are
command-line settable, and so have a globally unique key anyway.
Enforcing sections just makes things harder for the user and adds
complexity to the implementation.
6. if necessary, config-file-only args can be added later by
implementing a separate add method and using the namespace arg as in
appsettings_v0.5
Relevant sites:
- http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll
- http://tricksntweaks.blogspot.com/2013_05_01_archive.html
- http://www.youtube.com/watch?v=vvCwqHgZJc8#t=35
.. |Travis CI Status for bw2/ConfigArgParse| image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
Versioning
~~~~~~~~~~
This software follows `Semantic Versioning`_
.. _Semantic Versioning: http://semver.org/
Keywords: options,argparse,ConfigArgParse,config,environment variables,envvars,ENV,environment,optparse,YAML,INI
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*
Provides-Extra: test
Provides-Extra: yaml
ConfigArgParse-1.5.3/LICENSE 0000644 0000000 0000024 00000002056 13235644610 015452 0 ustar root staff 0000000 0000000 The MIT License (MIT)
Copyright (c) 2015 bw2
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
ConfigArgParse-1.5.3/tests/ 0000755 0000000 0000024 00000000000 14126173212 015577 5 ustar root staff 0000000 0000000 ConfigArgParse-1.5.3/tests/__init__.py 0000644 0000000 0000024 00000000056 13235644610 017716 0 ustar root staff 0000000 0000000 #!/usr/bin/env python
# -*- coding: utf-8 -*-
ConfigArgParse-1.5.3/tests/__init__.pyc 0000644 0000000 0000024 00000000214 13573223334 020056 0 ustar root staff 0000000 0000000 ó
ˆIwZc @ s d S( N( ( ( ( s5 /Users/weisburd/code/ConfigArgParse/tests/__init__.pyt t ConfigArgParse-1.5.3/tests/test_configargparse.py 0000644 0000000 0000024 00000210273 14126173075 022216 0 ustar root staff 0000000 0000000 import argparse
import configargparse
from contextlib import contextmanager
import inspect
import logging
import os
import sys
import tempfile
import types
import unittest
try:
import mock
except ImportError:
from unittest import mock
if sys.version_info >= (3, 0):
from io import StringIO
else:
from StringIO import StringIO
if sys.version_info >= (3, 10):
OPTIONAL_ARGS_STRING="options"
else:
OPTIONAL_ARGS_STRING="optional arguments"
# set COLUMNS to get expected wrapping
os.environ['COLUMNS'] = '80'
# enable logging to simplify debugging
logger = logging.getLogger()
logger.level = logging.DEBUG
stream_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(stream_handler)
def replace_error_method(arg_parser):
"""Swap out arg_parser's error(..) method so that instead of calling
sys.exit(..) it just raises an error.
"""
def error_method(self, message):
raise argparse.ArgumentError(None, message)
def exit_method(self, status, message=None):
self._exit_method_called = True
arg_parser._exit_method_called = False
arg_parser.error = types.MethodType(error_method, arg_parser)
arg_parser.exit = types.MethodType(exit_method, arg_parser)
return arg_parser
@contextmanager
def captured_output():
"""
swap stdout and stderr for StringIO so we can do asserts on outputs.
"""
new_out, new_err = StringIO(), StringIO()
old_out, old_err = sys.stdout, sys.stderr
try:
sys.stdout, sys.stderr = new_out, new_err
yield sys.stdout, sys.stderr
finally:
sys.stdout, sys.stderr = old_out, old_err
class TestCase(unittest.TestCase):
def initParser(self, *args, **kwargs):
p = configargparse.ArgParser(*args, **kwargs)
self.parser = replace_error_method(p)
self.add_arg = self.parser.add_argument
self.parse = self.parser.parse_args
self.parse_known = self.parser.parse_known_args
self.format_values = self.parser.format_values
self.format_help = self.parser.format_help
if not hasattr(self, "assertRegex"):
self.assertRegex = self.assertRegexpMatches
if not hasattr(self, "assertRaisesRegex"):
self.assertRaisesRegex = self.assertRaisesRegexp
return self.parser
def assertParseArgsRaises(self, regex, args, **kwargs):
self.assertRaisesRegex(argparse.ArgumentError, regex, self.parse,
args=args, **kwargs)
class TestBasicUseCases(TestCase):
def setUp(self):
self.initParser(args_for_setting_config_path=[])
def testBasicCase1(self):
## Test command line and config file values
self.add_arg("filenames", nargs="+", help="positional arg")
self.add_arg("-x", "--arg-x", action="store_true")
self.add_arg("-y", "--arg-y", dest="y1", type=int, required=True)
self.add_arg("--arg-z", action="append", type=float, required=True)
if sys.version_info >= (3, 9):
self.add_arg('--foo', action=argparse.BooleanOptionalAction, default=False)
else:
self.add_arg('--foo', action="store_true", default=False)
# make sure required args are enforced
self.assertParseArgsRaises("too few arg"
if sys.version_info.major < 3 else
"the following arguments are required", args="")
self.assertParseArgsRaises("argument -y/--arg-y is required"
if sys.version_info.major < 3 else
"the following arguments are required: -y/--arg-y",
args="-x --arg-z 11 file1.txt")
self.assertParseArgsRaises("argument --arg-z is required"
if sys.version_info.major < 3 else
"the following arguments are required: --arg-z",
args="file1.txt file2.txt file3.txt -x -y 1")
# check values after setting args on command line
ns = self.parse(args="file1.txt --arg-x -y 3 --arg-z 10 --foo",
config_file_contents="")
self.assertListEqual(ns.filenames, ["file1.txt"])
self.assertEqual(ns.arg_x, True)
self.assertEqual(ns.y1, 3)
self.assertEqual(ns.arg_z, [10])
self.assertEqual(ns.foo, True)
self.assertRegex(self.format_values(),
'Command Line Args: file1.txt --arg-x -y 3 --arg-z 10')
# check values after setting args in config file
ns = self.parse(args="file1.txt file2.txt", config_file_contents="""
# set all required args in config file
arg-x = True
arg-y = 10
arg-z = 30
arg-z = 40
foo = True
""")
self.assertListEqual(ns.filenames, ["file1.txt", "file2.txt"])
self.assertEqual(ns.arg_x, True)
self.assertEqual(ns.y1, 10)
self.assertEqual(ns.arg_z, [40])
self.assertEqual(ns.foo, True)
self.assertRegex(self.format_values(),
'Command Line Args: \\s+ file1.txt file2.txt\n'
'Config File \\(method arg\\):\n'
' arg-x: \\s+ True\n'
' arg-y: \\s+ 10\n'
' arg-z: \\s+ 40\n'
' foo: \\s+ True\n')
# check values after setting args in both command line and config file
ns = self.parse(args="file1.txt file2.txt --arg-x -y 3 --arg-z 100 ",
config_file_contents="""arg-y = 31.5
arg-z = 30
foo = True
""")
self.format_help()
self.format_values()
self.assertListEqual(ns.filenames, ["file1.txt", "file2.txt"])
self.assertEqual(ns.arg_x, True)
self.assertEqual(ns.y1, 3)
self.assertEqual(ns.arg_z, [100])
self.assertEqual(ns.foo, True)
self.assertRegex(self.format_values(),
"Command Line Args: file1.txt file2.txt --arg-x -y 3 --arg-z 100")
def testBasicCase2(self, use_groups=False):
## Test command line, config file and env var values
default_config_file = tempfile.NamedTemporaryFile(mode="w", delete=True)
default_config_file.flush()
p = self.initParser(default_config_files=['/etc/settings.ini',
'/home/jeff/.user_settings', default_config_file.name])
p.add_arg('vcf', nargs='+', help='Variant file(s)')
if not use_groups:
self.add_arg('--genome', help='Path to genome file', required=True)
self.add_arg('-v', dest='verbose', action='store_true')
self.add_arg('-g', '--my-cfg-file', required=True,
is_config_file=True)
self.add_arg('-d', '--dbsnp', env_var='DBSNP_PATH')
self.add_arg('-f', '--format',
choices=["BED", "MAF", "VCF", "WIG", "R"],
dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT",
default="BED")
else:
g = p.add_argument_group(title="g1")
g.add_arg('--genome', help='Path to genome file', required=True)
g.add_arg('-v', dest='verbose', action='store_true')
g.add_arg('-g', '--my-cfg-file', required=True,
is_config_file=True)
g = p.add_argument_group(title="g2")
g.add_arg('-d', '--dbsnp', env_var='DBSNP_PATH')
g.add_arg('-f', '--format',
choices=["BED", "MAF", "VCF", "WIG", "R"],
dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT",
default="BED")
# make sure required args are enforced
self.assertParseArgsRaises("too few arg"
if sys.version_info.major < 3 else
"the following arguments are required: vcf, -g/--my-cfg-file",
args="--genome hg19")
self.assertParseArgsRaises("Unable to open config file: file.txt. Error: No such file or director", args="-g file.txt")
# check values after setting args on command line
config_file2 = tempfile.NamedTemporaryFile(mode="w", delete=True)
config_file2.flush()
ns = self.parse(args="--genome hg19 -g %s bla.vcf " % config_file2.name)
self.assertEqual(ns.genome, "hg19")
self.assertEqual(ns.verbose, False)
self.assertIsNone(ns.dbsnp)
self.assertEqual(ns.fmt, "BED")
self.assertListEqual(ns.vcf, ["bla.vcf"])
self.assertRegex(self.format_values(),
'Command Line Args: --genome hg19 -g [^\\s]+ bla.vcf\n'
'Defaults:\n'
' --format: \\s+ BED\n')
# check precedence: args > env > config > default using the --format arg
default_config_file.write("--format MAF")
default_config_file.flush()
ns = self.parse(args="--genome hg19 -g %s f.vcf " % config_file2.name)
self.assertEqual(ns.fmt, "MAF")
self.assertRegex(self.format_values(),
'Command Line Args: --genome hg19 -g [^\\s]+ f.vcf\n'
'Config File \\([^\\s]+\\):\n'
' --format: \\s+ MAF\n')
config_file2.write("--format VCF")
config_file2.flush()
ns = self.parse(args="--genome hg19 -g %s f.vcf " % config_file2.name)
self.assertEqual(ns.fmt, "VCF")
self.assertRegex(self.format_values(),
'Command Line Args: --genome hg19 -g [^\\s]+ f.vcf\n'
'Config File \\([^\\s]+\\):\n'
' --format: \\s+ VCF\n')
ns = self.parse(env_vars={"OUTPUT_FORMAT":"R", "DBSNP_PATH":"/a/b.vcf"},
args="--genome hg19 -g %s f.vcf " % config_file2.name)
self.assertEqual(ns.fmt, "R")
self.assertEqual(ns.dbsnp, "/a/b.vcf")
self.assertRegex(self.format_values(),
'Command Line Args: --genome hg19 -g [^\\s]+ f.vcf\n'
'Environment Variables:\n'
' DBSNP_PATH: \\s+ /a/b.vcf\n'
' OUTPUT_FORMAT: \\s+ R\n')
ns = self.parse(env_vars={"OUTPUT_FORMAT":"R", "DBSNP_PATH":"/a/b.vcf",
"ANOTHER_VAR":"something"},
args="--genome hg19 -g %s --format WIG f.vcf" % config_file2.name)
self.assertEqual(ns.fmt, "WIG")
self.assertEqual(ns.dbsnp, "/a/b.vcf")
self.assertRegex(self.format_values(),
'Command Line Args: --genome hg19 -g [^\\s]+ --format WIG f.vcf\n'
'Environment Variables:\n'
' DBSNP_PATH: \\s+ /a/b.vcf\n')
if not use_groups:
self.assertRegex(self.format_help(),
'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE\n?'
'\\s+\\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n'
'positional arguments:\n'
' vcf \\s+ Variant file\\(s\\)\n\n'
'%s:\n'
' -h, --help \\s+ show this help message and exit\n'
' --genome GENOME \\s+ Path to genome file\n'
' -v\n'
' -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n'
' -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n'
' -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n\n'%OPTIONAL_ARGS_STRING +
7*r'(.+\s*)')
else:
self.assertRegex(self.format_help(),
'usage: .* \\[-h\\] --genome GENOME \\[-v\\] -g MY_CFG_FILE\n?'
'\\s+\\[-d DBSNP\\]\\s+\\[-f FRMT\\]\\s+vcf \\[vcf ...\\]\n\n'
'positional arguments:\n'
' vcf \\s+ Variant file\\(s\\)\n\n'
'%s:\n'
' -h, --help \\s+ show this help message and exit\n\n'
'g1:\n'
' --genome GENOME \\s+ Path to genome file\n'
' -v\n'
' -g MY_CFG_FILE, --my-cfg-file MY_CFG_FILE\n\n'
'g2:\n'
' -d DBSNP, --dbsnp DBSNP\\s+\\[env var: DBSNP_PATH\\]\n'
' -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n\n'%OPTIONAL_ARGS_STRING +
7*r'(.+\s*)')
self.assertParseArgsRaises("invalid choice: 'ZZZ'",
args="--genome hg19 -g %s --format ZZZ f.vcf" % config_file2.name)
self.assertParseArgsRaises("unrecognized arguments: --bla",
args="--bla --genome hg19 -g %s f.vcf" % config_file2.name)
default_config_file.close()
config_file2.close()
def testBasicCase2_WithGroups(self):
self.testBasicCase2(use_groups=True)
def testCustomOpenFunction(self):
expected_output = 'dummy open called'
def dummy_open(p):
print(expected_output)
return open(p)
self.initParser(config_file_open_func=dummy_open)
self.add_arg('--config', is_config_file=True)
self.add_arg('--arg1', default=1, type=int)
with tempfile.NamedTemporaryFile(mode='w', delete=False) as config_file:
config_file.write('arg1 2')
config_file_path = config_file.name
with captured_output() as (out, _):
args = self.parse('--config {}'.format(config_file_path))
self.assertTrue(hasattr(args, 'arg1'))
self.assertEqual(args.arg1, 2)
output = out.getvalue().strip()
self.assertEqual(output, expected_output)
def testIgnoreHelpArgs(self):
p = self.initParser()
self.add_arg('--arg1')
args, _ = self.parse_known('--arg2 --help', ignore_help_args=True)
self.assertEqual(args.arg1, None)
self.add_arg('--arg2')
args, _ = self.parse_known('--arg2 3 --help', ignore_help_args=True)
self.assertEqual(args.arg2, "3")
self.assertRaisesRegex(TypeError, "exit", self.parse_known, '--arg2 3 --help', ignore_help_args=False)
def testPositionalAndConfigVarLists(self):
self.initParser()
self.add_arg("a")
self.add_arg("-x", "--arg", nargs="+")
ns = self.parse("positional_value", config_file_contents="""arg = [Shell, someword, anotherword]""")
self.assertEqual(ns.arg, ['Shell', 'someword', 'anotherword'])
self.assertEqual(ns.a, "positional_value")
def testMutuallyExclusiveArgs(self):
config_file = tempfile.NamedTemporaryFile(mode="w", delete=True)
p = self.parser
g = p.add_argument_group(title="group1")
g.add_arg('--genome', help='Path to genome file', required=True)
g.add_arg('-v', dest='verbose', action='store_true')
g = p.add_mutually_exclusive_group(required=True)
g.add_arg('-f1', '--type1-cfg-file', is_config_file=True)
g.add_arg('-f2', '--type2-cfg-file', is_config_file=True)
g = p.add_mutually_exclusive_group(required=True)
g.add_arg('-f', '--format', choices=["BED", "MAF", "VCF", "WIG", "R"],
dest="fmt", metavar="FRMT", env_var="OUTPUT_FORMAT",
default="BED")
g.add_arg('-b', '--bam', dest='fmt', action="store_const", const="BAM",
env_var='BAM_FORMAT')
ns = self.parse(args="--genome hg19 -f1 %s --bam" % config_file.name)
self.assertEqual(ns.genome, "hg19")
self.assertEqual(ns.verbose, False)
self.assertEqual(ns.fmt, "BAM")
ns = self.parse(env_vars={"BAM_FORMAT" : "true"},
args="--genome hg19 -f1 %s" % config_file.name)
self.assertEqual(ns.genome, "hg19")
self.assertEqual(ns.verbose, False)
self.assertEqual(ns.fmt, "BAM")
self.assertRegex(self.format_values(),
'Command Line Args: --genome hg19 -f1 [^\\s]+\n'
'Environment Variables:\n'
' BAM_FORMAT: \\s+ true\n'
'Defaults:\n'
' --format: \\s+ BED\n')
self.assertRegex(self.format_help(),
r'usage: .* \[-h\] --genome GENOME \[-v\]\s+ \(-f1 TYPE1_CFG_FILE \|'
' \\s*-f2 TYPE2_CFG_FILE\\)\\s+\\(-f FRMT \\| -b\\)\n\n'
'%s:\n'
' -h, --help show this help message and exit\n'
' -f1 TYPE1_CFG_FILE, --type1-cfg-file TYPE1_CFG_FILE\n'
' -f2 TYPE2_CFG_FILE, --type2-cfg-file TYPE2_CFG_FILE\n'
' -f FRMT, --format FRMT\\s+\\[env var: OUTPUT_FORMAT\\]\n'
' -b, --bam\\s+\\[env var: BAM_FORMAT\\]\n\n'
'group1:\n'
' --genome GENOME Path to genome file\n'
' -v\n\n'%OPTIONAL_ARGS_STRING +
5*r'(.+\s*)')
config_file.close()
def testSubParsers(self):
config_file1 = tempfile.NamedTemporaryFile(mode="w", delete=True)
config_file1.write("--i = B")
config_file1.flush()
config_file2 = tempfile.NamedTemporaryFile(mode="w", delete=True)
config_file2.write("p = 10")
config_file2.flush()
parser = configargparse.ArgumentParser(prog="myProg")
subparsers = parser.add_subparsers(title="actions")
parent_parser = configargparse.ArgumentParser(add_help=False)
parent_parser.add_argument("-p", "--p", type=int, required=True,
help="set db parameter")
create_p = subparsers.add_parser("create", parents=[parent_parser],
help="create the orbix environment")
create_p.add_argument("--i", env_var="INIT", choices=["A","B"],
default="A")
create_p.add_argument("-config", is_config_file=True)
update_p = subparsers.add_parser("update", parents=[parent_parser],
help="update the orbix environment")
update_p.add_argument("-config2", is_config_file=True, required=True)
ns = parser.parse_args(args = "create -p 2 -config "+config_file1.name)
self.assertEqual(ns.p, 2)
self.assertEqual(ns.i, "B")
ns = parser.parse_args(args = "update -config2 " + config_file2.name)
self.assertEqual(ns.p, 10)
config_file1.close()
config_file2.close()
def testAddArgsErrors(self):
self.assertRaisesRegex(ValueError, "arg with "
"is_write_out_config_file_arg=True can't also have "
"is_config_file_arg=True", self.add_arg, "-x", "--X",
is_config_file=True, is_write_out_config_file_arg=True)
self.assertRaisesRegex(ValueError, "arg with "
"is_write_out_config_file_arg=True must have action='store'",
self.add_arg, "-y", "--Y", action="append",
is_write_out_config_file_arg=True)
def testConfigFileSyntax(self):
self.add_arg('-x', required=True, type=int)
self.add_arg('--y', required=True, type=float)
self.add_arg('--z')
self.add_arg('--c')
self.add_arg('--b', action="store_true")
self.add_arg('--a', action="append", type=int)
self.add_arg('--m', action="append", nargs=3, metavar=("", "", ""),)
ns = self.parse(args="-x 1", env_vars={}, config_file_contents="""
#inline comment 1
# inline comment 2
# inline comment 3
;inline comment 4
; inline comment 5
;inline comment 6
--- # separator 1
------------- # separator 2
y=1.1
y = 2.1
y= 3.1 # with comment
y= 4.1 ; with comment
---
y:5.1
y : 6.1
y: 7.1 # with comment
y: 8.1 ; with comment
---
y \t 9.1
y 10.1
y 11.1 # with comment
y 12.1 ; with comment
---
b
b = True
b: True
----
a = 33
---
z z 1
---
m = [[1, 2, 3], [4, 5, 6]]
""")
self.assertEqual(ns.x, 1)
self.assertEqual(ns.y, 12.1)
self.assertEqual(ns.z, 'z 1')
self.assertIsNone(ns.c)
self.assertEqual(ns.b, True)
self.assertEqual(ns.a, [33])
self.assertRegex(self.format_values(),
'Command Line Args: \\s+ -x 1\n'
'Config File \\(method arg\\):\n'
' y: \\s+ 12.1\n'
' b: \\s+ True\n'
' a: \\s+ 33\n'
' z: \\s+ z 1\n')
self.assertEqual(ns.m, [['1', '2', '3'], ['4', '5', '6']])
# -x is not a long arg so can't be set via config file
self.assertParseArgsRaises("argument -x is required"
if sys.version_info.major < 3 else
"the following arguments are required: -x, --y",
args="",
config_file_contents="-x 3")
self.assertParseArgsRaises("invalid float value: 'abc'",
args="-x 5",
config_file_contents="y: abc")
self.assertParseArgsRaises("argument --y is required"
if sys.version_info.major < 3 else
"the following arguments are required: --y",
args="-x 5",
config_file_contents="z: 1")
# test unknown config file args
self.assertParseArgsRaises("bla",
args="-x 1 --y 2.3",
config_file_contents="bla=3")
ns, args = self.parse_known("-x 10 --y 3.8",
config_file_contents="bla=3",
env_vars={"bla": "2"})
self.assertListEqual(args, ["--bla=3"])
self.initParser(ignore_unknown_config_file_keys=False)
ns, args = self.parse_known(args="-x 1", config_file_contents="bla=3",
env_vars={"bla": "2"})
self.assertEqual(set(args), {"--bla=3", "-x", "1"})
def testQuotedArgumentValues(self):
self.initParser()
self.add_arg("-a")
self.add_arg("--b")
self.add_arg("-c")
self.add_arg("--d")
self.add_arg("-e")
self.add_arg("-q")
self.add_arg("--quotes")
# sys.argv equivalent of -a="1" --b "1" -c= --d "" -e=: -q "\"'" --quotes "\"'"
ns = self.parse(args=['-a=1', '--b', '1', '-c=', '--d', '', '-e=:',
'-q', '"\'', '--quotes', '"\''],
env_vars={}, config_file_contents="")
self.assertEqual(ns.a, "1")
self.assertEqual(ns.b, "1")
self.assertEqual(ns.c, "")
self.assertEqual(ns.d, "")
self.assertEqual(ns.e, ":")
self.assertEqual(ns.q, '"\'')
self.assertEqual(ns.quotes, '"\'')
def testQuotedConfigFileValues(self):
self.initParser()
self.add_arg("--a")
self.add_arg("--b")
self.add_arg("--c")
ns = self.parse(args="", env_vars={}, config_file_contents="""
a="1"
b=:
c=
""")
self.assertEqual(ns.a, "1")
self.assertEqual(ns.b, ":")
self.assertEqual(ns.c, "")
def testBooleanValuesCanBeExpressedAsNumbers(self):
self.initParser()
store_true_env_var_name = "STORE_TRUE"
self.add_arg("--boolean_store_true", action="store_true", env_var=store_true_env_var_name)
result_namespace = self.parse("", config_file_contents="""boolean_store_true = 1""")
self.assertTrue(result_namespace.boolean_store_true)
result_namespace = self.parse("", config_file_contents="""boolean_store_true = 0""")
self.assertFalse(result_namespace.boolean_store_true)
result_namespace = self.parse("", env_vars={store_true_env_var_name: "1"})
self.assertTrue(result_namespace.boolean_store_true)
result_namespace = self.parse("", env_vars={store_true_env_var_name: "0"})
self.assertFalse(result_namespace.boolean_store_true)
self.initParser()
store_false_env_var_name = "STORE_FALSE"
self.add_arg("--boolean_store_false", action="store_false", env_var=store_false_env_var_name)
result_namespace = self.parse("", config_file_contents="""boolean_store_false = 1""")
self.assertFalse(result_namespace.boolean_store_false)
result_namespace = self.parse("", config_file_contents="""boolean_store_false = 0""")
self.assertTrue(result_namespace.boolean_store_false)
result_namespace = self.parse("", env_vars={store_false_env_var_name: "1"})
self.assertFalse(result_namespace.boolean_store_false)
result_namespace = self.parse("", env_vars={store_false_env_var_name: "0"})
self.assertTrue(result_namespace.boolean_store_false)
def testConfigOrEnvValueErrors(self):
# error should occur when a flag arg is set to something other than "true" or "false"
self.initParser()
self.add_arg("--height", env_var = "HEIGHT", required=True)
self.add_arg("--do-it", dest="x", env_var = "FLAG1", action="store_true")
self.add_arg("--dont-do-it", dest="y", env_var = "FLAG2", action="store_false")
ns = self.parse("", env_vars = {"HEIGHT": "tall", "FLAG1": "yes"})
self.assertEqual(ns.height, "tall")
self.assertEqual(ns.x, True)
ns = self.parse("", env_vars = {"HEIGHT": "tall", "FLAG2": "yes"})
self.assertEqual(ns.y, False)
ns = self.parse("", env_vars = {"HEIGHT": "tall", "FLAG2": "no"})
self.assertEqual(ns.y, True)
# error should occur when flag arg is given a value
self.initParser()
self.add_arg("-v", "--verbose", env_var="VERBOSE", action="store_true")
self.assertParseArgsRaises("Unexpected value for VERBOSE: 'bla'. "
"Expecting 'true', 'false', 'yes', 'no', '1' or '0'",
args="",
env_vars={"VERBOSE" : "bla"})
ns = self.parse("",
config_file_contents="verbose=true",
env_vars={"HEIGHT": "true"})
self.assertEqual(ns.verbose, True)
ns = self.parse("",
config_file_contents="verbose",
env_vars={"HEIGHT": "true"})
self.assertEqual(ns.verbose, True)
ns = self.parse("", env_vars = {"HEIGHT": "true", "VERBOSE": "true"})
self.assertEqual(ns.verbose, True)
ns = self.parse("", env_vars = {"HEIGHT": "true", "VERBOSE": "false"})
self.assertEqual(ns.verbose, False)
ns = self.parse("", config_file_contents="--verbose",
env_vars = {"HEIGHT": "true"})
self.assertEqual(ns.verbose, True)
# error should occur is non-append arg is given a list value
self.initParser()
self.add_arg("-f", "--file", env_var="FILES", action="append", type=int)
ns = self.parse("", env_vars = {"file": "[1,2,3]", "VERBOSE": "true"})
self.assertIsNone(ns.file)
def testValuesStartingWithDash(self):
self.initParser()
self.add_arg("--arg0")
self.add_arg("--arg1", env_var="ARG1")
self.add_arg("--arg2")
self.add_arg("--arg3", action='append')
self.add_arg("--arg4", action='append', env_var="ARG4")
self.add_arg("--arg5", action='append')
self.add_arg("--arg6")
ns = self.parse(
"--arg0=-foo --arg3=-foo --arg3=-bar --arg6=-test-more-dashes",
config_file_contents="arg2: -foo\narg5: [-foo, -bar]",
env_vars={"ARG1": "-foo", "ARG4": "[-foo, -bar]"}
)
self.assertEqual(ns.arg0, "-foo")
self.assertEqual(ns.arg1, "-foo")
self.assertEqual(ns.arg2, "-foo")
self.assertEqual(ns.arg3, ["-foo", "-bar"])
self.assertEqual(ns.arg4, ["-foo", "-bar"])
self.assertEqual(ns.arg5, ["-foo", "-bar"])
self.assertEqual(ns.arg6, "-test-more-dashes")
def testPriorityKnown(self):
self.initParser()
self.add_arg("--arg", env_var="ARG")
ns = self.parse(
"--arg command_line_val",
config_file_contents="arg: config_val",
env_vars={"ARG": "env_val"}
)
self.assertEqual(ns.arg, "command_line_val")
ns = self.parse(
"--arg=command_line_val",
config_file_contents="arg: config_val",
env_vars={"ARG": "env_val"}
)
self.assertEqual(ns.arg, "command_line_val")
ns = self.parse(
"",
config_file_contents="arg: config_val",
env_vars={"ARG": "env_val"}
)
self.assertEqual(ns.arg, "env_val")
def testPriorityUnknown(self):
self.initParser()
ns, args = self.parse_known(
"--arg command_line_val",
config_file_contents="arg: config_val",
env_vars={"arg": "env_val"}
)
self.assertListEqual(args, ["--arg", "command_line_val"])
ns, args = self.parse_known(
"--arg=command_line_val",
config_file_contents="arg: config_val",
)
self.assertListEqual(args, ["--arg=command_line_val"])
def testAutoEnvVarPrefix(self):
self.initParser(auto_env_var_prefix="TEST_")
self.add_arg("-a", "--arg0", is_config_file_arg=True)
self.add_arg("-b", "--arg1", is_write_out_config_file_arg=True)
self.add_arg("-x", "--arg2", env_var="TEST2", type=int)
self.add_arg("-y", "--arg3", action="append", type=int)
self.add_arg("-z", "--arg4", required=True)
self.add_arg("-w", "--arg4-more", required=True)
ns = self.parse("", env_vars = {
"TEST_ARG0": "0",
"TEST_ARG1": "1",
"TEST_ARG2": "2",
"TEST2": "22",
"TEST_ARG4": "arg4_value",
"TEST_ARG4_MORE": "magic"})
self.assertIsNone(ns.arg0)
self.assertIsNone(ns.arg1)
self.assertEqual(ns.arg2, 22)
self.assertEqual(ns.arg4, "arg4_value")
self.assertEqual(ns.arg4_more, "magic")
def testEnvVarLists(self):
self.initParser()
self.add_arg("-x", "--arg2", env_var="TEST2")
self.add_arg("-y", "--arg3", env_var="TEST3", type=int)
self.add_arg("-z", "--arg4", env_var="TEST4", nargs="+")
self.add_arg("-u", "--arg5", env_var="TEST5", nargs="+", type=int)
self.add_arg("--arg6", env_var="TEST6")
self.add_arg("--arg7", env_var="TEST7", action="append")
ns = self.parse("", env_vars={"TEST2": "22",
"TEST3": "22",
"TEST4": "[Shell, someword, anotherword]",
"TEST5": "[22, 99, 33]",
"TEST6": "[value6.1, value6.2, value6.3]",
"TEST7": "[value7.1, value7.2, value7.3]",
})
self.assertEqual(ns.arg2, "22")
self.assertEqual(ns.arg3, 22)
self.assertEqual(ns.arg4, ['Shell', 'someword', 'anotherword'])
self.assertEqual(ns.arg5, [22, 99, 33])
self.assertEqual(ns.arg6, "[value6.1, value6.2, value6.3]")
self.assertEqual(ns.arg7, ["value7.1", "value7.2", "value7.3"])
def testPositionalAndEnvVarLists(self):
self.initParser()
self.add_arg("a")
self.add_arg("-x", "--arg", env_var="TEST", nargs="+")
ns = self.parse("positional_value", env_vars={"TEST": "[Shell, someword, anotherword]"})
self.assertEqual(ns.arg, ['Shell', 'someword', 'anotherword'])
self.assertEqual(ns.a, "positional_value")
def testCounterCommandLine(self):
self.initParser()
self.add_arg("--verbose", "-v", action="count", default=0)
ns = self.parse(args="-v -v -v", env_vars={})
self.assertEqual(ns.verbose, 3)
ns = self.parse(args="-vvv", env_vars={})
self.assertEqual(ns.verbose, 3)
def testCounterConfigFile(self):
self.initParser()
self.add_arg("--verbose", "-v", action="count", default=0)
ns = self.parse(args="", env_vars={}, config_file_contents="""
verbose""")
self.assertEqual(ns.verbose, 1)
ns = self.parse(args="", env_vars={}, config_file_contents="""
verbose=3""")
self.assertEqual(ns.verbose, 3)
class TestMisc(TestCase):
# TODO test different action types with config file, env var
"""Test edge cases"""
def setUp(self):
self.initParser(args_for_setting_config_path=[])
@mock.patch('argparse.ArgumentParser.__init__')
def testKwrgsArePassedToArgParse(self, argparse_init):
kwargs_for_argparse = {"allow_abbrev": False, "whatever_other_arg": "something"}
parser = configargparse.ArgumentParser(add_config_file_help=False, **kwargs_for_argparse)
argparse_init.assert_called_with(parser, **kwargs_for_argparse)
def testGlobalInstances(self, name=None):
p = configargparse.getArgumentParser(name, prog="prog", usage="test")
self.assertEqual(p.usage, "test")
self.assertEqual(p.prog, "prog")
self.assertRaisesRegex(ValueError, "kwargs besides 'name' can only be "
"passed in the first time", configargparse.getArgumentParser, name,
prog="prog")
p2 = configargparse.getArgumentParser(name)
self.assertEqual(p, p2)
def testGlobalInstances_WithName(self):
self.testGlobalInstances("name1")
self.testGlobalInstances("name2")
def testAddArguments_ArgValidation(self):
self.assertRaises(ValueError, self.add_arg, 'positional', env_var="bla")
action = self.add_arg('positional')
self.assertIsNotNone(action)
self.assertEqual(action.dest, "positional")
def testAddArguments_IsConfigFilePathArg(self):
self.assertRaises(ValueError, self.add_arg, 'c', action="store_false",
is_config_file=True)
self.add_arg("-c", "--config", is_config_file=True)
self.add_arg("--x", required=True)
# verify parsing from config file
config_file = tempfile.NamedTemporaryFile(mode="w", delete=True)
config_file.write("x=bla")
config_file.flush()
ns = self.parse(args="-c %s" % config_file.name)
self.assertEqual(ns.x, "bla")
def testConstructor_ConfigFileArgs(self):
# Test constructor args:
# args_for_setting_config_path
# config_arg_is_required
# config_arg_help_message
temp_cfg = tempfile.NamedTemporaryFile(mode="w", delete=True)
temp_cfg.write("genome=hg19")
temp_cfg.flush()
self.initParser(args_for_setting_config_path=["-c", "--config"],
config_arg_is_required = True,
config_arg_help_message = "my config file",
default_config_files=[temp_cfg.name])
self.add_arg('--genome', help='Path to genome file', required=True)
self.assertParseArgsRaises("argument -c/--config is required"
if sys.version_info.major < 3 else
"arguments are required: -c/--config",
args="")
temp_cfg2 = tempfile.NamedTemporaryFile(mode="w", delete=True)
ns = self.parse("-c " + temp_cfg2.name)
self.assertEqual(ns.genome, "hg19")
# temp_cfg2 config file should override default config file values
temp_cfg2.write("genome=hg20")
temp_cfg2.flush()
ns = self.parse("-c " + temp_cfg2.name)
self.assertEqual(ns.genome, "hg20")
self.assertRegex(self.format_help(),
'usage: .* \\[-h\\] -c CONFIG_FILE --genome GENOME\n\n'
'%s:\n'
' -h, --help\\s+ show this help message and exit\n'
' -c CONFIG_FILE, --config CONFIG_FILE\\s+ my config file\n'
' --genome GENOME\\s+ Path to genome file\n\n'%OPTIONAL_ARGS_STRING +
5*r'(.+\s*)')
# just run print_values() to make sure it completes and returns None
output = StringIO()
self.assertIsNone(self.parser.print_values(file=output))
self.assertIn("Command Line Args:", output.getvalue())
# test ignore_unknown_config_file_keys=False
self.initParser(ignore_unknown_config_file_keys=False)
self.assertRaisesRegex(argparse.ArgumentError, "unrecognized arguments",
self.parse, config_file_contents="arg1 = 3")
ns, args = self.parse_known(config_file_contents="arg1 = 3")
self.assertEqual(getattr(ns, "arg1", ""), "")
# test ignore_unknown_config_file_keys=True
self.initParser(ignore_unknown_config_file_keys=True)
ns = self.parse(args="", config_file_contents="arg1 = 3")
self.assertEqual(getattr(ns, "arg1", ""), "")
ns, args = self.parse_known(config_file_contents="arg1 = 3")
self.assertEqual(getattr(ns, "arg1", ""), "")
def test_AbbrevConfigFileArgs(self):
"""Tests that abbreviated values don't get pulled from config file.
"""
temp_cfg = tempfile.NamedTemporaryFile(mode="w", delete=True)
temp_cfg.write("a2a = 0.5\n")
temp_cfg.write("a3a = 0.5\n")
temp_cfg.flush()
self.initParser()
self.add_arg('-c', '--config_file', required=False, is_config_file=True,
help='config file path')
self.add_arg('--hello', type=int, required=False)
command = '-c {} --hello 2'.format(temp_cfg.name)
known, unknown = self.parse_known(command)
self.assertListEqual(unknown, ['--a2a=0.5', '--a3a=0.5'])
def test_FormatHelp(self):
self.initParser(args_for_setting_config_path=["-c", "--config"],
config_arg_is_required = True,
config_arg_help_message = "my config file",
default_config_files=["~/.myconfig"],
args_for_writing_out_config_file=["-w", "--write-config"],
)
self.add_arg('--arg1', help='Arg1 help text', required=True)
self.add_arg('--flag', help='Flag help text', action="store_true")
self.assertRegex(self.format_help(),
r'usage: .* \[-h\] -c CONFIG_FILE\s+'
r'\[-w CONFIG_OUTPUT_PATH\]\s* --arg1\s+ARG1\s*\[--flag\]\s*'
'%s:\\s*'
'-h, --help \\s* show this help message and exit\n\\s*'
r'-c CONFIG_FILE, --config CONFIG_FILE\s+my config file\s*'
r'-w CONFIG_OUTPUT_PATH, --write-config CONFIG_OUTPUT_PATH\s*takes\s*'
r'the current command line args and writes them\s*'
r'out to a config file at the given path, then exits\s*'
r'--arg1 ARG1\s*Arg1 help text\s*'
r'--flag \s*Flag help text\s*'
'Args that start with \'--\' \\(eg. --arg1\\) can also be set in a '
r'config file\s*\(~/.myconfig or specified via -c\).\s*'
r'Config file syntax allows: key=value,\s*flag=true, stuff=\[a,b,c\] '
r'\(for details, see syntax at https://goo.gl/R74nmi\).\s*'
r'If an arg is specified in more than\s*one place, then '
r'commandline values\s*override config file values which override\s*'
r'defaults.'%OPTIONAL_ARGS_STRING
)
def test_FormatHelpProg(self):
self.initParser('format_help_prog')
self.assertRegex(self.format_help(), 'usage: format_help_prog .*')
def test_FormatHelpProgLib(self):
parser = argparse.ArgumentParser('format_help_prog')
self.assertRegex(parser.format_help(), 'usage: format_help_prog .*')
class CustomClass(object):
def __init__(self, name):
self.name = name
def __str__(self):
return self.name
@staticmethod
def valid_custom(s):
if s == "invalid": raise Exception("invalid name")
return TestMisc.CustomClass(s)
def testConstructor_WriteOutConfigFileArgs(self):
# Test constructor args:
# args_for_writing_out_config_file
# write_out_config_file_arg_help_message
cfg_f = tempfile.NamedTemporaryFile(mode="w+", delete=True)
self.initParser(args_for_writing_out_config_file=["-w"],
write_out_config_file_arg_help_message="write config")
self.add_arg("-not-config-file-settable")
self.add_arg("--config-file-settable-arg", type=int)
self.add_arg("--config-file-settable-arg2", type=int, default=3)
self.add_arg("--config-file-settable-flag", action="store_true")
self.add_arg("--config-file-settable-custom", type=TestMisc.valid_custom)
self.add_arg("-l", "--config-file-settable-list", action="append")
# write out a config file
command_line_args = "-w %s " % cfg_f.name
command_line_args += "--config-file-settable-arg 1 "
command_line_args += "--config-file-settable-flag "
command_line_args += "--config-file-settable-custom custom_value "
command_line_args += "-l a -l b -l c -l d "
self.assertFalse(self.parser._exit_method_called)
ns = self.parse(command_line_args)
self.assertTrue(self.parser._exit_method_called)
cfg_f.seek(0)
expected_config_file_contents = "config-file-settable-arg = 1\n"
expected_config_file_contents += "config-file-settable-flag = true\n"
expected_config_file_contents += "config-file-settable-custom = custom_value\n"
expected_config_file_contents += "config-file-settable-list = [a, b, c, d]\n"
expected_config_file_contents += "config-file-settable-arg2 = 3\n"
self.assertEqual(cfg_f.read().strip(),
expected_config_file_contents.strip())
self.assertRaisesRegex(ValueError, "Couldn't open / for writing:",
self.parse, args = command_line_args + " -w /")
cfg_f.close()
def testConstructor_WriteOutConfigFileArgs2(self):
# Test constructor args:
# args_for_writing_out_config_file
# write_out_config_file_arg_help_message
cfg_f = tempfile.NamedTemporaryFile(mode="w+", delete=True)
self.initParser(args_for_writing_out_config_file=["-w"],
write_out_config_file_arg_help_message="write config")
self.add_arg("-not-config-file-settable")
self.add_arg("-a", "--arg1", type=int, env_var="ARG1")
self.add_arg("-b", "--arg2", type=int, default=3)
self.add_arg("-c", "--arg3")
self.add_arg("-d", "--arg4")
self.add_arg("-e", "--arg5")
self.add_arg("--config-file-settable-flag", action="store_true",
env_var="FLAG_ARG")
self.add_arg("-l", "--config-file-settable-list", action="append")
# write out a config file
command_line_args = "-w %s " % cfg_f.name
command_line_args += "-l a -l b -l c -l d "
self.assertFalse(self.parser._exit_method_called)
ns = self.parse(command_line_args,
env_vars={"ARG1": "10", "FLAG_ARG": "true",
"SOME_OTHER_ENV_VAR": "2"},
config_file_contents="arg3 = bla3\narg4 = bla4")
self.assertTrue(self.parser._exit_method_called)
cfg_f.seek(0)
expected_config_file_contents = "config-file-settable-list = [a, b, c, d]\n"
expected_config_file_contents += "arg1 = 10\n"
expected_config_file_contents += "config-file-settable-flag = True\n"
expected_config_file_contents += "arg3 = bla3\n"
expected_config_file_contents += "arg4 = bla4\n"
expected_config_file_contents += "arg2 = 3\n"
self.assertEqual(cfg_f.read().strip(),
expected_config_file_contents.strip())
self.assertRaisesRegex(ValueError, "Couldn't open / for writing:",
self.parse, args = command_line_args + " -w /")
cfg_f.close()
def testConstructor_WriteOutConfigFileArgsLong(self):
"""Test config writing with long version of arg
There was a bug where the long version of the
args_for_writing_out_config_file was being dumped into the resultant
output config file
"""
# Test constructor args:
# args_for_writing_out_config_file
# write_out_config_file_arg_help_message
cfg_f = tempfile.NamedTemporaryFile(mode="w+", delete=True)
self.initParser(args_for_writing_out_config_file=["--write-config"],
write_out_config_file_arg_help_message="write config")
self.add_arg("-not-config-file-settable")
self.add_arg("--config-file-settable-arg", type=int)
self.add_arg("--config-file-settable-arg2", type=int, default=3)
self.add_arg("--config-file-settable-flag", action="store_true")
self.add_arg("-l", "--config-file-settable-list", action="append")
# write out a config file
command_line_args = "--write-config %s " % cfg_f.name
command_line_args += "--config-file-settable-arg 1 "
command_line_args += "--config-file-settable-flag "
command_line_args += "-l a -l b -l c -l d "
self.assertFalse(self.parser._exit_method_called)
ns = self.parse(command_line_args)
self.assertTrue(self.parser._exit_method_called)
cfg_f.seek(0)
expected_config_file_contents = "config-file-settable-arg = 1\n"
expected_config_file_contents += "config-file-settable-flag = true\n"
expected_config_file_contents += "config-file-settable-list = [a, b, c, d]\n"
expected_config_file_contents += "config-file-settable-arg2 = 3\n"
self.assertEqual(cfg_f.read().strip(),
expected_config_file_contents.strip())
self.assertRaisesRegex(ValueError, "Couldn't open / for writing:",
self.parse, args = command_line_args + " --write-config /")
cfg_f.close()
def testMethodAliases(self):
p = self.parser
p.add("-a", "--arg-a", default=3)
p.add_arg("-b", "--arg-b", required=True)
p.add_argument("-c")
g1 = p.add_argument_group(title="group1", description="descr")
g1.add("-d", "--arg-d", required=True)
g1.add_arg("-e", "--arg-e", required=True)
g1.add_argument("-f", "--arg-f", default=5)
g2 = p.add_mutually_exclusive_group(required=True)
g2.add("-x", "--arg-x")
g2.add_arg("-y", "--arg-y")
g2.add_argument("-z", "--arg-z", default=5)
# verify that flags must be globally unique
g2 = p.add_argument_group(title="group2", description="descr")
self.assertRaises(argparse.ArgumentError, g1.add, "-c")
self.assertRaises(argparse.ArgumentError, g2.add, "-f")
self.initParser()
p = self.parser
options = p.parse(args=[])
self.assertDictEqual(vars(options), {})
def testConfigOpenFuncError(self):
# test OSError
def error_func(path):
raise OSError(9, "some error")
self.initParser(config_file_open_func=error_func)
self.parser.add_argument('-g', is_config_file=True)
self.assertParseArgsRaises("Unable to open config file: file.txt. Error: some error", args="-g file.txt")
# test other error
def error_func(path):
raise Exception('custom error')
self.initParser(config_file_open_func=error_func)
self.parser.add_argument('-g', is_config_file=True)
self.assertParseArgsRaises("Unable to open config file: file.txt. Error: custom error", args="-g file.txt")
class TestConfigFileParsers(TestCase):
"""Test ConfigFileParser subclasses in isolation"""
def testDefaultConfigFileParser_Basic(self):
p = configargparse.DefaultConfigFileParser()
self.assertGreater(len(p.get_syntax_description()), 0)
# test the simplest case
input_config_str = StringIO("""a: 3\n""")
parsed_obj = p.parse(input_config_str)
output_config_str = p.serialize(parsed_obj)
self.assertEqual(input_config_str.getvalue().replace(": ", " = "),
output_config_str)
self.assertDictEqual(parsed_obj, {'a': '3'})
def testDefaultConfigFileParser_All(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
"# comment1 ",
"[ some section ]",
"----",
"---------",
"_a: 3",
"; comment2 ",
"_b = c",
"_list_arg1 = [a, b, c]",
"_str_arg = true",
"_list_arg2 = [1, 2, 3]",
]
# test parse
input_config_str = StringIO("\n".join(config_lines)+"\n")
parsed_obj = p.parse(input_config_str)
# test serialize
output_config_str = p.serialize(parsed_obj)
self.assertEqual("\n".join(
l.replace(': ', ' = ') for l in config_lines if l.startswith('_'))+"\n",
output_config_str)
self.assertDictEqual(parsed_obj, {
'_a': '3',
'_b': 'c',
'_list_arg1': ['a', 'b', 'c'],
'_str_arg': 'true',
'_list_arg2': [1, 2, 3],
})
self.assertListEqual(parsed_obj['_list_arg1'], ['a', 'b', 'c'])
self.assertListEqual(parsed_obj['_list_arg2'], [1, 2, 3])
def testDefaultConfigFileParser_BasicValues(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
{'line': 'key = value # comment # comment', 'expected': ('key', 'value', 'comment # comment')},
{'line': 'key=value#comment ', 'expected': ('key', 'value#comment', None)},
{'line': 'key=value', 'expected': ('key', 'value', None)},
{'line': 'key =value', 'expected': ('key', 'value', None)},
{'line': 'key= value', 'expected': ('key', 'value', None)},
{'line': 'key = value', 'expected': ('key', 'value', None)},
{'line': 'key = value', 'expected': ('key', 'value', None)},
{'line': ' key = value ', 'expected': ('key', 'value', None)},
{'line': 'key:value', 'expected': ('key', 'value', None)},
{'line': 'key :value', 'expected': ('key', 'value', None)},
{'line': 'key: value', 'expected': ('key', 'value', None)},
{'line': 'key : value', 'expected': ('key', 'value', None)},
{'line': 'key : value', 'expected': ('key', 'value', None)},
{'line': ' key : value ', 'expected': ('key', 'value', None)},
{'line': 'key value', 'expected': ('key', 'value', None)},
{'line': 'key value', 'expected': ('key', 'value', None)},
{'line': ' key value ', 'expected': ('key', 'value', None)},
]
for test in config_lines:
parsed_obj = p.parse(StringIO(test['line']))
parsed_obj = dict(parsed_obj)
expected = {test['expected'][0]: test['expected'][1]}
self.assertDictEqual(parsed_obj, expected,
msg="Line %r" % (test['line']))
def testDefaultConfigFileParser_QuotedValues(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
{'line': 'key="value"', 'expected': ('key', 'value', None)},
{'line': 'key = "value"', 'expected': ('key', 'value', None)},
{'line': ' key = "value" ', 'expected': ('key', 'value', None)},
{'line': 'key=" value "', 'expected': ('key', ' value ', None)},
{'line': 'key = " value "', 'expected': ('key', ' value ', None)},
{'line': ' key = " value " ', 'expected': ('key', ' value ', None)},
{'line': "key='value'", 'expected': ('key', 'value', None)},
{'line': "key = 'value'", 'expected': ('key', 'value', None)},
{'line': " key = 'value' ", 'expected': ('key', 'value', None)},
{'line': "key=' value '", 'expected': ('key', ' value ', None)},
{'line': "key = ' value '", 'expected': ('key', ' value ', None)},
{'line': " key = ' value ' ", 'expected': ('key', ' value ', None)},
{'line': 'key="', 'expected': ('key', '"', None)},
{'line': 'key = "', 'expected': ('key', '"', None)},
{'line': ' key = " ', 'expected': ('key', '"', None)},
{'line': 'key = \'"value"\'', 'expected': ('key', '"value"', None)},
{'line': 'key = "\'value\'"', 'expected': ('key', "'value'", None)},
{'line': 'key = ""value""', 'expected': ('key', '"value"', None)},
{'line': 'key = \'\'value\'\'', 'expected': ('key', "'value'", None)},
{'line': 'key="value', 'expected': ('key', '"value', None)},
{'line': 'key = "value', 'expected': ('key', '"value', None)},
{'line': ' key = "value ', 'expected': ('key', '"value', None)},
{'line': 'key=value"', 'expected': ('key', 'value"', None)},
{'line': 'key = value"', 'expected': ('key', 'value"', None)},
{'line': ' key = value " ', 'expected': ('key', 'value "', None)},
{'line': "key='value", 'expected': ('key', "'value", None)},
{'line': "key = 'value", 'expected': ('key', "'value", None)},
{'line': " key = 'value ", 'expected': ('key', "'value", None)},
{'line': "key=value'", 'expected': ('key', "value'", None)},
{'line': "key = value'", 'expected': ('key', "value'", None)},
{'line': " key = value ' ", 'expected': ('key', "value '", None)},
]
for test in config_lines:
parsed_obj = p.parse(StringIO(test['line']))
parsed_obj = dict(parsed_obj)
expected = {test['expected'][0]: test['expected'][1]}
self.assertDictEqual(parsed_obj, expected,
msg="Line %r" % (test['line']))
def testDefaultConfigFileParser_BlankValues(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
{'line': 'key=', 'expected': ('key', '', None)},
{'line': 'key =', 'expected': ('key', '', None)},
{'line': 'key= ', 'expected': ('key', '', None)},
{'line': 'key = ', 'expected': ('key', '', None)},
{'line': 'key = ', 'expected': ('key', '', None)},
{'line': ' key = ', 'expected': ('key', '', None)},
{'line': 'key:', 'expected': ('key', '', None)},
{'line': 'key :', 'expected': ('key', '', None)},
{'line': 'key: ', 'expected': ('key', '', None)},
{'line': 'key : ', 'expected': ('key', '', None)},
{'line': 'key : ', 'expected': ('key', '', None)},
{'line': ' key : ', 'expected': ('key', '', None)},
]
for test in config_lines:
parsed_obj = p.parse(StringIO(test['line']))
parsed_obj = dict(parsed_obj)
expected = {test['expected'][0]: test['expected'][1]}
self.assertDictEqual(parsed_obj, expected,
msg="Line %r" % (test['line']))
def testDefaultConfigFileParser_UnspecifiedValues(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
{'line': 'key ', 'expected': ('key', 'true', None)},
{'line': 'key', 'expected': ('key', 'true', None)},
{'line': 'key ', 'expected': ('key', 'true', None)},
{'line': ' key ', 'expected': ('key', 'true', None)},
]
for test in config_lines:
parsed_obj = p.parse(StringIO(test['line']))
parsed_obj = dict(parsed_obj)
expected = {test['expected'][0]: test['expected'][1]}
self.assertDictEqual(parsed_obj, expected,
msg="Line %r" % (test['line']))
def testDefaultConfigFileParser_ColonEqualSignValue(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
{'line': 'key=:', 'expected': ('key', ':', None)},
{'line': 'key =:', 'expected': ('key', ':', None)},
{'line': 'key= :', 'expected': ('key', ':', None)},
{'line': 'key = :', 'expected': ('key', ':', None)},
{'line': 'key = :', 'expected': ('key', ':', None)},
{'line': ' key = : ', 'expected': ('key', ':', None)},
{'line': 'key:=', 'expected': ('key', '=', None)},
{'line': 'key :=', 'expected': ('key', '=', None)},
{'line': 'key: =', 'expected': ('key', '=', None)},
{'line': 'key : =', 'expected': ('key', '=', None)},
{'line': 'key : =', 'expected': ('key', '=', None)},
{'line': ' key : = ', 'expected': ('key', '=', None)},
{'line': 'key==', 'expected': ('key', '=', None)},
{'line': 'key ==', 'expected': ('key', '=', None)},
{'line': 'key= =', 'expected': ('key', '=', None)},
{'line': 'key = =', 'expected': ('key', '=', None)},
{'line': 'key = =', 'expected': ('key', '=', None)},
{'line': ' key = = ', 'expected': ('key', '=', None)},
{'line': 'key::', 'expected': ('key', ':', None)},
{'line': 'key ::', 'expected': ('key', ':', None)},
{'line': 'key: :', 'expected': ('key', ':', None)},
{'line': 'key : :', 'expected': ('key', ':', None)},
{'line': 'key : :', 'expected': ('key', ':', None)},
{'line': ' key : : ', 'expected': ('key', ':', None)},
]
for test in config_lines:
parsed_obj = p.parse(StringIO(test['line']))
parsed_obj = dict(parsed_obj)
expected = {test['expected'][0]: test['expected'][1]}
self.assertDictEqual(parsed_obj, expected,
msg="Line %r" % (test['line']))
def testDefaultConfigFileParser_ValuesWithComments(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
{'line': 'key=value#comment ', 'expected': ('key', 'value#comment', None)},
{'line': 'key=value #comment', 'expected': ('key', 'value', 'comment')},
{'line': ' key = value # comment', 'expected': ('key', 'value', 'comment')},
{'line': 'key:value#comment', 'expected': ('key', 'value#comment', None)},
{'line': 'key:value #comment', 'expected': ('key', 'value', 'comment')},
{'line': ' key : value # comment', 'expected': ('key', 'value', 'comment')},
{'line': 'key=value;comment ', 'expected': ('key', 'value;comment', None)},
{'line': 'key=value ;comment', 'expected': ('key', 'value', 'comment')},
{'line': ' key = value ; comment', 'expected': ('key', 'value', 'comment')},
{'line': 'key:value;comment', 'expected': ('key', 'value;comment', None)},
{'line': 'key:value ;comment', 'expected': ('key', 'value', 'comment')},
{'line': ' key : value ; comment', 'expected': ('key', 'value', 'comment')},
{'line': 'key = value # comment # comment', 'expected': ('key', 'value', 'comment # comment')},
{'line': 'key = "value # comment" # comment', 'expected': ('key', 'value # comment', 'comment')},
{'line': 'key = "#" ; comment', 'expected': ('key', '#', 'comment')},
{'line': 'key = ";" # comment', 'expected': ('key', ';', 'comment')},
]
for test in config_lines:
parsed_obj = p.parse(StringIO(test['line']))
parsed_obj = dict(parsed_obj)
expected = {test['expected'][0]: test['expected'][1]}
self.assertDictEqual(parsed_obj, expected,
msg="Line %r" % (test['line']))
def testDefaultConfigFileParser_NegativeValues(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
{'line': 'key = -10', 'expected': ('key', '-10', None)},
{'line': 'key : -10', 'expected': ('key', '-10', None)},
{'line': 'key -10', 'expected': ('key', '-10', None)},
{'line': 'key = "-10"', 'expected': ('key', '-10', None)},
{'line': "key = '-10'", 'expected': ('key', '-10', None)},
{'line': 'key=-10', 'expected': ('key', '-10', None)},
]
for test in config_lines:
parsed_obj = p.parse(StringIO(test['line']))
parsed_obj = dict(parsed_obj)
expected = {test['expected'][0]: test['expected'][1]}
self.assertDictEqual(parsed_obj, expected,
msg="Line %r" % (test['line']))
def testDefaultConfigFileParser_KeySyntax(self):
p = configargparse.DefaultConfigFileParser()
# test the all syntax case
config_lines = [
{'line': 'key_underscore = value', 'expected': ('key_underscore', 'value', None)},
{'line': 'key_underscore=', 'expected': ('key_underscore', '', None)},
{'line': 'key_underscore', 'expected': ('key_underscore', 'true', None)},
{'line': '_key_underscore = value', 'expected': ('_key_underscore', 'value', None)},
{'line': '_key_underscore=', 'expected': ('_key_underscore', '', None)},
{'line': '_key_underscore', 'expected': ('_key_underscore', 'true', None)},
{'line': 'key_underscore_ = value', 'expected': ('key_underscore_', 'value', None)},
{'line': 'key_underscore_=', 'expected': ('key_underscore_', '', None)},
{'line': 'key_underscore_', 'expected': ('key_underscore_', 'true', None)},
{'line': 'key-dash = value', 'expected': ('key-dash', 'value', None)},
{'line': 'key-dash=', 'expected': ('key-dash', '', None)},
{'line': 'key-dash', 'expected': ('key-dash', 'true', None)},
{'line': 'key@word = value', 'expected': ('key@word', 'value', None)},
{'line': 'key@word=', 'expected': ('key@word', '', None)},
{'line': 'key@word', 'expected': ('key@word', 'true', None)},
{'line': 'key$word = value', 'expected': ('key$word', 'value', None)},
{'line': 'key$word=', 'expected': ('key$word', '', None)},
{'line': 'key$word', 'expected': ('key$word', 'true', None)},
{'line': 'key.word = value', 'expected': ('key.word', 'value', None)},
{'line': 'key.word=', 'expected': ('key.word', '', None)},
{'line': 'key.word', 'expected': ('key.word', 'true', None)},
]
for test in config_lines:
parsed_obj = p.parse(StringIO(test['line']))
parsed_obj = dict(parsed_obj)
expected = {test['expected'][0]: test['expected'][1]}
self.assertDictEqual(parsed_obj, expected,
msg="Line %r" % (test['line']))
def testYAMLConfigFileParser_Basic(self):
try:
import yaml
except:
logging.warning("WARNING: PyYAML not installed. "
"Couldn't test YAMLConfigFileParser")
return
p = configargparse.YAMLConfigFileParser()
self.assertGreater(len(p.get_syntax_description()), 0)
input_config_str = StringIO("""a: '3'\n""")
parsed_obj = p.parse(input_config_str)
output_config_str = p.serialize(dict(parsed_obj))
self.assertEqual(input_config_str.getvalue(), output_config_str)
self.assertDictEqual(parsed_obj, {'a': '3'})
def testYAMLConfigFileParser_All(self):
try:
import yaml
except:
logging.warning("WARNING: PyYAML not installed. "
"Couldn't test YAMLConfigFileParser")
return
p = configargparse.YAMLConfigFileParser()
# test the all syntax case
config_lines = [
"a: '3'",
"list_arg:",
"- 1",
"- 2",
"- 3",
]
# test parse
input_config_str = StringIO("\n".join(config_lines)+"\n")
parsed_obj = p.parse(input_config_str)
# test serialize
output_config_str = p.serialize(parsed_obj)
self.assertEqual(input_config_str.getvalue(), output_config_str)
self.assertDictEqual(parsed_obj, {'a': '3', 'list_arg': [1,2,3]})
################################################################################
# since configargparse should work as a drop-in replacement for argparse
# in all situations, run argparse unittests on configargparse by modifying
# their source code to use configargparse.ArgumentParser
try:
import test.test_argparse
#Sig = test.test_argparse.Sig
#NS = test.test_argparse.NS
except ImportError:
logging.error("\n\n"
"============================\n"
"ERROR: Many tests couldn't be run because 'import test.test_argparse' "
"failed. Try building/installing python from source rather than through"
" a package manager.\n"
"============================\n")
else:
test_argparse_source_code = inspect.getsource(test.test_argparse)
test_argparse_source_code = test_argparse_source_code.replace(
'argparse.ArgumentParser', 'configargparse.ArgumentParser').replace(
'TestHelpFormattingMetaclass', '_TestHelpFormattingMetaclass').replace(
'test_main', '_test_main')
# pytest tries to collect tests from TestHelpFormattingMetaclass, and
# test_main, and raises a warning when it finds it's not a test class
# nor test function. Renaming TestHelpFormattingMetaclass and test_main
# prevents pytest from trying.
# run or debug a subset of the argparse tests
#test_argparse_source_code = test_argparse_source_code.replace(
# "(TestCase)", "").replace(
# "(ParserTestCase)", "").replace(
# "(HelpTestCase)", "").replace(
# ", TestCase", "").replace(
# ", ParserTestCase", "")
#test_argparse_source_code = test_argparse_source_code.replace(
# "class TestMessageContentError", "class TestMessageContentError(TestCase)")
exec(test_argparse_source_code)
# print argparse unittest source code
def print_source_code(source_code, line_numbers, context_lines=10):
for n in line_numbers:
logging.debug("##### Code around line %s #####" % n)
lines_to_print = set(range(n - context_lines, n + context_lines))
for n2, line in enumerate(source_code.split("\n"), 1):
if n2 in lines_to_print:
logging.debug("%s %5d: %s" % (
"**" if n2 == n else " ", n2, line))
#print_source_code(test_argparse_source_code, [4540, 4565])
ConfigArgParse-1.5.3/MANIFEST.in 0000644 0000000 0000024 00000000063 13235644610 016177 0 ustar root staff 0000000 0000000 include LICENSE
include README.rst
include tests/*
ConfigArgParse-1.5.3/configargparse.py 0000644 0000000 0000024 00000141323 14126173075 020014 0 ustar root staff 0000000 0000000 """
A drop-in replacement for `argparse` that allows options to also be set via config files and/or environment variables.
:see: `configargparse.ArgumentParser`, `configargparse.add_argument`
"""
import argparse
import json
import glob
import os
import re
import sys
import types
from collections import OrderedDict
import textwrap
if sys.version_info >= (3, 0):
from io import StringIO
else:
from StringIO import StringIO
ACTION_TYPES_THAT_DONT_NEED_A_VALUE = [argparse._StoreTrueAction,
argparse._StoreFalseAction, argparse._CountAction,
argparse._StoreConstAction, argparse._AppendConstAction]
if sys.version_info >= (3, 9):
ACTION_TYPES_THAT_DONT_NEED_A_VALUE.append(argparse.BooleanOptionalAction)
is_boolean_optional_action = lambda action: isinstance(action, argparse.BooleanOptionalAction)
else:
is_boolean_optional_action = lambda action: False
ACTION_TYPES_THAT_DONT_NEED_A_VALUE = tuple(ACTION_TYPES_THAT_DONT_NEED_A_VALUE)
# global ArgumentParser instances
_parsers = {}
def init_argument_parser(name=None, **kwargs):
"""Creates a global ArgumentParser instance with the given name,
passing any args other than "name" to the ArgumentParser constructor.
This instance can then be retrieved using get_argument_parser(..)
"""
if name is None:
name = "default"
if name in _parsers:
raise ValueError(("kwargs besides 'name' can only be passed in the"
" first time. '%s' ArgumentParser already exists: %s") % (
name, _parsers[name]))
kwargs.setdefault('formatter_class', argparse.ArgumentDefaultsHelpFormatter)
kwargs.setdefault('conflict_handler', 'resolve')
_parsers[name] = ArgumentParser(**kwargs)
def get_argument_parser(name=None, **kwargs):
"""Returns the global ArgumentParser instance with the given name. The 1st
time this function is called, a new ArgumentParser instance will be created
for the given name, and any args other than "name" will be passed on to the
ArgumentParser constructor.
"""
if name is None:
name = "default"
if len(kwargs) > 0 or name not in _parsers:
init_argument_parser(name, **kwargs)
return _parsers[name]
class ArgumentDefaultsRawHelpFormatter(
argparse.ArgumentDefaultsHelpFormatter,
argparse.RawTextHelpFormatter,
argparse.RawDescriptionHelpFormatter):
"""HelpFormatter that adds default values AND doesn't do line-wrapping"""
pass
class ConfigFileParser(object):
"""This abstract class can be extended to add support for new config file
formats"""
def get_syntax_description(self):
"""Returns a string describing the config file syntax."""
raise NotImplementedError("get_syntax_description(..) not implemented")
def parse(self, stream):
"""Parses the keys and values from a config file.
NOTE: For keys that were specified to configargparse as
action="store_true" or "store_false", the config file value must be
one of: "yes", "no", "true", "false". Otherwise an error will be raised.
Args:
stream (IO): A config file input stream (such as an open file object).
Returns:
OrderedDict: Items where the keys are strings and the
values are either strings or lists (eg. to support config file
formats like YAML which allow lists).
"""
raise NotImplementedError("parse(..) not implemented")
def serialize(self, items):
"""Does the inverse of config parsing by taking parsed values and
converting them back to a string representing config file contents.
Args:
items: an OrderedDict of items to be converted to the config file
format. Keys should be strings, and values should be either strings
or lists.
Returns:
Contents of config file as a string
"""
raise NotImplementedError("serialize(..) not implemented")
class ConfigFileParserException(Exception):
"""Raised when config file parsing failed."""
class DefaultConfigFileParser(ConfigFileParser):
"""
Based on a simplified subset of INI and YAML formats. Here is the
supported syntax
.. code::
# this is a comment
; this is also a comment (.ini style)
--- # lines that start with --- are ignored (yaml style)
-------------------
[section] # .ini-style section names are treated as comments
# how to specify a key-value pair (all of these are equivalent):
name value # key is case sensitive: "Name" isn't "name"
name = value # (.ini style) (white space is ignored, so name = value same as name=value)
name: value # (yaml style)
--name value # (argparse style)
# how to set a flag arg (eg. arg which has action="store_true")
--name
name
name = True # "True" and "true" are the same
# how to specify a list arg (eg. arg which has action="append")
fruit = [apple, orange, lemon]
indexes = [1, 12, 35 , 40]
"""
def get_syntax_description(self):
msg = ("Config file syntax allows: key=value, flag=true, stuff=[a,b,c] "
"(for details, see syntax at https://goo.gl/R74nmi).")
return msg
def parse(self, stream):
# see ConfigFileParser.parse docstring
items = OrderedDict()
for i, line in enumerate(stream):
line = line.strip()
if not line or line[0] in ["#", ";", "["] or line.startswith("---"):
continue
match = re.match(r'^(?P[^:=;#\s]+)\s*'
r'(?:(?P[:=\s])\s*([\'"]?)(?P.+?)?\3)?'
r'\s*(?:\s[;#]\s*(?P.*?)\s*)?$', line)
if match:
key = match.group("key")
equal = match.group('equal')
value = match.group("value")
comment = match.group("comment")
if value is None and equal is not None and equal != ' ':
value = ''
elif value is None:
value = "true"
if value.startswith("[") and value.endswith("]"):
# handle special case of k=[1,2,3] or other json-like syntax
try:
value = json.loads(value)
except Exception as e:
# for backward compatibility with legacy format (eg. where config value is [a, b, c] instead of proper json ["a", "b", "c"]
value = [elem.strip() for elem in value[1:-1].split(",")]
if comment:
comment = comment.strip()[1:].strip()
items[key] = value
else:
raise ConfigFileParserException("Unexpected line {} in {}: {}".format(i,
getattr(stream, 'name', 'stream'), line))
return items
def serialize(self, items):
# see ConfigFileParser.serialize docstring
r = StringIO()
for key, value in items.items():
if isinstance(value, list):
# handle special case of lists
value = "["+", ".join(map(str, value))+"]"
r.write("{} = {}\n".format(key, value))
return r.getvalue()
class ConfigparserConfigFileParser(ConfigFileParser):
"""parses INI files using pythons configparser."""
def get_syntax_description(self):
msg = """Uses configparser module to parse an INI file which allows multi-line
values.
Allowed syntax is that for a ConfigParser with the following options:
allow_no_value = False,
inline_comment_prefixes = ("#",)
strict = True
empty_lines_in_values = False
See https://docs.python.org/3/library/configparser.html for details.
Note: INI file sections names are still treated as comments.
"""
return msg
def parse(self, stream):
# see ConfigFileParser.parse docstring
import configparser
from ast import literal_eval
# parse with configparser to allow multi-line values
config = configparser.ConfigParser(
delimiters=("=",":"),
allow_no_value=False,
comment_prefixes=("#",";"),
inline_comment_prefixes=("#",";"),
strict=True,
empty_lines_in_values=False,
)
try:
config.read_string(stream.read())
except Exception as e:
raise ConfigFileParserException("Couldn't parse config file: %s" % e)
# convert to dict and remove INI section names
result = OrderedDict()
for section in config.sections():
for k,v in config[section].items():
multiLine2SingleLine = v.replace('\n',' ').replace('\r',' ')
# handle special case for lists
if '[' in multiLine2SingleLine and ']' in multiLine2SingleLine:
# ensure not a dict with a list value
prelist_string = multiLine2SingleLine.split('[')[0]
if '{' not in prelist_string:
result[k] = literal_eval(multiLine2SingleLine)
else:
result[k] = multiLine2SingleLine
else:
result[k] = multiLine2SingleLine
return result
def serialize(self, items):
# see ConfigFileParser.serialize docstring
import configparser
import io
config = configparser.ConfigParser(
allow_no_value=False,
inline_comment_prefixes=("#",),
strict=True,
empty_lines_in_values=False,
)
items = {"DEFAULT": items}
config.read_dict(items)
stream = io.StringIO()
config.write(stream)
stream.seek(0)
return stream.read()
class YAMLConfigFileParser(ConfigFileParser):
"""Parses YAML config files. Depends on the PyYAML module.
https://pypi.python.org/pypi/PyYAML
"""
def get_syntax_description(self):
msg = ("The config file uses YAML syntax and must represent a YAML "
"'mapping' (for details, see http://learn.getgrav.org/advanced/yaml).")
return msg
def _load_yaml(self):
"""lazy-import PyYAML so that configargparse doesn't have to depend
on it unless this parser is used."""
try:
import yaml
except ImportError:
raise ConfigFileParserException("Could not import yaml. "
"It can be installed by running 'pip install PyYAML'")
return yaml
def parse(self, stream):
# see ConfigFileParser.parse docstring
yaml = self._load_yaml()
try:
parsed_obj = yaml.safe_load(stream)
except Exception as e:
raise ConfigFileParserException("Couldn't parse config file: %s" % e)
if not isinstance(parsed_obj, dict):
raise ConfigFileParserException("The config file doesn't appear to "
"contain 'key: value' pairs (aka. a YAML mapping). "
"yaml.load('%s') returned type '%s' instead of 'dict'." % (
getattr(stream, 'name', 'stream'), type(parsed_obj).__name__))
result = OrderedDict()
for key, value in parsed_obj.items():
if isinstance(value, list):
result[key] = value
elif value is None:
pass
else:
result[key] = str(value)
return result
def serialize(self, items, default_flow_style=False):
# see ConfigFileParser.serialize docstring
# lazy-import so there's no dependency on yaml unless this class is used
yaml = self._load_yaml()
# it looks like ordering can't be preserved: http://pyyaml.org/ticket/29
items = dict(items)
return yaml.dump(items, default_flow_style=default_flow_style)
# used while parsing args to keep track of where they came from
_COMMAND_LINE_SOURCE_KEY = "command_line"
_ENV_VAR_SOURCE_KEY = "environment_variables"
_CONFIG_FILE_SOURCE_KEY = "config_file"
_DEFAULTS_SOURCE_KEY = "defaults"
class ArgumentParser(argparse.ArgumentParser):
"""Drop-in replacement for `argparse.ArgumentParser` that adds support for
environment variables and ``.ini`` or ``.yaml-style`` config files.
"""
def __init__(self, *args, **kwargs):
r"""Supports args of the `argparse.ArgumentParser` constructor
as \*\*kwargs, as well as the following additional args.
Arguments:
add_config_file_help: Whether to add a description of config file
syntax to the help message.
add_env_var_help: Whether to add something to the help message for
args that can be set through environment variables.
auto_env_var_prefix: If set to a string instead of None, all config-
file-settable options will become also settable via environment
variables whose names are this prefix followed by the config
file key, all in upper case. (eg. setting this to ``foo_`` will
allow an arg like ``--my-arg`` to also be set via the FOO_MY_ARG
environment variable)
default_config_files: When specified, this list of config files will
be parsed in order, with the values from each config file
taking precedence over previous ones. This allows an application
to look for config files in multiple standard locations such as
the install directory, home directory, and current directory.
Also, shell \* syntax can be used to specify all conf files in a
directory. For example::
["/etc/conf/app_config.ini",
"/etc/conf/conf-enabled/*.ini",
"~/.my_app_config.ini",
"./app_config.txt"]
ignore_unknown_config_file_keys: If true, settings that are found
in a config file but don't correspond to any defined
configargparse args will be ignored. If false, they will be
processed and appended to the commandline like other args, and
can be retrieved using parse_known_args() instead of parse_args()
config_file_open_func: function used to open a config file for reading
or writing. Needs to return a file-like object.
config_file_parser_class: configargparse.ConfigFileParser subclass
which determines the config file format. configargparse comes
with DefaultConfigFileParser and YAMLConfigFileParser.
args_for_setting_config_path: A list of one or more command line
args to be used for specifying the config file path
(eg. ["-c", "--config-file"]). Default: []
config_arg_is_required: When args_for_setting_config_path is set,
set this to True to always require users to provide a config path.
config_arg_help_message: the help message to use for the
args listed in args_for_setting_config_path.
args_for_writing_out_config_file: A list of one or more command line
args to use for specifying a config file output path. If
provided, these args cause configargparse to write out a config
file with settings based on the other provided commandline args,
environment variants and defaults, and then to exit.
(eg. ["-w", "--write-out-config-file"]). Default: []
write_out_config_file_arg_help_message: The help message to use for
the args in args_for_writing_out_config_file.
"""
# This is the only way to make positional args (tested in the argparse
# main test suite) and keyword arguments work across both Python 2 and
# 3. This could be refactored to not need extra local variables.
add_config_file_help = kwargs.pop('add_config_file_help', True)
add_env_var_help = kwargs.pop('add_env_var_help', True)
auto_env_var_prefix = kwargs.pop('auto_env_var_prefix', None)
default_config_files = kwargs.pop('default_config_files', [])
ignore_unknown_config_file_keys = kwargs.pop(
'ignore_unknown_config_file_keys', False)
config_file_parser_class = kwargs.pop('config_file_parser_class',
DefaultConfigFileParser)
args_for_setting_config_path = kwargs.pop(
'args_for_setting_config_path', [])
config_arg_is_required = kwargs.pop('config_arg_is_required', False)
config_arg_help_message = kwargs.pop('config_arg_help_message',
"config file path")
args_for_writing_out_config_file = kwargs.pop(
'args_for_writing_out_config_file', [])
write_out_config_file_arg_help_message = kwargs.pop(
'write_out_config_file_arg_help_message', "takes the current "
"command line args and writes them out to a config file at the "
"given path, then exits")
self._config_file_open_func = kwargs.pop('config_file_open_func', open)
self._add_config_file_help = add_config_file_help
self._add_env_var_help = add_env_var_help
self._auto_env_var_prefix = auto_env_var_prefix
argparse.ArgumentParser.__init__(self, *args, **kwargs)
# parse the additional args
if config_file_parser_class is None:
self._config_file_parser = DefaultConfigFileParser()
else:
self._config_file_parser = config_file_parser_class()
self._default_config_files = default_config_files
self._ignore_unknown_config_file_keys = ignore_unknown_config_file_keys
if args_for_setting_config_path:
self.add_argument(*args_for_setting_config_path, dest="config_file",
required=config_arg_is_required, help=config_arg_help_message,
is_config_file_arg=True)
if args_for_writing_out_config_file:
self.add_argument(*args_for_writing_out_config_file,
dest="write_out_config_file_to_this_path",
metavar="CONFIG_OUTPUT_PATH",
help=write_out_config_file_arg_help_message,
is_write_out_config_file_arg=True)
def parse_args(self, args = None, namespace = None,
config_file_contents = None, env_vars = os.environ):
"""Supports all the same args as the `argparse.ArgumentParser.parse_args()`,
as well as the following additional args.
Arguments:
args: a list of args as in argparse, or a string (eg. "-x -y bla")
config_file_contents: String. Used for testing.
env_vars: Dictionary. Used for testing.
Returns:
argparse.Namespace: namespace
"""
args, argv = self.parse_known_args(
args=args,
namespace=namespace,
config_file_contents=config_file_contents,
env_vars=env_vars,
ignore_help_args=False)
if argv:
self.error('unrecognized arguments: %s' % ' '.join(argv))
return args
def parse_known_args(
self,
args=None,
namespace=None,
config_file_contents=None,
env_vars=os.environ,
ignore_help_args=False,
):
"""Supports all the same args as the `argparse.ArgumentParser.parse_args()`,
as well as the following additional args.
Arguments:
args: a list of args as in argparse, or a string (eg. "-x -y bla")
config_file_contents (str). Used for testing.
env_vars (dict). Used for testing.
ignore_help_args (bool): This flag determines behavior when user specifies ``--help`` or ``-h``. If False,
it will have the default behavior - printing help and exiting. If True, it won't do either.
Returns:
tuple[argparse.Namespace, list[str]]: tuple namescpace, unknown_args
"""
if args is None:
args = sys.argv[1:]
elif isinstance(args, str):
args = args.split()
else:
args = list(args)
for a in self._actions:
a.is_positional_arg = not a.option_strings
if ignore_help_args:
args = [arg for arg in args if arg not in ("-h", "--help")]
# maps a string describing the source (eg. env var) to a settings dict
# to keep track of where values came from (used by print_values()).
# The settings dicts for env vars and config files will then map
# the config key to an (argparse Action obj, string value) 2-tuple.
self._source_to_settings = OrderedDict()
if args:
a_v_pair = (None, list(args)) # copy args list to isolate changes
self._source_to_settings[_COMMAND_LINE_SOURCE_KEY] = {'': a_v_pair}
# handle auto_env_var_prefix __init__ arg by setting a.env_var as needed
if self._auto_env_var_prefix is not None:
for a in self._actions:
config_file_keys = self.get_possible_config_keys(a)
if config_file_keys and not (a.env_var or a.is_positional_arg
or a.is_config_file_arg or a.is_write_out_config_file_arg or
isinstance(a, argparse._VersionAction) or
isinstance(a, argparse._HelpAction)):
stripped_config_file_key = config_file_keys[0].strip(
self.prefix_chars)
a.env_var = (self._auto_env_var_prefix +
stripped_config_file_key).replace('-', '_').upper()
# add env var settings to the commandline that aren't there already
env_var_args = []
nargs = False
actions_with_env_var_values = [a for a in self._actions
if not a.is_positional_arg and a.env_var and a.env_var in env_vars
and not already_on_command_line(args, a.option_strings, self.prefix_chars)]
for action in actions_with_env_var_values:
key = action.env_var
value = env_vars[key]
# Make list-string into list.
if action.nargs or isinstance(action, argparse._AppendAction):
nargs = True
if value.startswith("[") and value.endswith("]"):
# handle special case of k=[1,2,3] or other json-like syntax
try:
value = json.loads(value)
except Exception:
# for backward compatibility with legacy format (eg. where config value is [a, b, c] instead of proper json ["a", "b", "c"]
value = [elem.strip() for elem in value[1:-1].split(",")]
env_var_args += self.convert_item_to_command_line_arg(
action, key, value)
if nargs:
args = args + env_var_args
else:
args = env_var_args + args
if env_var_args:
self._source_to_settings[_ENV_VAR_SOURCE_KEY] = OrderedDict(
[(a.env_var, (a, env_vars[a.env_var]))
for a in actions_with_env_var_values])
# before parsing any config files, check if -h was specified.
supports_help_arg = any(
a for a in self._actions if isinstance(a, argparse._HelpAction))
skip_config_file_parsing = supports_help_arg and (
"-h" in args or "--help" in args)
# prepare for reading config file(s)
known_config_keys = {config_key: action for action in self._actions
for config_key in self.get_possible_config_keys(action)}
# open the config file(s)
config_streams = []
if config_file_contents is not None:
stream = StringIO(config_file_contents)
stream.name = "method arg"
config_streams = [stream]
elif not skip_config_file_parsing:
config_streams = self._open_config_files(args)
# parse each config file
for stream in reversed(config_streams):
try:
config_items = self._config_file_parser.parse(stream)
except ConfigFileParserException as e:
self.error(e)
finally:
if hasattr(stream, "close"):
stream.close()
# add each config item to the commandline unless it's there already
config_args = []
nargs = False
for key, value in config_items.items():
if key in known_config_keys:
action = known_config_keys[key]
discard_this_key = already_on_command_line(
args, action.option_strings, self.prefix_chars)
else:
action = None
discard_this_key = self._ignore_unknown_config_file_keys or \
already_on_command_line(
args,
[self.get_command_line_key_for_unknown_config_file_setting(key)],
self.prefix_chars)
if not discard_this_key:
config_args += self.convert_item_to_command_line_arg(
action, key, value)
source_key = "%s|%s" %(_CONFIG_FILE_SOURCE_KEY, stream.name)
if source_key not in self._source_to_settings:
self._source_to_settings[source_key] = OrderedDict()
self._source_to_settings[source_key][key] = (action, value)
if (action and action.nargs or
isinstance(action, argparse._AppendAction)):
nargs = True
if nargs:
args = args + config_args
else:
args = config_args + args
# save default settings for use by print_values()
default_settings = OrderedDict()
for action in self._actions:
cares_about_default_value = (not action.is_positional_arg or
action.nargs in [OPTIONAL, ZERO_OR_MORE])
if (already_on_command_line(args, action.option_strings, self.prefix_chars) or
not cares_about_default_value or
action.default is None or
action.default == SUPPRESS or
isinstance(action, ACTION_TYPES_THAT_DONT_NEED_A_VALUE)):
continue
else:
if action.option_strings:
key = action.option_strings[-1]
else:
key = action.dest
default_settings[key] = (action, str(action.default))
if default_settings:
self._source_to_settings[_DEFAULTS_SOURCE_KEY] = default_settings
# parse all args (including commandline, config file, and env var)
namespace, unknown_args = argparse.ArgumentParser.parse_known_args(
self, args=args, namespace=namespace)
# handle any args that have is_write_out_config_file_arg set to true
# check if the user specified this arg on the commandline
output_file_paths = [getattr(namespace, a.dest, None) for a in self._actions
if getattr(a, "is_write_out_config_file_arg", False)]
output_file_paths = [a for a in output_file_paths if a is not None]
self.write_config_file(namespace, output_file_paths, exit_after=True)
return namespace, unknown_args
def get_source_to_settings_dict(self):
"""
If called after `parse_args()` or `parse_known_args()`, returns a dict that contains up to 4 keys corresponding
to where a given option's value is coming from:
- "command_line"
- "environment_variables"
- "config_file"
- "defaults"
Each such key, will be mapped to another dictionary containing the options set via that method. Here the key
will be the option name, and the value will be a 2-tuple of the form (`argparse.Action` obj, `str` value).
Returns:
dict[str, dict[str, tuple[argparse.Action, str]]]: source to settings dict
"""
return self._source_to_settings
def write_config_file(self, parsed_namespace, output_file_paths, exit_after=False):
"""Write the given settings to output files.
Args:
parsed_namespace: namespace object created within parse_known_args()
output_file_paths: any number of file paths to write the config to
exit_after: whether to exit the program after writing the config files
"""
for output_file_path in output_file_paths:
# validate the output file path
try:
with self._config_file_open_func(output_file_path, "w") as output_file:
pass
except IOError as e:
raise ValueError("Couldn't open {} for writing: {}".format(
output_file_path, e))
if output_file_paths:
# generate the config file contents
config_items = self.get_items_for_config_file_output(
self._source_to_settings, parsed_namespace)
file_contents = self._config_file_parser.serialize(config_items)
for output_file_path in output_file_paths:
with self._config_file_open_func(output_file_path, "w") as output_file:
output_file.write(file_contents)
print("Wrote config file to " + ", ".join(output_file_paths))
if exit_after:
self.exit(0)
def get_command_line_key_for_unknown_config_file_setting(self, key):
"""Compute a commandline arg key to be used for a config file setting
that doesn't correspond to any defined configargparse arg (and so
doesn't have a user-specified commandline arg key).
Args:
key: The config file key that was being set.
Returns:
str: command line key
"""
key_without_prefix_chars = key.strip(self.prefix_chars)
command_line_key = self.prefix_chars[0]*2 + key_without_prefix_chars
return command_line_key
def get_items_for_config_file_output(self, source_to_settings,
parsed_namespace):
"""Converts the given settings back to a dictionary that can be passed
to ConfigFormatParser.serialize(..).
Args:
source_to_settings: the dictionary described in parse_known_args()
parsed_namespace: namespace object created within parse_known_args()
Returns:
OrderedDict: where keys are strings and values are either strings
or lists
"""
config_file_items = OrderedDict()
for source, settings in source_to_settings.items():
if source == _COMMAND_LINE_SOURCE_KEY:
_, existing_command_line_args = settings['']
for action in self._actions:
config_file_keys = self.get_possible_config_keys(action)
if config_file_keys and not action.is_positional_arg and \
already_on_command_line(existing_command_line_args,
action.option_strings,
self.prefix_chars):
value = getattr(parsed_namespace, action.dest, None)
if value is not None:
if isinstance(value, bool):
value = str(value).lower()
config_file_items[config_file_keys[0]] = value
elif source == _ENV_VAR_SOURCE_KEY:
for key, (action, value) in settings.items():
config_file_keys = self.get_possible_config_keys(action)
if config_file_keys:
value = getattr(parsed_namespace, action.dest, None)
if value is not None:
config_file_items[config_file_keys[0]] = value
elif source.startswith(_CONFIG_FILE_SOURCE_KEY):
for key, (action, value) in settings.items():
config_file_items[key] = value
elif source == _DEFAULTS_SOURCE_KEY:
for key, (action, value) in settings.items():
config_file_keys = self.get_possible_config_keys(action)
if config_file_keys:
value = getattr(parsed_namespace, action.dest, None)
if value is not None:
config_file_items[config_file_keys[0]] = value
return config_file_items
def convert_item_to_command_line_arg(self, action, key, value):
"""Converts a config file or env var key + value to a list of
commandline args to append to the commandline.
Args:
action: The argparse Action object for this setting, or None if this
config file setting doesn't correspond to any defined
configargparse arg.
key: string (config file key or env var name)
value: parsed value of type string or list
Returns:
list[str]: args
"""
args = []
if action is None:
command_line_key = \
self.get_command_line_key_for_unknown_config_file_setting(key)
else:
if not is_boolean_optional_action(action):
command_line_key = action.option_strings[-1]
# handle boolean value
if action is not None and isinstance(action, ACTION_TYPES_THAT_DONT_NEED_A_VALUE):
if value.lower() in ("true", "yes", "1"):
if not is_boolean_optional_action(action):
args.append( command_line_key )
else:
# --foo
args.append(action.option_strings[0])
elif value.lower() in ("false", "no", "0"):
# don't append when set to "false" / "no"
if not is_boolean_optional_action(action):
pass
else:
# --no-foo
args.append(action.option_strings[1])
elif isinstance(action, argparse._CountAction):
for arg in args:
if any([arg.startswith(s) for s in action.option_strings]):
value = 0
args += [action.option_strings[0]] * int(value)
else:
self.error("Unexpected value for %s: '%s'. Expecting 'true', "
"'false', 'yes', 'no', '1' or '0'" % (key, value))
elif isinstance(value, list):
accepts_list_and_has_nargs = action is not None and action.nargs is not None and (
isinstance(action, argparse._StoreAction) or isinstance(action, argparse._AppendAction)
) and (
action.nargs in ('+', '*') or (isinstance(action.nargs, int) and action.nargs > 1)
)
if action is None or isinstance(action, argparse._AppendAction):
for list_elem in value:
if accepts_list_and_has_nargs and isinstance(list_elem, list):
args.append(command_line_key)
for sub_elem in list_elem:
args.append(str(sub_elem))
else:
args.append( "%s=%s" % (command_line_key, str(list_elem)) )
elif accepts_list_and_has_nargs:
args.append( command_line_key )
for list_elem in value:
args.append( str(list_elem) )
else:
self.error(("%s can't be set to a list '%s' unless its action type is changed "
"to 'append' or nargs is set to '*', '+', or > 1") % (key, value))
elif isinstance(value, str):
args.append( "%s=%s" % (command_line_key, value) )
else:
raise ValueError("Unexpected value type {} for value: {}".format(
type(value), value))
return args
def get_possible_config_keys(self, action):
"""This method decides which actions can be set in a config file and
what their keys will be. It returns a list of 0 or more config keys that
can be used to set the given action's value in a config file.
Returns:
list[str]: keys
"""
keys = []
# Do not write out the config options for writing out a config file
if getattr(action, 'is_write_out_config_file_arg', None):
return keys
for arg in action.option_strings:
if any(arg.startswith(2*c) for c in self.prefix_chars):
keys += [arg[2:], arg] # eg. for '--bla' return ['bla', '--bla']
return keys
def _open_config_files(self, command_line_args):
"""Tries to parse config file path(s) from within command_line_args.
Returns a list of opened config files, including files specified on the
commandline as well as any default_config_files specified in the
constructor that are present on disk.
Args:
command_line_args: List of all args
Returns:
list[IO]: open config files
"""
# open any default config files
config_files = []
for files in map(glob.glob, map(os.path.expanduser, self._default_config_files)):
for f in files:
config_files.append(self._config_file_open_func(f))
# list actions with is_config_file_arg=True. Its possible there is more
# than one such arg.
user_config_file_arg_actions = [
a for a in self._actions if getattr(a, "is_config_file_arg", False)]
if not user_config_file_arg_actions:
return config_files
for action in user_config_file_arg_actions:
# try to parse out the config file path by using a clean new
# ArgumentParser that only knows this one arg/action.
arg_parser = argparse.ArgumentParser(
prefix_chars=self.prefix_chars,
add_help=False)
arg_parser._add_action(action)
# make parser not exit on error by replacing its error method.
# Otherwise it sys.exits(..) if, for example, config file
# is_required=True and user doesn't provide it.
def error_method(self, message):
pass
arg_parser.error = types.MethodType(error_method, arg_parser)
# check whether the user provided a value
parsed_arg = arg_parser.parse_known_args(args=command_line_args)
if not parsed_arg:
continue
namespace, _ = parsed_arg
user_config_file = getattr(namespace, action.dest, None)
if not user_config_file:
continue
# open user-provided config file
user_config_file = os.path.expanduser(user_config_file)
try:
stream = self._config_file_open_func(user_config_file)
except Exception as e:
if len(e.args) == 2: # OSError
errno, msg = e.args
else:
msg = str(e)
# close previously opened config files
for config_file in config_files:
try:
config_file.close()
except Exception:
pass
self.error("Unable to open config file: %s. Error: %s" % (
user_config_file, msg
))
config_files += [stream]
return config_files
def format_values(self):
"""Returns a string with all args and settings and where they came from
(eg. commandline, config file, environment variable or default)
Returns:
str: source to settings string
"""
source_key_to_display_value_map = {
_COMMAND_LINE_SOURCE_KEY: "Command Line Args: ",
_ENV_VAR_SOURCE_KEY: "Environment Variables:\n",
_CONFIG_FILE_SOURCE_KEY: "Config File (%s):\n",
_DEFAULTS_SOURCE_KEY: "Defaults:\n"
}
r = StringIO()
for source, settings in self._source_to_settings.items():
source = source.split("|")
source = source_key_to_display_value_map[source[0]] % tuple(source[1:])
r.write(source)
for key, (action, value) in settings.items():
if key:
r.write(" {:<19}{}\n".format(key+":", value))
else:
if isinstance(value, str):
r.write(" %s\n" % value)
elif isinstance(value, list):
r.write(" %s\n" % ' '.join(value))
return r.getvalue()
def print_values(self, file = sys.stdout):
"""Prints the format_values() string (to sys.stdout or another file)."""
file.write(self.format_values())
def format_help(self):
msg = ""
added_config_file_help = False
added_env_var_help = False
if self._add_config_file_help:
default_config_files = self._default_config_files
cc = 2*self.prefix_chars[0] # eg. --
config_settable_args = [(arg, a) for a in self._actions for arg in
a.option_strings if self.get_possible_config_keys(a) and not
(a.dest == "help" or a.is_config_file_arg or
a.is_write_out_config_file_arg)]
config_path_actions = [a for a in
self._actions if getattr(a, "is_config_file_arg", False)]
if config_settable_args and (default_config_files or
config_path_actions):
self._add_config_file_help = False # prevent duplication
added_config_file_help = True
msg += ("Args that start with '%s' (eg. %s) can also be set in "
"a config file") % (cc, config_settable_args[0][0])
config_arg_string = " or ".join(a.option_strings[0]
for a in config_path_actions if a.option_strings)
if config_arg_string:
config_arg_string = "specified via " + config_arg_string
if default_config_files or config_arg_string:
msg += " (%s)." % " or ".join(tuple(default_config_files) +
tuple(filter(None, [config_arg_string])))
msg += " " + self._config_file_parser.get_syntax_description()
if self._add_env_var_help:
env_var_actions = [(a.env_var, a) for a in self._actions
if getattr(a, "env_var", None)]
for env_var, a in env_var_actions:
if a.help == SUPPRESS:
continue
env_var_help_string = " [env var: %s]" % env_var
if not a.help:
a.help = ""
if env_var_help_string not in a.help:
a.help += env_var_help_string
added_env_var_help = True
self._add_env_var_help = False # prevent duplication
if added_env_var_help or added_config_file_help:
value_sources = ["defaults"]
if added_config_file_help:
value_sources = ["config file values"] + value_sources
if added_env_var_help:
value_sources = ["environment variables"] + value_sources
msg += (" If an arg is specified in more than one place, then "
"commandline values override %s.") % (
" which override ".join(value_sources))
text_width = max(self._get_formatter()._width, 11)
msg = textwrap.fill(msg, text_width)
return (argparse.ArgumentParser.format_help(self)
+ ("\n{}\n".format(msg) if msg != "" else ""))
def add_argument(self, *args, **kwargs):
"""
This method supports the same args as ArgumentParser.add_argument(..)
as well as the additional args below.
Arguments:
env_var: If set, the value of this environment variable will override
any config file or default values for this arg (but can itself
be overridden on the commandline). Also, if auto_env_var_prefix is
set in the constructor, this env var name will be used instead of
the automatic name.
is_config_file_arg: If True, this arg is treated as a config file path
This provides an alternative way to specify config files in place of
the ArgumentParser(fromfile_prefix_chars=..) mechanism.
Default: False
is_write_out_config_file_arg: If True, this arg will be treated as a
config file path, and, when it is specified, will cause
configargparse to write all current commandline args to this file
as config options and then exit.
Default: False
Returns:
argparse.Action: the new argparse action
"""
env_var = kwargs.pop("env_var", None)
is_config_file_arg = kwargs.pop(
"is_config_file_arg", None) or kwargs.pop(
"is_config_file", None) # for backward compat.
is_write_out_config_file_arg = kwargs.pop(
"is_write_out_config_file_arg", None)
action = self.original_add_argument_method(*args, **kwargs)
action.is_positional_arg = not action.option_strings
action.env_var = env_var
action.is_config_file_arg = is_config_file_arg
action.is_write_out_config_file_arg = is_write_out_config_file_arg
if action.is_positional_arg and env_var:
raise ValueError("env_var can't be set for a positional arg.")
if action.is_config_file_arg and not isinstance(action, argparse._StoreAction):
raise ValueError("arg with is_config_file_arg=True must have "
"action='store'")
if action.is_write_out_config_file_arg:
error_prefix = "arg with is_write_out_config_file_arg=True "
if not isinstance(action, argparse._StoreAction):
raise ValueError(error_prefix + "must have action='store'")
if is_config_file_arg:
raise ValueError(error_prefix + "can't also have "
"is_config_file_arg=True")
return action
def already_on_command_line(existing_args_list, potential_command_line_args, prefix_chars):
"""Utility method for checking if any of the potential_command_line_args is
already present in existing_args.
Returns:
bool: already on command line?
"""
arg_names = []
for arg_string in existing_args_list:
if arg_string and arg_string[0] in prefix_chars and "=" in arg_string :
option_string, explicit_arg = arg_string.split("=", 1)
arg_names.append(option_string)
else:
arg_names.append(arg_string)
return any(
potential_arg in arg_names for potential_arg in potential_command_line_args
)
#TODO: Update to latest version of pydoctor when https://github.com/twisted/pydoctor/pull/414 has been merged
# such that the alises can be documented automatically.
# wrap ArgumentParser's add_argument(..) method with the one above
argparse._ActionsContainer.original_add_argument_method = argparse._ActionsContainer.add_argument
argparse._ActionsContainer.add_argument = add_argument
# add all public classes and constants from argparse module's namespace to this
# module's namespace so that the 2 modules are truly interchangeable
HelpFormatter = argparse.HelpFormatter
RawDescriptionHelpFormatter = argparse.RawDescriptionHelpFormatter
RawTextHelpFormatter = argparse.RawTextHelpFormatter
ArgumentDefaultsHelpFormatter = argparse.ArgumentDefaultsHelpFormatter
ArgumentError = argparse.ArgumentError
ArgumentTypeError = argparse.ArgumentTypeError
Action = argparse.Action
FileType = argparse.FileType
Namespace = argparse.Namespace
ONE_OR_MORE = argparse.ONE_OR_MORE
OPTIONAL = argparse.OPTIONAL
REMAINDER = argparse.REMAINDER
SUPPRESS = argparse.SUPPRESS
ZERO_OR_MORE = argparse.ZERO_OR_MORE
# deprecated PEP-8 incompatible API names.
initArgumentParser = init_argument_parser
getArgumentParser = get_argument_parser
getArgParser = get_argument_parser
getParser = get_argument_parser
# create shorter aliases for the key methods and class names
get_arg_parser = get_argument_parser
get_parser = get_argument_parser
ArgParser = ArgumentParser
Parser = ArgumentParser
argparse._ActionsContainer.add_arg = argparse._ActionsContainer.add_argument
argparse._ActionsContainer.add = argparse._ActionsContainer.add_argument
ArgumentParser.parse = ArgumentParser.parse_args
ArgumentParser.parse_known = ArgumentParser.parse_known_args
RawFormatter = RawDescriptionHelpFormatter
DefaultsFormatter = ArgumentDefaultsHelpFormatter
DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter
ConfigArgParse-1.5.3/setup.py 0000644 0000000 0000024 00000007467 14126173171 016171 0 ustar root staff 0000000 0000000 import logging
import os
import sys
try:
from setuptools import setup
except ImportError:
print("WARNING: setuptools not installed. Will try using distutils instead..")
from distutils.core import setup
def launch_http_server(directory):
assert os.path.isdir(directory)
try:
try:
from SimpleHTTPServer import SimpleHTTPRequestHandler
except ImportError:
from http.server import SimpleHTTPRequestHandler
try:
import SocketServer
except ImportError:
import socketserver as SocketServer
import socket
for port in [80] + list(range(8000, 8100)):
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('localhost', port))
s.close()
except socket.error as e:
logging.debug("Can't use port %d: %s" % (port, e.strerror))
continue
print("HTML coverage report now available at http://{}{}".format(
socket.gethostname(), (":%s" % port) if port != 80 else ""))
os.chdir(directory)
SocketServer.TCPServer(("", port),
SimpleHTTPRequestHandler).serve_forever()
else:
logging.debug("All network port. ")
except Exception as e:
logging.error("ERROR: while starting an HTTP server to serve "
"the coverage report: %s" % e)
command = sys.argv[-1]
if command == 'publish':
os.system('rm -rf dist')
os.system('python setup.py sdist')
os.system('python3 setup.py bdist_wheel')
os.system('twine upload dist/*whl dist/*gz')
sys.exit()
elif command == "coverage":
try:
import coverage
except:
sys.exit("coverage.py not installed (pip install --user coverage)")
setup_py_path = os.path.abspath(__file__)
os.system('coverage run --source=configargparse ' + setup_py_path +' test')
os.system('coverage report')
os.system('coverage html')
print("Done computing coverage")
launch_http_server(directory="htmlcov")
sys.exit()
long_description = ''
if command not in ['test', 'coverage']:
long_description = open('README.rst').read()
install_requires = []
tests_require = [
'mock',
'PyYAML',
'pytest',
]
setup(
name='ConfigArgParse',
version="1.5.3",
description='A drop-in replacement for argparse that allows options to '
'also be set via config files and/or environment variables.',
long_description=long_description,
url='https://github.com/bw2/ConfigArgParse',
py_modules=['configargparse'],
include_package_data=True,
license="MIT",
keywords='options, argparse, ConfigArgParse, config, environment variables, '
'envvars, ENV, environment, optparse, YAML, INI',
classifiers=[
'Development Status :: 4 - Beta',
'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License',
'Natural Language :: English',
"Programming Language :: Python :: 2",
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy',
],
test_suite='tests',
python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',
install_requires=install_requires,
tests_require=tests_require,
extras_require = {
'yaml': ["PyYAML"],
'test': tests_require,
}
)
ConfigArgParse-1.5.3/ConfigArgParse.egg-info/ 0000755 0000000 0000024 00000000000 14126173212 020761 5 ustar root staff 0000000 0000000 ConfigArgParse-1.5.3/ConfigArgParse.egg-info/PKG-INFO 0000644 0000000 0000024 00000053516 14126173212 022070 0 ustar root staff 0000000 0000000 Metadata-Version: 2.1
Name: ConfigArgParse
Version: 1.5.3
Summary: A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.
Home-page: https://github.com/bw2/ConfigArgParse
License: MIT
Description: ConfigArgParse
--------------
.. image:: https://img.shields.io/pypi/v/ConfigArgParse.svg?style=flat
:alt: PyPI version
:target: https://pypi.python.org/pypi/ConfigArgParse
.. image:: https://img.shields.io/pypi/pyversions/ConfigArgParse.svg
:alt: Supported Python versions
:target: https://pypi.python.org/pypi/ConfigArgParse
.. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
:alt: Travis CI build
:target: https://travis-ci.org/bw2/ConfigArgParse
.. image:: https://img.shields.io/badge/-API_Documentation-blue
:alt: API Documentation
:target: https://bw2.github.io/ConfigArgParse/
Overview
~~~~~~~~
Applications with more than a handful of user-settable options are best
configured through a combination of command line args, config files,
hard-coded defaults, and in some cases, environment variables.
Python's command line parsing modules such as argparse have very limited
support for config files and environment variables, so this module
extends argparse to add these features.
Available on PyPI: http://pypi.python.org/pypi/ConfigArgParse
.. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
:target: https://travis-ci.org/bw2/ConfigArgParse
Features
~~~~~~~~
- command-line, config file, env var, and default settings can now be
defined, documented, and parsed in one go using a single API (if a
value is specified in more than one way then: command line >
environment variables > config file values > defaults)
- config files can have .ini or .yaml style syntax (eg. key=value or
key: value)
- user can provide a config file via a normal-looking command line arg
(eg. -c path/to/config.txt) rather than the argparse-style @config.txt
- one or more default config file paths can be specified
(eg. ['/etc/bla.conf', '~/.my_config'] )
- all argparse functionality is fully supported, so this module can
serve as a drop-in replacement (verified by argparse unittests).
- env vars and config file keys & syntax are automatically documented
in the -h help message
- new method :code:`print_values()` can report keys & values and where
they were set (eg. command line, env var, config file, or default).
- lite-weight (no 3rd-party library dependencies except (optionally) PyYAML)
- extensible (:code:`ConfigFileParser` can be subclassed to define a new
config file format)
- unittested by running the unittests that came with argparse but on
configargparse, and using tox to test with Python 2.7 and Python 3+
Example
~~~~~~~
*config_test.py*:
Script that defines 4 options and a positional arg and then parses and prints the values. Also,
it prints out the help message as well as the string produced by :code:`format_values()` to show
what they look like.
.. code:: py
import configargparse
p = configargparse.ArgParser(default_config_files=['/etc/app/conf.d/*.conf', '~/.my_settings'])
p.add('-c', '--my-config', required=True, is_config_file=True, help='config file path')
p.add('--genome', required=True, help='path to genome file') # this option can be set in a config file because it starts with '--'
p.add('-v', help='verbose', action='store_true')
p.add('-d', '--dbsnp', help='known variants .vcf', env_var='DBSNP_PATH') # this option can be set in a config file because it starts with '--'
p.add('vcf', nargs='+', help='variant file(s)')
options = p.parse_args()
print(options)
print("----------")
print(p.format_help())
print("----------")
print(p.format_values()) # useful for logging where different settings came from
*config.txt:*
Since the script above set the config file as required=True, lets create a config file to give it:
.. code:: py
# settings for config_test.py
genome = HCMV # cytomegalovirus genome
dbsnp = /data/dbsnp/variants.vcf
*command line:*
Now run the script and pass it the config file:
.. code:: bash
DBSNP_PATH=/data/dbsnp/variants_v2.vcf python config_test.py --my-config config.txt f1.vcf f2.vcf
*output:*
Here is the result:
.. code:: bash
Namespace(dbsnp='/data/dbsnp/variants_v2.vcf', genome='HCMV', my_config='config.txt', v=False, vcf=['f1.vcf', 'f2.vcf'])
----------
usage: config_test.py [-h] -c MY_CONFIG --genome GENOME [-v] [-d DBSNP]
vcf [vcf ...]
Args that start with '--' (eg. --genome) can also be set in a config file
(/etc/app/conf.d/*.conf or ~/.my_settings or specified via -c). Config file
syntax allows: key=value, flag=true, stuff=[a,b,c] (for details, see syntax at
https://goo.gl/R74nmi). If an arg is specified in more than one place, then
commandline values override environment variables which override config file
values which override defaults.
positional arguments:
vcf variant file(s)
optional arguments:
-h, --help show this help message and exit
-c MY_CONFIG, --my-config MY_CONFIG
config file path
--genome GENOME path to genome file
-v verbose
-d DBSNP, --dbsnp DBSNP
known variants .vcf [env var: DBSNP_PATH]
----------
Command Line Args: --my-config config.txt f1.vcf f2.vcf
Environment Variables:
DBSNP_PATH: /data/dbsnp/variants_v2.vcf
Config File (config.txt):
genome: HCMV
Special Values
~~~~~~~~~~~~~~
Under the hood, configargparse handles environment variables and config file
values by converting them to their corresponding command line arg. For
example, "key = value" will be processed as if "--key value" was specified
on the command line.
Also, the following special values (whether in a config file or an environment
variable) are handled in a special way to support booleans and lists:
- :code:`key = true` is handled as if "--key" was specified on the command line.
In your python code this key must be defined as a boolean flag
(eg. action="store_true" or similar).
- :code:`key = [value1, value2, ...]` is handled as if "--key value1 --key value2"
etc. was specified on the command line. In your python code this key must
be defined as a list (eg. action="append").
Config File Syntax
~~~~~~~~~~~~~~~~~~
Only command line args that have a long version (eg. one that starts with '--')
can be set in a config file. For example, "--color" can be set by putting
"color=green" in a config file. The config file syntax depends on the constuctor
arg: :code:`config_file_parser_class` which can be set to one of the provided
classes: :code:`DefaultConfigFileParser`, :code:`YAMLConfigFileParser`,
:code:`ConfigparserConfigFileParser` or to your own subclass of the
:code:`ConfigFileParser` abstract class.
*DefaultConfigFileParser* - the full range of valid syntax is:
.. code:: yaml
# this is a comment
; this is also a comment (.ini style)
--- # lines that start with --- are ignored (yaml style)
-------------------
[section] # .ini-style section names are treated as comments
# how to specify a key-value pair (all of these are equivalent):
name value # key is case sensitive: "Name" isn't "name"
name = value # (.ini style) (white space is ignored, so name = value same as name=value)
name: value # (yaml style)
--name value # (argparse style)
# how to set a flag arg (eg. arg which has action="store_true")
--name
name
name = True # "True" and "true" are the same
# how to specify a list arg (eg. arg which has action="append")
fruit = [apple, orange, lemon]
indexes = [1, 12, 35 , 40]
*YAMLConfigFileParser* - allows a subset of YAML syntax (http://goo.gl/VgT2DU)
.. code:: yaml
# a comment
name1: value
name2: true # "True" and "true" are the same
fruit: [apple, orange, lemon]
indexes: [1, 12, 35, 40]
*ConfigparserConfigFileParser* - allows a subset of python's configparser
module syntax (https://docs.python.org/3.7/library/configparser.html). In
particular the following configparser options are set:
.. code:: py
config = configparser.ArgParser(
delimiters=("=",":"),
allow_no_value=False,
comment_prefixes=("#",";"),
inline_comment_prefixes=("#",";"),
strict=True,
empty_lines_in_values=False,
)
Once configparser parses the config file all section names are removed, thus all
keys must have unique names regardless of which INI section they are defined
under. Also, any keys which have python list syntax are converted to lists by
evaluating them as python code using ast.literal_eval
(https://docs.python.org/3/library/ast.html#ast.literal_eval). To facilitate
this all multi-line values are converted to single-line values. Thus multi-line
string values will have all new-lines converted to spaces. Note, since key-value
pairs that have python dictionary syntax are saved as single-line strings, even
if formatted across multiple lines in the config file, dictionaries can be read
in and converted to valid python dictionaries with PyYAML's safe_load. Example
given below:
.. code:: py
# inside your config file (e.g. config.ini)
[section1] # INI sections treated as comments
system1_settings: { # start of multi-line dictionary
'a':True,
'b':[2, 4, 8, 16],
'c':{'start':0, 'stop':1000},
'd':'experiment 32 testing simulation with parameter a on'
} # end of multi-line dictionary value
.......
# in your configargparse setup
import configargparse
import yaml
parser = configargparse.ArgParser(
config_file_parser_class=configargparse.ConfigparserConfigFileParser
)
parser.add_argument('--system1_settings', type=yaml.safe_load)
args = parser.parse_args() # now args.system1 is a valid python dict
ArgParser Singletons
~~~~~~~~~~~~~~~~~~~~~~~~~
To make it easier to configure different modules in an application,
configargparse provides globally-available ArgumentParser instances
via configargparse.get_argument_parser('name') (similar to
logging.getLogger('name')).
Here is an example of an application with a utils module that also
defines and retrieves its own command-line args.
*main.py*
.. code:: py
import configargparse
import utils
p = configargparse.get_argument_parser()
p.add_argument("-x", help="Main module setting")
p.add_argument("--m-setting", help="Main module setting")
options = p.parse_known_args() # using p.parse_args() here may raise errors.
*utils.py*
.. code:: py
import configargparse
p = configargparse.get_argument_parser()
p.add_argument("--utils-setting", help="Config-file-settable option for utils")
if __name__ == "__main__":
options = p.parse_known_args()
Help Formatters
~~~~~~~~~~~~~~~
:code:`ArgumentDefaultsRawHelpFormatter` is a new HelpFormatter that both adds
default values AND disables line-wrapping. It can be passed to the constructor:
:code:`ArgParser(.., formatter_class=ArgumentDefaultsRawHelpFormatter)`
Aliases
~~~~~~~
The configargparse.ArgumentParser API inherits its class and method
names from argparse and also provides the following shorter names for
convenience:
- p = configargparse.get_arg_parser() # get global singleton instance
- p = configargparse.get_parser()
- p = configargparse.ArgParser() # create a new instance
- p = configargparse.Parser()
- p.add_arg(..)
- p.add(..)
- options = p.parse(..)
HelpFormatters:
- RawFormatter = RawDescriptionHelpFormatter
- DefaultsFormatter = ArgumentDefaultsHelpFormatter
- DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter
API Documentation
~~~~~~~~~~~~~~~~~
You can review the generated API Documentation for the ``configargparse`` module: `HERE `_
Design Notes
~~~~~~~~~~~~
Unit tests:
tests/test_configargparse.py contains custom unittests for features
specific to this module (such as config file and env-var support), as
well as a hook to load and run argparse unittests (see the built-in
test.test_argparse module) but on configargparse in place of argparse.
This ensures that configargparse will work as a drop in replacement for
argparse in all usecases.
Previously existing modules (PyPI search keywords: config argparse):
- argparse (built-in module Python v2.7+)
- Good:
- fully featured command line parsing
- can read args from files using an easy to understand mechanism
- Bad:
- syntax for specifying config file path is unusual (eg.
@file.txt)and not described in the user help message.
- default config file syntax doesn't support comments and is
unintuitive (eg. --namevalue)
- no support for environment variables
- ConfArgParse v1.0.15
(https://pypi.python.org/pypi/ConfArgParse)
- Good:
- extends argparse with support for config files parsed by
ConfigParser
- clear documentation in README
- Bad:
- config file values are processed using
ArgumentParser.set_defaults(..) which means "required" and
"choices" are not handled as expected. For example, if you
specify a required value in a config file, you still have to
specify it again on the command line.
- doesn't work with Python 3 yet
- no unit tests, code not well documented
- appsettings v0.5 (https://pypi.python.org/pypi/appsettings)
- Good:
- supports config file (yaml format) and env_var parsing
- supports config-file-only setting for specifying lists and
dicts
- Bad:
- passes in config file and env settings via parse_args
namespace param
- tests not finished and don't work with Python 3 (import
StringIO)
- argparse_config v0.5.1
(https://pypi.python.org/pypi/argparse_config)
- Good:
- similar features to ConfArgParse v1.0.15
- Bad:
- doesn't work with Python 3 (error during pip install)
- yconf v0.3.2 - (https://pypi.python.org/pypi/yconf) - features
and interface not that great
- hieropt v0.3 - (https://pypi.python.org/pypi/hieropt) - doesn't
appear to be maintained, couldn't find documentation
- configurati v0.2.3 - (https://pypi.python.org/pypi/configurati)
- Good:
- JSON, YAML, or Python configuration files
- handles rich data structures such as dictionaries
- can group configuration names into sections (like .ini files)
- Bad:
- doesn't work with Python 3
- 2+ years since last release to PyPI
- apparently unmaintained
Design choices:
1. all options must be settable via command line. Having options that
can only be set using config files or env. vars adds complexity to
the API, and is not a useful enough feature since the developer can
split up options into sections and call a section "config file keys",
with command line args that are just "--" plus the config key.
2. config file and env. var settings should be processed by appending
them to the command line (another benefit of #1). This is an
easy-to-implement solution and implicitly takes care of checking that
all "required" args are provied, etc., plus the behavior should be
easy for users to understand.
3. configargparse shouldn't override argparse's
convert_arg_line_to_args method so that all argparse unit tests
can be run on configargparse.
4. in terms of what to allow for config file keys, the "dest" value of
an option can't serve as a valid config key because many options can
have the same dest. Instead, since multiple options can't use the
same long arg (eg. "--long-arg-x"), let the config key be either
"--long-arg-x" or "long-arg-x". This means the developer can allow
only a subset of the command-line args to be specified via config
file (eg. short args like -x would be excluded). Also, that way
config keys are automatically documented whenever the command line
args are documented in the help message.
5. don't force users to put config file settings in the right .ini
[sections]. This doesn't have a clear benefit since all options are
command-line settable, and so have a globally unique key anyway.
Enforcing sections just makes things harder for the user and adds
complexity to the implementation.
6. if necessary, config-file-only args can be added later by
implementing a separate add method and using the namespace arg as in
appsettings_v0.5
Relevant sites:
- http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll
- http://tricksntweaks.blogspot.com/2013_05_01_archive.html
- http://www.youtube.com/watch?v=vvCwqHgZJc8#t=35
.. |Travis CI Status for bw2/ConfigArgParse| image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
Versioning
~~~~~~~~~~
This software follows `Semantic Versioning`_
.. _Semantic Versioning: http://semver.org/
Keywords: options,argparse,ConfigArgParse,config,environment variables,envvars,ENV,environment,optparse,YAML,INI
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*
Provides-Extra: test
Provides-Extra: yaml
ConfigArgParse-1.5.3/ConfigArgParse.egg-info/SOURCES.txt 0000644 0000000 0000024 00000000470 14126173212 022646 0 ustar root staff 0000000 0000000 LICENSE
MANIFEST.in
README.rst
configargparse.py
setup.py
ConfigArgParse.egg-info/PKG-INFO
ConfigArgParse.egg-info/SOURCES.txt
ConfigArgParse.egg-info/dependency_links.txt
ConfigArgParse.egg-info/requires.txt
ConfigArgParse.egg-info/top_level.txt
tests/__init__.py
tests/__init__.pyc
tests/test_configargparse.py ConfigArgParse-1.5.3/ConfigArgParse.egg-info/requires.txt 0000644 0000000 0000024 00000000052 14126173212 023356 0 ustar root staff 0000000 0000000
[test]
mock
PyYAML
pytest
[yaml]
PyYAML
ConfigArgParse-1.5.3/ConfigArgParse.egg-info/top_level.txt 0000644 0000000 0000024 00000000017 14126173212 023511 0 ustar root staff 0000000 0000000 configargparse
ConfigArgParse-1.5.3/ConfigArgParse.egg-info/dependency_links.txt 0000644 0000000 0000024 00000000001 14126173212 025027 0 ustar root staff 0000000 0000000
ConfigArgParse-1.5.3/setup.cfg 0000644 0000000 0000024 00000000046 14126173212 016256 0 ustar root staff 0000000 0000000 [egg_info]
tag_build =
tag_date = 0
ConfigArgParse-1.5.3/README.rst 0000644 0000000 0000024 00000041603 14103070316 016124 0 ustar root staff 0000000 0000000 ConfigArgParse
--------------
.. image:: https://img.shields.io/pypi/v/ConfigArgParse.svg?style=flat
:alt: PyPI version
:target: https://pypi.python.org/pypi/ConfigArgParse
.. image:: https://img.shields.io/pypi/pyversions/ConfigArgParse.svg
:alt: Supported Python versions
:target: https://pypi.python.org/pypi/ConfigArgParse
.. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
:alt: Travis CI build
:target: https://travis-ci.org/bw2/ConfigArgParse
.. image:: https://img.shields.io/badge/-API_Documentation-blue
:alt: API Documentation
:target: https://bw2.github.io/ConfigArgParse/
Overview
~~~~~~~~
Applications with more than a handful of user-settable options are best
configured through a combination of command line args, config files,
hard-coded defaults, and in some cases, environment variables.
Python's command line parsing modules such as argparse have very limited
support for config files and environment variables, so this module
extends argparse to add these features.
Available on PyPI: http://pypi.python.org/pypi/ConfigArgParse
.. image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
:target: https://travis-ci.org/bw2/ConfigArgParse
Features
~~~~~~~~
- command-line, config file, env var, and default settings can now be
defined, documented, and parsed in one go using a single API (if a
value is specified in more than one way then: command line >
environment variables > config file values > defaults)
- config files can have .ini or .yaml style syntax (eg. key=value or
key: value)
- user can provide a config file via a normal-looking command line arg
(eg. -c path/to/config.txt) rather than the argparse-style @config.txt
- one or more default config file paths can be specified
(eg. ['/etc/bla.conf', '~/.my_config'] )
- all argparse functionality is fully supported, so this module can
serve as a drop-in replacement (verified by argparse unittests).
- env vars and config file keys & syntax are automatically documented
in the -h help message
- new method :code:`print_values()` can report keys & values and where
they were set (eg. command line, env var, config file, or default).
- lite-weight (no 3rd-party library dependencies except (optionally) PyYAML)
- extensible (:code:`ConfigFileParser` can be subclassed to define a new
config file format)
- unittested by running the unittests that came with argparse but on
configargparse, and using tox to test with Python 2.7 and Python 3+
Example
~~~~~~~
*config_test.py*:
Script that defines 4 options and a positional arg and then parses and prints the values. Also,
it prints out the help message as well as the string produced by :code:`format_values()` to show
what they look like.
.. code:: py
import configargparse
p = configargparse.ArgParser(default_config_files=['/etc/app/conf.d/*.conf', '~/.my_settings'])
p.add('-c', '--my-config', required=True, is_config_file=True, help='config file path')
p.add('--genome', required=True, help='path to genome file') # this option can be set in a config file because it starts with '--'
p.add('-v', help='verbose', action='store_true')
p.add('-d', '--dbsnp', help='known variants .vcf', env_var='DBSNP_PATH') # this option can be set in a config file because it starts with '--'
p.add('vcf', nargs='+', help='variant file(s)')
options = p.parse_args()
print(options)
print("----------")
print(p.format_help())
print("----------")
print(p.format_values()) # useful for logging where different settings came from
*config.txt:*
Since the script above set the config file as required=True, lets create a config file to give it:
.. code:: py
# settings for config_test.py
genome = HCMV # cytomegalovirus genome
dbsnp = /data/dbsnp/variants.vcf
*command line:*
Now run the script and pass it the config file:
.. code:: bash
DBSNP_PATH=/data/dbsnp/variants_v2.vcf python config_test.py --my-config config.txt f1.vcf f2.vcf
*output:*
Here is the result:
.. code:: bash
Namespace(dbsnp='/data/dbsnp/variants_v2.vcf', genome='HCMV', my_config='config.txt', v=False, vcf=['f1.vcf', 'f2.vcf'])
----------
usage: config_test.py [-h] -c MY_CONFIG --genome GENOME [-v] [-d DBSNP]
vcf [vcf ...]
Args that start with '--' (eg. --genome) can also be set in a config file
(/etc/app/conf.d/*.conf or ~/.my_settings or specified via -c). Config file
syntax allows: key=value, flag=true, stuff=[a,b,c] (for details, see syntax at
https://goo.gl/R74nmi). If an arg is specified in more than one place, then
commandline values override environment variables which override config file
values which override defaults.
positional arguments:
vcf variant file(s)
optional arguments:
-h, --help show this help message and exit
-c MY_CONFIG, --my-config MY_CONFIG
config file path
--genome GENOME path to genome file
-v verbose
-d DBSNP, --dbsnp DBSNP
known variants .vcf [env var: DBSNP_PATH]
----------
Command Line Args: --my-config config.txt f1.vcf f2.vcf
Environment Variables:
DBSNP_PATH: /data/dbsnp/variants_v2.vcf
Config File (config.txt):
genome: HCMV
Special Values
~~~~~~~~~~~~~~
Under the hood, configargparse handles environment variables and config file
values by converting them to their corresponding command line arg. For
example, "key = value" will be processed as if "--key value" was specified
on the command line.
Also, the following special values (whether in a config file or an environment
variable) are handled in a special way to support booleans and lists:
- :code:`key = true` is handled as if "--key" was specified on the command line.
In your python code this key must be defined as a boolean flag
(eg. action="store_true" or similar).
- :code:`key = [value1, value2, ...]` is handled as if "--key value1 --key value2"
etc. was specified on the command line. In your python code this key must
be defined as a list (eg. action="append").
Config File Syntax
~~~~~~~~~~~~~~~~~~
Only command line args that have a long version (eg. one that starts with '--')
can be set in a config file. For example, "--color" can be set by putting
"color=green" in a config file. The config file syntax depends on the constuctor
arg: :code:`config_file_parser_class` which can be set to one of the provided
classes: :code:`DefaultConfigFileParser`, :code:`YAMLConfigFileParser`,
:code:`ConfigparserConfigFileParser` or to your own subclass of the
:code:`ConfigFileParser` abstract class.
*DefaultConfigFileParser* - the full range of valid syntax is:
.. code:: yaml
# this is a comment
; this is also a comment (.ini style)
--- # lines that start with --- are ignored (yaml style)
-------------------
[section] # .ini-style section names are treated as comments
# how to specify a key-value pair (all of these are equivalent):
name value # key is case sensitive: "Name" isn't "name"
name = value # (.ini style) (white space is ignored, so name = value same as name=value)
name: value # (yaml style)
--name value # (argparse style)
# how to set a flag arg (eg. arg which has action="store_true")
--name
name
name = True # "True" and "true" are the same
# how to specify a list arg (eg. arg which has action="append")
fruit = [apple, orange, lemon]
indexes = [1, 12, 35 , 40]
*YAMLConfigFileParser* - allows a subset of YAML syntax (http://goo.gl/VgT2DU)
.. code:: yaml
# a comment
name1: value
name2: true # "True" and "true" are the same
fruit: [apple, orange, lemon]
indexes: [1, 12, 35, 40]
*ConfigparserConfigFileParser* - allows a subset of python's configparser
module syntax (https://docs.python.org/3.7/library/configparser.html). In
particular the following configparser options are set:
.. code:: py
config = configparser.ArgParser(
delimiters=("=",":"),
allow_no_value=False,
comment_prefixes=("#",";"),
inline_comment_prefixes=("#",";"),
strict=True,
empty_lines_in_values=False,
)
Once configparser parses the config file all section names are removed, thus all
keys must have unique names regardless of which INI section they are defined
under. Also, any keys which have python list syntax are converted to lists by
evaluating them as python code using ast.literal_eval
(https://docs.python.org/3/library/ast.html#ast.literal_eval). To facilitate
this all multi-line values are converted to single-line values. Thus multi-line
string values will have all new-lines converted to spaces. Note, since key-value
pairs that have python dictionary syntax are saved as single-line strings, even
if formatted across multiple lines in the config file, dictionaries can be read
in and converted to valid python dictionaries with PyYAML's safe_load. Example
given below:
.. code:: py
# inside your config file (e.g. config.ini)
[section1] # INI sections treated as comments
system1_settings: { # start of multi-line dictionary
'a':True,
'b':[2, 4, 8, 16],
'c':{'start':0, 'stop':1000},
'd':'experiment 32 testing simulation with parameter a on'
} # end of multi-line dictionary value
.......
# in your configargparse setup
import configargparse
import yaml
parser = configargparse.ArgParser(
config_file_parser_class=configargparse.ConfigparserConfigFileParser
)
parser.add_argument('--system1_settings', type=yaml.safe_load)
args = parser.parse_args() # now args.system1 is a valid python dict
ArgParser Singletons
~~~~~~~~~~~~~~~~~~~~~~~~~
To make it easier to configure different modules in an application,
configargparse provides globally-available ArgumentParser instances
via configargparse.get_argument_parser('name') (similar to
logging.getLogger('name')).
Here is an example of an application with a utils module that also
defines and retrieves its own command-line args.
*main.py*
.. code:: py
import configargparse
import utils
p = configargparse.get_argument_parser()
p.add_argument("-x", help="Main module setting")
p.add_argument("--m-setting", help="Main module setting")
options = p.parse_known_args() # using p.parse_args() here may raise errors.
*utils.py*
.. code:: py
import configargparse
p = configargparse.get_argument_parser()
p.add_argument("--utils-setting", help="Config-file-settable option for utils")
if __name__ == "__main__":
options = p.parse_known_args()
Help Formatters
~~~~~~~~~~~~~~~
:code:`ArgumentDefaultsRawHelpFormatter` is a new HelpFormatter that both adds
default values AND disables line-wrapping. It can be passed to the constructor:
:code:`ArgParser(.., formatter_class=ArgumentDefaultsRawHelpFormatter)`
Aliases
~~~~~~~
The configargparse.ArgumentParser API inherits its class and method
names from argparse and also provides the following shorter names for
convenience:
- p = configargparse.get_arg_parser() # get global singleton instance
- p = configargparse.get_parser()
- p = configargparse.ArgParser() # create a new instance
- p = configargparse.Parser()
- p.add_arg(..)
- p.add(..)
- options = p.parse(..)
HelpFormatters:
- RawFormatter = RawDescriptionHelpFormatter
- DefaultsFormatter = ArgumentDefaultsHelpFormatter
- DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter
API Documentation
~~~~~~~~~~~~~~~~~
You can review the generated API Documentation for the ``configargparse`` module: `HERE `_
Design Notes
~~~~~~~~~~~~
Unit tests:
tests/test_configargparse.py contains custom unittests for features
specific to this module (such as config file and env-var support), as
well as a hook to load and run argparse unittests (see the built-in
test.test_argparse module) but on configargparse in place of argparse.
This ensures that configargparse will work as a drop in replacement for
argparse in all usecases.
Previously existing modules (PyPI search keywords: config argparse):
- argparse (built-in module Python v2.7+)
- Good:
- fully featured command line parsing
- can read args from files using an easy to understand mechanism
- Bad:
- syntax for specifying config file path is unusual (eg.
@file.txt)and not described in the user help message.
- default config file syntax doesn't support comments and is
unintuitive (eg. --namevalue)
- no support for environment variables
- ConfArgParse v1.0.15
(https://pypi.python.org/pypi/ConfArgParse)
- Good:
- extends argparse with support for config files parsed by
ConfigParser
- clear documentation in README
- Bad:
- config file values are processed using
ArgumentParser.set_defaults(..) which means "required" and
"choices" are not handled as expected. For example, if you
specify a required value in a config file, you still have to
specify it again on the command line.
- doesn't work with Python 3 yet
- no unit tests, code not well documented
- appsettings v0.5 (https://pypi.python.org/pypi/appsettings)
- Good:
- supports config file (yaml format) and env_var parsing
- supports config-file-only setting for specifying lists and
dicts
- Bad:
- passes in config file and env settings via parse_args
namespace param
- tests not finished and don't work with Python 3 (import
StringIO)
- argparse_config v0.5.1
(https://pypi.python.org/pypi/argparse_config)
- Good:
- similar features to ConfArgParse v1.0.15
- Bad:
- doesn't work with Python 3 (error during pip install)
- yconf v0.3.2 - (https://pypi.python.org/pypi/yconf) - features
and interface not that great
- hieropt v0.3 - (https://pypi.python.org/pypi/hieropt) - doesn't
appear to be maintained, couldn't find documentation
- configurati v0.2.3 - (https://pypi.python.org/pypi/configurati)
- Good:
- JSON, YAML, or Python configuration files
- handles rich data structures such as dictionaries
- can group configuration names into sections (like .ini files)
- Bad:
- doesn't work with Python 3
- 2+ years since last release to PyPI
- apparently unmaintained
Design choices:
1. all options must be settable via command line. Having options that
can only be set using config files or env. vars adds complexity to
the API, and is not a useful enough feature since the developer can
split up options into sections and call a section "config file keys",
with command line args that are just "--" plus the config key.
2. config file and env. var settings should be processed by appending
them to the command line (another benefit of #1). This is an
easy-to-implement solution and implicitly takes care of checking that
all "required" args are provied, etc., plus the behavior should be
easy for users to understand.
3. configargparse shouldn't override argparse's
convert_arg_line_to_args method so that all argparse unit tests
can be run on configargparse.
4. in terms of what to allow for config file keys, the "dest" value of
an option can't serve as a valid config key because many options can
have the same dest. Instead, since multiple options can't use the
same long arg (eg. "--long-arg-x"), let the config key be either
"--long-arg-x" or "long-arg-x". This means the developer can allow
only a subset of the command-line args to be specified via config
file (eg. short args like -x would be excluded). Also, that way
config keys are automatically documented whenever the command line
args are documented in the help message.
5. don't force users to put config file settings in the right .ini
[sections]. This doesn't have a clear benefit since all options are
command-line settable, and so have a globally unique key anyway.
Enforcing sections just makes things harder for the user and adds
complexity to the implementation.
6. if necessary, config-file-only args can be added later by
implementing a separate add method and using the namespace arg as in
appsettings_v0.5
Relevant sites:
- http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll
- http://tricksntweaks.blogspot.com/2013_05_01_archive.html
- http://www.youtube.com/watch?v=vvCwqHgZJc8#t=35
.. |Travis CI Status for bw2/ConfigArgParse| image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master
Versioning
~~~~~~~~~~
This software follows `Semantic Versioning`_
.. _Semantic Versioning: http://semver.org/