pax_global_header00006660000000000000000000000064141532636120014515gustar00rootroot0000000000000052 comment=5002045740734ccc61e796698480a94f7a1383ff plac-1.3.4/000077500000000000000000000000001415326361200124415ustar00rootroot00000000000000plac-1.3.4/.github/000077500000000000000000000000001415326361200140015ustar00rootroot00000000000000plac-1.3.4/.github/workflows/000077500000000000000000000000001415326361200160365ustar00rootroot00000000000000plac-1.3.4/.github/workflows/python-package.yml000066400000000000000000000023201415326361200214700ustar00rootroot00000000000000# This workflow will install Python dependencies, run tests and lint with a variety of Python versions # For more information see: https://docs.github.com/actions/automating-builds-and-tests/building-and-testing-python name: Test plac on: push: branches: [ master ] pull_request: branches: [ master ] jobs: build: runs-on: ubuntu-latest strategy: matrix: python-version: ['2.7', '3.5', '3.6', '3.7', '3.8'] steps: - uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - name: Install dependencies run: | python -m pip install --upgrade pip python -m pip install flake8 python -m pip install -e . - name: Lint with flake8 run: | # stop the build if there are Python syntax errors or undefined names flake8 *.py --count --select=E9,F63,F7,F82 --show-source --statistics # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide flake8 *.py --count --exit-zero --max-line-length=127 --statistics - name: Tests run: | python doc/test_plac.py plac-1.3.4/.gitignore000066400000000000000000000000311415326361200144230ustar00rootroot00000000000000doc/conf.shelve.db docs/ plac-1.3.4/.travis.yml000066400000000000000000000002441415326361200145520ustar00rootroot00000000000000language: python dist: bionic python: - "2.7" - "3.5" - "3.6" - "3.7" - "3.8" - "3.9" - "3.10" install: - python setup.py install script: pytest -v doc plac-1.3.4/CHANGES.md000066400000000000000000000135221415326361200140360ustar00rootroot00000000000000HISTORY ------- ## [Unreleased] ## 1.3.0 (200-12-27) Thanks to Istvan Albert, it is now possible to use language keywords and builtins as option/flag names. Some broken links were fixed and the documentation has been moved to https://plac.readthedocs.io, while the CI framework has changed from Travis to GitHub actions. ## 1.2.0 (2020-06-05) Added dedenting of usage docstrings, as requested by Istvan Albert. Added new decorators `plac.pos`, `plac.opt`, `plac.flg` and an example using them in a section "For the impatient". Added tests on travis for Python 3.8. ## 1.1.3 (2018-10-27) Fixed some issues with kwargs parsing, docstring formatting and empty string defaults reported by the user https://github.com/isaacto. Changed the testing framework on travis from nosetest to pytest. Ported the documentation to sphinx. ## 1.1.0 (2018-07-28) Extended the recognition of default types to date and datetime in ISO format. Fixed a bug when running plac scripts from Jupyter notebooks, signaled by https://github.com/ursachi and https://github.com/rkpatel33. Moreover, at user request, removed a Python 3.7 deprecation warning, added a LICENSE.txt file and a Quickstart section to the README. plac is tested on Travis for Python 2.7 and 3.4+ but it should work also for all the other 3.X releases. ## 1.0.0 (2018-08-03) New feature, requested by John Didion: if the type of an argument is not specified but there is a default value, it is inferred from it. This is experimental and works only for Python literal types. Fixed a bug caused by arguments with default None in newer versions of argparse. Added a `gh-pages` branch with the documentation, as suggested by Ryan Gonzalez. Extended the Travis testing to Python 3.6. Python 2.6 still works but it is untested and therefore deprecated. ## 0.9.6 (2016-07-09) Solved an issue with non-ASCII characters; now any UTF-8 character can go in the help message. Added support for `--version` in plac.call. Modernized the changelog https://keepachangelog.com/ ## 0.9.5 (2016-06-09) Removed a usage of `print >>` that was breaking Python 3, signaled by Quentin Pradet ## 0.9.4 (2016-06-09) Removed use_2to3 in setup.py which was breaking Python 2, signaled by Quentin Pradet ## 0.9.3 (2016-06-07) Fixed the tests on Python 3 and produced a universal wheel instead of relying on 2to3. Enabled Travis builds for Python 3.3, 3.4, 3.5 ## 0.9.2 (2016-06-07) Moved the repository from GoogleCode to GitHub. Included the doc fixes by Nicola Larosa and polished the code base to be PEP 8 compliant. Enabled Travis builds for Python 2.6 and 2.7 ## 0.9.1 (2012-04-23) Options and flags can now contain dashes (i.e. ``--dry-run`` is valid and translated into dry_run, you are not forced to use ``--dry-run`` anymore); restored the monitor support temporarily removed in 0.9.0, fixed an issue with tuple defaults and fixed the display of the help command; specified which features are experimental and which features are fully supported ## 0.9.0 (2011-06-19) Default values are now displayed in the help message by default; removed .help and introduced help; removed the special dotted commands from the usage message; added an ``Interpreter.Exit`` exception; removed the experimental monitor framework because it is too much platform-dependent; added a reference to Argh; now plac has its own space on Google Code ## 0.8.1 (2011-04-11) Removed a stray newline in the output of plac, as signaled by Daniele Pighin; fixed a bug in the doctest method raising non-existing exceptions; turned the notification messages into unicode strings; removed an ugly SystemExit message for invalid commands, signaled by Tuk Bredsdorff ## 0.8.0 (2011-02-16) Added a monitor framework and a TkMonitor ## 0.7.6 (2011-01-13) Fixed the error propagation in ``Interpreter.__exit__``. Added a note about commandline and marrow.script in the documentation ## 0.7.5 (2011-01-01) Fixed a bug with the help of subcommands, signaled by Paul Jacobson; added the ability to save the output of a command into a file; postponed the import of the readline module to avoid buffering issues; fixed a bug with the traceback when in multiprocessing mode ## 0.7.4 (2010-09-04) Fixed the plac_runner switches -i and -s; fixed a bug with multiline output and issue with nosetest ## 0.7.3 (2010-08-31) Put the documentation in a single document; added runp ## 0.7.2 (2010-08-11) Interpreter.call does not start an interpreter automagically anymore; better documented and added tests for the metavar concept (2010-08-31) ## 0.7.1 (2010-08-11) A few bug fixes ## 0.7.0 (2010-08-07) Improved and documented the support for parallel programming; added an asynchronous server; added plac.Interpreter.call ## 0.6.1 (2010-07-12) Fixed the history file location; added the ability to pass a split function; added two forgotten files; added a reference to cmd2 by Catherine Devlin ## 0.6.0 (2010-07-11) Improved the interactive experience with full readline support and custom help. Added support for long running command, via threads and processes ## 0.5.0 (2010-06-20) Gigantic release. Introduced smart options, added an Interpreter class and the command container concept. Made the split plac/plac_core/plac_ext and added a plac runner, able to run scripts, batch files and doctests. Removed the default formatter class ## 0.4.3 (2010-06-11) Fixed the installation procedure to automatically download argparse if needed ## 0.4.2 (2010-06-04) Added missing .help files, made the tests generative and added a note about Clap in the documentation ## 0.4.1 (2010-06-03) Changed the default formatter class and fixed a bug in the display of the default arguments. Added more stringent tests. ## 0.4.0 (2010-06-03) abbrev is now optional. Added a note about CLIArgs and opterate. Added keyword arguments recognition. ``plac.call`` now returns the the output of the main function. ## 0.3.0 (2010-06-02) First released version. plac-1.3.4/LICENSE.txt000066400000000000000000000024541415326361200142710ustar00rootroot00000000000000Copyright (c) 2010-2021, Michele Simionato, Istvan Albert All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in bytecode form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. plac-1.3.4/MANIFEST.in000066400000000000000000000001321415326361200141730ustar00rootroot00000000000000include *.md *.rst doc/*.py doc/*.help doc/*.txt doc/*.html doc/*.pdf include LICENSE.txt plac-1.3.4/Makefile000066400000000000000000000005171415326361200141040ustar00rootroot00000000000000.PHONY: \ default \ dist \ upload \ test \ clean default: sphinx-build doc docs dist: plac_core.py plac_ext.py python setup.py build sdist bdist_wheel upload: rm -rf build/* dist/* python setup.py sdist bdist_wheel python -m twine upload --repository pypi dist/* test: python doc/test_plac.py clean: rm -rf docs/ plac-1.3.4/README.md000066400000000000000000000122131415326361200137170ustar00rootroot00000000000000# Plac: Parsing the Command Line the Easy Way `plac` is a Python package that can generate command line parameters from function signatures. `plac` works on Python 2.6 through all versions of Python 3. `plac` has no dependencies beyond modules already present in the Python standard library. `plac` implements most of its functionality in a single file that may be included in your source code. # Quickstart Here is how to turn a script that does some processing on a database table into a full, command-line enabled program: ```python # updatedb.py from datetime import datetime def main(dsn, table='product', today=datetime.today()): "Do something on the database" print(dsn, table, today) if __name__ == '__main__': import plac plac.call(main) ``` Here is the help message automatically generated by plac: ``` python updatedb.py -h ``` prints: ``` usage: updatedb.py [-h] dsn [table] [today] Do something on the database positional arguments: dsn table [product] today [2019-07-28 07:18:20.054708] optional arguments: -h, --help show this help message and exit ``` # Next steps The automatic inference takes us only so far, usually we need more control over the parameters. `plac` offers simple decorator helpers for positional, option and flag type parameters: ```python import plac from pathlib import Path @plac.pos('model', "Model name", choices=['A', 'B', 'C']) @plac.opt('output_dir', "Optional output directory", type=Path) @plac.opt('n_iter', "Number of training iterations", type=int) @plac.flg('debug', "Enable debug mode") def main(model, output_dir='.', n_iter=100, debug=False): """A script for machine learning""" pass if __name__ == '__main__': plac.call(main) ``` Running the script with `$ python example.py -h` will give you the following help message: : ``` usage: example.py [-h] [-o .] [-n 100] [-d] {A,B,C} A script for machine learning positional arguments: {A,B,C} Model name optional arguments: -h, --help show this help message and exit -o ., --output-dir . Optional output directory -n 100, --n-iter 100 Number of training iterations -d, --debug Enable debug mode ``` # Quick reference The following decorator reference helps you recall what parameters are valid for each decorator type: ```python # Positional parameters. def pos(arg, help=None, type=None, choices=None, metavar=None): # Option parameters. def opt(arg, help=None, type=None, abbrev=None, choices=None, metavar=None): # Flag parameters. def flg(arg, help=None, abbrev=None): ``` Notably, the main functionality of `plac` is implemented in a single module called `plac_core.py` that, if necessary, may be included and distributed with your source code thus reducing external dependencies in your code. # Avoiding name clashes Python syntax, or your variable naming may impose constraints on what words may be used as parameters. To circumvent that limitation append a trailing underscore to the name. `plac` will strip that underscore from the command line parameter name: ```python import plac @plac.flg('list_') # avoid clash with builtin @plac.flg('yield_') # avoid clash with keyword @plac.opt('sys_') # avoid clash with a very common name def main(list_, yield_=False, sys_=100): print(list_) print(yield_) print(sys_) if __name__ == '__main__': plac.call(main) ``` produces the usage: ``` usage: example13.py [-h] [-l] [-y] [-s 100] optional arguments: -h, --help show this help message and exit -l, --list -y, --yield [False] -s 100, --sys 100 [100] ``` # Variable arguments Your `plac` enabled program may accept multiple positional arguments and even additional key=value pairs: ```python import plac @plac.pos('args', help="words") @plac.opt('kwds', help="key=value", ) def main(*args, **kwds): print(args) print(kwds) if __name__ == '__main__': plac.call(main) ``` the usage will be: ``` usage: example15.py [-h] [args ...] [kwds ...] positional arguments: args words kwds key=value optional arguments: -h, --help show this help message and exit ``` when running it as: python example15.py A B x=10 y=20 the program prints: ('A', 'B') {'x': '10', 'y': '20'} # Documentation In addition, plac can do a lot more, up to the creation of domain-specific languages(!). See the full documentation for more details. - # Installation If you wish to install the package do pip install plac If you prefer to install the full distribution from source, including the documentation, download the [tarball](https://pypi.org/project/plac/#files), unpack it and run python setup.py install # Testing Run python doc/test_plac.py You will see several apparent errors, but this is right, since the tests are checking for several error conditions. The important thing is that you get at the a line like `Executed XX tests OK` # Code - Author: Michele Simionato, Maintainer: Istvan Albert, # Issues - # License BSD License plac-1.3.4/RELEASE.md000066400000000000000000000005161415326361200140450ustar00rootroot00000000000000How to make a new release ========================= 1. Update the changelog (CHANGES.md) 2. Update the version number in plac.py and doc/plac_core.rst 3. Build the docs with `make` 4. Make a tag on the repo (i.e. git tag plac-1.3.0), commit and push 5. Make a source tarball with `make dist` and upload to PyPI with `make upload` plac-1.3.4/doc/000077500000000000000000000000001415326361200132065ustar00rootroot00000000000000plac-1.3.4/doc/annotations.py000066400000000000000000000004371415326361200161210ustar00rootroot00000000000000# annotations.py class Positional(object): def __init__(self, help='', type=None, choices=None, metavar=None): self.help = help self.kind = 'positional' self.abbrev = None self.type = type self.choices = choices self.metavar = metavar plac-1.3.4/doc/conf.py000066400000000000000000000035601415326361200145110ustar00rootroot00000000000000# Configuration file for the Sphinx documentation builder. # # This file only contains a selection of the most common options. For a full # list see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html # -- Path setup -------------------------------------------------------------- # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # # import os # import sys # sys.path.insert(0, os.path.abspath('.')) # -- Project information ----------------------------------------------------- project = 'plac' copyright = '2010-2021, Michele Simionato' author = 'Michele Simionato' # -- General configuration --------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ 'sphinx.ext.autosectionlabel', ] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # html_theme = 'alabaster' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] plac-1.3.4/doc/dry_run.py000066400000000000000000000003041415326361200152370ustar00rootroot00000000000000def main(dry_run: ('Dry run', 'flag', 'd')): if dry_run: print('Doing nothing') else: print('Doing something') if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example1.py000066400000000000000000000005321415326361200152740ustar00rootroot00000000000000# example1.py def main(dsn): "Do something with the database" print("ok") if __name__ == '__main__': import sys n = len(sys.argv[1:]) if n == 0: sys.exit('usage: python %s dsn' % sys.argv[0]) elif n == 1: main(sys.argv[1]) else: sys.exit('Unrecognized arguments: %s' % ' '.join(sys.argv[2:])) plac-1.3.4/doc/example10.help000066400000000000000000000003511415326361200156530ustar00rootroot00000000000000usage: example10.py [-h] {add,mul} [n ...] A script to add and multiply numbers positional arguments: {add,mul} The name of an operator n Zero or more numbers options: -h, --help show this help message and exit plac-1.3.4/doc/example10.py000066400000000000000000000012131415326361200153510ustar00rootroot00000000000000# example10.py import plac # example with full annotations (help, kind, abbrev, type, choices, metavar) @plac.annotations( operator=("The name of an operator", 'positional', None, str, ['add', 'mul']), numbers=("Zero or more numbers", 'positional', None, float, None, 'n')) def main(operator, *numbers): "A script to add and multiply numbers" if operator == 'mul': op = float.__mul__ result = 1.0 else: # operator == 'add' op = float.__add__ result = 0.0 for n in numbers: result = op(result, n) return result if __name__ == '__main__': print(plac.call(main)) plac-1.3.4/doc/example11.help000066400000000000000000000003201415326361200156500ustar00rootroot00000000000000usage: example11.py [-h] i n [rest ...] positional arguments: i This is an int n This is a float rest Other arguments options: -h, --help show this help message and exit plac-1.3.4/doc/example11.py000066400000000000000000000004711415326361200153570ustar00rootroot00000000000000# example11.py import plac from annotations import Positional @plac.annotations( i=Positional("This is an int", int), n=Positional("This is a float", float), rest=Positional("Other arguments")) def main(i, n, *rest): print(i, n, rest) if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example12.help000066400000000000000000000003411415326361200156540ustar00rootroot00000000000000usage: example12.py [-h] [-opt OPT] [args ...] [kw ...] positional arguments: args default arguments kw keyword arguments options: -h, --help show this help message and exit -opt OPT some option plac-1.3.4/doc/example12.py000066400000000000000000000005671415326361200153660ustar00rootroot00000000000000# example12.py import plac @plac.annotations( opt=('some option', 'option'), args='default arguments', kw='keyword arguments') def main(opt, *args, **kw): if opt: yield 'opt=%s' % opt if args: yield 'args=%s' % str(args) if kw: yield 'kw=%s' % kw if __name__ == '__main__': for output in plac.call(main): print(output) plac-1.3.4/doc/example13.help000066400000000000000000000002601415326361200156550ustar00rootroot00000000000000usage: example13.py [-h] [-l] [-y] [-s 100] options: -h, --help show this help message and exit -l, --list -y, --yield [False] -s 100, --sys 100 [100] plac-1.3.4/doc/example13.py000066400000000000000000000005021415326361200153540ustar00rootroot00000000000000# example13.py import plac @plac.flg('list_') # avoid clash with builtin @plac.flg('yield_') # avoid clash with keyword @plac.opt('sys_') # avoid clash with a very common name def main(list_, yield_=False, sys_=100): print(list_) print(yield_) print(sys_) if __name__ == '__main__': plac.call(main) plac-1.3.4/doc/example14.help000066400000000000000000000002161415326361200156570ustar00rootroot00000000000000usage: example14.py [-h] [words ...] positional arguments: words Input words options: -h, --help show this help message and exit plac-1.3.4/doc/example14.py000066400000000000000000000003051415326361200153560ustar00rootroot00000000000000# example14.py # Tests choices on variable number of arguments import plac @plac.pos('words', help="Input words") def main(*words): print(words) if __name__ == '__main__': plac.call(main)plac-1.3.4/doc/example15.help000066400000000000000000000002521415326361200156600ustar00rootroot00000000000000usage: example15.py [-h] [args ...] [kwds ...] positional arguments: args words kwds key=value options: -h, --help show this help message and exit plac-1.3.4/doc/example15.py000066400000000000000000000004011415326361200153540ustar00rootroot00000000000000# example15.py # Tests choices on variable number of option arguments import plac @plac.pos('args', help="words") @plac.opt('kwds', help="key=value", ) def main(*args, **kwds): print(args) print(kwds) if __name__ == '__main__': plac.call(main)plac-1.3.4/doc/example16.help000066400000000000000000000002061415326361200156600ustar00rootroot00000000000000usage: example16.py [-h] a positional arguments: a options: -h, --help show this help message and exit plac-1.3.4/doc/example16.py000066400000000000000000000001661415326361200153650ustar00rootroot00000000000000import plac def main(a: str): return p = plac.parser_from(main) if __name__ == '__main__': plac.call(main) plac-1.3.4/doc/example2.py000066400000000000000000000003601415326361200152740ustar00rootroot00000000000000# example2.py def main(dsn): "Do something on the database" print(dsn) # ... if __name__ == '__main__': import argparse p = argparse.ArgumentParser() p.add_argument('dsn') arg = p.parse_args() main(arg.dsn) plac-1.3.4/doc/example3.help000066400000000000000000000002211415326361200155710ustar00rootroot00000000000000usage: example3.py [-h] dsn Do something with the database positional arguments: dsn options: -h, --help show this help message and exit plac-1.3.4/doc/example3.py000066400000000000000000000002301415326361200152710ustar00rootroot00000000000000# example3.py def main(dsn): "Do something with the database" print(dsn) # ... if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example4.py000066400000000000000000000007001415326361200152740ustar00rootroot00000000000000# example4.py from datetime import datetime def main(dsn, table='product', today=datetime.today()): "Do something on the database" print(dsn, table, today) if __name__ == '__main__': # manual management before argparse import sys args = sys.argv[1:] if not args: sys.exit('usage: python %s dsn' % sys.argv[0]) elif len(args) > 2: sys.exit('Unrecognized arguments: %s' % ' '.join(argv[2:])) main(*args) plac-1.3.4/doc/example5.help000066400000000000000000000003221415326361200155750ustar00rootroot00000000000000usage: example5.py [-h] dsn [table] [today] Do something on the database positional arguments: dsn table [product] today [YYYY-MM-DD] options: -h, --help show this help message and exit plac-1.3.4/doc/example5.py000066400000000000000000000003321415326361200152760ustar00rootroot00000000000000# example5.py from datetime import date def main(dsn, table='product', today=date.today()): "Do something on the database" print(dsn, table, today) if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example5_.py000066400000000000000000000004361415326361200154420ustar00rootroot00000000000000# example5_.py from datetime import date # the first example with a function annotation def main(dsn: "the database dsn", table='product', today=date.today()): "Do something on the database" print(dsn, table, today) if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example6.help000066400000000000000000000003331415326361200156000ustar00rootroot00000000000000usage: example6.py [-h] [-command select * from table] dsn positional arguments: dsn options: -h, --help show this help message and exit -command select * from table SQL query plac-1.3.4/doc/example6.py000066400000000000000000000003031415326361200152750ustar00rootroot00000000000000# example6.py def main(dsn, command: ("SQL query", 'option')='select * from table'): print('executing %r on %s' % (command, dsn)) if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example7.help000066400000000000000000000002601415326361200156000ustar00rootroot00000000000000usage: example7.py [-h] dsn [scripts ...] Run the given scripts on the database positional arguments: dsn scripts options: -h, --help show this help message and exit plac-1.3.4/doc/example7.py000066400000000000000000000003771415326361200153110ustar00rootroot00000000000000# example7.py from datetime import datetime def main(dsn, *scripts): "Run the given scripts on the database" for script in scripts: print('executing %s' % script) # ... if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example7_.help000066400000000000000000000003261415326361200157420ustar00rootroot00000000000000usage: example7_.py [-h] dsn [scripts ...] Run the given scripts on the database positional arguments: dsn Database dsn scripts SQL scripts options: -h, --help show this help message and exit plac-1.3.4/doc/example7_.py000066400000000000000000000004371415326361200154450ustar00rootroot00000000000000# example7_.py from datetime import datetime def main(dsn: "Database dsn", *scripts: "SQL scripts"): "Run the given scripts on the database" for script in scripts: print('executing %s' % script) # ... if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example8.help000066400000000000000000000003121415326361200155770ustar00rootroot00000000000000usage: example8.py [-h] [-c COMMAND] dsn positional arguments: dsn options: -h, --help show this help message and exit -c COMMAND, --command COMMAND SQL query plac-1.3.4/doc/example8.py000066400000000000000000000003241415326361200153020ustar00rootroot00000000000000# example8.py def main(command: ("SQL query", 'option', 'c'), dsn): if command: print('executing %s on %s' % (command, dsn)) # ... if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example8_.help000066400000000000000000000003571415326361200157470ustar00rootroot00000000000000usage: example8_.py [-h] [-c select * from table] dsn positional arguments: dsn options: -h, --help show this help message and exit -c select * from table, --command select * from table SQL query plac-1.3.4/doc/example8_.py000066400000000000000000000003111415326361200154350ustar00rootroot00000000000000# example8_.py def main(dsn, command: ("SQL query", 'option', 'c')='select * from table'): print('executing %r on %s' % (command, dsn)) if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example9.help000066400000000000000000000002701415326361200156030ustar00rootroot00000000000000usage: example9.py [-h] [-v] dsn positional arguments: dsn connection string options: -h, --help show this help message and exit -v, --verbose prints more info plac-1.3.4/doc/example9.py000066400000000000000000000003361415326361200153060ustar00rootroot00000000000000# example9.py def main(verbose: ('prints more info', 'flag', 'v'), dsn: 'connection string'): if verbose: print('connecting to %s' % dsn) # ... if __name__ == '__main__': import plac; plac.call(main) plac-1.3.4/doc/example_all.help000066400000000000000000000005451415326361200163470ustar00rootroot00000000000000usage: example_all.py [-h] [-o .] [-n 100] [-d] {A,B,C} A script for machine learning positional arguments: {A,B,C} Model name options: -h, --help show this help message and exit -o ., --output-dir . Optional output directory -n 100, --n-iter 100 Number of training iterations -d, --debug Enable debug mode plac-1.3.4/doc/example_all.py000066400000000000000000000007301415326361200160430ustar00rootroot00000000000000import plac try: from pathlib import Path except ImportError: # in Python 2.7 Path = str @plac.pos('model', "Model name", choices=['A', 'B', 'C']) @plac.opt('output_dir', "Optional output directory", type=Path) @plac.opt('n_iter', "Number of training iterations", type=int) @plac.flg('debug', "Enable debug mode") def main(model, output_dir='.', n_iter=100, debug=False): """A script for machine learning""" if __name__ == '__main__': plac.call(main) plac-1.3.4/doc/importer1.py000066400000000000000000000010511415326361200154770ustar00rootroot00000000000000import time import plac class FakeImporter(object): "A fake importer with an import_file command" commands = ['import_file'] def __init__(self, dsn): self.dsn = dsn def import_file(self, fname): "Import a file into the database" try: for n in range(10000): time.sleep(.01) if n % 100 == 99: yield 'Imported %d lines' % (n+1) finally: print('closing the file') if __name__ == '__main__': plac.Interpreter.call(FakeImporter) plac-1.3.4/doc/importer2.py000066400000000000000000000012671415326361200155110ustar00rootroot00000000000000import time import plac class FakeImporter(object): "A fake importer with an import_file command" thcommands = ['import_file'] def __init__(self, dsn): self.dsn = dsn def import_file(self, fname): "Import a file into the database" try: for n in range(10000): time.sleep(.02) if n % 100 == 99: # every two seconds yield 'Imported %d lines' % (n+1) if n % 10 == 9: # every 0.2 seconds yield # go back and check the TOBEKILLED status finally: print('closing the file') if __name__ == '__main__': plac.Interpreter.call(FakeImporter) plac-1.3.4/doc/importer3.py000066400000000000000000000010531415326361200155030ustar00rootroot00000000000000import time import plac class FakeImporter(object): "A fake importer with an import_file command" mpcommands = ['import_file'] def __init__(self, dsn): self.dsn = dsn def import_file(self, fname): "Import a file into the database" try: for n in range(10000): time.sleep(.02) if n % 100 == 99: yield 'Imported %d lines' % (n+1) finally: print('closing the file') if __name__ == '__main__': plac.Interpreter.call(FakeImporter) plac-1.3.4/doc/importer_ui.py000066400000000000000000000015251415326361200161210ustar00rootroot00000000000000from __future__ import with_statement from Tkinter import * from importer3 import FakeImporter def taskwidget(root, task, tick=500): "A Label widget showing the output of a task every 500 ms" sv = StringVar(root) lb = Label(root, textvariable=sv) def show_outlist(): try: out = task.outlist[-1] except IndexError: # no output yet out = '' sv.set('%s %s' % (task, out)) root.after(tick, show_outlist) root.after(0, show_outlist) return lb def monitor(tasks): root = Tk() for task in tasks: task.run() taskwidget(root, task).pack() root.mainloop() if __name__ == '__main__': import plac with plac.Interpreter(plac.call(FakeImporter)) as i: tasks = [i.submit('import_file f1'), i.submit('import_file f2')] monitor(tasks) plac-1.3.4/doc/index.rst000066400000000000000000000000651415326361200150500ustar00rootroot00000000000000.. include:: plac_core.rst .. include:: plac_adv.rst plac-1.3.4/doc/ishelve.help000066400000000000000000000011001415326361200155070ustar00rootroot00000000000000usage: ishelve.py [.help] [.showall] [.clear] [.delete DELETE] [.filename conf.shelve] [params ...] [setters ...] Simple interface to a shelve positional arguments: params names of the parameters in the shelve setters setters param=value optional arguments: .help show help .showall show all parameters in the shelve .clear clear the shelve .delete DELETE delete an element .filename conf.shelve filename of the shelve plac-1.3.4/doc/ishelve.plac000066400000000000000000000000651415326361200155070ustar00rootroot00000000000000#!ishelve.py .clear a=1 b=2 .show .del a .dl b .show plac-1.3.4/doc/ishelve.placet000066400000000000000000000002601415326361200160350ustar00rootroot00000000000000#!ishelve.py i> .clear # start from a clean state cleared the shelve i> a=1 setting a=1 i> a 1 i> .del a deleted a i> a a: not found i> .cler # spelling error .cler: not found plac-1.3.4/doc/ishelve.py000066400000000000000000000035441415326361200152250ustar00rootroot00000000000000# ishelve.py import os import shelve import plac DEFAULT_SHELVE = 'conf.shelve' @plac.annotations( help=('show help', 'flag'), showall=('show all parameters in the shelve', 'flag'), clear=('clear the shelve', 'flag'), delete=('delete an element', 'option'), filename=('filename of the shelve', 'option'), params='names of the parameters in the shelve', setters='setters param=value') def main(help, showall, clear, delete, filename=DEFAULT_SHELVE, *params, **setters): "A simple interface to a shelve. Use .help to see the available commands." sh = shelve.open(filename) try: if not any([help, showall, clear, delete, params, setters]): yield ('no arguments passed, use .help to see the ' 'available commands') elif help: # custom help yield 'Commands: .help, .showall, .clear, .delete' yield ' ...' yield ' ...' elif showall: for param, name in sh.items(): yield '%s=%s' % (param, name) elif clear: sh.clear() yield 'cleared the shelve' elif delete: try: del sh[delete] except KeyError: yield '%s: not found' % delete else: yield 'deleted %s' % delete for param in params: try: yield sh[param] except KeyError: yield '%s: not found' % param for param, value in setters.items(): sh[param] = value yield 'setting %s=%s' % (param, value) finally: sh.close() main.add_help = False # there is a custom help, remove the default one main.prefix_chars = '.' # use dot-prefixed commands if __name__ == '__main__': for output in plac.call(main): print(output) plac-1.3.4/doc/ishelve2.hel000066400000000000000000000003621415326361200154220ustar00rootroot00000000000000usage: ishelve2.py [-h] [-configfile CONFIGFILE] A minimal interface over a shelve object. optional arguments: -h, --help show this help message and exit -configfile CONFIGFILE path name of the shelve plac-1.3.4/doc/ishelve2.plac000066400000000000000000000001251415326361200155660ustar00rootroot00000000000000#!ishelve2.py:ShelveInterface -c conf.shelve set a 1 del a del a # intentional error plac-1.3.4/doc/ishelve2.placet000066400000000000000000000003061415326361200161200ustar00rootroot00000000000000#!ishelve2.py:ShelveInterface -configfile=test.shelve # an example of a .placet file for the ShelveInterface i> del deleting everything i> set a 1 setting a=1 i> set b 2 setting b=2 i> show a a = 1 plac-1.3.4/doc/ishelve2.py000066400000000000000000000027211415326361200153030ustar00rootroot00000000000000# ishelve2.py import os import shelve import plac class ShelveInterface(object): "A minimal interface over a shelve object." commands = 'set', 'show', 'showall', 'delete' @plac.annotations( configfile=('path name of the shelve', 'option')) def __init__(self, configfile): self.configfile = configfile or 'conf.shelve' self.fname = os.path.expanduser(self.configfile) self.__doc__ += ('\nOperating on %s.\nUse help to see ' 'the available commands.\n' % self.fname) def __enter__(self): self.sh = shelve.open(self.fname) return self def __exit__(self, etype, exc, tb): self.sh.close() def set(self, name, value): "set name value" yield 'setting %s=%s' % (name, value) self.sh[name] = value def show(self, *names): "show given parameters" for name in names: yield '%s = %s' % (name, self.sh[name]) # no error checking def showall(self): "show all parameters" for name in self.sh: yield '%s = %s' % (name, self.sh[name]) def delete(self, name=''): "delete given parameter (or everything)" if name == '': yield 'deleting everything' self.sh.clear() else: yield 'deleting %s' % name del self.sh[name] # no error checking if __name__ == '__main__': plac.Interpreter(plac.call(ShelveInterface)).interact() plac-1.3.4/doc/ishelve3.py000066400000000000000000000003721415326361200153040ustar00rootroot00000000000000# ishelve3.py from ishelve2 import ShelveInterface if __name__ == '__main__': import plac; plac.Interpreter.call(ShelveInterface) ## try the following: # $ python ishelve3.py delete # $ python ishelve3.py set a 1 # $ python ishelve3.py showall plac-1.3.4/doc/picalculator.py000066400000000000000000000036721415326361200162520ustar00rootroot00000000000000# -*- coding: utf-8 -*- from __future__ import with_statement from __future__ import division import math from random import random import multiprocessing import plac class PiCalculator(object): """Compute \u03C0 in parallel with threads or processes""" @plac.annotations( npoints=('number of integration points', 'positional', None, int), mode=('sequential|parallel|threaded', 'option', 'm', str, 'SPT')) def __init__(self, npoints, mode='S'): self.npoints = npoints if mode == 'P': self.mpcommands = ['calc_pi'] elif mode == 'T': self.thcommands = ['calc_pi'] elif mode == 'S': self.commands = ['calc_pi'] self.n_cpu = multiprocessing.cpu_count() def submit_tasks(self): npoints = math.ceil(self.npoints / self.n_cpu) self.i = plac.Interpreter(self).__enter__() return [self.i.submit('calc_pi %d' % npoints) for _ in range(self.n_cpu)] def close(self): self.i.close() @plac.annotations(npoints=('npoints', 'positional', None, int)) def calc_pi(self, npoints): counts = 0 for j in range(npoints): n, r = divmod(j, 1000000) if r == 0: yield '%dM iterations' % n x, y = random(), random() if x*x + y*y < 1: counts += 1 yield (4.0 * counts) / npoints def run(self): tasks = self.i.tasks() for t in tasks: t.run() try: total = 0 for task in tasks: total += task.result except: # the task was killed print(tasks) return return total / self.n_cpu if __name__ == '__main__': pc = plac.call(PiCalculator) pc.submit_tasks() try: import time t0 = time.time() print('%f in %f seconds ' % (pc.run(), time.time() - t0)) finally: pc.close() plac-1.3.4/doc/plac.el000066400000000000000000000050141415326361200144470ustar00rootroot00000000000000;;; Emacs-plac integration: add the following to your .emacs (define-generic-mode 'plac-mode '("#") ; comment chars '(); highlighted commands nil '(".plac\\'"); file extensions nil) (add-hook 'plac-mode-hook (lambda () (local-set-key [f4] 'plac-start))) (add-hook 'plac-mode-hook (lambda () (local-set-key [f5] 'plac-send))) (add-hook 'plac-mode-hook (lambda () (local-set-key [f6] 'plac-stop))) (defconst terminator 59); ASCII code for the semicolon (defvar *plac-process* nil) (defun plac-start () "Start an inferior plac process by inferring the script to use from the shebang line" (interactive) (let ((shebang-line (save-excursion (goto-line 1) (end-of-line) (buffer-substring-no-properties 3 (point))))) (if *plac-process* (princ "plac already started") (setq *plac-process* (start-process "plac" "*plac*" "plac_runner.py" "-m" shebang-line)))) (display-buffer "*plac*")) ;(defun plac-send () ; "Send the current region to the inferior plac process" ; (interactive) ; (save-excursion (set-buffer "*plac*") (erase-buffer)) ; (process-send-region *plac-process* (region-beginning) (region-end))) (defun current-paragraph-beg-end () "Returns the extrema of the current paragraph, delimited by semicolons" (interactive) (save-excursion (let ((beg (save-excursion (goto-line 2) (point))); skip the shebang (end (point-max))) ;; go backward (while (> (point) beg) (goto-char (1- (point))) (if (= terminator (following-char)) (setq beg (point)))) (if (= terminator (following-char)) (setq beg (1+ beg))) ;; go forward (while (< (point) end) (goto-char (1+ (point))) (if (= 59 (following-char)) (setq end (point)))) (if (= 59 (following-char)) (setq end (1+ end))) (list beg end)))) (defun plac-send () "Send the current region to the inferior plac process" (interactive) (save-excursion (set-buffer "*plac*") (erase-buffer)) (let ((p (apply 'buffer-substring-no-properties (current-paragraph-beg-end)))) (message p) (process-send-string *plac-process* (concat p "\n")))) ;(switch-to-buffer-other-window "*plac*"))) ;(save-excursion (set-buffer "*plac*") ; (set-window-start (selected-window) 1 nil)))) (defun plac-stop () "Stop the inferior plac process by sending to it an EOF" (interactive) (process-send-eof *plac-process*) (setq *plac-process* nil) "killed *plac-process*") (provide 'plac) plac-1.3.4/doc/plac_adv.rst000066400000000000000000001277721415326361200155310ustar00rootroot00000000000000Advanced usages of plac ======================= Introduction ------------ One of the design goals of plac_ is to make it dead easy to write a scriptable and testable interface for an application. You can use plac_ whenever you have an API with strings in input and strings in output, and that includes a *huge* domain of applications. A string-oriented interface is a scriptable interface by construction. That means that you can define a command language for your application and that it is possible to write scripts which are interpretable by plac_ and can be run as batch scripts. Actually, at the most general level, you can see plac_ as a generic tool to write domain specific languages (DSL). With plac_ you can test your application interactively as well as with batch scripts, and even with the analogous of Python doctests for your defined language. You can easily replace the ``cmd`` module of the standard library and you could easily write an application like twill_ with plac_. Or you could use it to script your building procedure. plac_ also supports parallel execution of multiple commands and can be used as task manager. It is also quite easy to build a GUI or a Web application on top of plac_. When speaking of things you can do with plac_, your imagination is the only limit! From scripts to interactive applications ---------------------------------------- Command-line scripts have many advantages, but they are no substitute for interactive applications. In particular, if you have a script with a large startup time which must be run multiple times, it is best to turn it into an interactive application, so that the startup is performed only once. ``plac`` provides an ``Interpreter`` class just for this purpose. The ``Interpreter`` class wraps the main function of a script and provides an ``.interact`` method to start an interactive interpreter reading commands from the console. The ``.interact`` method reads commands from the console and send them to the underlying interpreter, until the user send a CTRL-D command (CTRL-Z in Windows). There is a default argument ``prompt='i> '`` which can be used to change the prompt. The text displayed at the beginning of the interactive session is the docstring of the main function. ``plac`` also understands command abbreviations: in this example ``del`` is an abbreviation for ``delete``. In case of ambiguous abbreviations plac_ raises a ``NameError``. Finally I must notice that ``plac.Interpreter`` is available only if you are using a recent version of Python (>= 2.5), because it is a context manager object which uses extended generators internally. Testing a plac application -------------------------- You can conveniently test your application in interactive mode. However manual testing is a poor substitute for automatic testing. In principle, one could write automatic tests for the ``ishelve`` application by using ``plac.call`` directly: .. include:: test_ishelve.py :literal: However, using ``plac.call`` is not especially nice. The big issue is that ``plac.call`` responds to invalid input by printing an error message on stderr and by raising a ``SystemExit``: this is certainly not a nice thing to do in a test. As a consequence of this behavior it is impossible to test for invalid commands, unless you wrap the ``SystemExit`` exception by hand each time (and possibly you do something with the error message in stderr too). Luckily, ``plac`` offers a better testing support through the ``check`` method of ``Interpreter`` objects: .. include:: test_ishelve_more.py :literal: The method ``.check(given_input, expected_output)`` works on strings and raises an ``AssertionError`` if the output produced by the interpreter is different from the expected output for the given input. Notice that ``AssertionError`` is caught by tools like ``pytest`` and ``nosetests`` and actually ``plac`` tests are intended to be run with such tools. Interpreters offer a minor syntactic advantage with respect to calling ``plac.call`` directly, but they offer a *major* semantic advantage when things go wrong (read exceptions): an ``Interpreter`` object internally invokes something like ``plac.call``, but it wraps all exceptions, so that ``i.check`` is guaranteed not to raise any exception except ``AssertionError``. Even the ``SystemExit`` exception is captured and you can write your test as ``i.check('-cler', 'SystemExit: unrecognized arguments: -cler')`` without risk of exiting from the Python interpreter. There is a second advantage of interpreters: if the main function contains some initialization code and finalization code (``__enter__`` and ``__exit__`` functions) they will be run at the beginning and at the end of the interpreter loop, whereas ``plac.call`` ignores the initialization/finalization code. Plac easy tests --------------- Writing your tests in terms of ``Interpreter.check`` is certainly an improvement over writing them in terms of ``plac.call``, but they are still too low-level for my taste. The ``Interpreter`` class provides support for doctest-style tests, a.k.a. *plac easy tests*. By using plac easy tests you can cut and paste your interactive session and turn it into a runnable automatics test. Consider for instance the following file ``ishelve.placet`` (the ``.placet`` extension is a mnemonic for "plac easy tests"): .. include:: ishelve.placet :literal: Notice the presence of the shebang line containing the name of the plac_ tool to test (a plac_ tool is just a Python module with a function called ``main``). The shebang is ignored by the interpreter (it looks like a comment to it) but it is there so that external tools (say a test runner) can infer the plac interpreter to use to test the file. You can run the ``ishelve.placet`` file by calling the ``.doctest`` method of the interpreter, as in this example:: $ python -c "import plac, ishelve plac.Interpreter(ishelve.main).doctest(open('ishelve.placet'), verbose=True)" Internally ``Interpreter.doctests`` invokes things like ``Interpreter.check`` multiple times inside the same context and compares the output with the expected output: if even one check fails, the whole test fails. You should realize that the easy tests supported by ``plac`` are *not* unittests: they are functional tests. They model the user interaction and the order of the operations generally matters. The single subtests in a ``.placet`` file are not independent and it makes sense to exit immediately at the first failure. The support for doctests in plac_ comes nearly for free, thanks to the shlex_ module in the standard library, which is able to parse simple languages as the ones you can implement with plac_. In particular, thanks to shlex_, plac_ is able to recognize comments (the default comment character is ``#``), escape sequences and more. Look at the shlex_ documentation if you need to customize how the language is interpreted. For more flexibility, it is even possible to pass the interpreter a custom split function with signature ``split(line, commentchar)``. In addition, I have implemented some support for line number recognition, so that if a test fails you get the line number of the failing command. This is especially useful if your tests are stored in external files, though they do not need to be in a file: you can just pass to the ``.doctest`` method a list of strings corresponding to the lines of the file. At the present plac_ does not use any code from the doctest module, but the situation may change in the future (it would be nice if plac_ could reuse doctests directives like ELLIPSIS). It is straightforward to integrate your ``.placet`` tests with standard testing tools. For instance, you can integrate your doctests with ``nose`` or ``py.test`` as follow:: import os, shlex, plac def test_doct(): """ Find all the doctests in the current directory and run them with the corresponding plac interpreter (the shebang rules!) """ placets = [f for f in os.listdir('.') if f.endswith('.placet')] for placet in placets: lines = list(open(placet)) assert lines[0].startswith('#!'), 'Missing or incorrect shebang line!' firstline = lines[0][2:] # strip the shebang main = plac.import_main(*shlex.split(firstline)) yield plac.Interpreter(main).doctest, lines[1:] Here you should notice that usage of ``plac.import_main``, a utility which is able to import the main function of the script specified in the shebang line. You can use both the full path name of the tool, or a relative path name. In this case the runner looks at the environment variable ``PLACPATH`` and it searches the plac tool in the directories specified there (``PLACPATH`` is just a string containing directory names separated by colons). If the variable ``PLACPATH`` is not defined, it just looks in the current directory. If the plac tool is not found, an ``ImportError`` is raised. Plac batch scripts ------------------ It is pretty easy to realize that an interactive interpreter can also be used to run batch scripts: instead of reading the commands from the console, it is enough to read the commands from a file. plac_ interpreters provide an ``.execute`` method to perform just that. There is just a subtle point to notice: whereas in an interactive loop one wants to manage all exceptions, a batch script should not continue in the background in case of unexpected errors. The implementation of ``Interpreter.execute`` makes sure that any error raised by ``plac.call`` internally is re-raised. In other words, plac_ interpreters *wrap the errors, but does not eat them*: the errors are always accessible and can be re-raised on demand. The exception is the case of invalid commands, which are skipped. Consider for instance the following batch file, which contains a misspelled command (``.dl`` instead of ``.del``): .. include:: ishelve.plac :literal: If you execute the batch file, the interpreter will print a ``.dl: not found`` at the ``.dl`` line and will continue:: $ python -c "import plac, ishelve plac.Interpreter(ishelve.main).execute(open('ishelve.plac'), verbose=True)" i> .clear cleared the shelve i> a=1 b=2 setting a=1 setting b=2 i> .show b=2 a=1 i> .del a deleted a i> .dl b 2 .dl: not found i> .show b=2 The ``verbose`` flag is there to show the lines which are being interpreted (prefixed by ``i>``). This is done on purpose, so that you can cut and paste the output of the batch script and turn it into a ``.placet`` test (cool, isn't it?). Implementing subcommands ------------------------ When I discussed the ``ishelve`` implementation, I said that it looked like the poor man implementation of an object system as a chain of elifs; I also said that plac_ was able to do much better than that. Here I will substantiate my claim. plac_ is actually able to infer a set of subparsers from a generic container of commands. This is useful if you want to implement *subcommands* (a familiar example of a command-line application featuring subcommands is version control system). \ Technically a container of commands is any object with a ``.commands`` attribute listing a set of functions or methods which are valid commands. A command container may have initialization/finalization hooks (``__enter__/__exit__``) and dispatch hooks (``__missing__``, invoked for invalid command names). Moreover, only when using command containers is plac_ able to provide automatic *autocompletion* of commands. The shelve interface can be rewritten in an object-oriented way as follows: .. include:: ishelve2.py :literal: ``plac.Interpreter`` objects wrap context manager objects consistently. In other words, if you wrap an object with ``__enter__`` and ``__exit__`` methods, they are invoked in the right order (``__enter__`` before the interpreter loop starts and ``__exit__`` after the interpreter loop ends, both in the regular and in the exceptional case). In our example, the methods ``__enter__`` and ``__exit__`` make sure the the shelve is opened and closed correctly even in the case of exceptions. Notice that I have not implemented any error checking in the ``show`` and ``delete`` methods on purpose, to verify that plac_ works correctly in the presence of exceptions. When working with command containers, plac_ automatically adds two special commands to the set of provided commands: ``help`` and ``.last_tb``. The ``help`` command is the easier to understand: when invoked without arguments it displays the list of available commands with the same formatting of the cmd_ module; when invoked with the name of a command it displays the usage message for that command. The ``.last_tb`` command is useful when debugging: in case of errors, it allows you to display the traceback of the last executed command. Here is the usage message: .. include:: ishelve2.hel :literal: Here is a session of usage on a Unix-like operating system:: $ python ishelve2.py -c test.shelve A minimal interface over a shelve object. Operating on test.shelve. Use help to see the available commands. i> help special commands ================ .last_tb custom commands =============== delete set show showall i> delete deleting everything i> set a pippo setting a=pippo i> set b lippo setting b=lippo i> showall b = lippo a = pippo i> show a b a = pippo b = lippo i> del a deleting a i> showall b = lippo i> delete a deleting a KeyError: 'a' i> .last_tb File "/usr/local/lib/python2.6/dist-packages/plac-0.6.0-py2.6.egg/plac_ext.py", line 190, in _wrap for value in genobj: File "./ishelve2.py", line 37, in delete del self.sh[name] # no error checking File "/usr/lib/python2.6/shelve.py", line 136, in __delitem__ del self.dict[key] i> Notice that in interactive mode the traceback is hidden, unless you pass the ``verbose`` flag to the ``Interpreter.interact`` method. CHANGED IN VERSION 0.9: if you have an old version of plac_ the ``help`` command must be prefixed with a dot, i.e. you must write ``.help``. The old behavior was more consistent in my opinion, since it made it clear that the ``help`` command was special and threated differently from the regular commands. Notice that if you implement a custom ``help`` command in the commander class the default help will not be added, as you would expect. In version 0.9 an exception ```plac.Interpreter.Exit`` was added. Its purpose is to make it easy to define commands to exit from the command loop. Just define something like:: def quit(self): raise plac.Interpreter.Exit and the interpreter will be closed properly when the ``quit`` command is entered. plac.Interpreter.call --------------------- At the core of ``plac`` there is the ``call`` function which invokes a callable with the list of arguments passed at the command-line (``sys.argv[1:]``). Thanks to ``plac.call`` you can launch your module by simply adding the lines:: if __name__ == '__main__': plac.call(main) Everything works fine if ``main`` is a simple callable performing some action; however, in many cases, one has a ``main`` "function" which is actually a factory returning a command container object. For instance, in my second shelve example the main function is the class ``ShelveInterface``, and the two lines needed to run the module are a bit ugly:: if __name__ == '__main__': plac.Interpreter(plac.call(ShelveInterface)).interact() Moreover, now the program runs, but only in interactive mode, i.e. it is not possible to run it as a script. Instead, it would be nice to be able to specify the command to execute on the command-line and have the interpreter start, execute the command and finish properly (I mean by calling ``__enter__`` and ``__exit__``) without needing user input. Then the script could be called from a batch shell script working in the background. In order to provide such functionality ``plac.Interpreter`` provides a classmethod named ``.call`` which takes the factory, instantiates it with the arguments read from the command line, wraps the resulting container object as an interpreter and runs it with the remaining arguments found in the command line. Here is the code to turn the ``ShelveInterface`` into a script .. include:: ishelve3.py :literal: and here are a few examples of usage:: $ python ishelve3.py help special commands ================ .last_tb custom commands =============== delete set show showall $ python ishelve3.py set a 1 setting a=1 $ python ishelve3.py show a a = 1 If you pass the ``-i`` flag in the command line, then the script will enter in interactive mode and ask the user for the commands to execute:: $ python ishelve3.py -i A minimal interface over a shelve object. Operating on conf.shelve. Use help to see the available commands. i> In a sense, I have closed the circle: at the beginning of this document I discussed how to turn a script into an interactive application (the ``shelve_interpreter.py`` example), whereas here I have show how to turn an interactive application into a script. The complete signature of ``plac.Interpreter.call`` is the following:: call(factory, arglist=sys.argv[1:], commentchar='#', split=shlex.split, stdin=sys.stdin, prompt='i> ', verbose=False) The factory must have a fixed number of positional arguments (no default arguments, no varargs, no kwargs), otherwise a ``TypeError`` is raised: the reason is that we want to be able to distinguish the command-line arguments needed to instantiate the factory from the remaining arguments that must be sent to the corresponding interpreter object. It is also possible to specify a list of arguments different from ``sys.argv[1:]`` (useful in tests), the character to be recognized as a comment, the splitting function, the input source, the prompt to use while in interactive mode, and a verbose flag. Readline support ---------------- Starting from release 0.6 plac_ offers full readline support. That means that if your Python was compiled with readline support you get autocompletion and persistent command history for free. By default all commands autocomplete in a case sensitive way. If you want to add new words to the autocompletion set, or you want to change the location of the ``.history`` file, or to change the case sensitivity, the way to go is to pass a ``plac.ReadlineInput`` object to the interpreter. If the readline library is not available, my suggestion is to use the rlwrap_ tool which provides similar features, at least on Unix-like platforms. plac_ should also work fine on Windows with the pyreadline_ library (I do not use Windows, so this part is very little tested: I tried it only once and it worked, but your mileage may vary). For people worried about licenses, I will notice that plac_ uses the readline library only if available, it does not include it and it does not rely on it in any fundamental way, so that the plac_ licence does not need to be the GPL (actually it is a BSD do-whatever-you-want-with-it licence). The interactive mode of ``plac`` can be used as a replacement of the cmd_ module in the standard library. It is actually better than cmd_: for instance, the ``help`` command is more powerful, since it provides information about the arguments accepted by the given command:: i> help set usage: set name value set name value positional arguments: name value i> help delete usage: delete [name] delete given parameter (or everything) positional arguments: name [None] i> help show usage: show [names ...] show given parameters positional arguments: names As you can imagine, the help message is provided by the underlying argparse_ subparser: there is a subparser for each command. plac_ commands accept options, flags, varargs, keyword arguments, arguments with defaults, arguments with a fixed number of choices, type conversion and all the features provided of argparse_ . Moreover at the moment ``plac`` also understands command abbreviations. However, this feature may disappear in future releases. It was meaningful in the past, when plac_ did not support readline. Notice that if an abbreviation is ambiguous, plac_ warns you:: i> sh NameError: Ambiguous command 'sh': matching ['showall', 'show'] The plac runner --------------- The distribution of plac_ includes a runner script named ``plac_runner.py``, which will be installed in a suitable directory in your system by distutils_ (say in ``/usr/local/bin/plac_runner.py`` in a Unix-like operative system). The runner provides many facilities to run ``.plac`` scripts and ``.placet`` files, as well as Python modules containing a ``main`` object, which can be a function, a command container object or even a command container class. For instance, suppose you want to execute a script containing commands defined in the ``ishelve2`` module like the following one: .. include:: ishelve2.plac :literal: The first line of the ``.plac`` script contains the name of the python module containing the plac interpreter and the arguments which must be passed to its main function in order to be able to instantiate an interpreter object. In this case I appended ``:ShelveInterface`` to the name of the module to specify the object that must be imported: if not specified, by default the object named 'main' is imported. The other lines contains commands. You can run the script as follows:: $ plac_runner.py --batch ishelve2.plac setting a=1 deleting a Traceback (most recent call last): ... _bsddb.DBNotFoundError: (-30988, 'DB_NOTFOUND: No matching key/data pair found') The last command intentionally contained an error, to show that the plac runner does not eat the traceback. The runner can also be used to run Python modules in interactive mode and non-interactive mode. If you put this alias in your bashrc ``alias plac="plac_runner.py"`` (or you define a suitable ``plac.bat`` script in Windows) you can run the ``ishelve2.py`` script in interactive mode as follows:: $ plac -i ishelve2.py:ShelveInterface A minimal interface over a shelve object. Operating on conf.shelve. .help to see the available commands. i> del deleting everything i> set a 1 setting a=1 i> set b 2 setting b=2 i> show b b = 2 Now you can cut and paste the interactive session and turn it into a ``.placet`` file like the following: .. include:: ishelve2.placet :literal: Notice that the first line specifies a test database ``test.shelve``, to avoid clobbering your default shelve. If you misspell the arguments in the first line plac will give you an argparse_ error message (just try). You can run placets following the shebang convention directly with the plac runner:: $ plac --test ishelve2.placet run 1 plac test(s) If you want to see the output of the tests, pass the ``-v/--verbose`` flag. Notice that he runner ignores the extension, so you can actually use any extension your like, but *it relies on the first line of the file to invoke the corresponding plac tool with the given arguments*. The plac runner does not provide any test discovery facility, but you can use standard Unix tools to help. For instance, you can run all the ``.placet`` files into a directory and its subdirectories as follows:: $ find . -name \*.placet | xargs plac_runner.py -t The plac runner expects the main function of your script to return a plac tool, i.e. a function or an object with a ``.commands`` attribute. If this is not the case the runner exits gracefully. It also works in non-interactive mode, if you call it as ``$ plac module.py args ...`` Here is an example:: $ plac ishelve.py a=1 setting a=1 $ plac ishelve.py .show a=1 Notice that in non-interactive mode the runner just invokes ``plac.call`` on the ``main`` object of the Python module. A non class-based example ------------------------- plac_ does not force you to use classes to define command containers. Even a simple function can be a valid command container, it is enough to add a ``.commands`` attribute to it, and possibly ``__enter__`` and/or ``__exit__`` attributes too. In particular, a Python module is a perfect container of commands. As an example, consider the following module implementing a fake Version Control System: .. include:: vcs.py :literal: Notice that I have defined both an ``__exit__`` hook and a ``__missing__`` hook, invoked for non-existing commands. The real trick here is the line ``main = __import__(__name__)``, which define ``main`` to be an alias for the current module. The ``vcs`` module can be run through the plac runner (try ``plac vcs.py -h``): .. include:: vcs.help :literal: You can get help for the subcommands by inserting an ``-h`` after the name of the command:: $ plac vcs.py status -h usage: plac_runner.py vcs.py status [-h] [-q] A fake status command optional arguments: -h, --help show this help message and exit -q, --quiet summary information Notice how the docstring of the command is automatically shown in the usage message, as well as the documentation for the sub flag ``-q``. Here is an example of a non-interactive session:: $ plac vcs.py check url checkout url $ plac vcs.py st -q status True $ plac vcs.py co commit None and here is an interactive session:: $ plac -i vcs.py usage: plac_runner.py vcs.py [-h] {status,commit,checkout} ... i> check url checkout url i> st -q status True i> co commit None i> sto Command 'sto' does not exist i> [CTRL-D] ok Notice the invocation of the ``__missing__`` hook for non-existing commands. Notice also that the ``__exit__`` hook gets called only in interactive mode. If the commands are completely independent, a module is a good fit for a method container. In other situations, it is best to use a custom class. Writing your own plac runner ---------------------------- The runner included in the plac_ distribution is intentionally kept small (around 50 lines of code) so that you can study it and write your own runner if you want to. If you need to go to such level of detail, you should know that the most important method of the ``Interpreter`` class is the ``.send`` method, which takes strings as input and returns a four elements tuple with attributes ``.str``, ``.etype``, ``.exc`` and ``.tb``: - ``.str`` is the output of the command, if successful (a string); - ``.etype`` is the class of the exception, if the command fails; - ``.exc`` is the exception instance; - ``.tb`` is the traceback. Moreover, the ``__str__`` representation of the output object is redefined to return the output string if the command was successful, or the error message (preceded by the name of the exception class) if the command failed. For instance, if you send a misspelled option to the interpreter a ``SystemExit`` will be trapped: >>> import plac >>> from ishelve import ishelve >>> with plac.Interpreter(ishelve) as i: ... print(i.send('.cler')) ... SystemExit: unrecognized arguments: .cler It is important to invoke the ``.send`` method inside the context manager, otherwise you will get a ``RuntimeError``. For instance, suppose you want to implement a graphical runner for a plac-based interpreter with two text widgets: one to enter the commands and one to display the results. Suppose you want to display the errors with tracebacks in red. You will need to code something like that (pseudocode follows):: input_widget = WidgetReadingInput() output_widget = WidgetDisplayingOutput() def send(interpreter, line): out = interpreter.send(line) if out.tb: # there was an error output_widget.display(out.tb, color='red') else: output_widget.display(out.str) main = plac.import_main(tool_path) # get the main object with plac.Interpreter(main) as i: def callback(event): if event.user_pressed_ENTER(): send(i, input_widget.last_line) input_widget.addcallback(callback) gui_mainloop.start() You can adapt the pseudocode to your GUI toolkit of choice and you can also change the file associations in such a way that the graphical user interface starts when clicking on a plac tool file. An example of a GUI program built on top of plac_ is given later on, in the paragraph *Managing the output of concurrent commands* (using Tkinter for simplicity and portability). There is a final *caveat*: since the plac interpreter loop is implemented via extended generators, plac interpreters are single threaded: you will get an error if you ``.send`` commands from separated threads. You can circumvent the problem by using a queue. If EXIT is a sentinel value to signal exiting from the interpreter loop, you can write code like this:: with interpreter: for input_value in iter(input_queue.get, EXIT): output_queue.put(interpreter.send(input_value)) The same trick also works for processes; you could run the interpreter loop in a separate process and send commands to it via the Queue class provided by the multiprocessing_ module. Long running commands --------------------- As we saw, by default a plac_ interpreter blocks until the command terminates. This is an issue, in the sense that it makes the interactive experience quite painful for long running commands. An example is better than a thousand words, so consider the following fake importer: .. include:: importer1.py :literal: If you run the ``import_file`` command, you will have to wait for 200 seconds before entering a new command:: $ python importer1.py dsn -i A fake importer with an import_file command i> import_file file1 ... Imported 100 lines Imported 200 lines Imported 300 lines ... Imported 10000 lines closing the file Being unable to enter any other command is quite annoying: in those situations one would like to run the long running commands in the background, to keep the interface responsive. plac_ provides two ways to reach this goal: threads and processes. Threaded commands ----------------- The most familiar way to execute a task in the background (even if not necessarily the best way) is to run it into a separate thread. In our example it is sufficient to replace the line ``commands = ['import_file']`` with ``thcommands = ['import_file']`` to tell to the plac_ interpreter that the command ``import_file`` should be run into a separated thread. Here is an example session:: i> import_file file1 The import task started in a separated thread. You can see the progress of the task by using the special command ``.output``:: i> .output 1 Imported 100 lines Imported 200 lines If you look after a while, you will get more lines of output:: i> .output 1 Imported 100 lines Imported 200 lines Imported 300 lines Imported 400 lines If you look after a time long enough, the task will be finished:: i> .output 1 It is possible to store the output of a task into a file, to be read later (this is useful for tasks with a large output):: i> .output 1 out.txt saved output of 1 into out.txt You can even skip the number argument: then ``.output`` will the return the output of the last launched command (the special commands like .output do not count). You can launch many tasks one after the other:: i> import_file file2 i> import_file file3 The ``.list`` command displays all the running tasks:: i> .list It is even possible to kill a task:: i> .kill 5 # wait a bit ... closing the file i> .output 5 Note that since at the Python level it is impossible to kill a thread, the ``.kill`` command works by setting the status of the task to ``TOBEKILLED``. Internally the generator corresponding to the command is executed in the thread and the status is checked at each iteration: when the status becomes ``TOBEKILLED``, a ``GeneratorExit`` exception is raised and the thread terminates (softly, so that the ``finally`` clause is honored). In our example the generator is yielding back control once every 100 iterations, i.e. every two seconds (not much). In order to get a responsive interface it is a good idea to yield more often, for instance every 10 iterations (i.e. 5 times per second), as in the following code: .. include:: importer2.py :literal: Running commands as external processes -------------------------------------- Threads are not loved much in the Python world and actually most people prefer to use processes instead. For this reason plac_ provides the option to execute long running commands as external processes. Unfortunately the current implementation only works on Unix-like operating systems (including Mac OS/X) because it relies on fork via the multiprocessing_ module. In our example, to enable the feature it is sufficient to replace the line ``thcommands = ['import_file']`` with ``mpcommands = ['import_file']``. The user experience is exactly the same as with threads and you will not see any difference at the user interface level:: i> import_file file3 i> .kill 1 closing the file i> .output 1 Imported 100 lines Imported 200 lines i> Still, using processes is quite different than using threads: in particular, when using processes you can only yield pickleable values and you cannot re-raise an exception first raised in a different process, because traceback objects are not pickleable. Moreover, you cannot rely on automatic sharing of your objects. On the plus side, when using processes you do not need to worry about killing a command: they are killed immediately using a SIGTERM signal, and there is no ``TOBEKILLED`` mechanism. Moreover, the killing is guaranteed to be soft: internally a command receiving a SIGTERM raises a ``TerminatedProcess`` exception which is trapped in the generator loop, so that the command is closed properly. Using processes allows one to take full advantage of multicore machines and it is safer than using threads, so it is the recommended approach unless you are working on Windows. Managing the output of concurrent commands ------------------------------------------ plac_ acts as a command-line task launcher and can be used as the base to build a GUI-based task launcher and task monitor. To this aim the interpreter class provides a ``.submit`` method which returns a task object and a ``.tasks`` method returning the list of all the tasks submitted to the interpreter. The ``submit`` method does not start the task and thus it is nonblocking. Each task has an ``.outlist`` attribute which is a list storing the value yielded by the generator underlying the task (the ``None`` values are skipped though): the ``.outlist`` grows as the task runs and more values are yielded. Accessing the ``.outlist`` is nonblocking and can be done freely. Finally there is a ``.result`` property which waits for the task to finish and returns the last yielded value or raises an exception. The code below provides an example of how you could implement a GUI over the importer example: .. include:: importer_ui.py :literal: Experimental features ===================== The distribution of plac_ includes a few experimental features which I am not committed to fully support and that may go away in future versions. They are included as examples of things that you may build on top of plac_: the aim is to give you ideas. Some of the experimental features might grow to become external projects built on plac_. Parallel computing with plac ---------------------------- plac_ is certainly not intended as a tool for parallel computing, but still you can use it to launch a set of commands and collect the results, similarly to the MapReduce pattern popularized by Google. In order to give an example, I will consider the "Hello World" of parallel computing, i.e. the computation of pi with independent processes. There is a huge number of algorithms to compute pi; here I will describe a trivial one chosen for simplicity, not for efficiency. The trick is to consider the first quadrant of a circle with radius 1 and to extract a number of points ``(x, y)`` with ``x`` and ``y`` random variables in the interval ``[0,1]``. The probability of extracting a number inside the quadrant (i.e. with ``x^2 + y^2 < 1``) is proportional to the area of the quadrant (i.e. ``pi/4``). The value of ``pi`` therefore can be extracted by multiplying by 4 the ratio between the number of points in the quadrant versus the total number of points ``N``, for ``N`` large:: def calc_pi(N): inside = 0 for j in xrange(N): x, y = random(), random() if x*x + y*y < 1: inside += 1 return (4.0 * inside) / N The algorithm is trivially parallelizable: if you have n CPUs, you can compute pi n times with N/n iterations, sum the results and divide the total by n. I have a Macbook with two cores, therefore I would expect a speedup factor of 2 with respect to a sequential computation. Moreover, I would expect a threaded computation to be even slower than a sequential computation, due to the GIL and the scheduling overhead. Here is a script implementing the algorithm and working in three different modes (parallel mode, threaded mode and sequential mode) depending on a ``mode`` option: .. include:: picalculator.py :literal: Notice the ``submit_tasks`` method, which instantiates and initializes a ``plac.Interpreter`` object and submits a number of commands corresponding to the number of available CPUs. The ``calc_pi`` command yields a log message for each million interactions, in order to monitor the progress of the computation. The ``run`` method starts all the submitted commands in parallel and sums the results. It returns the average value of ``pi`` after the slowest CPU has finished its job (if the CPUs are equal and equally busy they should finish more or less at the same time). Here are the results on my old Macbook with Ubuntu 10.04 and Python 2.6, for 10 million of iterations:: $ python picalculator.py -mP 10000000 # two processes 3.141904 in 5.744545 seconds $ python picalculator.py -mT 10000000 # two threads 3.141272 in 13.875645 seconds $ python picalculator.py -mS 10000000 # sequential 3.141586 in 11.353841 seconds As you see using processes one gets a 2x speedup indeed, where the threaded mode is some 20% slower than the sequential mode. Since the pattern "submit a bunch of tasks, start them and collect the results" is so common, plac_ provides a utility function ``runp(genseq, mode='p')`` to start a bunch of generators and return a list of results. By default ``runp`` use processes, but you can use threads by passing ``mode='t'``. With ``runp`` the parallel pi calculation becomes a one-liner:: sum(task.result for task in plac.runp(calc_pi(N) for i in range(ncpus)))/ncpus The file ``test_runp`` in the ``doc`` directory of the plac distribution shows another usage example. Note that if one of the tasks fails for some reason, you will get the exception object instead of the result. Monitor support --------------- plac_ provides experimental support for monitoring the output of concurrent commands, at least for platforms where multiprocessing is fully supported. You can define your own monitor class, simply by inheriting from ``plac.Monitor`` and overriding the methods ``add_listener(self, taskno)``, ``del_listener(self, taskno)``, ``notify_listener(self, taskno, msg)``, ``read_queue(self)``, ``start(self)`` and ``stop(self)``. Then you can add a monitor object to any ``plac.Interpreter`` object by calling the ``add_monitor`` method. For convenience, ``plac`` comes with a very simple ``TkMonitor`` based on Tkinter (I chose Tkinter because it is easy to use and in the standard library, but you can use any GUI): you can look at how the ``TkMonitor`` is implemented in ``plac_tk.py`` and adapt it. Here is a usage example of the ``TkMonitor``: .. include:: tkmon.py :literal: Try to run the ``hello`` command in the interactive interpreter: each time, a new text widget will be added displaying the output of the command. Note that if ``Tkinter`` is not installed correctly on your system, the ``TkMonitor`` class will not be available. The plac server --------------- A command-line oriented interface can be easily converted into a socket-based interface. Starting from release 0.7 plac_ features a built-in server which is able to accept commands from multiple clients and execute them. The server works by instantiating a separate interpreter for each client, so that if a client interpreter dies for any reason, the other interpreters keep working. To avoid external dependencies the server is based on the ``asynchat`` module in the standard library, but it would not be difficult to replace the server with a different one (for instance, a Twisted server). Notice that at the moment the plac_ server does not work with to Python 3.2+ due to changes to ``asynchat``. In time I will fix this and other known issues. You should consider the server functionality still experimental and subject to changes. Also, notice that since ``asynchat``-based servers are asynchronous, any blocking command in the interpreter should be run in a separated process or thread. The default port for the plac_ server is 2199, and the command to signal end-of-connection is EOF. For instance, here is how you could manage remote import on a database (say an SQLite db): .. include:: server_ex.py :literal: You can connect to the server with ``telnet`` on port 2199, as follows:: $ telnet localhost 2199 Trying ::1... Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. i> import_file f1 i> .list i> .out Imported 100 lines Imported 200 lines i> EOF Connection closed by foreign host. Summary ------- Once plac_ claimed to be the easiest command-line arguments parser in the world. Having read this document you may think that it is not so easy after all. But it is a false impression. Actually the rules are quite simple: 1. if you want to implement a command-line script, use ``plac.call``; 2. if you want to implement a command interpreter, use ``plac.Interpreter``: - for an interactive interpreter, call the ``.interact`` method; - for a batch interpreter, call the ``.execute`` method; 3. for testing call the ``Interpreter.check`` method in the appropriate context or use the ``Interpreter.doctest`` feature; 4. if you need to go to a lower level, you may need to call the ``Interpreter.send`` method which returns a (finished) ``Task`` object; 5. long running commands can be executed in the background as threads or processes: just declare them in the lists ``thcommands`` and ``mpcommands`` respectively; 6. the ``.start_server`` method starts an asynchronous server on the given port number (default 2199). Moreover, remember that ``plac_runner.py`` is your friend. ---- Appendix: custom annotation objects ----------------------------------- Internally plac_ uses an ``Annotation`` class to convert the tuples in the function signature to annotation objects, i.e. objects with six attributes: ``help, kind, short, type, choices, metavar``. Advanced users can implement their own annotation objects. For instance, here is an example of how you could implement annotations for positional arguments: .. include:: annotations.py :literal: You can use such annotation objects as follows: .. include:: example11.py :literal: Here is the usage message you get: .. include:: example11.help :literal: You can go on and define ``Option`` and ``Flag`` classes, if you like. Using custom annotation objects you could do advanced things like extracting the annotations from a configuration file or from a database, but I expect such use cases to be quite rare: the default mechanism should work pretty well for most users. .. _plac: https://pypi.org/project/plac/ .. _argparse: https://docs.python.org/library/argparse.html .. _twill: https://github.com/twill-tools/twill .. _shlex: https://docs.python.org/library/shlex.html .. _multiprocessing: https://docs.python.org/library/multiprocessing.html .. _distutils: https://docs.python.org/distutils/ .. _cmd: https://docs.python.org/library/cmd.html .. _rlwrap: https://github.com/hanslub42/rlwrap .. _pyreadline: https://ipython.org/pyreadline.html plac-1.3.4/doc/plac_core.rst000066400000000000000000000760011415326361200156730ustar00rootroot00000000000000Plac: Parsing the Command Line the Easy Way =========================================== :Author: Michele Simionato :E-mail: michele.simionato@gmail.com :Version: 1.3.0 :Date: December 2020 :Download page: https://pypi.org/project/plac/ :Project page: https://github.com/ialbert/plac :Requires: Python from 2.6 up :Installation: ``pip install plac`` :License: BSD license .. contents:: For the impatient ------------------------------ Here is how you would write a command-line script with plac, taken from a real life machine learning script that I found on the net: .. include:: example_all.py :literal: Running the script with ``$ python example_all.py -h`` will give you the following help message: .. include:: example_all.help :literal: The patient readers will find all the explanations in the sections below. The importance of scaling down ------------------------------ There is no want of command-line arguments parsers in the Python world. The standard library alone contains three different modules: getopt_ (from the stone age), optparse_ (from Python 2.3) and argparse_ (from Python 2.7). All of them are quite powerful and especially argparse_ is an industrial strength solution; unfortunately, all of them feature a non-negligible learning curve and a certain verbosity. They do not scale down well enough, at least in my opinion. It should not be necessary to stress the importance of `scaling down`_; nevertheless, a lot of people are obsessed with features and concerned with the possibility of scaling up, forgetting the equally important issue of scaling down. This is an old meme in the computing world: programs should address the common cases simply and simple things should be kept simple, while at the same time keeping difficult things possible. plac_ adhere as much as possible to this philosophy and it is designed to handle well the simple cases, while retaining the ability to handle complex cases by relying on the underlying power of argparse_. Technically plac_ is just a simple wrapper over argparse_ which hides most of its complexity by using a declarative interface: the argument parser is inferred rather than written down by imperatively. Still, plac_ is surprisingly scalable upwards, even without using the underlying argparse_. I have been using Python for 9 years and in my experience it is extremely unlikely that you will ever need to go beyond the features provided by the declarative interface of plac_: they should be more than enough for 99.9% of the use cases. plac_ is targeting especially unsophisticated users, programmers, sys-admins, scientists and in general people writing throw-away scripts for themselves, choosing the command-line interface because it is quick and simple. Such users are not interested in features, they are interested in a small learning curve: they just want to be able to write a simple command line tool from a simple specification, not to build a command-line parser by hand. Unfortunately, the modules in the standard library forces them to go the hard way. They are designed to implement power user tools and they have a non-trivial learning curve. On the contrary, plac_ is designed to be simple to use and extremely concise, as the examples below will show. Scripts with required arguments ------------------------------- Let me start with the simplest possible thing: a script that takes a single argument and does something to it. It cannot get simpler than that, unless you consider the case of a script without command-line arguments, where there is nothing to parse. Still, it is a use case *extremely common*: I need to write scripts like that nearly every day, I wrote hundreds of them in the last few years and I have never been happy. Here is a typical example of code I have been writing by hand for years: .. include:: example1.py :literal: As you see the whole ``if __name__ == '__main__'`` block (nine lines) is essentially boilerplate that should not exist. Actually I think the language should recognize the main function and pass the command-line arguments automatically; unfortunately this is unlikely to happen. I have been writing boilerplate like this in hundreds of scripts for years, and every time I *hate* it. The purpose of using a scripting language is convenience and trivial things should be trivial. Unfortunately the standard library does not help for this incredibly common use case. Using getopt_ and optparse_ does not help, since they are intended to manage options and not positional arguments; the argparse_ module helps a bit and it is able to reduce the boilerplate from nine lines to six lines: .. include:: example2.py :literal: However, it just feels too complex to instantiate a class and to define a parser by hand for such a trivial task. The plac_ module is designed to manage well such use cases, and it is able to reduce the original nine lines of boiler plate to two lines. With the plac_ module all you need to write is .. include:: example3.py :literal: The plac_ module provides for free (actually the work is done by the underlying argparse_ module) a nice usage message:: $ python example3.py -h .. include:: example3.help :literal: Moreover plac_ manages the case of missing arguments and of too many arguments. This is only the tip of the iceberg: plac_ is able to do much more than that. Scripts with default arguments ------------------------------ The need to have suitable defaults for command-line scripts is quite common. For instance I have encountered this use case at work hundreds of times: .. include:: example4.py :literal: Here I want to perform a query on a database table, by extracting the most recent data: it makes sense for ``today`` to be a default argument. If there is a most used table (in this example a table called ``'product'``) it also makes sense to make it a default argument. Performing the parsing of the command-line arguments by hand takes 8 ugly lines of boilerplate (using argparse_ would require about the same number of lines). With plac_ the entire ``__main__`` block reduces to the usual two lines:: if __name__ == '__main__': import plac; plac.call(main) In other words, six lines of boilerplate have been removed, and we get the usage message for free: .. include:: example5.help :literal: Notice that by default plac_ prints the string representation of the default values (with square brackets) in the usage message. plac_ manages transparently even the case when you want to pass a variable number of arguments. Here is an example, a script running on a database a series of SQL scripts: .. include:: example7.py :literal: Here is the usage message: .. include:: example7.help :literal: The examples here should have made clear that *plac is able to figure out the command-line arguments parser to use from the signature of the main function*. This is the whole idea behind plac_: if the intent is clear, let's the machine take care of the details. plac_ is inspired to an old Python Cookbook recipe of mine (optionparse_), in the sense that it delivers the programmer from the burden of writing the parser, but is less of a hack: instead of extracting the parser from the docstring of the module, it extracts it from the signature of the ``main`` function. The idea comes from the `function annotations` concept, a new feature of Python 3. An example is worth a thousand words, so here it is: .. include:: example7_.py :literal: Here the arguments of the ``main`` function have been annotated with strings which are intended to be used in the help message: .. include:: example7_.help :literal: plac_ is able to recognize much more complex annotations, as I will show in the next paragraphs. Scripts with options (and smart options) ---------------------------------------- It is surprising how few command-line scripts with options I have written over the years (probably less than a hundred), compared to the number of scripts with positional arguments I wrote (certainly more than a thousand of them). Still, this use case cannot be neglected. The standard library modules (all of them) are quite verbose when it comes to specifying the options and frankly I have never used them directly. Instead, I have always relied on the optionparse_ recipe, which provides a convenient wrapper over argparse_. Alternatively, in the simplest cases, I have just performed the parsing by hand. In plac_ the parser is inferred by the function annotations. Here is an example: .. include:: example8.py :literal: Here the argument ``command`` has been annotated with the tuple ``("SQL query", 'option', 'c')``: the first string is the help string which will appear in the usage message, the second string tells plac_ that ``command`` is an option and the third string that there is also a short form of the option ``-c``, the long form being ``--command``. The usage message is the following: .. include:: example8.help :literal: Here are two examples of usage:: $ python3 example8.py -c "select * from table" dsn executing select * from table on dsn $ python3 example8.py --command="select * from table" dsn executing select * from table on dsn The third argument in the function annotation can be omitted: in such case it will be assumed to be ``None``. The consequence is that the usual dichotomy between long and short options (GNU-style options) disappears: we get *smart options*, which have the single character prefix of short options and behave like both long and short options, since they can be abbreviated. Here is an example featuring smart options: .. include:: example6.py :literal: .. include:: example6.help :literal: The following are all valid invocations of the script:: $ python3 example6.py -c "select" dsn executing 'select' on dsn $ python3 example6.py -com "select" dsn executing 'select' on dsn $ python3 example6.py -command="select" dsn executing 'select' on dsn Notice that the form ``-command=SQL`` (with the ``=`` sign) is recognized only for the full option, not for its abbreviations:: $ python3 example6.py -com="select" dsn usage: example6.py [-h] [-command COMMAND] dsn example6.py: error: unrecognized arguments: -com=select If the option is not passed, the variable ``command`` will get the value ``None``. However, it is possible to specify a non-trivial default. Here is an example: .. include:: example8_.py :literal: Notice that the default value appears in the help message: .. include:: example8_.help :literal: When you run the script and you do not pass the ``-command`` option, the default query will be executed:: $ python3 example8_.py dsn executing 'select * from table' on dsn Scripts with flags ------------------ plac_ is able to recognize flags, i.e. boolean options which are ``True`` if they are passed to the command line and ``False`` if they are absent. Here is an example: .. include:: example9.py :literal: .. include:: example9.help :literal: :: $ python3 example9.py -v dsn connecting to dsn Notice that it is an error trying to specify a default for flags: the default value for a flag is always ``False``. If you feel the need to implement non-boolean flags, you should use an option with two choices, as explained in the "more features" section. For consistency with the way the usage message is printed, I suggest you to follow the Flag-Option-Required-Default (FORD) convention: in the ``main`` function write first the flag arguments, then the option arguments, then the required arguments and finally the default arguments. This is just a convention and you are not forced to use it, except for the default arguments (including the varargs) which must stay at the end as it is required by the Python syntax. I also suggests to specify a one-character abbreviation for flags: in this way you can use the GNU-style composition of flags (i.e. ``-zxvf`` is an abbreviation of ``-z -x -v -f``). I usually do not provide the one-character abbreviation for options, since it does not make sense to compose them. Starting from plac_ 0.9.1 underscores in options and flags are automatically turned into dashes. This feature was implemented at user request, to make it possible to use a more traditional naming. For instance now you can have a ``--dry-run`` flag, whereas before you had to use ``--dry_run``. .. include:: dry_run.py :literal: Here is an example of usage:: $ python3.2 dry_run.py -h usage: dry_run.py [-h] [-d] optional arguments: -h, --help show this help message and exit -d, --dry-run Dry run plac for Python 2.X users ------------------------- Even if Python 2 has reached its end of life, plac_ still provides a way to work with function annotations by means of decorators. For instance the annotated function declaration :: def main(dsn: "Database dsn", *scripts: "SQL scripts"): ... is equivalent to the following code:: @plac.annotations( dsn="Database dsn", scripts="SQL scripts") def main(dsn, *scripts): ... In the rest of this article I will assume that you are using Python 2.X with X >= 4 and I will use the ``plac.annotations`` decorator. Notice however that the core features of plac_ run even on Python 2.3. More features ------------- One of the goals of plac_ is to have a learning curve of *minutes* for its core features, compared to the learning curve of *hours* of argparse_. In order to reach this goal, I have *not* sacrificed all the features of argparse_. Actually a lot of the argparse_ power persists in plac_. Until now, I have only showed simple annotations, but in general an annotation is a 6-tuple of the form ``(help, kind, abbrev, type, choices, metavar)`` where ``help`` is the help message, ``kind`` is a string in the set { ``"flag"``, ``"option"``, ``"positional"``}, ``abbrev`` is a one-character string or ``None``, ``type`` is a callable taking a string in input, ``choices`` is a discrete sequence of values and ``metavar`` is a string. ``type`` is used to automagically convert the command line arguments from the string type to any Python type; by default there is no conversion and ``type=None``. ``choices`` is used to restrict the number of the valid options; by default there is no restriction i.e. ``choices=None``. ``metavar`` has two meanings. For a positional argument it is used to change the argument name in the usage message (and only there). By default the metavar is ``None`` and the name in the usage message is the same as the argument name. For an option the ``metavar`` is used differently in the usage message, which has now the form ``[--option-name METAVAR]``. If the ``metavar`` is ``None``, then it is equal to the uppercased name of the argument, unless the argument has a default: then it is equal to the stringified form of the default. Here is an example showing all of the features, including the metavar, copied from the argparse_ documentation: .. include:: example10.py :literal: If you cannot remember the order of the annotations you can use the ``plac.Annotation`` class (there is an example in the next section) or the alternative decoration syntax introduced in version 1.2: .. code-block:: python @plac.pos('operator', "The name of an operator", choices=['add', 'mul']) @plac.pos('numbers', "Zero or more numbers", float, metavar='n') def main(operator, *numbers): ... which is more compact. There are also a ``plac.opt`` decorator for options and ``plac.flg`` decorator for flags and they can be stacked together at will. Here is the usage: .. include:: example10.help :literal: Notice that the docstring of the ``main`` function has been automatically added to the usage message. Here are a couple of examples of usage:: $ python example10.py add 1 2 3 4 10.0 $ python example10.py mul 1 2 3 4 24.0 $ python example10.py ad 1 2 3 4 # a misspelling error usage: example10.py [-h] {add,mul} [n ...] example10.py: error: argument operator: invalid choice: 'ad' (choose from 'add', 'mul') ``plac.call`` can also be used in doctests like this: >>> import plac, example10 >>> plac.call(example10.main, ['add', '1', '2']) 3.0 ``plac.call`` works for generators too: >>> def main(n): ... for i in range(int(n)): ... yield i >>> plac.call(main, ['3']) [0, 1, 2] Internally ``plac.call`` tries to convert the output of the main function into a list, if possible. If the output is not iterable or it is a string, it is left unchanged, but if it is iterable it is converted. In particular, generator objects are exhausted by ``plac.call``. This behavior avoids mistakes like forgetting of applying ``list(result)`` to the result of ``plac.call``; moreover it makes errors visible early, and avoids mistakes in code like the following:: try: result = plac.call(main, args) except: # do something Without eagerness, a main function returning a generator object would not raise any exception until the generator is iterated over. If you are a fan of laziness, you can still have it by setting the ``eager`` flag to ``False``, as in the following example:: for line in plac.call(main, args, eager=False): print(line) If ``main`` returns a generator object this example will print each line as soon as available, whereas the default behaviour is to print all the lines together and the end of the computation. What to do if an argument name clashes with a Python builtin/keyword? ---------------------------------------------------------------------- Since version 1.3, thanks to a contribution of `Istvan Albert`_, plac_ manages such cases easily. The trick is to add one (or more) trailing underscores to the arguments that would clash; plac_ will automatically strip the underscores: .. include:: example13.py :literal: The usage message will be as you would expect:: $ python doc/example13.py -h usage: example13.py [-h] [-l] [-y] [-s 100] optional arguments: -h, --help show this help message and exit -l, --list -y, --yield [False] -s 100, --sys 100 [100] Keyword arguments ----------------- Starting from release 0.4, plac_ supports keyword arguments. In practice that means that if your main function has keyword arguments, plac_ treats specially arguments of the form ``"name=value"`` in the command line. Here is an example: .. include:: example12.py :literal: Here is the generated usage message: .. include:: example12.help :literal: Here is how you call the script:: $ python example12.py -o X a1 a2 name=value opt=X args=('a1', 'a2') kw={'name': 'value'} When using keyword arguments, one must be careful to use names which are not already taken; for instance in this examples the name ``opt`` is taken:: $ python example12.py 1 2 kw1=1 kw2=2 opt=0 usage: example12.py [-h] [-o OPT] [args ...] [kw ...] example12.py: error: colliding keyword arguments: opt The names taken are the names of the flags, of the options, and of the positional arguments, excepted varargs and keywords. This limitation is a consequence of the way the argument names are managed in function calls by the Python language. plac vs argparse ---------------- plac_ is opinionated and by design it does not try to make available all of the features of argparse_ in an easy way. In particular you should be aware of the following limitations/differences (the following assumes knowledge of argparse_): - plac does not support the destination concept: the destination coincides with the name of the argument, always. This restriction has some drawbacks. For instance, suppose you want to define a long option called ``--yield``. In this case the destination would be ``yield``, which is a Python keyword, and since you cannot introduce an argument with that name in a function definition, it is impossible to implement it. Your choices are to change the name of the long option, or to use argparse_ with a suitable destination. - plac_ does not support "required options". As the argparse_ documentation puts it: *Required options are generally considered bad form - normal users expect options to be optional. You should avoid the use of required options whenever possible.* Notice that since argparse_ supports them, plac_ can manage them too, but not directly. - plac_ supports only regular boolean flags. argparse_ has the ability to define generalized two-value flags with values different from ``True`` and ``False``. An earlier version of plac_ had this feature too, but since you can use options with two choices instead, and in any case the conversion from ``{True, False}`` to any couple of values can be trivially implemented with a ternary operator (``value1 if flag else value2``), I have removed it (KISS rules!). - plac_ does not support ``nargs`` options directly (it uses them internally, though, to implement flag recognition). The reason it that all the use cases of interest to me are covered by plac_ and I did not feel the need to increase the learning curve by adding direct support for ``nargs``. - plac_ does support subparsers, but you must read the section :ref:`Implementing subcommands`_ to see how it works. - plac_ does not support actions directly. This also looks like a feature too advanced for the goals of plac_. Notice, however, that the ability to define your own annotation objects (again, see the section :ref:`Implementing subcommand`_) may mitigate the need for custom actions. On the plus side, plac_ can directly leverage a number of argparse_ features. For instance, you can use argparse.FileType_ directly. Moreover, it is possible to pass options to the underlying ``argparse.ArgumentParser`` object (currently it accepts the default arguments ``description``, ``epilog``, ``prog``, ``usage``, ``add_help``, ``argument_default``, ``parents``, ``prefix_chars``, ``fromfile_prefix_chars``, ``conflict_handler``, ``formatter_class``). It is enough to set such attributes on the ``main`` function. For instance writing :: def main(...): pass main.add_help = False disables the recognition of the help flag ``-h, --help``. This mechanism does not look particularly elegant, but it works well enough. I assume that the typical user of plac_ will be happy with the defaults and would not want to change them; still it is possible if she wants to. For instance, by setting the ``description`` attribute, it is possible to add a comment to the usage message (by default the docstring of the ``main`` function is used as description). It is also possible to change the option prefix; for instance if your script must run under Windows and you want to use "/" as option prefix you can add the line:: main.prefix_chars='/-' The first prefix char (``/``) is used as the default for the recognition of options and flags; the second prefix char (``-``) is kept to keep the ``-h/--help`` option working: however you can disable it and reimplement it, if you like. It is possible to access directly the underlying ArgumentParser_ object, by invoking the ``plac.parser_from`` utility function: >>> import plac >>> def main(arg): ... pass ... >>> print(plac.parser_from(main)) #doctest: +ELLIPSIS ArgumentParser(prog=...) Internally ``plac.call`` uses ``plac.parser_from``. Notice that when ``plac.call(func)`` is invoked multiple time, the parser is re-used and not rebuilt from scratch again. I use ``plac.parser_from`` in the unit tests of the module, but regular users should not need to use it, unless they want to access *all* of the features of argparse_ directly without calling the main function. Interested readers should read the documentation of argparse_ to understand the meaning of the other options. If there is a set of options that you use very often, you may consider writing a decorator adding such options to the ``main`` function for you. For simplicity, plac_ does not perform any magic. Final example: a shelve interface --------------------------------- Here is a nontrivial example showing off many plac_ feature, including keyword arguments recognition. The use case is the following: suppose we have stored the configuration parameters of a given application into a Python shelve and we need a command-line tool to edit the shelve. A possible implementation using plac_ could be the following: .. include:: ishelve.py :literal: A few notes are in order: 1. I have disabled the ordinary help provided by argparse_ and I have implemented a custom help command. 2. I have changed the prefix character used to recognize the options to a dot. 3. Keyword arguments recognition (in the ``**setters``) is used to make it possible to store a value in the shelve with the syntax ``param_name=param_value``. 4. ``*params`` are used to retrieve parameters from the shelve and some error checking is performed in the case of missing parameters 5. A command to clear the shelve is implemented as a flag (``.clear``). 6. A command to delete a given parameter is implemented as an option (``.delete``). 7. There is an option with default (``.filename=conf.shelve``) to set the filename of the shelve. 8. All things considered, the code looks like a poor man's object oriented interface implemented with a chain of elifs instead of methods. Of course, plac_ can do better than that, but let me start from a low-level approach first. If you run ``ishelve.py`` without arguments you get the following message:: $ python ishelve.py no arguments passed, use .help to see the available commands If you run ``ishelve.py`` with the option ``.h`` (or any abbreviation of ``.help``) you get:: $ python ishelve.py .h Commands: .help, .showall, .clear, .delete ... ... You can check by hand that the tool works:: $ python ishelve.py .clear # start from an empty shelve cleared the shelve $ python ishelve.py a=1 b=2 setting a=1 setting b=2 $ python ishelve.py .showall b=2 a=1 $ python ishelve.py .del b # abbreviation for .delete deleted b $ python ishelve.py a 1 $ python ishelve.py b b: not found $ python ishelve.py .cler # misspelled command usage: ishelve.py [.help] [.showall] [.clear] [.delete DELETE] [.filename conf.shelve] [params ...] [setters ...] ishelve.py: error: unrecognized arguments: .cler plac vs the rest of the world ----------------------------- Originally plac_ boasted about being "the easiest command-line arguments parser in the world". Since then, people started pointing out to me various projects which are based on the same idea (extracting the parser from the main function signature) and are arguably even easier than plac_: - opterator_ by Dusty Phillips - CLIArgs_ by Pavel Panchekha - commandline_ by David Laban Luckily for me none of such projects had the idea of using function annotations and argparse_; as a consequence, they are no match for the capabilities of plac_. Of course, there are tons of other libraries to parse the command line. For instance Clap_ by Matthew Frazier which appeared on PyPI just the day before plac_; Clap_ is fine but it is certainly not easier than plac_. plac_ can also be used as a replacement of the cmd_ module in the standard library and as such it shares many features with the module cmd2_ by Catherine Devlin. However, this is completely coincidental, since I became aware of the cmd2_ module only after writing plac_. Command-line argument parsers keep coming out; between the newcomers I will notice `marrow.script`_ by Alice Bevan-McGregor, which is quite similar to plac_ in spirit, but does not rely on argparse_ at all. Argh_ by Andrey Mikhaylenko is also worth mentioning: it is based on argparse_, it came after plac_ and I must give credit to the author for the choice of the name, much funnier than plac! The future ---------- Currently the core of plac_ is around 200 lines of code, not counting blanks, comments and docstrings. I do not plan to extend the core much in the future. The idea is to keep the module short: it is and it should remain a little wrapper over argparse_. Actually I have thought about contributing the core back to argparse_ if plac_ becomes successful and gains a reasonable number of users. For the moment it should be considered in a frozen status. Notice that even if plac_ has been designed to be simple to use for simple stuff, its power should not be underestimated; it is actually a quite advanced tool with a domain of applicability which far exceeds the realm of command-line arguments parsers. Version 0.5 of plac_ doubled the code base and the documentation: it is based on the idea of using plac_ to implement command-line interpreters, i.e. something akin to the ``cmd`` module in the standard library, only better. The new features are implemented in a separated module (``plac_ext.py``), since they require Python 2.5 to work, whereas ``plac_core.py`` only requires Python 2.3. Trivia: the story behind the name --------------------------------- The plac_ project started very humbly: I just wanted to make my old optionparse_ recipe easy_installable, and to publish it on PyPI. The original name of plac_ was optionparser and the idea behind it was to build an OptionParser_ object from the docstring of the module. However, before doing that, I decided to check out the argparse_ module, since I knew it was going into Python 2.7 and Python 2.7 was coming out. Soon enough I realized two things: 1. the single greatest idea of argparse_ was unifying the positional arguments and the options in a single namespace object; 2. parsing the docstring was so old-fashioned, considering the existence of functions annotations in Python 3. Putting together these two observations with the original idea of inferring the parser I decided to build an ArgumentParser_ object from function annotations. The ``optionparser`` name was ruled out, since I was now using argparse_; a name like ``argparse_plus`` was also ruled out, since the typical usage was completely different from the argparse_ usage. I made a research on PyPI and the name *clap* (Command Line Arguments Parser) was not taken, so I renamed everything to clap. After two days a Clap_ module appeared on PyPI ! Having little imagination, I decided to rename everything again to plac, an anagram of clap: since it is a non-existing English name, I hope nobody will steal it from me! That concludes the section about the basic usage of plac_. You are now ready to read about the advanced usage. .. _argparse: https://docs.python.org/library/argparse.html .. _optparse: https://docs.python.org/library/optparse.html .. _getopt: https://docs.python.org/library/getopt.html .. _optionparse: https://code.activestate.com/recipes/278844-parsing-the-command-line/ .. _plac: https://pypi.org/project/plac/ .. _scaling down: https://www.welton.it/articles/scalable_systems.html .. _ArgumentParser: https://docs.python.org/library/argparse.html#argparse.ArgumentParser .. _argparse.FileType: https://docs.python.org/library/argparse.html#argparse.FileType .. _Clap: https://pypi.org/project/Clap/ .. _OptionParser: https://docs.python.org/library/optparse.html#optparse.OptionParser .. _CLIArgs: https://pypi.org/project/CLIArgs/ .. _opterator: https://pypi.org/project/opterator/ .. _cmd2: https://github.com/python-cmd2/cmd2 .. _cmd: https://docs.python.org/library/cmd.html .. _commandline: https://pypi.org/project/commandline/ .. _marrow.script: https://github.com/marrow/script .. _argh: https://pythonhosted.org/argh/ .. _Istvan Albert: https://github.com/ialbert plac-1.3.4/doc/read_stdin.py000066400000000000000000000004451415326361200156770ustar00rootroot00000000000000""" You can run this script as $ python read_stdin.py < ishelve.bat """ from __future__ import with_statement import sys from ishelve import ishelve import plac if __name__ == '__main__': with plac.Interpreter(ishelve) as i: for line in sys.stdin: print(i.send(line)) plac-1.3.4/doc/server_ex.py000066400000000000000000000003011415326361200155540ustar00rootroot00000000000000import plac from importer2 import FakeImporter def main(port=2199): main = FakeImporter('dsn') plac.Interpreter(main).start_server(port) if __name__ == '__main__': plac.call(main) plac-1.3.4/doc/test_ishelve.py000066400000000000000000000006371415326361200162640ustar00rootroot00000000000000# test_ishelve.py import plac import ishelve def test(): assert plac.call(ishelve.main, ['.clear']) == ['cleared the shelve'] assert plac.call(ishelve.main, ['a=1']) == ['setting a=1'] assert plac.call(ishelve.main, ['a']) == ['1'] assert plac.call(ishelve.main, ['.delete=a']) == ['deleted a'] assert plac.call(ishelve.main, ['a']) == ['a: not found'] if __name__ == '__main__': test() plac-1.3.4/doc/test_ishelve_more.py000066400000000000000000000005231415326361200173000ustar00rootroot00000000000000# test_ishelve_more.py from __future__ import with_statement import ishelve import plac def test(): with plac.Interpreter(ishelve.main) as i: i.check('.clear', 'cleared the shelve') i.check('a=1', 'setting a=1') i.check('a', '1') i.check('.delete=a', 'deleted a') i.check('a', 'a: not found') plac-1.3.4/doc/test_pi.py000066400000000000000000000003431415326361200152270ustar00rootroot00000000000000from picalculator import PiCalculator def test(): pc = PiCalculator(10, 'T') tasks = pc.submit_tasks() for task in tasks: task.run() print(sum(task.result for task in tasks) / pc.n_cpu) pc.close() plac-1.3.4/doc/test_plac.py000066400000000000000000000233731415326361200155460ustar00rootroot00000000000000""" The tests should be run as standalone script """ import os import sys import argparse import datetime import doctest import subprocess import plac import plac_core version = sys.version_info[:2] sys_argv0 = sys.argv[0] docdir = os.path.dirname(os.path.abspath(__file__)) os.chdir(docdir) PLAC_RUNNER = os.path.join(os.path.dirname(docdir), 'plac_runner.py') # ####################### helpers ###################### # def fix_today(text): return text.replace('YYYY-MM-DD', str(datetime.date.today())) def expect(errclass, func, *args, **kw): try: func(*args, **kw) except errclass: pass else: raise RuntimeError('%s expected, got none!' % errclass.__name__) def parser_from(f, **kw): f.__annotations__ = kw return plac.parser_from(f) # FIXME: Remove this when removing support for Python 3.8 class PlacTestFormatter(argparse.RawDescriptionHelpFormatter): if version < (3, 9): def _format_args(self, action, default_metavar): get_metavar = self._metavar_formatter(action, default_metavar) if action.nargs == argparse.ZERO_OR_MORE: metavar = get_metavar(1) if len(metavar) == 2: result = '[%s [%s ...]]' % metavar else: result = '[%s ...]' % metavar else: result = super(PlacTestFormatter, self)._format_args( action, default_metavar) return result def check_help(name): sys.argv[0] = name + '.py' # avoid issue with pytest plac_core._parser_registry.clear() # makes different imports independent try: try: main = plac.import_main(name + '.py') except SyntaxError: if sys.version < '3': # expected for Python 2.X return else: # not expected for Python 3.X raise p = plac.parser_from(main, formatter_class=PlacTestFormatter) expected = fix_today(open(name + '.help').read()).strip() got = p.format_help().strip() assert got == expected, got finally: sys.argv[0] = sys_argv0 # ###################### tests ########################### # def test_expected_help(): for fname in os.listdir('.'): if fname.endswith('.help'): name = fname[:-5] if name not in ('vcs', 'ishelve'): check_help(fname[:-5]) p1 = parser_from(lambda delete, *args: None, delete=('delete a file', 'option')) def test_p1(): arg = p1.parse_args(['-d', 'foo', 'arg1', 'arg2']) assert arg.delete == 'foo' assert arg.args == ['arg1', 'arg2'] arg = p1.parse_args([]) assert arg.delete is None, arg.delete assert arg.args == [], arg.args p2 = parser_from(lambda arg1, delete, *args: None, delete=('delete a file', 'option', 'd')) def test_p2(): arg = p2.parse_args(['-d', 'foo', 'arg1', 'arg2']) assert arg.delete == 'foo', arg.delete assert arg.arg1 == 'arg1', arg.arg1 assert arg.args == ['arg2'], arg.args arg = p2.parse_args(['arg1']) assert arg.delete is None, arg.delete assert arg.args == [], arg.args assert arg, arg expect(SystemExit, p2.parse_args, []) p3 = parser_from(lambda arg1, delete: None, delete=('delete a file', 'option', 'd')) def test_p3(): arg = p3.parse_args(['arg1']) assert arg.delete is None, arg.delete assert arg.arg1 == 'arg1', arg.args expect(SystemExit, p3.parse_args, ['arg1', 'arg2']) expect(SystemExit, p3.parse_args, []) p4 = parser_from(lambda delete, delete_all, color="black": None, delete=('delete a file', 'option', 'd'), delete_all=('delete all files', 'flag', 'a'), color=('color', 'option', 'c')) def test_p4(): arg = p4.parse_args(['-a']) assert arg.delete_all is True, arg.delete_all arg = p4.parse_args([]) arg = p4.parse_args(['--color=black']) assert arg.color == 'black' arg = p4.parse_args(['--color=red']) assert arg.color == 'red' p5 = parser_from(lambda dry_run=False: None, dry_run=('Dry run', 'flag', 'x')) def test_p5(): arg = p5.parse_args(['--dry-run']) assert arg.dry_run is True, arg.dry_run p_global = parser_from(lambda reserved_=False: None, reserved_=('Reserved word', 'flag', 'g')) def test_global(): arg = p_global.parse_args(['--reserved']) assert arg.reserved is True, arg.reserved def test_flag_with_default(): expect(TypeError, parser_from, lambda yes_or_no='no': None, yes_or_no=('A yes/no flag', 'flag', 'f')) def assert_usage(parser, expected): usage = parser.format_usage() assert usage == expected, usage def test_metavar_no_defaults(): sys.argv[0] = 'test_plac.py' # positional p = parser_from(lambda x: None, x=('first argument', 'positional', None, str, [], 'METAVAR')) assert_usage(p, 'usage: test_plac.py [-h] METAVAR\n') # option p = parser_from(lambda x: None, x=('first argument', 'option', None, str, [], 'METAVAR')) assert_usage(p, 'usage: test_plac.py [-h] [-x METAVAR]\n') sys.argv[0] = sys_argv0 def test_metavar_with_defaults(): sys.argv[0] = 'test_plac.py' # positional p = parser_from(lambda x='a': None, x=('first argument', 'positional', None, str, [], 'METAVAR')) assert_usage(p, 'usage: test_plac.py [-h] [METAVAR]\n') # option p = parser_from(lambda x='a': None, x=('first argument', 'option', None, str, [], 'METAVAR')) assert_usage(p, 'usage: test_plac.py [-h] [-x METAVAR]\n') p = parser_from(lambda x='a': None, x=('first argument', 'option', None, str, [])) assert_usage(p, 'usage: test_plac.py [-h] [-x a]\n') sys.argv[0] = sys_argv0 def test_metavar_empty_string(): # see https://github.com/ialbert/plac/issues/36 def main(arg=''): pass sys.argv[0] = 'test_plac.py' p = parser_from(main) assert_usage(p, "usage: test_plac.py [-h] ['']\n") sys.argv[0] = sys_argv0 def test_kwargs(): def main(opt, arg1, *args, **kw): print(opt, arg1) return args, kw main.__annotations__ = dict(opt=('Option', 'option')) argskw = plac.call(main, ['arg1', 'arg2', 'a=1', 'b=2']) assert argskw == [('arg2',), {'a': '1', 'b': '2'}], argskw argskw = plac.call(main, ['arg1', 'arg2', 'a=1', '-o', '2']) assert argskw == [('arg2',), {'a': '1'}], argskw expect(SystemExit, plac.call, main, ['arg1', 'arg2', 'a=1', 'opt=2']) def test_kwargs2(): # see https://github.com/ialbert/plac/issues/39 def main(**kw): return kw.items() assert plac.call(main, ['a=1']) == [('a', '1')] expect(SystemExit, plac.call, main, ['foo']) expect(SystemExit, plac.call, main, ['foo', 'a=1']) def test_kwargs3(): # see https://github.com/ialbert/plac/issues/38 def main(opt='foo', **kw): return opt, kw main.__annotations__ = dict(opt=('Option', 'option')) assert plac.call(main, ['-o', 'abc=']) == ['abc=', {}] assert plac.call(main, ['-o', 'abc=', 'z=1']) == ['abc=', {'z': '1'}] assert plac.call(main, ['z=1']) == ['foo', {'z': '1'}] def test_date_default(): p = parser_from(lambda day=datetime.date.today(): day) arg = p.parse_args(['2019-11-19']) assert arg.day == datetime.date(2019, 11, 19) def test_int_default(): p = parser_from(lambda number=42: number) arg = p.parse_args([]) assert arg.number == 42 arg = p.parse_args(['424242']) assert arg.number == 424242 def test_none_default(): p = parser_from(lambda nonable=None: arg) arg = p.parse_args([]) assert arg.nonable is None arg = p.parse_args(['somestring']) assert arg.nonable == 'somestring' class Cmds(object): add_help = False commands = 'help', 'commit' def help(self, name): return 'help', name def commit(self): return 'commit' cmds = Cmds() def test_cmds(): assert 'commit' == plac.call(cmds, ['commit']) assert ['help', 'foo'] == plac.call(cmds, ['help', 'foo']) expect(SystemExit, plac.call, cmds, []) def test_cmd_abbrevs(): assert 'commit' == plac.call(cmds, ['comm']) assert ['help', 'foo'] == plac.call(cmds, ['h', 'foo']) expect(SystemExit, plac.call, cmds, ['foo']) def test_sub_help(): c = Cmds() c.add_help = True expect(SystemExit, plac.call, c, ['commit', '-h']) def test_yield(): def main(): for i in (1, 2, 3): yield i assert plac.call(main, []) == [1, 2, 3] def test_doctest(): failure, tot = doctest.testfile('index.rst', module_relative=False) assert not failure, failure failing_scripts = set(['ishelve2.plac']) def check_script(args): if failing_scripts.intersection(args): assert subprocess.call(args) > 0, ( # expected failure 'Unexpected success for %s' % ' '.join(args)) else: assert subprocess.call(args) == 0, 'Failed %s' % ' '.join(args) ''' # Disabling unused functionality def test_batch(): for batch in os.listdir('.'): if batch.endswith('.plac'): check_script([sys.executable, PLAC_RUNNER, '-b', batch]) def test_placet(): for placet in os.listdir('.'): if placet.endswith('.placet'): check_script([sys.executable, PLAC_RUNNER, '-t', placet]) ''' if __name__ == '__main__': n = 0 for name, test in sorted(globals().items()): if name.startswith('test_'): print('Running ' + name) maybegen = test() if hasattr(maybegen, '__iter__'): for func, arg in maybegen: func(arg) n += 1 else: n += 1 print('Executed %d tests OK' % n) plac-1.3.4/doc/test_runp.py000066400000000000000000000011061415326361200156010ustar00rootroot00000000000000""" This test should work on Linux if you have Tkinter installed. """ from __future__ import with_statement import plac, time def gen(n): for i in range(n + 1): yield str(i) time.sleep(.1) def err(): yield 1/0 def test1(): assert ['3', '5', '10'] == plac.runp([gen(3), gen(5), gen(10)]) def test2(): result, error = plac.runp([gen(3), err()]) assert result == '3' and error.__class__ == ZeroDivisionError def test3(): t0 = time.time() plac.runp([gen(9), gen(9)]) assert int(time.time() - t0) == 1 # it must take 1 second, not 2 plac-1.3.4/doc/test_server.py000066400000000000000000000017461415326361200161350ustar00rootroot00000000000000import multiprocessing, subprocess, random, time import plac from ishelve2 import ShelveInterface i = plac.Interpreter(ShelveInterface(configfile=None)) COMMANDS = ['''\ help set a 1 ''', '''\ set b 1 wrong command showall '''] def telnet(commands, port): po = subprocess.Popen(['telnet', 'localhost', str(port)], stdin=subprocess.PIPE) try: for cmd in commands.splitlines(): po.stdin.write((cmd + '\n').encode('ascii')) time.sleep(.1) # wait a bit for the server to answer finally: po.stdin.close() def test(): port = random.choice(range(2000, 20000)) server = multiprocessing.Process(target=i.start_server, args=(port,)) server.start() clients = [] for cmds in COMMANDS: cl = multiprocessing.Process(target=telnet, args=(cmds, port)) clients.append(cl) cl.start() for cl in clients: cl.join() server.terminate() # should trap the output and check it plac-1.3.4/doc/tkmon.py000066400000000000000000000005101415326361200147040ustar00rootroot00000000000000from __future__ import with_statement import plac class Hello(object): mpcommands = ['hello', 'quit'] def hello(self): yield 'hello' def quit(self): raise plac.Interpreter.Exit if __name__ == '__main__': i = plac.Interpreter(Hello()) i.add_monitor(plac.TkMonitor('tkmon')) i.interact() plac-1.3.4/doc/vcs.help000066400000000000000000000005401415326361200146520ustar00rootroot00000000000000usage: plac_runner.py vcs.py [-h] {status,commit,checkout} ... A Fake Version Control System optional arguments: -h, --help show this help message and exit subcommands: {status,commit,checkout} checkout A fake checkout command commit A fake commit command status A fake status command plac-1.3.4/doc/vcs.py000066400000000000000000000017041415326361200143550ustar00rootroot00000000000000"A Fake Version Control System" import plac # this implementation also works with Python 2.4 commands = 'checkout', 'commit', 'status' @plac.annotations(url='url of the source code') def checkout(url): "A fake checkout command" return ('checkout ', url) @plac.annotations(message=('commit message', 'option')) def commit(message): "A fake commit command" return ('commit ', message) @plac.annotations(quiet=('summary information', 'flag', 'q')) def status(quiet): "A fake status command" return ('status ', quiet) def __missing__(name): return ('Command %r does not exist' % name,) def __exit__(etype, exc, tb): "Will be called automatically at the end of the interpreter loop" if etype in (None, GeneratorExit): # success print('ok') main = __import__(__name__) # the module imports itself! if __name__ == '__main__': import plac for out in plac.call(main, version='0.1.0'): print(out) plac-1.3.4/plac.py000066400000000000000000000031011415326361200137250ustar00rootroot00000000000000# ######################### LICENSE ############################### # # Copyright (c) 2010-2021, Michele Simionato # All rights reserved. # # Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # Redistributions in bytecode form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, # INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, # BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS # OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND # ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR # TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE # USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH # DAMAGE. """ See docs/index.html for the documentation. """ from plac_core import * from plac_ext import (Interpreter, import_main, ReadlineInput, stdout, runp, Monitor, default_help) __version__ = '1.3.4' try: from plac_tk import TkMonitor except ImportError: pass plac-1.3.4/plac_core.py000066400000000000000000000367451415326361200147610ustar00rootroot00000000000000# this module should be kept Python 2.3 compatible import re import sys import time import inspect import textwrap import functools import argparse from datetime import datetime, date from gettext import gettext as _ version = sys.version_info[:2] if sys.version >= '3': from inspect import getfullargspec else: class getfullargspec(object): "A quick and dirty replacement for getfullargspec for Python 2.X" def __init__(self, f): self.args, self.varargs, self.varkw, self.defaults = \ inspect.getargspec(f) self.annotations = getattr(f, '__annotations__', {}) def to_date(s): """Returns year-month-day""" return date(*time.strptime(s, "%Y-%m-%d")[0:3]) def to_datetime(s): """Returns year-month-day hour-minute-second""" return datetime(*time.strptime(s, "%Y-%m-%d %H-%M-%S")[0:6]) def getargspec(callableobj): """Given a callable return an object with attributes .args, .varargs, .varkw, .defaults. It tries to do the "right thing" with functions, methods, classes and generic callables.""" if inspect.isfunction(callableobj): argspec = getfullargspec(callableobj) elif inspect.ismethod(callableobj): argspec = getfullargspec(callableobj) del argspec.args[0] # remove first argument elif inspect.isclass(callableobj): if callableobj.__init__ is object.__init__: # to avoid an error argspec = getfullargspec(lambda self: None) else: argspec = getfullargspec(callableobj.__init__) del argspec.args[0] # remove first argument elif hasattr(callableobj, '__call__'): argspec = getfullargspec(callableobj.__call__) del argspec.args[0] # remove first argument else: raise TypeError(_('Could not determine the signature of ') + str(callableobj)) return argspec def annotations(**ann): """ Returns a decorator annotating a function with the given annotations. This is a trick to support function annotations in Python 2.X. """ def annotate(f): fas = getfullargspec(f) args = fas.args if fas.varargs: args.append(fas.varargs) if fas.varkw: args.append(fas.varkw) for argname in ann: if argname not in args: raise NameError( _('Annotating non-existing argument: %s') % argname) f.__annotations__ = ann return f return annotate def _annotate(arg, ann, f): try: f.__annotations__[arg] = ann except AttributeError: # Python 2.7 f.__annotations__ = {arg: ann} return f def pos(arg, help=None, type=None, choices=None, metavar=None): """ Decorator for annotating positional arguments """ return functools.partial( _annotate, arg, (help, 'positional', None, type, choices, metavar)) def opt(arg, help=None, type=None, abbrev=None, choices=None, metavar=None): """ Decorator for annotating optional arguments """ abbrev = abbrev or arg[0] return functools.partial( _annotate, arg, (help, 'option', abbrev, type, choices, metavar)) def flg(arg, help=None, abbrev=None): """ Decorator for annotating flags """ return functools.partial( _annotate, arg, (help, 'flag', abbrev or arg[0], None, None, None)) def is_annotation(obj): """ An object is an annotation object if it has the attributes help, kind, abbrev, type, choices, metavar. """ return (hasattr(obj, 'help') and hasattr(obj, 'kind') and hasattr(obj, 'abbrev') and hasattr(obj, 'type') and hasattr(obj, 'choices') and hasattr(obj, 'metavar')) class Annotation(object): def __init__(self, help=None, kind="positional", abbrev=None, type=None, choices=None, metavar=None): assert kind in ('positional', 'option', 'flag'), kind if kind == "positional": assert abbrev is None, abbrev self.help = help self.kind = kind self.abbrev = abbrev self.type = type self.choices = choices self.metavar = metavar def from_(cls, obj): "Helper to convert an object into an annotation, if needed" if is_annotation(obj): return obj # do nothing elif inspect.isclass(obj): obj = str(obj) elif iterable(obj): return cls(*obj) return cls(obj) from_ = classmethod(from_) NONE = object() # sentinel use to signal the absence of a default PARSER_CFG = getfullargspec(argparse.ArgumentParser.__init__).args[1:] # the default arguments accepted by an ArgumentParser object def pconf(obj): """ Extracts the configuration of the underlying ArgumentParser from obj """ cfg = dict(description=(textwrap.dedent(obj.__doc__.rstrip()) if obj.__doc__ else None), formatter_class=argparse.RawDescriptionHelpFormatter) for name in dir(obj): if name in PARSER_CFG: # argument of ArgumentParser cfg[name] = getattr(obj, name) return cfg _parser_registry = {} def parser_from(obj, **confparams): """ obj can be a callable or an object with a .commands attribute. Returns an ArgumentParser. """ try: # the underlying parser has been generated already return _parser_registry[obj] except KeyError: # generate a new parser pass conf = pconf(obj).copy() conf.update(confparams) _parser_registry[obj] = parser = ArgumentParser(**conf) parser.obj = obj parser.case_sensitive = confparams.get( 'case_sensitive', getattr(obj, 'case_sensitive', True)) if hasattr(obj, 'commands') and not inspect.isclass(obj): # a command container instance parser.addsubcommands(obj.commands, obj, 'subcommands') else: parser.populate_from(obj) return parser def _extract_kwargs(args): """ Returns two lists: regular args and name=value args """ arglist = [] kwargs = {} for arg in args: match = re.match(r'([a-zA-Z_]\w*)=', arg) if match: name = match.group(1) kwargs[name] = arg[len(name)+1:] else: arglist.append(arg) return arglist, kwargs def _match_cmd(abbrev, commands, case_sensitive=True): """ Extract the command name from an abbreviation or raise a NameError """ if not case_sensitive: abbrev = abbrev.upper() commands = [c.upper() for c in commands] perfect_matches = [name for name in commands if name == abbrev] if len(perfect_matches) == 1: return perfect_matches[0] matches = [name for name in commands if name.startswith(abbrev)] n = len(matches) if n == 1: return matches[0] elif n > 1: raise NameError( _('Ambiguous command %r: matching %s' % (abbrev, matches))) class ArgumentParser(argparse.ArgumentParser): """ An ArgumentParser with .func and .argspec attributes, and possibly .commands and .subparsers. """ case_sensitive = True if version < (3, 10): def __init__(self, *args, **kwargs): super(ArgumentParser, self).__init__(*args, **kwargs) if self._action_groups[1].title == _('optional arguments'): self._action_groups[1].title = _('options') def alias(self, arg): "Can be overridden to preprocess command-line arguments" return arg def consume(self, args): """ Call the underlying function with the args. Works also for command containers, by dispatching to the right subparser. """ arglist = [self.alias(a) for a in args] cmd = None if hasattr(self, 'subparsers'): subp, cmd = self._extract_subparser_cmd(arglist) if subp is None and cmd is not None: return cmd, self.missing(cmd) elif subp is not None: # use the subparser self = subp if hasattr(self, 'argspec') and self.argspec.varargs: # ignore unrecognized arguments ns, extraopts = self.parse_known_args(arglist) else: ns, extraopts = self.parse_args(arglist), [] # may raise an exit if not hasattr(self, 'argspec'): raise SystemExit if hasattr(self, 'argspec') and self.argspec.varkw: v = self.argspec.varargs varkw = self.argspec.varkw if v in ns.__dict__: lst = ns.__dict__.pop(v) lst, kwargs = _extract_kwargs(lst) ns.__dict__[v] = lst elif varkw in ns.__dict__: lst = ns.__dict__.pop(varkw) lst, kwargs = _extract_kwargs(lst) ns.__dict__[varkw] = lst if lst and not v: self.error(_('Unrecognized arguments: %s') % arglist) else: kwargs = {} collision = set(self.argspec.args) & set(kwargs) if collision: self.error( _('colliding keyword arguments: %s') % ' '.join(collision)) # Correct options with trailing undescores args = [getattr(ns, a.rstrip('_')) for a in self.argspec.args] varargs = getattr(ns, self.argspec.varargs or '', []) return cmd, self.func(*(args + varargs + extraopts), **kwargs) def _extract_subparser_cmd(self, arglist): """ Extract the right subparser from the first recognized argument """ optprefix = self.prefix_chars[0] name_parser_map = self.subparsers._name_parser_map for i, arg in enumerate(arglist): if not arg.startswith(optprefix): cmd = _match_cmd(arg, name_parser_map, self.case_sensitive) del arglist[i] return name_parser_map.get(cmd), cmd or arg return None, None def addsubcommands(self, commands, obj, title=None, cmdprefix=''): """ Extract a list of subcommands from obj and add them to the parser """ if hasattr(obj, cmdprefix) and obj.cmdprefix in self.prefix_chars: raise ValueError(_('The prefix %r is already taken!' % cmdprefix)) if not hasattr(self, 'subparsers'): self.subparsers = self.add_subparsers(title=title) elif title: self.add_argument_group(title=title) # populate ._action_groups prefixlen = len(getattr(obj, 'cmdprefix', '')) add_help = getattr(obj, 'add_help', True) for cmd in commands: func = getattr(obj, cmd[prefixlen:]) # strip the prefix doc = (textwrap.dedent(func.__doc__.rstrip()) if func.__doc__ else None) self.subparsers.add_parser( cmd, add_help=add_help, help=doc, **pconf(func) ).populate_from(func) def _set_func_argspec(self, obj): """ Extracts the signature from a callable object and adds an .argspec attribute to the parser. Also adds a .func reference to the object. """ self.func = obj self.argspec = getargspec(obj) _parser_registry[obj] = self def populate_from(self, func): """ Extract the arguments from the attributes of the passed function and return a populated ArgumentParser instance. """ self._set_func_argspec(func) f = self.argspec defaults = f.defaults or () n_args = len(f.args) n_defaults = len(defaults) alldefaults = (NONE,) * (n_args - n_defaults) + defaults prefix = self.prefix = getattr(func, 'prefix_chars', '-')[0] for name, default in zip(f.args, alldefaults): ann = f.annotations.get(name, ()) a = Annotation.from_(ann) metavar = a.metavar if default is NONE: dflt = None else: dflt = default if a.help is None: a.help = '[%s]' % str(dflt) # dflt can be a tuple if a.type is None: # try to infer the type from the default argument if isinstance(default, datetime): a.type = to_datetime elif isinstance(default, date): a.type = to_date elif default is not None: a.type = type(default) if not metavar and default == '': metavar = "''" if a.kind in ('option', 'flag'): if name.endswith("_"): # allows reserved words to be specified with underscores suffix = name.rstrip('_') else: # convert undescores to dashes. suffix = name.replace('_', '-') if a.abbrev: shortlong = (prefix + a.abbrev, prefix*2 + suffix) else: shortlong = (prefix + suffix,) elif default is NONE: # required argument self.add_argument(name, help=a.help, type=a.type, choices=a.choices, metavar=metavar) else: # default argument self.add_argument( name, nargs='?', help=a.help, default=dflt, type=a.type, choices=a.choices, metavar=metavar) if a.kind == 'option': if default is not NONE: metavar = metavar or str(default) self.add_argument( help=a.help, default=dflt, type=a.type, choices=a.choices, metavar=metavar, *shortlong) elif a.kind == 'flag': if default is not NONE and default is not False: raise TypeError(_('Flag %r wants default False, got %r') % (name, default)) self.add_argument(action='store_true', help=a.help, *shortlong) if f.varargs: a = Annotation.from_(f.annotations.get(f.varargs, ())) self.add_argument(f.varargs, nargs='*', help=a.help, default=[], type=a.type, metavar=a.metavar) if f.varkw: a = Annotation.from_(f.annotations.get(f.varkw, ())) self.add_argument(f.varkw, nargs='*', help=a.help, default={}, type=a.type, metavar=a.metavar) def missing(self, name): "May raise a SystemExit" miss = getattr(self.obj, '__missing__', lambda name: self.error('No command %r' % name)) return miss(name) def print_actions(self): "Useful for debugging" print(self) for a in self._actions: print(a) def iterable(obj): "Any object with an __iter__ method which is not a string or class" return hasattr(obj, '__iter__') and not inspect.isclass(obj) and not isinstance(obj, (str, bytes)) def call(obj, arglist=None, eager=True, version=None): """ If obj is a function or a bound method, parse the given arglist by using the parser inferred from the annotations of obj and call obj with the parsed arguments. If obj is an object with attribute .commands, dispatch to the associated subparser. """ if arglist is None: arglist = sys.argv[1:] parser = parser_from(obj) if version: parser.add_argument( '--version', '-v', action='version', version=version) cmd, result = parser.consume(arglist) if iterable(result) and eager: # listify the result return list(result) return result plac-1.3.4/plac_ext.py000066400000000000000000001145221415326361200146170ustar00rootroot00000000000000# this module requires Python 2.6+ from __future__ import with_statement from contextlib import contextmanager from operator import attrgetter from gettext import gettext as _ import inspect import os import sys import cmd import shlex import subprocess import argparse import itertools import traceback import multiprocessing import signal import threading import plac_core version = sys.version_info[:2] if version < (3, 5): from imp import load_source else: import importlib.util def load_source(dotname, path): spec = importlib.util.spec_from_file_location(dotname, path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod if sys.version < '3': def exec_(_code_, _globs_=None, _locs_=None): if _globs_ is None: frame = sys._getframe(1) _globs_ = frame.f_globals if _locs_ is None: _locs_ = frame.f_locals del frame elif _locs_ is None: _locs_ = _globs_ exec("""exec _code_ in _globs_, _locs_""") exec(''' def raise_(tp, value=None, tb=None): raise tp, value, tb ''') else: exec_ = eval('exec') def raise_(tp, value=None, tb=None): """ A function that matches the Python 2.x ``raise`` statement. This allows re-raising exceptions with the cls value and traceback on Python 2 and 3. """ if value is not None and isinstance(tp, Exception): raise TypeError("instance exception may not have a separate value") if value is not None: exc = tp(value) else: exc = tp if exc.__traceback__ is not tb: raise exc.with_traceback(tb) raise exc try: raw_input except NameError: # Python 3 raw_input = input def decode(val): """ Decode an object assuming the encoding is UTF-8. """ try: # assume it is an encoded bytes object return val.decode('utf-8') except AttributeError: # it was an already decoded unicode object return str(val) # ############################ generic utils ############################### # @contextmanager def stdout(fileobj): "usage: with stdout(file('out.txt', 'a')): do_something()" orig_stdout = sys.stdout sys.stdout = fileobj try: yield finally: sys.stdout = orig_stdout def write(x): "Write str(x) on stdout and flush, no newline added" sys.stdout.write(str(x)) sys.stdout.flush() def gen_val(value): "Return a generator object with a single element" yield value def gen_exc(etype, exc, tb): "Return a generator object raising an exception" raise_(etype, exc, tb) yield def less(text): "Send a text to less via a pipe" # -c clear the screen before starting less po = subprocess.Popen(['less', '-c'], stdin=subprocess.PIPE) try: po.stdin.write(text) except IOError: pass po.stdin.close() po.wait() use_less = (sys.platform != 'win32') # unices class TerminatedProcess(Exception): pass def terminatedProcess(signum, frame): raise TerminatedProcess # ########################## readline support ############################ # def read_line(stdin, prompt=''): "Read a line from stdin, using readline when possible" if isinstance(stdin, ReadlineInput): return stdin.readline(prompt) else: write(prompt) return stdin.readline() def read_long_line(stdin, terminator): """ Read multiple lines from stdin until the terminator character is found, then yield a single space-separated long line. """ while True: lines = [] while True: line = stdin.readline() # ends with \n if not line: # EOF return line = line.strip() if not line: continue elif line[-1] == terminator: lines.append(line[:-1]) break else: lines.append(line) yield ' '.join(lines) class ReadlineInput(object): """ An iterable with a .readline method reading from stdin. """ def __init__(self, completions, case_sensitive=True, histfile=None): self.completions = completions self.case_sensitive = case_sensitive self.histfile = histfile if not case_sensitive: self.completions = [c.upper() for c in completions] import readline self.rl = readline readline.parse_and_bind("tab: complete") readline.set_completer(self.complete) def __enter__(self): self.old_completer = self.rl.get_completer() try: if self.histfile: self.rl.read_history_file(self.histfile) except IOError: # the first time pass return self def __exit__(self, etype, exc, tb): self.rl.set_completer(self.old_completer) if self.histfile: self.rl.write_history_file(self.histfile) def complete(self, kw, state): # state is 0, 1, 2, ... and increases by hitting TAB if not self.case_sensitive: kw = kw.upper() try: return [k for k in self.completions if k.startswith(kw)][state] except IndexError: # no completions return # exit def readline(self, prompt=''): try: return raw_input(prompt) + '\n' except EOFError: return '' def __iter__(self): return iter(self.readline, '') # ################# help functionality in plac interpreters ################# # class HelpSummary(object): "Build the help summary consistently with the cmd module" @classmethod def add(cls, obj, specialcommands): p = plac_core.parser_from(obj) c = cmd.Cmd(stdout=cls()) c.stdout.write('\n') c.print_topics('special commands', sorted(specialcommands), 15, 80) c.print_topics('custom commands', sorted(obj.commands), 15, 80) c.print_topics('commands run in external processes', sorted(obj.mpcommands), 15, 80) c.print_topics('threaded commands', sorted(obj.thcommands), 15, 80) p.helpsummary = str(c.stdout) def __init__(self): self._ls = [] def write(self, s): self._ls.append(s) def __str__(self): return ''.join(self._ls) class PlacFormatter(argparse.RawDescriptionHelpFormatter): def _metavar_formatter(self, action, default_metavar): 'Remove special commands from the usage message' choices = action.choices or {} action.choices = dict((n, c) for n, c in choices.items() if not n.startswith('.')) return super(PlacFormatter, self)._metavar_formatter( action, default_metavar) def format_help(self): "Attached to plac_core.ArgumentParser for plac interpreters" try: return self.helpsummary except AttributeError: return super(plac_core.ArgumentParser, self).format_help() plac_core.ArgumentParser.format_help = format_help def default_help(obj, cmd=None): "The default help functionality in plac interpreters" parser = plac_core.parser_from(obj) if cmd is None: yield parser.format_help() return subp = parser.subparsers._name_parser_map.get(cmd) if subp is None: yield _('Unknown command %s' % cmd) elif getattr(obj, '_interact_', False): # in interactive mode formatter = subp._get_formatter() formatter._prog = cmd # remove the program name from the usage formatter.add_usage( subp.usage, [a for a in subp._actions if a.dest != 'help'], subp._mutually_exclusive_groups) formatter.add_text(subp.description) for action_group in subp._action_groups: formatter.start_section(action_group.title) formatter.add_text(action_group.description) formatter.add_arguments(a for a in action_group._group_actions if a.dest != 'help') formatter.end_section() yield formatter.format_help() else: # regular argparse help yield subp.format_help() # ######################## import management ############################## # try: PLACDIRS = os.environ.get('PLACPATH', '.').split(':') except: raise ValueError(_('Ill-formed PLACPATH: got %PLACPATHs') % os.environ) def partial_call(factory, arglist): "Call a container factory with the arglist and return a plac object" a = plac_core.parser_from(factory).argspec if a.defaults or a.varargs or a.varkw: raise TypeError('Interpreter.call must be invoked on ' 'factories with required arguments only') required_args = ', '.join(a.args) if required_args: required_args += ',' # trailing comma code = '''def makeobj(interact, %s *args): obj = factory(%s) obj._interact_ = interact obj._args_ = args return obj\n''' % (required_args, required_args) dic = dict(factory=factory) exec_(code, dic) makeobj = dic['makeobj'] makeobj.add_help = False if inspect.isclass(factory): makeobj.__annotations__ = getattr( factory.__init__, '__annotations__', {}) else: makeobj.__annotations__ = getattr( factory, '__annotations__', {}) makeobj.__annotations__['interact'] = ( 'start interactive interpreter', 'flag', 'i') return plac_core.call(makeobj, arglist) def import_main(path, *args): """ A utility to import the main function of a plac tool. It also works with command container factories. """ if ':' in path: # importing a factory path, factory_name = path.split(':') else: # importing the main function factory_name = None if not os.path.isabs(path): # relative path, look at PLACDIRS for placdir in PLACDIRS: fullpath = os.path.join(placdir, path) if os.path.exists(fullpath): break else: # no break raise ImportError(_('Cannot find %s' % path)) else: fullpath = path name, ext = os.path.splitext(os.path.basename(fullpath)) module = load_source(name, fullpath) if factory_name: tool = partial_call(getattr(module, factory_name), args) else: tool = module.main return tool # ############################ Task classes ############################# # # base class not instantiated directly class BaseTask(object): """ A task is a wrapper over a generator object with signature Task(no, arglist, genobj), attributes .no .arglist .outlist .str .etype .exc .tb .status and methods .run and .kill. """ STATES = ('SUBMITTED', 'RUNNING', 'TOBEKILLED', 'KILLED', 'FINISHED', 'ABORTED') def __init__(self, no, arglist, genobj): self.no = no self.arglist = arglist self._genobj = self._wrap(genobj) self.str, self.etype, self.exc, self.tb = '', None, None, None self.status = 'SUBMITTED' self.outlist = [] def notify(self, msg): "Notifies the underlying monitor. To be implemented" def _wrap(self, genobj, stringify_tb=False): """ Wrap the genobj into a generator managing the exceptions, populating the .outlist, setting the .status and yielding None. stringify_tb must be True if the traceback must be sent to a process. """ self.status = 'RUNNING' try: for value in genobj: if self.status == 'TOBEKILLED': # exit from the loop raise GeneratorExit if value is not None: # add output self.outlist.append(value) self.notify(decode(value)) yield except Interpreter.Exit: # wanted exit self._regular_exit() raise except (GeneratorExit, TerminatedProcess, KeyboardInterrupt): # soft termination self.status = 'KILLED' except Exception: # unexpected exception self.etype, self.exc, tb = sys.exc_info() self.tb = ''.join(traceback.format_tb(tb)) if stringify_tb else tb self.status = 'ABORTED' else: self._regular_exit() def _regular_exit(self): self.status = 'FINISHED' try: self.str = '\n'.join(map(decode, self.outlist)) except IndexError: self.str = 'no result' def run(self): "Run the inner generator" for none in self._genobj: pass def kill(self): "Set a TOBEKILLED status" self.status = 'TOBEKILLED' def wait(self): "Wait for the task to finish: to be overridden" @property def traceback(self): "Return the traceback as a (possibly empty) string" if self.tb is None: return '' elif isinstance(self.tb, (str, bytes)): return self.tb else: return ''.join(traceback.format_tb(self.tb)) @property def result(self): self.wait() if self.exc: if isinstance(self.tb, (str, bytes)): raise self.etype(self.tb) else: raise_(self.etype, self.exc, self.tb or None) if not self.outlist: return None return self.outlist[-1] def __repr__(self): "String representation containing class name, number, arglist, status" return '<%s %d [%s] %s>' % ( self.__class__.__name__, self.no, ' '.join(self.arglist), self.status) nulltask = BaseTask(0, [], ('skip' for dummy in (1,))) # ######################## synchronous tasks ############################## # class SynTask(BaseTask): """ Synchronous task running in the interpreter loop and displaying its output as soon as available. """ def __str__(self): "Return the output string or the error message" if self.etype: # there was an error return '%s: %s' % (self.etype.__name__, self.exc) else: return '\n'.join(map(str, self.outlist)) class ThreadedTask(BaseTask): """ A task running in a separated thread. """ def __init__(self, no, arglist, genobj): BaseTask.__init__(self, no, arglist, genobj) self.thread = threading.Thread(target=super(ThreadedTask, self).run) def run(self): "Run the task into a thread" self.thread.start() def wait(self): "Block until the thread ends" self.thread.join() # ######################## multiprocessing tasks ######################### # def sharedattr(name, on_error): "Return a property to be attached to an MPTask" def get(self): try: return getattr(self.ns, name) except: # the process was killed or died hard return on_error def set(self, value): try: setattr(self.ns, name, value) except: # the process was killed or died hard pass return property(get, set) class MPTask(BaseTask): """ A task running as an external process. The current implementation only works on Unix-like systems, where multiprocessing use forks. """ str = sharedattr('str', '') etype = sharedattr('etype', None) exc = sharedattr('exc', None) tb = sharedattr('tb', None) status = sharedattr('status', 'ABORTED') @property def outlist(self): try: return self._outlist except: # the process died hard return [] def notify(self, msg): self.man.notify_listener(self.no, msg) def __init__(self, no, arglist, genobj, manager): """ The monitor has a .send method and a .man multiprocessing.Manager """ self.no = no self.arglist = arglist self._genobj = self._wrap(genobj, stringify_tb=True) self.man = manager self._outlist = manager.mp.list() self.ns = manager.mp.Namespace() self.status = 'SUBMITTED' self.etype, self.exc, self.tb = None, None, None self.str = repr(self) self.proc = multiprocessing.Process(target=super(MPTask, self).run) def run(self): "Run the task into an external process" self.proc.start() def wait(self): "Block until the external process ends or is killed" self.proc.join() def kill(self): """Kill the process with a SIGTERM inducing a TerminatedProcess exception in the children""" self.proc.terminate() # ######################## Task Manager ###################### # class TaskManager(object): """ Store the given commands into a task registry. Provides methods to manage the submitted tasks. """ cmdprefix = '.' specialcommands = set(['.last_tb']) def __init__(self, obj): self.obj = obj self.registry = {} # {taskno : task} if obj.mpcommands or obj.thcommands: self.specialcommands.update(['.kill', '.list', '.output']) interact = getattr(obj, '_interact_', False) self.parser = plac_core.parser_from( obj, prog='' if interact else None, formatter_class=PlacFormatter) HelpSummary.add(obj, self.specialcommands) self.man = Manager() if obj.mpcommands else None signal.signal(signal.SIGTERM, terminatedProcess) def close(self): "Kill all the running tasks" for task in self.registry.values(): try: if task.status == 'RUNNING': task.kill() task.wait() except: # task killed, nothing to wait pass if self.man: self.man.stop() def _get_latest(self, taskno=-1, status=None): "Get the latest submitted task from the registry" assert taskno < 0, 'You must pass a negative number' if status: tasks = [t for t in self.registry.values() if t.status == status] else: tasks = [t for t in self.registry.values()] tasks.sort(key=attrgetter('no')) if len(tasks) >= abs(taskno): return tasks[taskno] # ########################## special commands ######################## # @plac_core.annotations( taskno=('task to kill', 'positional', None, int)) def kill(self, taskno=-1): 'kill the given task (-1 to kill the latest running task)' if taskno < 0: task = self._get_latest(taskno, status='RUNNING') if task is None: yield 'Nothing to kill' return elif taskno not in self.registry: yield 'Unknown task %d' % taskno return else: task = self.registry[taskno] if task.status in ('ABORTED', 'KILLED', 'FINISHED'): yield 'Already finished %s' % task return task.kill() yield task @plac_core.annotations( status=('', 'positional', None, str, BaseTask.STATES)) def list(self, status='RUNNING'): 'list tasks with a given status' for task in self.registry.values(): if task.status == status: yield task @plac_core.annotations( taskno=('task number', 'positional', None, int)) def output(self, taskno=-1, fname=None): 'show the output of a given task (and optionally save it to a file)' if taskno < 0: task = self._get_latest(taskno) if task is None: yield 'Nothing to show' return elif taskno not in self.registry: yield 'Unknown task %d' % taskno return else: task = self.registry[taskno] outstr = '\n'.join(map(str, task.outlist)) if fname: open(fname, 'w').write(outstr) yield 'saved output of %d into %s' % (taskno, fname) return yield task if len(task.outlist) > 20 and use_less: less(outstr) # has no meaning for a plac server else: yield outstr @plac_core.annotations( taskno=('task number', 'positional', None, int)) def last_tb(self, taskno=-1): "show the traceback of a given task, if any" task = self._get_latest(taskno) if task: yield task.traceback else: yield 'Nothing to show' # ########################## SyncProcess ############################# # class Process(subprocess.Popen): "Start the interpreter specified by the params in a subprocess" def __init__(self, params): signal.signal(signal.SIGPIPE, signal.SIG_DFL) # to avoid broken pipe messages code = '''import plac, sys sys.argv[0] = '<%s>' plac.Interpreter(plac.import_main(*%s)).interact(prompt='i>\\n') ''' % (params[0], params) subprocess.Popen.__init__( self, [sys.executable, '-u', '-c', code], stdin=subprocess.PIPE, stdout=subprocess.PIPE) self.man = multiprocessing.Manager() def close(self): "Close stdin and stdout" self.stdin.close() self.stdout.close() self.man.shutdown() def recv(self): # char-by-char cannot work "Return the output of the subprocess, line-by-line until the prompt" lines = [] while True: lines.append(self.stdout.readline()) if lines[-1] == 'i>\n': out = ''.join(lines) return out[:-1] + ' ' # remove last newline def send(self, line): """Send a line (adding a newline) to the underlying subprocess and wait for the answer""" self.stdin.write(line + os.linesep) return self.recv() class StartStopObject(object): started = False def start(self): pass def stop(self): pass class Monitor(StartStopObject): """ Base monitor class with methods add_listener/del_listener/notify_listener read_queue and and start/stop. """ def __init__(self, name, queue=None): self.name = name self.queue = queue or multiprocessing.Queue() def add_listener(self, taskno): pass def del_listener(self, taskno): pass def notify_listener(self, taskno, msg): pass def start(self): pass def stop(self): pass def read_queue(self): pass class Manager(StartStopObject): """ The plac Manager contains a multiprocessing.Manager and a set of slave monitor processes to which we can send commands. There is a manager for each interpreter with mpcommands. """ def __init__(self): self.registry = {} self.started = False self.mp = None def add(self, monitor): 'Add or replace a monitor in the registry' proc = multiprocessing.Process(None, monitor.start, monitor.name) proc.queue = monitor.queue self.registry[monitor.name] = proc def delete(self, name): 'Remove a named monitor from the registry' del self.registry[name] # can be called more than once def start(self): if self.mp is None: self.mp = multiprocessing.Manager() for monitor in self.registry.values(): monitor.start() self.started = True def stop(self): for monitor in self.registry.values(): monitor.queue.close() monitor.terminate() if self.mp: self.mp.shutdown() self.mp = None self.started = False def notify_listener(self, taskno, msg): for monitor in self.registry.values(): monitor.queue.put(('notify_listener', taskno, msg)) def add_listener(self, no): for monitor in self.registry.values(): monitor.queue.put(('add_listener', no)) # ######################### plac server ############################# # import asyncore import asynchat import socket class _AsynHandler(asynchat.async_chat): "asynchat handler starting a new interpreter loop for each connection" terminator = '\r\n' # the standard one for telnet prompt = 'i> ' def __init__(self, socket, interpreter): asynchat.async_chat.__init__(self, socket) self.set_terminator(self.terminator) self.i = interpreter self.i.__enter__() self.data = [] self.write(self.prompt) def write(self, data, *args): "Push a string back to the client" if args: data %= args if data.endswith('\n') and not data.endswith(self.terminator): data = data[:-1] + self.terminator # fix newlines self.push(data) def collect_incoming_data(self, data): "Collect one character at the time" self.data.append(data) def found_terminator(self): "Put in the queue the line received from the client" line = ''.join(self.data) self.log('Received line %r from %s' % (line, self.addr)) if line == 'EOF': self.i.__exit__(None, None, None) self.handle_close() else: task = self.i.submit(line) task.run() # synchronous or not if task.etype: # manage exception error = '%s: %s\nReceived: %s' % ( task.etype.__name__, task.exc, ' '.join(task.arglist)) self.log_info(task.traceback + error) # on the server self.write(error + self.terminator) # back to the client else: # no exception self.write(task.str + self.terminator) self.data = [] self.write(self.prompt) class _AsynServer(asyncore.dispatcher): "asyncore-based server spawning AsynHandlers" def __init__(self, interpreter, newhandler, port, listen=5): self.interpreter = interpreter self.newhandler = newhandler self.port = port asyncore.dispatcher.__init__(self) self.create_socket(socket.AF_INET, socket.SOCK_STREAM) self.bind(('', port)) self.listen(listen) def handle_accept(self): clientsock, clientaddr = self.accept() self.log('Connected from %s' % str(clientaddr)) i = self.interpreter.__class__(self.interpreter.obj) # new interpreter self.newhandler(clientsock, i) # spawn a new handler # ########################## the Interpreter ############################ # class Interpreter(object): """ A context manager with a .send method and a few utility methods: execute, test and doctest. """ class Exit(Exception): pass def __init__(self, obj, commentchar='#', split=shlex.split): self.obj = obj try: self.name = obj.__module__ except AttributeError: self.name = 'plac' self.commentchar = commentchar self.split = split self._set_commands(obj) self.tm = TaskManager(obj) self.man = self.tm.man self.parser = self.tm.parser if self.commands: self.parser.addsubcommands( self.tm.specialcommands, self.tm, title='special commands') if obj.mpcommands: self.parser.addsubcommands( obj.mpcommands, obj, title='commands run in external processes') if obj.thcommands: self.parser.addsubcommands( obj.thcommands, obj, title='threaded commands') self.parser.error = lambda msg: sys.exit(msg) # patch the parser self._interpreter = None def _set_commands(self, obj): "Make sure obj has the right command attributes as Python sets" for attrname in ('commands', 'mpcommands', 'thcommands'): setattr(self, attrname, set(getattr(self.__class__, attrname, []))) setattr(obj, attrname, set(getattr(obj, attrname, []))) self.commands = obj.commands self.mpcommands.update(obj.mpcommands) self.thcommands.update(obj.thcommands) if (obj.commands or obj.mpcommands or obj.thcommands) and \ not hasattr(obj, 'help'): # add default help obj.help = default_help.__get__(obj, obj.__class__) self.commands.add('help') def __enter__(self): "Start the inner interpreter loop" self._interpreter = self._make_interpreter() self._interpreter.send(None) return self def __exit__(self, exctype, exc, tb): "Close the inner interpreter and the task manager" self.close(exctype, exc, tb) def submit(self, line): "Send a line to the underlying interpreter and return a task object" if self._interpreter is None: raise RuntimeError(_('%r not initialized: probably you forgot to ' 'use the with statement') % self) if isinstance(line, (str, bytes)): arglist = self.split(line, self.commentchar) else: # expects a list of strings arglist = line if not arglist: return nulltask m = self.tm.man # manager if m and not m.started: m.start() task = self._interpreter.send(arglist) # nonblocking if not plac_core._match_cmd(arglist[0], self.tm.specialcommands): self.tm.registry[task.no] = task if m: m.add_listener(task.no) return task def send(self, line): """Send a line to the underlying interpreter and return the finished task""" task = self.submit(line) BaseTask.run(task) # blocking return task def tasks(self): "The full lists of the submitted tasks" return self.tm.registry.values() def close(self, exctype=None, exc=None, tb=None): "Can be called to close the interpreter prematurely" self.tm.close() if exctype is not None: self._interpreter.throw(exctype, exc, tb) else: self._interpreter.close() def _make_interpreter(self): "The interpreter main loop, from lists of arguments to task objects" enter = getattr(self.obj, '__enter__', lambda: None) exit = getattr(self.obj, '__exit__', lambda et, ex, tb: None) enter() task = None try: for no in itertools.count(1): arglist = yield task try: cmd, result = self.parser.consume(arglist) except SystemExit as e: # for invalid commands if e.args == (0,): # raised as sys.exit(0) errlist = [] else: errlist = [str(e)] task = SynTask(no, arglist, iter(errlist)) continue except: # anything else task = SynTask(no, arglist, gen_exc(*sys.exc_info())) continue if not plac_core.iterable(result): # atomic result task = SynTask(no, arglist, gen_val(result)) elif cmd in self.obj.mpcommands: task = MPTask(no, arglist, result, self.tm.man) elif cmd in self.obj.thcommands: task = ThreadedTask(no, arglist, result) else: # blocking task task = SynTask(no, arglist, result) except GeneratorExit: # regular exit exit(None, None, None) except: # exceptional exit exit(*sys.exc_info()) raise def check(self, given_input, expected_output): "Make sure you get the expected_output from the given_input" output = self.send(given_input).str # blocking ok = (output == expected_output) if not ok: # the message here is not internationalized on purpose msg = 'input: %s\noutput: %s\nexpected: %s' % ( given_input, output, expected_output) raise AssertionError(msg) def _parse_doctest(self, lineiter): "Returns the lines of input, the lines of output, and the line number" lines = [line.strip() for line in lineiter] inputs = [] positions = [] for i, line in enumerate(lines): if line.startswith('i> '): inputs.append(line[3:]) positions.append(i) positions.append(len(lines) + 1) # last position outputs = [] for i, start in enumerate(positions[:-1]): end = positions[i + 1] outputs.append('\n'.join(lines[start+1:end])) return zip(inputs, outputs, positions) def doctest(self, lineiter, verbose=False): """ Parse a text containing doctests in a context and tests of all them. Raise an error even if a single doctest if broken. Use this for sequential tests which are logically grouped. """ with self: try: for input, output, no in self._parse_doctest(lineiter): if verbose: write('i> %s\n' % input) write('-> %s\n' % output) task = self.send(input) # blocking if not str(task) == output: msg = ('line %d: input: %s\noutput: %s\nexpected: %s\n' % (no + 1, input, task, output)) write(msg) if task.exc: raise_(task.etype, task.exc, task.tb) except self.Exit: pass def execute(self, lineiter, verbose=False): "Execute a lineiter of commands in a context and print the output" with self: try: for line in lineiter: if verbose: write('i> ' + line) task = self.send(line) # finished task if task.etype: # there was an error raise_(task.etype, task.exc, task.tb) write('%s\n' % task.str) except self.Exit: pass def multiline(self, stdin=sys.stdin, terminator=';', verbose=False): "The multiline mode is especially suited for usage with emacs" with self: try: for line in read_long_line(stdin, terminator): task = self.submit(line) task.run() write('%s\n' % task.str) if verbose and task.traceback: write(task.traceback) except self.Exit: pass def interact(self, stdin=sys.stdin, prompt='i> ', verbose=False): "Starts an interactive command loop reading commands from the console" try: import readline readline_present = True except ImportError: readline_present = False if stdin is sys.stdin and readline_present: # use readline histfile = os.path.expanduser('~/.%s.history' % self.name) completions = list(self.commands) + list(self.mpcommands) + \ list(self.thcommands) + list(self.tm.specialcommands) self.stdin = ReadlineInput(completions, histfile=histfile) else: self.stdin = stdin self.prompt = prompt self.verbose = verbose intro = self.obj.__doc__ or '' write(intro + '\n') with self: self.obj._interact_ = True if self.stdin is sys.stdin: # do not close stdin automatically self._manage_input() else: with self.stdin: # close stdin automatically self._manage_input() def _manage_input(self): "Convert input lines into task which are then executed" try: for line in iter(lambda: read_line(self.stdin, self.prompt), ''): line = line.strip() if not line: continue task = self.submit(line) task.run() # synchronous or not write(str(task) + '\n') if self.verbose and task.etype: write(task.traceback) except self.Exit: pass def start_server(self, port=2199, **kw): """Starts an asyncore server reading commands for clients and opening a new interpreter for each connection.""" _AsynServer(self, _AsynHandler, port) # register the server try: asyncore.loop(**kw) except (KeyboardInterrupt, TerminatedProcess): pass finally: asyncore.close_all() def add_monitor(self, mon): self.man.add(mon) def del_monitor(self, name): self.man.delete(name) @classmethod def call(cls, factory, arglist=sys.argv[1:], commentchar='#', split=shlex.split, stdin=sys.stdin, prompt='i> ', verbose=False): """ Call a container factory with the arglist and instantiate an interpreter object. If there are remaining arguments, send them to the interpreter, else start an interactive session. """ obj = partial_call(factory, arglist) i = cls(obj, commentchar, split) if i.obj._args_: with i: task = i.send(i.obj._args_) # synchronous if task.exc: raise_(task.etype, task.exc, task.tb) out = str(task) if out: print(out) elif i.obj._interact_: i.interact(stdin, prompt, verbose) else: i.parser.print_usage() # ################################## runp ################################### # class _TaskLauncher(object): "Helper for runp" def __init__(self, genseq, mode): if mode == 'p': self.mpcommands = ['rungen'] else: self.thcommands = ['rungen'] self.genlist = list(genseq) def rungen(self, i): for out in self.genlist[int(i) - 1]: yield out def runp(genseq, mode='p'): """Run a sequence of generators in parallel. Mode can be 'p' (use processes) or 't' (use threads). After all of them are finished, return a list of task objects. """ assert mode in 'pt', mode launcher = _TaskLauncher(genseq, mode) res = [] with Interpreter(launcher) as inter: for i in range(len(launcher.genlist)): inter.submit('rungen %d' % (i + 1)).run() for task in inter.tasks(): try: res.append(task.result) except Exception as e: res.append(e) return res plac-1.3.4/plac_runner.py000077500000000000000000000045161415326361200153340ustar00rootroot00000000000000#!/usr/bin/env python from __future__ import with_statement import os import sys import shlex import plac def run(fnames, cmd, verbose): "Run batch scripts and tests" for fname in fnames: with open(fname) as f: lines = list(f) if not lines[0].startswith('#!'): sys.exit('Missing or incorrect shebang line!') firstline = lines[0][2:] # strip the shebang init_args = shlex.split(firstline) tool = plac.import_main(*init_args) command = getattr(plac.Interpreter(tool), cmd) # doctest or execute if verbose: sys.stdout.write('Running %s with %s' % (fname, firstline)) command(lines[1:], verbose=verbose) @plac.annotations( verbose=('verbose mode', 'flag', 'v'), interactive=('run plac tool in interactive mode', 'flag', 'i'), multiline=('run plac tool in multiline mode', 'flag', 'm'), serve=('run plac server', 'option', 's', int), batch=('run plac batch files', 'flag', 'b'), test=('run plac test files', 'flag', 't'), fname='script to run (.py or .plac or .placet)', extra='additional arguments', ) def main(verbose, interactive, multiline, serve, batch, test, fname='', *extra): "Runner for plac tools, plac batch files and plac tests" baseparser = plac.parser_from(main) if not fname: baseparser.print_help() elif sys.argv[1] == fname: # script mode plactool = plac.import_main(fname) plactool.prog = os.path.basename(sys.argv[0]) + ' ' + fname out = plac.call(plactool, sys.argv[2:], eager=False) if plac.iterable(out): for output in out: print(output) else: print(out) elif interactive or multiline or serve: plactool = plac.import_main(fname, *extra) plactool.prog = '' i = plac.Interpreter(plactool) if interactive: i.interact(verbose=verbose) elif multiline: i.multiline(verbose=verbose) elif serve: i.start_server(serve) elif batch: run((fname,) + extra, 'execute', verbose) elif test: run((fname,) + extra, 'doctest', verbose) print('run %s plac test(s)' % (len(extra) + 1)) else: baseparser.print_usage() main.add_help = False if __name__ == '__main__': plac.call(main) plac-1.3.4/plac_tk.py000066400000000000000000000036401415326361200144330ustar00rootroot00000000000000from __future__ import print_function import os import sys if sys.version_info < (3,): import Queue as queue else: import queue import plac_core from Tkinter import Tk from ScrolledText import ScrolledText from plac_ext import Monitor, TerminatedProcess class TkMonitor(Monitor): """ An interface over a dictionary {taskno: scrolledtext widget}, with methods add_listener, del_listener, notify_listener and start/stop. """ def __init__(self, name, queue=None): Monitor.__init__(self, name, queue) self.widgets = {} @plac_core.annotations(taskno=('task number', 'positional', None, int)) def add_listener(self, taskno): "There is a ScrolledText for each task" st = ScrolledText(self.root, height=5) st.insert('end', 'Output of task %d\n' % taskno) st.pack() self.widgets[taskno] = st @plac_core.annotations(taskno=('task number', 'positional', None, int)) def del_listener(self, taskno): del self.widgets[taskno] @plac_core.annotations(taskno=('task number', 'positional', None, int)) def notify_listener(self, taskno, msg): w = self.widgets[taskno] w.insert('end', msg + '\n') w.update() def start(self): 'Start the mainloop' self.root = Tk() self.root.title(self.name) self.root.wm_protocol("WM_DELETE_WINDOW", self.stop) self.root.after(0, self.read_queue) try: self.root.mainloop() except KeyboardInterrupt: print('Process %d killed by CTRL-C' % os.getpid(), file=sys.stderr) except TerminatedProcess: pass def stop(self): self.root.quit() def read_queue(self): try: cmd_args = self.queue.get_nowait() except queue.Empty: pass else: getattr(self, cmd_args[0])(*cmd_args[1:]) self.root.after(100, self.read_queue) plac-1.3.4/setup.cfg000066400000000000000000000000761415326361200142650ustar00rootroot00000000000000[bdist_wheel] universal = 1 #[upload_docs] #upload-dir = doc plac-1.3.4/setup.py000066400000000000000000000035121415326361200141540ustar00rootroot00000000000000from setuptools import setup import os.path def require(*modules): """Check if the given modules are already available; if not add them to the dependency list.""" deplist = [] for module in modules: try: __import__(module) except ImportError: deplist.append(module) return deplist def getversion(fname): "Get the __version__ without importing plac" for line in open(fname): if line.startswith('__version__'): return eval(line[13:]) if __name__ == '__main__': setup(name='plac', version=getversion( os.path.join(os.path.dirname(__file__), 'plac.py')), description=('The smartest command line arguments parser ' 'in the world'), long_description=open('README.md').read(), long_description_content_type="text/markdown", author='Michele Simionato', author_email='michele.simionato@gmail.com', url='https://github.com/ialbert/plac', license="BSD License", py_modules=['plac_core', 'plac_ext', 'plac_tk', 'plac'], scripts=['plac_runner.py'], install_requires=require('argparse'), keywords="command line arguments parser", platforms=["All"], classifiers=['Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'License :: OSI Approved :: BSD License', 'Natural Language :: English', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Topic :: Software Development :: Libraries', 'Topic :: Utilities'], zip_safe=False)