pax_global_header00006660000000000000000000000064122207036610014511gustar00rootroot0000000000000052 comment=89f93f4a641cb8134313ad6f3aa08aa0d1642928 voluptuous-0.8.2/000077500000000000000000000000001222070366100137655ustar00rootroot00000000000000voluptuous-0.8.2/.gitignore000077500000000000000000000001421222070366100157550ustar00rootroot00000000000000*.gem *.swp *.pyc *#* build dist .svn/* .DS_Store *.so .Python *.egg-info .coverage .tox MANIFEST voluptuous-0.8.2/.travis.yml000066400000000000000000000003671222070366100161040ustar00rootroot00000000000000language: python python: - "2.6" - "2.7" # Not quite ready for prime time... - "3.2" - "3.3" - "pypy" # command to install dependencies #install: "pip install -r requirements.txt --use-mirrors" # command to run tests script: nosetests voluptuous-0.8.2/COPYING000066400000000000000000000027161222070366100150260ustar00rootroot00000000000000Copyright (c) 2010, Alec Thomas All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - Neither the name of SwapOff.org nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. voluptuous-0.8.2/MANIFEST.in000066400000000000000000000000161222070366100155200ustar00rootroot00000000000000include *.rst voluptuous-0.8.2/README.md000066400000000000000000000301701222070366100152450ustar00rootroot00000000000000# Voluptuous is a Python data validation library [![Build Status](https://travis-ci.org/alecthomas/voluptuous.png)](https://travis-ci.org/alecthomas/voluptuous) Voluptuous, *despite* the name, is a Python data validation library. It is primarily intended for validating data coming into Python as JSON, YAML, etc. It has three goals: 1. Simplicity. 2. Support for complex data structures. 3. Provide useful error messages. ## Contact Voluptuous now has a mailing list! Send a mail to [](mailto:voluptuous@librelist.com) to subscribe. Instructions will follow. You can also contact me directly via [email](mailto:alec@swapoff.org) or [Twitter](https://twitter.com/alecthomas). To file a bug, create a [new issue](https://github.com/alecthomas/voluptuous/issues/new) on GitHub with a short example of how to replicate the issue. ## Show me an example Twitter's [user search API](https://dev.twitter.com/docs/api/1/get/users/search) accepts query URLs like: ``` $ curl 'http://api.twitter.com/1/users/search.json?q=python&per_page=20&page=1 ``` To validate this we might use a schema like: ```pycon >>> from voluptuous import Schema >>> schema = Schema({ ... 'q': str, ... 'per_page': int, ... 'page': int, ... }) ``` This schema very succinctly and roughly describes the data required by the API, and will work fine. But it has a few problems. Firstly, it doesn't fully express the constraints of the API. According to the API, `per_page` should be restricted to at most 20, defaulting to 5, for example. To describe the semantics of the API more accurately, our schema will need to be more thoroughly defined: ```pycon >>> from voluptuous import Required, All, Length, Range >>> schema = Schema({ ... Required('q'): All(str, Length(min=1)), ... Required('per_page', default=5): All(int, Range(min=1, max=20)), ... 'page': All(int, Range(min=0)), ... }) ``` This schema fully enforces the interface defined in Twitter's documentation, and goes a little further for completeness. "q" is required: ```pycon >>> from voluptuous import MultipleInvalid >>> try: ... schema({}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "required key not provided @ data['q']" True ``` ...must be a string: ```pycon >>> try: ... schema({'q': 123}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "expected str for dictionary value @ data['q']" True ``` ...and must be at least one character in length: ```pycon >>> try: ... schema({'q': ''}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "length of value must be at least 1 for dictionary value @ data['q']" True >>> schema({'q': '#topic'}) == {'q': '#topic', 'per_page': 5} True ``` "per\_page" is a positive integer no greater than 20: ```pycon >>> try: ... schema({'q': '#topic', 'per_page': 900}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "value must be at most 20 for dictionary value @ data['per_page']" True >>> try: ... schema({'q': '#topic', 'per_page': -10}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "value must be at least 1 for dictionary value @ data['per_page']" True ``` "page" is an integer \>= 0: ```pycon >>> try: ... schema({'q': '#topic', 'per_page': 'one'}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) "expected int for dictionary value @ data['per_page']" >>> schema({'q': '#topic', 'page': 1}) == {'q': '#topic', 'page': 1, 'per_page': 5} True ``` ## Defining schemas Schemas are nested data structures consisting of dictionaries, lists, scalars and *validators*. Each node in the input schema is pattern matched against corresponding nodes in the input data. ### Literals Literals in the schema are matched using normal equality checks: ```pycon >>> schema = Schema(1) >>> schema(1) 1 >>> schema = Schema('a string') >>> schema('a string') 'a string' ``` ### Types Types in the schema are matched by checking if the corresponding value is an instance of the type: ```pycon >>> schema = Schema(int) >>> schema(1) 1 >>> try: ... schema('one') ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "expected int" True ``` ### Lists Lists in the schema are treated as a set of valid values. Each element in the schema list is compared to each value in the input data: ```pycon >>> schema = Schema([1, 'a', 'string']) >>> schema([1]) [1] >>> schema([1, 1, 1]) [1, 1, 1] >>> schema(['a', 1, 'string', 1, 'string']) ['a', 1, 'string', 1, 'string'] ``` ### Validation functions Validators are simple callables that raise an `Invalid` exception when they encounter invalid data. The criteria for determining validity is entirely up to the implementation; it may check that a value is a valid username with `pwd.getpwnam()`, it may check that a value is of a specific type, and so on. The simplest kind of validator is a Python function that raises ValueError when its argument is invalid. Conveniently, many builtin Python functions have this property. Here's an example of a date validator: ```pycon >>> from datetime import datetime >>> def Date(fmt='%Y-%m-%d'): ... return lambda v: datetime.strptime(v, fmt) ``` ```pycon >>> schema = Schema(Date()) >>> schema('2013-03-03') datetime.datetime(2013, 3, 3, 0, 0) >>> try: ... schema('2013-03') ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "not a valid value" True ``` In addition to simply determining if a value is valid, validators may mutate the value into a valid form. An example of this is the `Coerce(type)` function, which returns a function that coerces its argument to the given type: ```python def Coerce(type, msg=None): """Coerce a value to a type. If the type constructor throws a ValueError, the value will be marked as Invalid. """ def f(v): try: return type(v) except ValueError: raise Invalid(msg or ('expected %s' % type.__name__)) return f ``` This example also shows a common idiom where an optional human-readable message can be provided. This can vastly improve the usefulness of the resulting error messages. ### Dictionaries Each key-value pair in a schema dictionary is validated against each key-value pair in the corresponding data dictionary: ```pycon >>> schema = Schema({1: 'one', 2: 'two'}) >>> schema({1: 'one'}) {1: 'one'} ``` #### Extra dictionary keys By default any additional keys in the data, not in the schema will trigger exceptions: ```pycon >>> schema = Schema({2: 3}) >>> try: ... schema({1: 2, 2: 3}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "extra keys not allowed @ data[1]" True ``` This behaviour can be altered on a per-schema basis with `Schema(..., extra=True)`: ```pycon >>> schema = Schema({2: 3}, extra=True) >>> schema({1: 2, 2: 3}) {1: 2, 2: 3} ``` It can also be overridden per-dictionary by using the catch-all marker token `extra` as a key: ```pycon >>> from voluptuous import Extra >>> schema = Schema({1: {Extra: object}}) >>> schema({1: {'foo': 'bar'}}) {1: {'foo': 'bar'}} ``` #### Required dictionary keys By default, keys in the schema are not required to be in the data: ```pycon >>> schema = Schema({1: 2, 3: 4}) >>> schema({3: 4}) {3: 4} ``` Similarly to how extra\_ keys work, this behaviour can be overridden per-schema: ```pycon >>> schema = Schema({1: 2, 3: 4}, required=True) >>> try: ... schema({3: 4}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "required key not provided @ data[1]" True ``` And per-key, with the marker token `Required(key)`: ```pycon >>> schema = Schema({Required(1): 2, 3: 4}) >>> try: ... schema({3: 4}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "required key not provided @ data[1]" True >>> schema({1: 2}) {1: 2} ``` #### Optional dictionary keys If a schema has `required=True`, keys may be individually marked as optional using the marker token `Optional(key)`: ```pycon >>> from voluptuous import Optional >>> schema = Schema({1: 2, Optional(3): 4}, required=True) >>> try: ... schema({}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "required key not provided @ data[1]" True >>> schema({1: 2}) {1: 2} >>> try: ... schema({1: 2, 4: 5}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "extra keys not allowed @ data[4]" True ``` ```pycon >>> schema({1: 2, 3: 4}) {1: 2, 3: 4} ``` ### Objects Each key-value pair in a schema dictionary is validated against each attribute-value pair in the corresponding object: ```pycon >>> from voluptuous import Object >>> class Structure(object): ... def __init__(self, q=None): ... self.q = q ... def __repr__(self): ... return ''.format(self) ... >>> schema = Schema(Object({'q': 'one'}, cls=Structure)) >>> schema(Structure(q='one')) ``` ## Error reporting Validators must throw an `Invalid` exception if invalid data is passed to them. All other exceptions are treated as errors in the validator and will not be caught. Each `Invalid` exception has an associated `path` attribute representing the path in the data structure to our currently validating value. This is used during error reporting, but also during matching to determine whether an error should be reported to the user or if the next match should be attempted. This is determined by comparing the depth of the path where the check is, to the depth of the path where the error occurred. If the error is more than one level deeper, it is reported. The upshot of this is that *matching is depth-first and fail-fast*. To illustrate this, here is an example schema: ```pycon >>> schema = Schema([[2, 3], 6]) ``` Each value in the top-level list is matched depth-first in-order. Given input data of `[[6]]`, the inner list will match the first element of the schema, but the literal `6` will not match any of the elements of that list. This error will be reported back to the user immediately. No backtracking is attempted: ```pycon >>> try: ... schema([[6]]) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == "invalid list value @ data[0][0]" True ``` If we pass the data `[6]`, the `6` is not a list type and so will not recurse into the first element of the schema. Matching will continue on to the second element in the schema, and succeed: ```pycon >>> schema([6]) [6] ``` ## Why use Voluptuous over another validation library? **Validators are simple callables** : No need to subclass anything, just use a function. **Errors are simple exceptions.** : A validator can just `raise Invalid(msg)` and expect the user to get useful messages. **Schemas are basic Python data structures.** : Should your data be a dictionary of integer keys to strings? `{int: str}` does what you expect. List of integers, floats or strings? `[int, float, str]`. **Designed from the ground up for validating more than just forms.** : Nested data structures are treated in the same way as any other type. Need a list of dictionaries? `[{}]` **Consistency.** : Types in the schema are checked as types. Values are compared as values. Callables are called to validate. Simple. ## Other libraries and inspirations Voluptuous is heavily inspired by [Validino](http://code.google.com/p/validino/), and to a lesser extent, [jsonvalidator](http://code.google.com/p/jsonvalidator/) and [json\_schema](http://blog.sendapatch.se/category/json_schema.html). I greatly prefer the light-weight style promoted by these libraries to the complexity of libraries like FormEncode. voluptuous-0.8.2/setup.cfg000066400000000000000000000000761222070366100156110ustar00rootroot00000000000000[nosetests] doctest-extension = md with-doctest = 1 where = . voluptuous-0.8.2/setup.py000066400000000000000000000016061222070366100155020ustar00rootroot00000000000000try: from setuptools import setup except ImportError: from distutils.core import setup import sys sys.path.insert(0, '.') version = __import__('voluptuous').__version__ try: import pypandoc long_description = pypandoc.convert('README.md', 'rst') except ImportError: print('WARNING: Could not locate pandoc, using Markdown long_description.') long_description = open('README.md').read() description = long_description.splitlines()[0].strip() setup( name='voluptuous', url='http://github.com/alecthomas/voluptuous', download_url='http://pypi.python.org/pypi/voluptuous', version=version, description=description, long_description=long_description, license='BSD', platforms=['any'], py_modules=['voluptuous'], author='Alec Thomas', author_email='alec@swapoff.org', install_requires=[ 'setuptools >= 0.6b1', ], ) voluptuous-0.8.2/tests.md000066400000000000000000000164571222070366100154660ustar00rootroot00000000000000Error reporting should be accurate: >>> from voluptuous import * >>> schema = Schema(['one', {'two': 'three', 'four': ['five'], ... 'six': {'seven': 'eight'}}]) >>> schema(['one']) ['one'] >>> schema([{'two': 'three'}]) [{'two': 'three'}] It should show the exact index and container type, in this case a list value: >>> try: ... schema(['one', 'two']) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) == 'invalid list value @ data[1]' True It should also be accurate for nested values: >>> try: ... schema([{'two': 'nine'}]) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) "not a valid value for dictionary value @ data[0]['two']" >>> try: ... schema([{'four': ['nine']}]) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) "invalid list value @ data[0]['four'][0]" >>> try: ... schema([{'six': {'seven': 'nine'}}]) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) "not a valid value for dictionary value @ data[0]['six']['seven']" Errors should be reported depth-first: >>> validate = Schema({'one': {'two': 'three', 'four': 'five'}}) >>> try: ... validate({'one': {'four': 'six'}}) ... except Invalid as e: ... print(e) ... print(e.path) not a valid value for dictionary value @ data['one']['four'] ['one', 'four'] Voluptuous supports validation when extra fields are present in the data: >>> schema = Schema({'one': 1, Extra: object}) >>> schema({'two': 'two', 'one': 1}) == {'two': 'two', 'one': 1} True >>> schema = Schema({'one': 1}) >>> try: ... schema({'two': 2}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) "extra keys not allowed @ data['two']" dict, list, and tuple should be available as type validators: >>> Schema(dict)({'a': 1, 'b': 2}) == {'a': 1, 'b': 2} True >>> Schema(list)([1,2,3]) [1, 2, 3] >>> Schema(tuple)((1,2,3)) (1, 2, 3) Validation should return instances of the right types when the types are subclasses of dict or list: >>> class Dict(dict): ... pass >>> >>> d = Schema(dict)(Dict(a=1, b=2)) >>> d == {'a': 1, 'b': 2} True >>> type(d) is Dict True >>> class List(list): ... pass >>> >>> l = Schema(list)(List([1,2,3])) >>> l [1, 2, 3] >>> type(l) is List True Multiple errors are reported: >>> schema = Schema({'one': 1, 'two': 2}) >>> try: ... schema({'one': 2, 'two': 3, 'three': 4}) ... except MultipleInvalid as e: ... errors = sorted(e.errors, key=lambda k: str(k)) ... print([str(i) for i in errors]) # doctest: +NORMALIZE_WHITESPACE ["extra keys not allowed @ data['three']", "not a valid value for dictionary value @ data['one']", "not a valid value for dictionary value @ data['two']"] >>> schema = Schema([[1], [2], [3]]) >>> try: ... schema([1, 2, 3]) ... except MultipleInvalid as e: ... print([str(i) for i in e.errors]) # doctest: +NORMALIZE_WHITESPACE ['invalid list value @ data[0]', 'invalid list value @ data[1]', 'invalid list value @ data[2]'] Multiple errors for nested fields in dicts and objects: > \>\>\> from collections import namedtuple \>\>\> validate = Schema({ > ... 'anobject': Object({ ... 'strfield': str, ... 'intfield': int ... > }) ... }) \>\>\> try: ... SomeObj = namedtuple('SomeObj', ('strfield', > 'intfield')) ... validate({'anobject': SomeObj(strfield=123, > intfield='one')}) ... except MultipleInvalid as e: ... > print(sorted(str(i) for i in e.errors)) \# doctest: > +NORMALIZE\_WHITESPACE ["expected int for object value @ > data['anobject']['intfield']", "expected str for object value @ > data['anobject']['strfield']"] Custom classes validate as schemas: >>> class Thing(object): ... pass >>> schema = Schema(Thing) >>> t = schema(Thing()) >>> type(t) is Thing True Classes with custom metaclasses should validate as schemas: >>> class MyMeta(type): ... pass >>> class Thing(object): ... __metaclass__ = MyMeta >>> schema = Schema(Thing) >>> t = schema(Thing()) >>> type(t) is Thing True Schemas built with All() should give the same error as the original validator (Issue \#26): >>> schema = Schema({ ... Required('items'): All([{ ... Required('foo'): str ... }]) ... }) >>> try: ... schema({'items': [{}]}) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) "required key not provided @ data['items'][0]['foo']" Validator should return same instance of the same type for object: >>> class Structure(object): ... def __init__(self, q=None): ... self.q = q ... def __repr__(self): ... return '{0.__name__}(q={1.q!r})'.format(type(self), self) ... >>> schema = Schema(Object({'q': 'one'}, cls=Structure)) >>> type(schema(Structure(q='one'))) is Structure True Object validator should treat cls argument as optional. In this case it shouldn't check object type: >>> from collections import namedtuple >>> NamedTuple = namedtuple('NamedTuple', ('q',)) >>> schema = Schema(Object({'q': 'one'})) >>> named = NamedTuple(q='one') >>> schema(named) == named True >>> schema(named) NamedTuple(q='one') If cls argument passed to object validator we should check object type: >>> schema = Schema(Object({'q': 'one'}, cls=Structure)) >>> schema(NamedTuple(q='one')) # doctest: +IGNORE_EXCEPTION_DETAIL Traceback (most recent call last): ... MultipleInvalid: expected a >>> schema = Schema(Object({'q': 'one'}, cls=NamedTuple)) >>> schema(NamedTuple(q='one')) NamedTuple(q='one') Ensure that objects with \_\_slots\_\_ supported properly: >>> class SlotsStructure(Structure): ... __slots__ = ['q'] ... >>> schema = Schema(Object({'q': 'one'})) >>> schema(SlotsStructure(q='one')) SlotsStructure(q='one') >>> class DictStructure(object): ... __slots__ = ['q', '__dict__'] ... def __init__(self, q=None, page=None): ... self.q = q ... self.page = page ... def __repr__(self): ... return '{0.__name__}(q={1.q!r}, page={1.page!r})'.format(type(self), self) ... >>> structure = DictStructure(q='one') >>> structure.page = 1 >>> try: ... schema(structure) ... raise AssertionError('MultipleInvalid not raised') ... except MultipleInvalid as e: ... exc = e >>> str(exc) "extra keys not allowed @ data['page']" >>> schema = Schema(Object({'q': 'one', Extra: object})) >>> schema(structure) DictStructure(q='one', page=1) Ensure that objects can be used with other validators: >>> schema = Schema({'meta': Object({'q': 'one'})}) >>> schema({'meta': Structure(q='one')}) {'meta': Structure(q='one')} voluptuous-0.8.2/tox.ini000066400000000000000000000005351222070366100153030ustar00rootroot00000000000000[tox] envlist = py26,py27 [testenv] distribute = True sitepackages = False deps = nose nose-cover3 coverage>=3.0 commands = nosetests \ --with-coverage3 \ --cover3-package=voluptuous \ --cover3-branch \ --verbose [testenv:py26] basepython = python2.6 [testenv:py27] basepython = python2.7 voluptuous-0.8.2/voluptuous.py000066400000000000000000000730151222070366100166120ustar00rootroot00000000000000# encoding: utf-8 # # Copyright (C) 2010-2013 Alec Thomas # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. # # Author: Alec Thomas """Schema validation for Python data structures. Given eg. a nested data structure like this: { 'exclude': ['Users', 'Uptime'], 'include': [], 'set': { 'snmp_community': 'public', 'snmp_timeout': 15, 'snmp_version': '2c', }, 'targets': { 'localhost': { 'exclude': ['Uptime'], 'features': { 'Uptime': { 'retries': 3, }, 'Users': { 'snmp_community': 'monkey', 'snmp_port': 15, }, }, 'include': ['Users'], 'set': { 'snmp_community': 'monkeys', }, }, }, } A schema like this: >>> settings = { ... 'snmp_community': str, ... 'retries': int, ... 'snmp_version': All(Coerce(str), Any('3', '2c', '1')), ... } >>> features = ['Ping', 'Uptime', 'Http'] >>> schema = Schema({ ... 'exclude': features, ... 'include': features, ... 'set': settings, ... 'targets': { ... 'exclude': features, ... 'include': features, ... 'features': { ... str: settings, ... }, ... }, ... }) Validate like so: >>> schema({ ... 'set': { ... 'snmp_community': 'public', ... 'snmp_version': '2c', ... }, ... 'targets': { ... 'exclude': ['Ping'], ... 'features': { ... 'Uptime': {'retries': 3}, ... 'Users': {'snmp_community': 'monkey'}, ... }, ... }, ... }) == { ... 'set': {'snmp_version': '2c', 'snmp_community': 'public'}, ... 'targets': { ... 'exclude': ['Ping'], ... 'features': {'Uptime': {'retries': 3}, ... 'Users': {'snmp_community': 'monkey'}}}} True """ from functools import wraps import os import re import sys from contextlib import contextmanager if sys.version > '3': import urllib.parse as urlparse long = int unicode = str basestring = str ifilter = filter iteritems = dict.items else: from itertools import ifilter import urlparse iteritems = dict.iteritems __author__ = 'Alec Thomas ' __version__ = '0.8.2' @contextmanager def raises(exc, msg=None): try: yield except exc as e: if msg is not None: assert str(e) == msg, '%r != %r' % (str(e), msg) class Undefined(object): def __nonzero__(self): return False def __repr__(self): return '...' UNDEFINED = Undefined() class Error(Exception): """Base validation exception.""" class SchemaError(Error): """An error was encountered in the schema.""" class Invalid(Error): """The data was invalid. :attr msg: The error message. :attr path: The path to the error, as a list of keys in the source data. """ def __init__(self, message, path=None): Error.__init__(self, message) self.path = path or [] @property def msg(self): return self.args[0] def __str__(self): path = ' @ data[%s]' % ']['.join(map(repr, self.path)) \ if self.path else '' return Exception.__str__(self) + path class MultipleInvalid(Invalid): def __init__(self, errors=None): self.errors = errors[:] if errors else [] def __repr__(self): return 'MultipleInvalid(%r)' % self.errors @property def msg(self): return self.errors[0].msg @property def path(self): return self.errors[0].path def add(self, error): self.errors.append(error) def __str__(self): return str(self.errors[0]) class Schema(object): """A validation schema. The schema is a Python tree-like structure where nodes are pattern matched against corresponding trees of values. Nodes can be values, in which case a direct comparison is used, types, in which case an isinstance() check is performed, or callables, which will validate and optionally convert the value. """ def __init__(self, schema, required=False, extra=False): """Create a new Schema. :param schema: Validation schema. See :module:`voluptuous` for details. :param required: Keys defined in the schema must be in the data. :param extra: Keys in the data need not have keys in the schema. """ self.schema = schema self.required = required self.extra = extra self._compiled = self._compile(schema) def __call__(self, data): """Validate data against this schema.""" try: return self._compiled([], data) except MultipleInvalid: raise except Invalid as e: raise MultipleInvalid([e]) # return self.validate([], self.schema, data) def _compile(self, schema): if schema is Extra: return lambda _, v: v if isinstance(schema, Object): return self._compile_object(schema) if isinstance(schema, dict): return self._compile_dict(schema) elif isinstance(schema, list): return self._compile_list(schema) elif isinstance(schema, tuple): return self._compile_tuple(schema) type_ = type(schema) if type_ is type: type_ = schema if type_ in (int, long, str, unicode, float, complex, object, list, dict, type(None)) or callable(schema): return _compile_scalar(schema) raise SchemaError('unsupported schema data type %r' % type(schema).__name__) def _compile_mapping(self, schema, invalid_msg=None): """Create validator for given mapping.""" invalid_msg = ' ' + (invalid_msg or 'for mapping value') default_required_keys = set(key for key in schema if (self.required and not isinstance(key, Optional)) or isinstance(key, Required)) _compiled_schema = {} for skey, svalue in iteritems(schema): new_key = self._compile(skey) new_value = self._compile(svalue) _compiled_schema[skey] = (new_key, new_value) def validate_mapping(path, iterable, out): required_keys = default_required_keys.copy() error = None errors = [] for key, value in iterable: key_path = path + [key] for skey, (ckey, cvalue) in iteritems(_compiled_schema): try: new_key = ckey(key_path, key) except Invalid as e: if len(e.path) > len(key_path): raise if not error or len(e.path) > len(error.path): error = e continue # Backtracking is not performed once a key is selected, so if # the value is invalid we immediately throw an exception. exception_errors = [] try: out[new_key] = cvalue(key_path, value) except MultipleInvalid as e: exception_errors.extend(e.errors) except Invalid as e: exception_errors.append(e) if exception_errors: for err in exception_errors: if len(err.path) > len(key_path): errors.append(err) else: errors.append( Invalid(err.msg + invalid_msg, err.path)) break # Key and value okay, mark any Required() fields as found. required_keys.discard(skey) break else: if self.extra: out[key] = value else: errors.append(Invalid('extra keys not allowed', key_path)) for key in required_keys: if getattr(key, 'default', UNDEFINED) is not UNDEFINED: out[key.schema] = key.default else: msg = key.msg if hasattr(key, 'msg') and key.msg else 'required key not provided' errors.append(Invalid(msg, path + [key])) if errors: raise MultipleInvalid(errors) return out return validate_mapping def _compile_object(self, schema): """Validate an object. Has the same behavior as dictionary validator but work with object attributes. For example: >>> class Structure(object): ... def __init__(self, one=None, three=None): ... self.one = one ... self.three = three ... >>> validate = Schema(Object({'one': 'two', 'three': 'four'}, cls=Structure)) >>> with raises(MultipleInvalid, "not a valid value for object value @ data['one']"): ... validate(Structure(one='three')) """ base_validate = self._compile_mapping(schema, invalid_msg='for object value') def validate_object(path, data): if schema.cls is not UNDEFINED and not isinstance(data, schema.cls): raise Invalid('expected a {0!r}'.format(schema.cls), path) iterable = _iterate_object(data) iterable = ifilter(lambda item: item[1] is not None, iterable) out = base_validate(path, iterable, {}) return type(data)(**out) return validate_object def _compile_dict(self, schema): """Validate a dictionary. A dictionary schema can contain a set of values, or at most one validator function/type. A dictionary schema will only validate a dictionary: >>> validate = Schema({}) >>> with raises(MultipleInvalid, 'expected a dictionary'): ... validate([]) An invalid dictionary value: >>> validate = Schema({'one': 'two', 'three': 'four'}) >>> with raises(MultipleInvalid, "not a valid value for dictionary value @ data['one']"): ... validate({'one': 'three'}) An invalid key: >>> with raises(MultipleInvalid, "extra keys not allowed @ data['two']"): ... validate({'two': 'three'}) Validation function, in this case the "int" type: >>> validate = Schema({'one': 'two', 'three': 'four', int: str}) Valid integer input: >>> validate({10: 'twenty'}) {10: 'twenty'} By default, a "type" in the schema (in this case "int") will be used purely to validate that the corresponding value is of that type. It will not Coerce the value: >>> with raises(MultipleInvalid, "extra keys not allowed @ data['10']"): ... validate({'10': 'twenty'}) Wrap them in the Coerce() function to achieve this: >>> validate = Schema({'one': 'two', 'three': 'four', ... Coerce(int): str}) >>> validate({'10': 'twenty'}) {10: 'twenty'} Custom message for required key >>> validate = Schema({Required('one', 'required'): 'two'}) >>> with raises(MultipleInvalid, "required @ data['one']"): ... validate({}) (This is to avoid unexpected surprises.) Multiple errors for nested field in a dict: >>> validate = Schema({ ... 'adict': { ... 'strfield': str, ... 'intfield': int ... } ... }) >>> try: ... validate({ ... 'adict': { ... 'strfield': 123, ... 'intfield': 'one' ... } ... }) ... except MultipleInvalid as e: ... print(sorted(str(i) for i in e.errors)) # doctest: +NORMALIZE_WHITESPACE ["expected int for dictionary value @ data['adict']['intfield']", "expected str for dictionary value @ data['adict']['strfield']"] """ base_validate = self._compile_mapping(schema, invalid_msg='for dictionary value') def validate_dict(path, data): if not isinstance(data, dict): raise Invalid('expected a dictionary', path) out = type(data)() return base_validate(path, iteritems(data), out) return validate_dict def _compile_sequence(self, schema, seq_type): """Validate a sequence type. This is a sequence of valid values or validators tried in order. >>> validator = Schema(['one', 'two', int]) >>> validator(['one']) ['one'] >>> with raises(MultipleInvalid, 'invalid list value @ data[0]'): ... validator([3.5]) >>> validator([1]) [1] """ _compiled = [self._compile(s) for s in schema] seq_type_name = seq_type.__name__ def validate_sequence(path, data): if not isinstance(data, seq_type): raise Invalid('expected a %s' % seq_type_name, path) # Empty seq schema, allow any data. if not schema: return data out = [] invalid = None errors = [] index_path = UNDEFINED for i, value in enumerate(data): index_path = path + [i] invalid = None for validate in _compiled: try: out.append(validate(index_path, value)) break except Invalid as e: if len(e.path) > len(index_path): raise invalid = e else: if len(invalid.path) <= len(index_path): invalid = Invalid('invalid %s value' % seq_type_name, index_path) errors.append(invalid) if errors: raise MultipleInvalid(errors) return type(data)(out) return validate_sequence def _compile_tuple(self, schema): """Validate a tuple. A tuple is a sequence of valid values or validators tried in order. >>> validator = Schema(('one', 'two', int)) >>> validator(('one',)) ('one',) >>> with raises(MultipleInvalid, 'invalid tuple value @ data[0]'): ... validator((3.5,)) >>> validator((1,)) (1,) """ return self._compile_sequence(schema, tuple) def _compile_list(self, schema): """Validate a list. A list is a sequence of valid values or validators tried in order. >>> validator = Schema(['one', 'two', int]) >>> validator(['one']) ['one'] >>> with raises(MultipleInvalid, 'invalid list value @ data[0]'): ... validator([3.5]) >>> validator([1]) [1] """ return self._compile_sequence(schema, list) def _compile_scalar(schema): """A scalar value. The schema can either be a value or a type. >>> _compile_scalar(int)([], 1) 1 >>> with raises(Invalid, 'expected float'): ... _compile_scalar(float)([], '1') Callables have >>> _compile_scalar(lambda v: float(v))([], '1') 1.0 As a convenience, ValueError's are trapped: >>> with raises(Invalid, 'not a valid value'): ... _compile_scalar(lambda v: float(v))([], 'a') """ if isinstance(schema, type): def validate_instance(path, data): if isinstance(data, schema): return data else: raise Invalid('expected %s' % schema.__name__, path) return validate_instance if callable(schema): def validate_callable(path, data): try: return schema(data) except ValueError as e: raise Invalid('not a valid value', path) except Invalid as e: raise Invalid(e.msg, path + e.path) return validate_callable def validate_value(path, data): if data != schema: raise Invalid('not a valid value', path) return data return validate_value def _iterate_object(obj): """Return iterator over object attributes. Respect objects with defined __slots__. """ d = {} try: d = vars(obj) except TypeError: # maybe we have named tuple here? if hasattr(obj, '_asdict'): d = obj._asdict() for item in iteritems(d): yield item try: slots = obj.__slots__ except AttributeError: pass else: for key in slots: if key != '__dict__': yield (key, getattr(obj, key)) raise StopIteration() class Object(dict): """Indicate that we should work with attributes, not keys.""" def __init__(self, schema, cls=UNDEFINED): self.cls = cls super(Object, self).__init__(schema) class Marker(object): """Mark nodes for special treatment.""" def __init__(self, schema, msg=None): self.schema = schema self._schema = Schema(schema) self.msg = msg def __call__(self, v): try: return self._schema(v) except Invalid as e: if not self.msg or len(e.path) > 1: raise raise Invalid(self.msg) def __str__(self): return str(self.schema) def __repr__(self): return repr(self.schema) class Optional(Marker): """Mark a node in the schema as optional.""" class Required(Marker): """Mark a node in the schema as being required, and optionally provide a default value. >>> schema = Schema({Required('key'): str}) >>> with raises(MultipleInvalid, "required key not provided @ data['key']"): ... schema({}) >>> schema = Schema({Required('key', default='value'): str}) >>> schema({}) {'key': 'value'} """ def __init__(self, schema, msg=None, default=UNDEFINED): super(Required, self).__init__(schema, msg=msg) self.default = default def Extra(_): """Allow keys in the data that are not present in the schema.""" raise SchemaError('"Extra" should never be called') # As extra() is never called there's no way to catch references to the # deprecated object, so we just leave an alias here instead. extra = Extra def Msg(schema, msg): """Report a user-friendly message if a schema fails to validate. >>> validate = Schema( ... Msg(['one', 'two', int], ... 'should be one of "one", "two" or an integer')) >>> with raises(MultipleInvalid, 'should be one of "one", "two" or an integer'): ... validate(['three']) Messages are only applied to invalid direct descendants of the schema: >>> validate = Schema(Msg([['one', 'two', int]], 'not okay!')) >>> with raises(MultipleInvalid, 'invalid list value @ data[0][0]'): ... validate([['three']]) """ schema = Schema(schema) @wraps(Msg) def f(v): try: return schema(v) except Invalid as e: if len(e.path) > 1: raise e else: raise Invalid(msg) return f def message(default=None): """Convenience decorator to allow functions to provide a message. Set a default message: >>> @message('not an integer') ... def isint(v): ... return int(v) >>> validate = Schema(isint()) >>> with raises(MultipleInvalid, 'not an integer'): ... validate('a') The message can be overridden on a per validator basis: >>> validate = Schema(isint('bad')) >>> with raises(MultipleInvalid, 'bad'): ... validate('a') """ def decorator(f): @wraps(f) def check(msg=None): @wraps(f) def wrapper(*args, **kwargs): try: return f(*args, **kwargs) except ValueError: raise Invalid(msg or default or 'invalid value') return wrapper return check return decorator def truth(f): """Convenience decorator to convert truth functions into validators. >>> @truth ... def isdir(v): ... return os.path.isdir(v) >>> validate = Schema(isdir) >>> validate('/') '/' >>> with raises(MultipleInvalid, 'not a valid value'): ... validate('/notavaliddir') """ @wraps(f) def check(v): t = f(v) if not t: raise ValueError return v return check def Coerce(type, msg=None): """Coerce a value to a type. If the type constructor throws a ValueError or TypeError, the value will be marked as Invalid. Default behavior: >>> validate = Schema(Coerce(int)) >>> with raises(MultipleInvalid, 'expected int'): ... validate(None) >>> with raises(MultipleInvalid, 'expected int'): ... validate('foo') With custom message: >>> validate = Schema(Coerce(int, "moo")) >>> with raises(MultipleInvalid, 'moo'): ... validate('foo') """ @wraps(Coerce) def f(v): try: return type(v) except (ValueError, TypeError): raise Invalid(msg or ('expected %s' % type.__name__)) return f @message('value was not true') @truth def IsTrue(v): """Assert that a value is true, in the Python sense. >>> validate = Schema(IsTrue()) "In the Python sense" means that implicitly false values, such as empty lists, dictionaries, etc. are treated as "false": >>> with raises(MultipleInvalid, "value was not true"): ... validate([]) >>> validate([1]) [1] >>> with raises(MultipleInvalid, "value was not true"): ... validate(False) ...and so on. """ return v @message('value was not false') def IsFalse(v): """Assert that a value is false, in the Python sense. (see :func:`IsTrue` for more detail) >>> validate = Schema(IsFalse()) >>> validate([]) [] """ if v: raise ValueError return v @message('expected boolean') def Boolean(v): """Convert human-readable boolean values to a bool. Accepted values are 1, true, yes, on, enable, and their negatives. Non-string values are cast to bool. >>> validate = Schema(Boolean()) >>> validate(True) True >>> with raises(MultipleInvalid, "expected boolean"): ... validate('moo') """ if isinstance(v, basestring): v = v.lower() if v in ('1', 'true', 'yes', 'on', 'enable'): return True if v in ('0', 'false', 'no', 'off', 'disable'): return False raise ValueError return bool(v) def Any(*validators, **kwargs): """Use the first validated value. :param msg: Message to deliver to user if validation fails. :param kwargs: All other keyword arguments are passed to the sub-Schema constructors. :returns: Return value of the first validator that passes. >>> validate = Schema(Any('true', 'false', ... All(Any(int, bool), Coerce(bool)))) >>> validate('true') 'true' >>> validate(1) True >>> with raises(MultipleInvalid, "not a valid value"): ... validate('moo') """ msg = kwargs.pop('msg', None) schemas = [Schema(val, **kwargs) for val in validators] @wraps(Any) def f(v): error = None for schema in schemas: try: return schema(v) except Invalid as e: if error is None or len(e.path) > len(error.path): error = e else: if error: raise error raise Invalid(msg or 'no valid value found') return f def All(*validators, **kwargs): """Value must pass all validators. The output of each validator is passed as input to the next. :param msg: Message to deliver to user if validation fails. :param kwargs: All other keyword arguments are passed to the sub-Schema constructors. >>> validate = Schema(All('10', Coerce(int))) >>> validate('10') 10 """ msg = kwargs.pop('msg', None) schemas = [Schema(val, **kwargs) for val in validators] def f(v): try: for schema in schemas: v = schema(v) except Invalid as e: raise e if msg is None else Invalid(msg) return v return f def Match(pattern, msg=None): """Value must be a string that matches the regular expression. >>> validate = Schema(Match(r'^0x[A-F0-9]+$')) >>> validate('0x123EF4') '0x123EF4' >>> with raises(MultipleInvalid, "does not match regular expression"): ... validate('123EF4') >>> with raises(MultipleInvalid, 'expected string or buffer'): ... validate(123) Pattern may also be a _compiled regular expression: >>> validate = Schema(Match(re.compile(r'0x[A-F0-9]+', re.I))) >>> validate('0x123ef4') '0x123ef4' """ if isinstance(pattern, basestring): pattern = re.compile(pattern) def f(v): try: match = pattern.match(v) except TypeError: raise Invalid("expected string or buffer") if not match: raise Invalid(msg or 'does not match regular expression') return v return f def Replace(pattern, substitution, msg=None): """Regex substitution. >>> validate = Schema(All(Replace('you', 'I'), ... Replace('hello', 'goodbye'))) >>> validate('you say hello') 'I say goodbye' """ if isinstance(pattern, basestring): pattern = re.compile(pattern) def f(v): return pattern.sub(substitution, v) return f @message('expected a URL') def Url(v): """Verify that the value is a URL.""" try: urlparse.urlparse(v) return v except: raise ValueError @message('not a file') @truth def IsFile(v): """Verify the file exists.""" return os.path.isfile(v) @message('not a directory') @truth def IsDir(v): """Verify the directory exists. >>> IsDir()('/') '/' """ return os.path.isdir(v) @message('path does not exist') @truth def PathExists(v): """Verify the path exists, regardless of its type.""" return os.path.exists(v) def Range(min=None, max=None, min_included=True, max_included=True, msg=None): """Limit a value to a range. Either min or max may be omitted. Either min or max can be excluded from the range of accepted values. :raises Invalid: If the value is outside the range. >>> s = Schema(Range(min=1, max=10, min_included=False)) >>> s(5) 5 >>> s(10) 10 >>> with raises(MultipleInvalid, 'value must be at most 10'): ... s(20) >>> with raises(MultipleInvalid, 'value must be higher than 1'): ... s(1) """ @wraps(Range) def f(v): if min_included: if min is not None and v < min: raise Invalid(msg or 'value must be at least %s' % min) else: if min is not None and v <= min: raise Invalid(msg or 'value must be higher than %s' % min) if max_included: if max is not None and v > max: raise Invalid(msg or 'value must be at most %s' % max) else: if max is not None and v >= max: raise Invalid(msg or 'value must be lower than %s' % max) return v return f def Clamp(min=None, max=None, msg=None): """Clamp a value to a range. Either min or max may be omitted. """ @wraps(Clamp) def f(v): if min is not None and v < min: v = min if max is not None and v > max: v = max return v return f def Length(min=None, max=None, msg=None): """The length of a value must be in a certain range.""" @wraps(Length) def f(v): if min is not None and len(v) < min: raise Invalid(msg or 'length of value must be at least %s' % min) if max is not None and len(v) > max: raise Invalid(msg or 'length of value must be at most %s' % max) return v return f def Lower(v): """Transform a string to lower case. >>> s = Schema(Lower) >>> s('HI') 'hi' """ return str(v).lower() def Upper(v): """Transform a string to upper case. >>> s = Schema(Upper) >>> s('hi') 'HI' """ return str(v).upper() def Capitalize(v): """Capitalise a string. >>> s = Schema(Capitalize) >>> s('hello world') 'Hello world' """ return str(v).capitalize() def Title(v): """Title case a string. >>> s = Schema(Title) >>> s('hello world') 'Hello World' """ return str(v).title() def DefaultTo(default_value, msg=None): """Sets a value to default_value if none provided. >>> s = Schema(DefaultTo(42)) >>> s(None) 42 """ @wraps(DefaultTo) def f(v): if v is None: v = default_value return v return f if __name__ == '__main__': import doctest doctest.testmod()