anymarkup-core-0.8.1/ 0000755 0000765 0000024 00000000000 13623502530 016307 5 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/LICENSE 0000644 0000765 0000024 00000002667 13515571465 017344 0 ustar slavek.kabrda staff 0000000 0000000 Copyright (c) 2015, Slavek Kabrda
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ''AS IS'' AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
anymarkup-core-0.8.1/MANIFEST.in 0000644 0000765 0000024 00000000172 13515571465 020062 0 ustar slavek.kabrda staff 0000000 0000000 include README.rst
include LICENSE
include requirements.txt
recursive-include test *.py
recursive-include test/fixtures *
anymarkup-core-0.8.1/PKG-INFO 0000644 0000765 0000024 00000003651 13623502530 017411 0 ustar slavek.kabrda staff 0000000 0000000 Metadata-Version: 1.1
Name: anymarkup-core
Version: 0.8.1
Summary: Core library for anymarkup
Home-page: https://github.com/bkabrda/anymarkup-core
Author: Slavek Kabrda
Author-email: slavek.kabrda@gmail.com
License: BSD
Description: anymarkup-core
==============
.. image:: https://travis-ci.org/bkabrda/anymarkup-core.svg?branch=master
:target: https://travis-ci.org/bkabrda/anymarkup-core
:alt: Build Status
.. image:: https://landscape.io/github/bkabrda/anymarkup-core/master/landscape.svg?style=flat
:target: https://landscape.io/github/bkabrda/anymarkup-core/master
:alt: Code Health
.. image:: https://coveralls.io/repos/bkabrda/anymarkup-core/badge.svg?branch=master
:target: https://coveralls.io/r/bkabrda/anymarkup-core?branch=master
:alt: Coverage
This is the core library that implements functionality of https://github.com/bkabrda/anymarkup.
You can install this if you only want to use a subset of anymarkup parsers. For example, you
can do this::
$ pip install anymarkup-core PyYAML
$ python -c "import anymarkup_core; print(anymarkup_core.parse('foo: bar'))"
... and you don't need `xmltodict` installed, for example. You can use anymarkup-core
in the same way you use anymarkup, except you have to import from `anymarkup_core`, obviously.
Keywords: xml,yaml,toml,json,json5,ini
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
anymarkup-core-0.8.1/README.rst 0000644 0000765 0000024 00000002032 13515571465 020010 0 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core
==============
.. image:: https://travis-ci.org/bkabrda/anymarkup-core.svg?branch=master
:target: https://travis-ci.org/bkabrda/anymarkup-core
:alt: Build Status
.. image:: https://landscape.io/github/bkabrda/anymarkup-core/master/landscape.svg?style=flat
:target: https://landscape.io/github/bkabrda/anymarkup-core/master
:alt: Code Health
.. image:: https://coveralls.io/repos/bkabrda/anymarkup-core/badge.svg?branch=master
:target: https://coveralls.io/r/bkabrda/anymarkup-core?branch=master
:alt: Coverage
This is the core library that implements functionality of https://github.com/bkabrda/anymarkup.
You can install this if you only want to use a subset of anymarkup parsers. For example, you
can do this::
$ pip install anymarkup-core PyYAML
$ python -c "import anymarkup_core; print(anymarkup_core.parse('foo: bar'))"
... and you don't need `xmltodict` installed, for example. You can use anymarkup-core
in the same way you use anymarkup, except you have to import from `anymarkup_core`, obviously.
anymarkup-core-0.8.1/anymarkup_core/ 0000755 0000765 0000024 00000000000 13623502530 021326 5 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/anymarkup_core/__init__.py 0000644 0000765 0000024 00000042110 13623502423 023436 0 ustar slavek.kabrda staff 0000000 0000000 # -*- coding: utf-8 -*-
import collections
import datetime
import io
import json
import os
import re
import traceback
import six
try:
import configobj
except ImportError:
configobj = None
try:
import json5
except ImportError:
json5 = None
try:
import toml
except ImportError:
toml = None
try:
import xmltodict
except ImportError:
xmltodict = None
try:
import yaml
except ImportError:
yaml = None
__all__ = ['AnyMarkupError', 'parse', 'parse_file', 'serialize', 'serialize_file']
__version__ = '0.8.1'
fmt_to_exts = {'ini': ['ini'],
'json': ['json'],
'json5': ['json5'],
'toml': ['toml'],
'xml': ['xml'],
'yaml': ['yaml', 'yml']}
fmt_to_lib = {'ini': (configobj, 'configobj'),
'json': (json, 'json'),
'json5': (json5, 'json5'),
'toml': (toml, 'toml'),
'xml': (xmltodict, 'xmltodict'),
'yaml': (yaml, 'PyYAML')}
def _is_utf8(enc_str):
return enc_str.lower() in ['utf8', 'utf-8']
class AnyMarkupError(Exception):
def __init__(self, cause, original_tb=''):
"""Wrapper for all errors that occur during anymarkup calls.
Args:
cause: either a reraised exception or a string with cause
"""
super(AnyMarkupError, self).__init__()
self.cause = cause
self.original_tb = original_tb
def __str__(self):
cause = str(self.cause)
if isinstance(self.cause, Exception):
cause = 'caught {0}: {1}'.format(type(self.cause), cause)
msg = 'AnyMarkupError: {0}'.format(cause)
if self.original_tb:
msg += '\nOriginal traceback:\n{0}'.format(self.original_tb)
return msg
def parse(inp, format=None, encoding='utf-8', force_types=True, interpolate=True):
"""Parse input from file-like object, unicode string or byte string.
Args:
inp: file-like object, unicode string or byte string with the markup
format: explicitly override the guessed `inp` markup format
encoding: `inp` encoding, defaults to utf-8
force_types:
if `True`, integers, floats, booleans and none/null
are recognized and returned as proper types instead of strings;
if `False`, everything is converted to strings
if `None`, backend return value is used
interpolate: turn on interpolation for INI files (defaults to True)
Returns:
parsed input (dict or list) containing unicode values
Raises:
AnyMarkupError if a problem occurs while parsing or inp
"""
proper_inp = inp
if hasattr(inp, 'read'):
proper_inp = inp.read()
# if proper_inp is unicode, encode it
if isinstance(proper_inp, six.text_type):
proper_inp = proper_inp.encode(encoding)
# try to guess markup type
fname = None
if hasattr(inp, 'name'):
fname = inp.name
fmt = _get_format(format, fname, proper_inp)
# make it look like file-like bytes-yielding object
proper_inp = six.BytesIO(proper_inp)
try:
res = _do_parse(proper_inp, fmt, encoding, force_types, interpolate)
except Exception as e:
# I wish there was only Python 3 and I could just use "raise ... from e"
raise AnyMarkupError(e, traceback.format_exc())
if res is None:
res = {}
return res
def parse_file(path, format=None, encoding='utf-8', force_types=True, interpolate=True):
"""A convenience wrapper of parse, which accepts path of file to parse.
Args:
path: path to file to parse
format: explicitly override the guessed `inp` markup format
encoding: file encoding, defaults to utf-8
force_types:
if `True`, integers, floats, booleans and none/null
are recognized and returned as proper types instead of strings;
if `False`, everything is converted to strings
if `None`, backend return value is used
interpolate: turn on interpolation for INI files (defaults to True)
Returns:
parsed `inp` (dict or list) containing unicode values
Raises:
AnyMarkupError if a problem occurs while parsing
"""
try:
with open(path, 'rb') as f:
return parse(f, format, encoding, force_types, interpolate)
except EnvironmentError as e:
raise AnyMarkupError(e, traceback.format_exc())
def serialize(struct, format, target=None, encoding='utf-8'):
"""Serialize given structure and return it as encoded string or write it to file-like object.
Args:
struct: structure (dict or list) with unicode members to serialize; note that list
can only be serialized to json
format: specify markup format to serialize structure as
target: binary-opened file-like object to serialize to; if None (default),
the result will be returned instead of writing to `target`
encoding: encoding to use when serializing, defaults to utf-8
Returns:
bytestring with serialized structure if `target` is None; return value of
`target.write` otherwise
Raises:
AnyMarkupError if a problem occurs while serializing
"""
# raise if "unicode-opened"
if hasattr(target, 'encoding') and target.encoding:
raise AnyMarkupError('Input file must be opened in binary mode')
fname = None
if hasattr(target, 'name'):
fname = target.name
fmt = _get_format(format, fname)
try:
serialized = _do_serialize(struct, fmt, encoding)
if target is None:
return serialized
else:
return target.write(serialized)
except Exception as e:
raise AnyMarkupError(e, traceback.format_exc())
def serialize_file(struct, path, format=None, encoding='utf-8'):
"""A convenience wrapper of serialize, which accepts path of file to serialize to.
Args:
struct: structure (dict or list) with unicode members to serialize; note that list
can only be serialized to json
path: path of the file to serialize to
format: override markup format to serialize structure as (taken from filename
by default)
encoding: encoding to use when serializing, defaults to utf-8
Returns:
number of bytes written
Raises:
AnyMarkupError if a problem occurs while serializing
"""
try:
with open(path, 'wb') as f:
return serialize(struct, format, f, encoding)
except EnvironmentError as e:
raise AnyMarkupError(e, traceback.format_exc())
def _check_lib_installed(fmt, action):
if fmt_to_lib[fmt][0] is None:
raise ImportError('Can\'t {action} {fmt}: {name} not installed'.
format(action=action, fmt=fmt, name=fmt_to_lib[fmt][1]))
def _do_parse(inp, fmt, encoding, force_types, interpolate):
"""Actually parse input.
Args:
inp: bytes yielding file-like object
fmt: format to use for parsing
encoding: encoding of `inp`
force_types:
if `True`, integers, floats, booleans and none/null
are recognized and returned as proper types instead of strings;
if `False`, everything is converted to strings
if `None`, backend return value is used
interpolate: turn on interpolation for INI files
Returns:
parsed `inp` (dict or list) containing unicode values
Raises:
various sorts of errors raised by used libraries while parsing
"""
res = {}
_check_lib_installed(fmt, 'parse')
if fmt == 'ini':
cfg = configobj.ConfigObj(inp, encoding=encoding, interpolation=interpolate)
res = cfg.dict()
elif fmt == 'json':
if six.PY3:
# python 3 json only reads from unicode objects
inp = io.TextIOWrapper(inp, encoding=encoding)
res = json.load(inp)
else:
res = json.load(inp, encoding=encoding)
elif fmt == 'json5':
if six.PY3:
inp = io.TextIOWrapper(inp, encoding=encoding)
res = json5.load(inp, encoding=encoding)
elif fmt == 'toml':
if not _is_utf8(encoding):
raise AnyMarkupError('toml is always utf-8 encoded according to specification')
if six.PY3:
# python 3 toml prefers unicode objects
inp = io.TextIOWrapper(inp, encoding=encoding)
res = toml.load(inp)
elif fmt == 'xml':
res = xmltodict.parse(inp, encoding=encoding)
elif fmt == 'yaml':
# guesses encoding by its own, there seems to be no way to pass
# it explicitly
res = yaml.safe_load(inp)
else:
raise # unknown format
# make sure it's all unicode and all int/float values were parsed correctly
# the unicode part is here because of yaml on PY2 and also as workaround for
# https://github.com/DiffSK/configobj/issues/18#issuecomment-76391689
return _ensure_proper_types(res, encoding, force_types)
def _do_serialize(struct, fmt, encoding):
"""Actually serialize input.
Args:
struct: structure to serialize to
fmt: format to serialize to
encoding: encoding to use while serializing
Returns:
encoded serialized structure
Raises:
various sorts of errors raised by libraries while serializing
"""
res = None
_check_lib_installed(fmt, 'serialize')
if fmt == 'ini':
config = configobj.ConfigObj(encoding=encoding)
for k, v in struct.items():
config[k] = v
res = b'\n'.join(config.write())
elif fmt in ['json', 'json5']:
# specify separators to get rid of trailing whitespace
# specify ensure_ascii to make sure unicode is serialized in \x... sequences,
# not in \u sequences
res = (json if fmt == 'json' else json5).dumps(struct,
indent=2,
separators=(',', ': '),
ensure_ascii=False).encode(encoding)
elif fmt == 'toml':
if not _is_utf8(encoding):
raise AnyMarkupError('toml must always be utf-8 encoded according to specification')
res = toml.dumps(struct).encode(encoding)
elif fmt == 'xml':
# passing encoding argument doesn't encode, just sets the xml property
res = xmltodict.unparse(struct, pretty=True, encoding='utf-8').encode('utf-8')
elif fmt == 'yaml':
res = yaml.safe_dump(struct, encoding='utf-8', default_flow_style=False)
else:
raise # unknown format
return res
def _ensure_proper_types(struct, encoding, force_types):
"""A convenience function that recursively makes sure the given structure
contains proper types according to value of `force_types`.
Args:
struct: a structure to check and fix
encoding: encoding to use on found bytestrings
force_types:
if `True`, integers, floats, booleans and none/null
are recognized and returned as proper types instead of strings;
if `False`, everything is converted to strings
if `None`, unmodified `struct` is returned
Returns:
a fully decoded copy of given structure
"""
if force_types is None:
return struct
# if it's an empty value
res = None
if isinstance(struct, (dict, collections.OrderedDict)):
res = type(struct)()
for k, v in struct.items():
res[_ensure_proper_types(k, encoding, force_types)] = \
_ensure_proper_types(v, encoding, force_types)
elif isinstance(struct, list):
res = []
for i in struct:
res.append(_ensure_proper_types(i, encoding, force_types))
elif isinstance(struct, six.binary_type):
res = struct.decode(encoding)
elif isinstance(struct, (six.text_type, type(None), type(True), six.integer_types, float)):
res = struct
elif isinstance(struct, datetime.datetime):
# toml can parse datetime natively
res = struct
else:
raise AnyMarkupError('internal error - unexpected type {0} in parsed markup'.
format(type(struct)))
if force_types and isinstance(res, six.text_type):
res = _recognize_basic_types(res)
elif not (force_types or
isinstance(res, (dict, collections.OrderedDict, list, six.text_type))):
res = six.text_type(res)
return res
def _recognize_basic_types(s):
"""If value of given string `s` is an integer (or long), float or boolean, convert it
to a proper type and return it.
"""
tps = [int, float]
if not six.PY3: # compat for older versions of six that don't have PY2
tps.append(long)
for tp in tps:
try:
return tp(s)
except ValueError:
pass
if s.lower() == 'true':
return True
if s.lower() == 'false':
return False
if s.lower() in ['none', 'null']:
return None
return s
def _get_format(format, fname, inp=None):
"""Try to guess markup format of given input.
Args:
format: explicit format override to use
fname: name of file, if a file was used to read `inp`
inp: optional bytestring to guess format of (can be None, if markup
format is to be guessed only from `format` and `fname`)
Returns:
guessed format (a key of fmt_to_exts dict)
Raises:
AnyMarkupError if explicit format override has unsupported value
or if it's impossible to guess the format
"""
fmt = None
err = True
if format is not None:
if format in fmt_to_exts:
fmt = format
err = False
elif fname:
# get file extension without leading dot
file_ext = os.path.splitext(fname)[1][len(os.path.extsep):]
for fmt_name, exts in fmt_to_exts.items():
if file_ext in exts:
fmt = fmt_name
err = False
if fmt is None:
if inp is not None:
fmt = _guess_fmt_from_bytes(inp)
err = False
if err:
err_string = 'Failed to guess markup format based on: '
what = []
for k, v in {format: 'specified format argument',
fname: 'filename', inp: 'input string'}.items():
if k:
what.append(v)
if not what:
what.append('nothing to guess format from!')
err_string += ', '.join(what)
raise AnyMarkupError(err_string)
return fmt
def _guess_fmt_from_bytes(inp):
"""Try to guess format of given bytestring.
Args:
inp: byte string to guess format of
Returns:
guessed format
"""
stripped = inp.strip()
fmt = None
ini_section_header_re = re.compile(br'^\[([\w-]+)\]')
if len(stripped) == 0:
# this can be anything, so choose yaml, for example
fmt = 'yaml'
else:
if stripped.startswith(b'<'):
fmt = 'xml'
else:
for l in stripped.splitlines():
line = l.strip()
# there are C-style comments in json5, but we don't auto-detect it,
# so it doesn't matter here
if not line.startswith(b'#') and line:
break
# json, ini or yaml => skip comments and then determine type
if ini_section_header_re.match(line):
fmt = 'ini'
else:
# we assume that yaml is superset of json
# TODO: how do we figure out it's not yaml?
fmt = 'yaml'
return fmt
# following code makes it possible to use OrderedDict with PyYAML
# based on https://bitbucket.org/xi/pyyaml/issue/13
def construct_ordereddict(loader, node):
try:
omap = loader.construct_yaml_omap(node)
return collections.OrderedDict(*omap)
except yaml.constructor.ConstructorError:
return loader.construct_yaml_seq(node)
def represent_ordereddict(dumper, data):
# NOTE: For block style this uses the compact omap notation, but for flow style
# it does not.
values = []
node = yaml.SequenceNode(u'tag:yaml.org,2002:omap', values, flow_style=False)
if dumper.alias_key is not None:
dumper.represented_objects[dumper.alias_key] = node
for key, value in data.items():
key_item = dumper.represent_data(key)
value_item = dumper.represent_data(value)
node_item = yaml.MappingNode(u'tag:yaml.org,2002:map', [(key_item, value_item)],
flow_style=False)
values.append(node_item)
return node
def represent_str(dumper, data):
# borrowed from http://stackoverflow.com/a/33300001
if len(data.splitlines()) > 1:
return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
return dumper.represent_scalar('tag:yaml.org,2002:str', data)
if yaml is not None:
yaml.SafeLoader.add_constructor(u'tag:yaml.org,2002:omap', construct_ordereddict)
yaml.SafeDumper.add_representer(collections.OrderedDict, represent_ordereddict)
yaml.SafeDumper.add_representer(str, represent_str)
if six.PY2:
yaml.SafeDumper.add_representer(unicode, represent_str)
anymarkup-core-0.8.1/anymarkup_core.egg-info/ 0000755 0000765 0000024 00000000000 13623502530 023020 5 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/anymarkup_core.egg-info/PKG-INFO 0000644 0000765 0000024 00000003651 13623502527 024130 0 ustar slavek.kabrda staff 0000000 0000000 Metadata-Version: 1.1
Name: anymarkup-core
Version: 0.8.1
Summary: Core library for anymarkup
Home-page: https://github.com/bkabrda/anymarkup-core
Author: Slavek Kabrda
Author-email: slavek.kabrda@gmail.com
License: BSD
Description: anymarkup-core
==============
.. image:: https://travis-ci.org/bkabrda/anymarkup-core.svg?branch=master
:target: https://travis-ci.org/bkabrda/anymarkup-core
:alt: Build Status
.. image:: https://landscape.io/github/bkabrda/anymarkup-core/master/landscape.svg?style=flat
:target: https://landscape.io/github/bkabrda/anymarkup-core/master
:alt: Code Health
.. image:: https://coveralls.io/repos/bkabrda/anymarkup-core/badge.svg?branch=master
:target: https://coveralls.io/r/bkabrda/anymarkup-core?branch=master
:alt: Coverage
This is the core library that implements functionality of https://github.com/bkabrda/anymarkup.
You can install this if you only want to use a subset of anymarkup parsers. For example, you
can do this::
$ pip install anymarkup-core PyYAML
$ python -c "import anymarkup_core; print(anymarkup_core.parse('foo: bar'))"
... and you don't need `xmltodict` installed, for example. You can use anymarkup-core
in the same way you use anymarkup, except you have to import from `anymarkup_core`, obviously.
Keywords: xml,yaml,toml,json,json5,ini
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
anymarkup-core-0.8.1/anymarkup_core.egg-info/SOURCES.txt 0000644 0000765 0000024 00000001432 13623502527 024712 0 ustar slavek.kabrda staff 0000000 0000000 LICENSE
MANIFEST.in
README.rst
requirements.txt
setup.py
anymarkup_core/__init__.py
anymarkup_core.egg-info/PKG-INFO
anymarkup_core.egg-info/SOURCES.txt
anymarkup_core.egg-info/dependency_links.txt
anymarkup_core.egg-info/requires.txt
anymarkup_core.egg-info/top_level.txt
test/__init__.py
test/test_libs_not_installed.py
test/test_parse.py
test/test_serialize.py
test/fixtures/bad_extension.xml
test/fixtures/empty.ini
test/fixtures/empty.json
test/fixtures/empty.json5
test/fixtures/empty.toml
test/fixtures/empty.xml
test/fixtures/empty.yaml
test/fixtures/example.ini
test/fixtures/example.json
test/fixtures/example.json5
test/fixtures/example.toml
test/fixtures/example.xml
test/fixtures/example.yaml
test/fixtures/example_omap.yaml
test/fixtures/types.json
test/fixtures/without_extension anymarkup-core-0.8.1/anymarkup_core.egg-info/dependency_links.txt 0000644 0000765 0000024 00000000001 13623502527 027074 0 ustar slavek.kabrda staff 0000000 0000000
anymarkup-core-0.8.1/anymarkup_core.egg-info/requires.txt 0000644 0000765 0000024 00000000004 13623502527 025420 0 ustar slavek.kabrda staff 0000000 0000000 six
anymarkup-core-0.8.1/anymarkup_core.egg-info/top_level.txt 0000644 0000765 0000024 00000000017 13623502527 025556 0 ustar slavek.kabrda staff 0000000 0000000 anymarkup_core
anymarkup-core-0.8.1/requirements.txt 0000644 0000765 0000024 00000000004 13515571465 021602 0 ustar slavek.kabrda staff 0000000 0000000 six
anymarkup-core-0.8.1/setup.cfg 0000644 0000765 0000024 00000000046 13623502530 020130 0 ustar slavek.kabrda staff 0000000 0000000 [egg_info]
tag_build =
tag_date = 0
anymarkup-core-0.8.1/setup.py 0000644 0000765 0000024 00000001747 13623502410 020027 0 ustar slavek.kabrda staff 0000000 0000000 #!/usr/bin/env python3
# -*- coding: utf-8 -*-
from setuptools import setup, find_packages
setup(
name='anymarkup-core',
version='0.8.1',
description='Core library for anymarkup',
long_description=''.join(open('README.rst').readlines()),
keywords='xml, yaml, toml, json, json5, ini',
author='Slavek Kabrda',
author_email='slavek.kabrda@gmail.com',
url='https://github.com/bkabrda/anymarkup-core',
license='BSD',
packages=['anymarkup_core'],
install_requires=open('requirements.txt').read().splitlines(),
classifiers=[
'Development Status :: 4 - Beta',
'Intended Audience :: Developers',
'License :: OSI Approved :: BSD License',
'Operating System :: POSIX :: Linux',
'Programming Language :: Python',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
]
)
anymarkup-core-0.8.1/test/ 0000755 0000765 0000024 00000000000 13623502530 017266 5 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/test/__init__.py 0000644 0000765 0000024 00000007142 13515571465 021420 0 ustar slavek.kabrda staff 0000000 0000000 # -*- coding: utf-8 -*-
from collections import OrderedDict
from copy import deepcopy
from datetime import datetime, tzinfo
from toml.tz import TomlTz
example_ini = u"""\
[foo]
bar = ěšč
spam = 1
baz = 1.1
[[blah]]
blahblah = True, text4
nothing = None\
"""
example_ini_with_interpolation = u"""\
[foo]
bar = ěšč
spam = 1 [%%(test)s]
baz = 1.1
[[blah]]
blahblah = True, text4
nothing = None\
"""
example_json = u"""\
{
"foo": {
"bar": "ěšč",
"spam": 1,
"baz": 1.1,
"blah": {
"blahblah": [
true,
"text4"
],
"nothing": null
}
}
}"""
example_json5 = u"""\
{foo: {
bar: 'ěšč',
spam: 1,
baz: 1.1,
blah: {
blahblah: [true, "text4",],
nothing: null
}
}}"""
# toml is special, since it e.g. only allows lists to have same types of values
example_toml = u"""\
title = "TOML Example"
[foo]
name = "barů"
dob = 1987-07-05T17:45:00Z
[spam]
spam2 = "spam"
ham = [1, 2, 3]
[spam.spam]
foo = [["bar"]]
"""
example_xml = u"""\
\těšč
\t1
\t1.1
\t
\t\ttrue
\t\ttext4
\t\t
\t
"""
example_yaml_map = u"""\
foo:
bar: ěšč
spam: 1
baz: 1.1
blah:
blahblah:
- True
- text4
nothing:"""
# for testing OrderedDict parsing/serializing with PyYAML
# TODO: what about "nothing: null"? it's not there for normal map
example_yaml_omap = u"""\
!!omap
- foo: !!omap
- bar: ěšč
- spam: 1
- baz: 1.1
- blah: !!omap
- blahblah:
- True
- text4
- nothing: null"""
example_as_ordered_dict = OrderedDict(
[(u'foo', OrderedDict([
(u'bar', u'ěšč'),
(u'spam', 1),
(u'baz', 1.1),
(u'blah', OrderedDict([
(u'blahblah', [True, u'text4']),
(u'nothing', None)
]))
]))]
)
example_as_dict = {
u'foo': {
u'bar': u'ěšč',
u'spam': 1,
u'baz': 1.1,
u'blah': {
u'blahblah': [True, u'text4'],
u'nothing': None
}
}
}
toml_example_as_dict = {
u'foo': {
u'dob': datetime(1987, 7, 5, 17, 45, tzinfo=TomlTz('Z')),
u'name': u'barů'},
u'spam': {u'ham': [1, 2, 3],
u'spam': {u'foo': [[u'bar']]},
u'spam2': u'spam'},
u'title': u'TOML Example'
}
# ini loading doesn't yield any ints/floats/NoneTypes/bools, so it's ideal
# to test our custom convertors; for other types, some of these values
# are pre-converted by the used parsers
types_ini = u"""
[x]
a=1
b=1.1
c=None
d=True"""
types_json = u"""
{
"x":
{
"a": 1,
"b": 1.1,
"c": null,
"d": true,
}
}"""
types_json5 = types_json
types_toml = u"""
[x]
a=1
b=1.1
c=1987-07-05T17:45:00Z
d=true
"""
types_yaml = u"""
x:
a: 1
b: 1.1
c: None
d: True
"""
types_xml = u"""\
\t1
\t1.1
\tNone
\tTrue
"""
types_as_struct_with_objects = {
'x': {
'a': 1,
'b': 1.1,
'c': None,
'd': True,
}
}
toml_types_as_struct_with_objects = deepcopy(types_as_struct_with_objects)
toml_types_as_struct_with_objects['x']['c'] = datetime(1987, 7, 5, 17, 45, tzinfo=TomlTz('Z'))
types_as_struct_with_strings = {
'x': {
'a': "1",
'b': "1.1",
'c': "None",
'd': "True",
}
}
toml_types_as_struct_with_strings = deepcopy(types_as_struct_with_strings)
toml_types_as_struct_with_strings['x']['c'] = '1987-07-05 17:45:00+00:00'
anymarkup-core-0.8.1/test/fixtures/ 0000755 0000765 0000024 00000000000 13623502530 021137 5 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/test/fixtures/bad_extension.xml 0000644 0000765 0000024 00000000010 13515571465 024507 0 ustar slavek.kabrda staff 0000000 0000000 foo:bar
anymarkup-core-0.8.1/test/fixtures/empty.ini 0000644 0000765 0000024 00000000000 13515571465 023001 0 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/test/fixtures/empty.json 0000644 0000765 0000024 00000000000 13515571465 023173 0 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/test/fixtures/empty.json5 0000644 0000765 0000024 00000000000 13515571465 023260 0 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/test/fixtures/empty.toml 0000644 0000765 0000024 00000000000 13515571465 023175 0 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/test/fixtures/empty.xml 0000644 0000765 0000024 00000000000 13515571465 023022 0 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/test/fixtures/empty.yaml 0000644 0000765 0000024 00000000000 13515571465 023164 0 ustar slavek.kabrda staff 0000000 0000000 anymarkup-core-0.8.1/test/fixtures/example.ini 0000644 0000765 0000024 00000000125 13515571465 023306 0 ustar slavek.kabrda staff 0000000 0000000 [foo]
bar=ěšč
spam=1
baz=1.1
[[blah]]
blahblah = True, text4
nothing= None
anymarkup-core-0.8.1/test/fixtures/example.json 0000644 0000765 0000024 00000000222 13515571465 023476 0 ustar slavek.kabrda staff 0000000 0000000 {"foo": {
"bar": "ěšč",
"spam": 1,
"baz": 1.1,
"blah": {
"blahblah": [true, "text4"],
"nothing": null
}
}}
anymarkup-core-0.8.1/test/fixtures/example.json5 0000644 0000765 0000024 00000000205 13515571465 023564 0 ustar slavek.kabrda staff 0000000 0000000 {foo: {
bar: 'ěšč',
spam: 1,
baz: 1.1,
blah: {
blahblah: [true, "text4",],
nothing: null
}
}}
anymarkup-core-0.8.1/test/fixtures/example.toml 0000644 0000765 0000024 00000000217 13515571465 023504 0 ustar slavek.kabrda staff 0000000 0000000 title = "TOML Example"
[foo]
name = "barů"
dob = 1987-07-05T17:45:00Z
[spam]
spam2 = "spam"
ham = [1, 2, 3]
[spam.spam]
foo = [["bar"]]
anymarkup-core-0.8.1/test/fixtures/example.xml 0000644 0000765 0000024 00000000337 13515571465 023334 0 ustar slavek.kabrda staff 0000000 0000000
ěšč
1
1.1
True
text4
anymarkup-core-0.8.1/test/fixtures/example.yaml 0000644 0000765 0000024 00000000172 13515571465 023473 0 ustar slavek.kabrda staff 0000000 0000000 foo:
bar: ěšč
spam: 1
baz: 1.1
blah:
blahblah:
- True
- text4
nothing:
anymarkup-core-0.8.1/test/fixtures/example_omap.yaml 0000644 0000765 0000024 00000000206 13515571465 024505 0 ustar slavek.kabrda staff 0000000 0000000 !!omap
- foo: !!omap
- bar: ěšč
- spam: text2
- blah: !!omap
- blahblah:
- text3
- text4
- nothing: null
anymarkup-core-0.8.1/test/fixtures/types.json 0000644 0000765 0000024 00000000031 13515571465 023205 0 ustar slavek.kabrda staff 0000000 0000000 {
"a": 1,
"b": "1"
}
anymarkup-core-0.8.1/test/fixtures/without_extension 0000644 0000765 0000024 00000000125 13515571465 024674 0 ustar slavek.kabrda staff 0000000 0000000 [foo]
bar=ěšč
spam=1
baz=1.1
[[blah]]
blahblah = True, text4
nothing= None
anymarkup-core-0.8.1/test/test_libs_not_installed.py 0000644 0000765 0000024 00000002621 13515571465 024565 0 ustar slavek.kabrda staff 0000000 0000000 from flexmock import flexmock
import pytest
import yaml
import anymarkup_core
class TestLibsNotInstalled(object):
# json is always there, since we only support Python >= 2.7
@pytest.mark.parametrize(('fmt', 'lib'), [
('ini', 'configobj'),
('xml', 'xmltodict'),
('yaml', 'yaml'),
])
def test_raises_proper_error(self, fmt, lib):
flexmock(anymarkup_core).should_receive(lib).and_return(None)
flexmock(anymarkup_core).should_receive('fmt_to_lib').and_return({fmt: (None, lib)})
with pytest.raises(anymarkup_core.AnyMarkupError):
anymarkup_core.parse('', format=fmt)
with pytest.raises(anymarkup_core.AnyMarkupError):
anymarkup_core.serialize('', format=fmt)
def test_uninstalled_dep_doesnt_make_parsing_fail_for_installed_deps(self):
flexmock(anymarkup_core).should_receive('configobj').and_return(None)
flexmock(anymarkup_core).should_receive('fmt_to_lib').\
and_return({'ini': (None, ''), 'yaml': (yaml, '')})
with pytest.raises(anymarkup_core.AnyMarkupError):
anymarkup_core.parse('', format='ini')
assert anymarkup_core.parse('foo: bar') == {'foo': 'bar'}
with pytest.raises(anymarkup_core.AnyMarkupError):
anymarkup_core.serialize('', format='ini')
assert anymarkup_core.serialize({'foo': 'bar'}, format='yaml') == b'foo: bar\n'
anymarkup-core-0.8.1/test/test_parse.py 0000644 0000765 0000024 00000016614 13515571465 022036 0 ustar slavek.kabrda staff 0000000 0000000 # -*- coding: utf-8 -*-
from datetime import datetime
import io
import os
import pytest
import six
import toml
from anymarkup_core import *
from test import *
class TestParse(object):
fixtures = os.path.join(os.path.dirname(__file__), 'fixtures')
def assert_unicode(self, struct):
if isinstance(struct, dict):
for k, v in struct.items():
self.assert_unicode(k)
self.assert_unicode(v)
elif isinstance(struct, list):
for i in struct:
self.assert_unicode(i)
elif isinstance(struct, (six.string_types, type(None), type(True), \
six.integer_types, float, datetime)):
pass
else:
raise AssertionError('Unexpected type {0} in parsed structure'.format(type(struct)))
@pytest.mark.parametrize(('str', 'fmt', 'expected'), [
('', None, {}),
('{}', None, {}),
('[]', None, []),
(example_ini, None, example_as_dict),
(example_json, None, example_as_dict),
(example_json5, 'json5', example_as_dict),
(example_toml, 'toml', toml_example_as_dict), # we can't tell toml from ini
(example_xml, None, example_as_ordered_dict),
(example_yaml_map, None, example_as_dict),
(example_yaml_omap, None, example_as_ordered_dict),
])
def test_parse_basic(self, str, fmt, expected):
parsed = parse(str, fmt)
assert parsed == expected
assert type(parsed) == type(expected)
self.assert_unicode(parsed)
@pytest.mark.parametrize(('str', 'fmt', 'expected'), [
('', None, {}),
('{}', None, {}),
('[]', None, []),
(example_ini, None, example_as_dict),
(example_json, None, example_as_dict),
(example_json5, 'json5', example_as_dict),
(example_toml, 'toml', toml_example_as_dict), # we can't tell toml from ini
(example_xml, None, example_as_ordered_dict),
(example_yaml_map, None, example_as_dict),
(example_yaml_omap, None, example_as_ordered_dict),
])
def test_parse_basic_interpolation_is_false(self, str, fmt, expected):
parsed = parse(str, fmt, interpolate=False)
assert parsed == expected
assert type(parsed) == type(expected)
self.assert_unicode(parsed)
def test_parse_interpolation_fail(self):
with pytest.raises(AnyMarkupError):
parse(example_ini_with_interpolation)
def test_parse_interpolation_pass_when_false(self):
parsed = parse(example_ini_with_interpolation, interpolate=False)
assert type(parsed) == dict
@pytest.mark.parametrize(('str', 'expected'), [
('# comment', {}),
('# comment\n', {}),
('# comment\n' + example_ini, example_as_dict),
('# comment\n' + example_json, example_as_dict),
('# comment\n' + example_json5, example_as_dict),
('# comment\n' + example_yaml_map, example_as_dict),
('# comment\n' + example_yaml_omap, example_as_ordered_dict),
# no test for toml, since it's not auto-recognized
])
def test_parse_recognizes_comments_in_ini_json_yaml(self, str, expected):
parsed = parse(str)
assert parsed == expected
assert type(parsed) == type(expected)
self.assert_unicode(parsed)
@pytest.mark.parametrize(('str, fmt, expected'), [
(types_ini, None, types_as_struct_with_objects),
(types_json, None, types_as_struct_with_objects),
(types_json5, 'json5', types_as_struct_with_objects),
(types_toml, 'toml', toml_types_as_struct_with_objects),
(types_xml, None, types_as_struct_with_objects),
(types_yaml, None, types_as_struct_with_objects),
])
def test_parse_force_types_true(self, str, fmt, expected):
assert parse(str, fmt) == expected
@pytest.mark.parametrize(('str', 'fmt', 'expected'), [
(types_ini, None, types_as_struct_with_strings),
(types_json, None, types_as_struct_with_strings),
(types_json5, 'json5', types_as_struct_with_strings),
(types_toml, 'toml', toml_types_as_struct_with_strings),
(types_xml, None, types_as_struct_with_strings),
(types_yaml, None, types_as_struct_with_strings),
])
def test_parse_force_types_false(self, str, fmt, expected):
assert parse(str, fmt, force_types=False) == expected
@pytest.mark.parametrize(('str', 'fmt', 'expected'), [
# Note: the expected result is backend-specific
(types_ini, None, {'x': {'a': '1', 'b': '1.1', 'c': 'None', 'd': 'True'}}),
(types_json, None, {'x': {'a': 1, 'b': 1.1, 'c': None, 'd': True}}),
(types_json5, 'json5', {'x': {'a': 1, 'b': 1.1, 'c': None, 'd': True}}),
(types_toml, 'toml', {'x': {'a': 1, 'b': 1.1,
'c': datetime(1987, 7, 5, 17, 45, tzinfo=TomlTz('Z')),
'd': True}}),
(types_xml, None, {'x': {'a': '1', 'b': '1.1', 'c': 'None', 'd': 'True'}}),
(types_yaml, None, {'x': {'a': 1, 'b': 1.1, 'c': 'None', 'd': True}}),
])
def test_parse_force_types_none(self, str, fmt, expected):
assert parse(str, fmt, force_types=None) == expected
def test_parse_works_with_bytes_yielding_file(self):
f = open(os.path.join(self.fixtures, 'empty.ini'), 'rb')
parsed = parse(f)
assert parsed == {}
def test_parse_works_with_unicode_yielding_file(self):
# on Python 2, this can only be simulated with io.open
f = io.open(os.path.join(self.fixtures, 'empty.ini'), encoding='utf-8')
parsed = parse(f)
assert parsed == {}
def test_parse_fails_on_wrong_format(self):
with pytest.raises(AnyMarkupError):
parse('foo: bar', format='xml')
@pytest.mark.parametrize(('file', 'expected'), [
# TODO: some parsers allow empty files, others don't - this should be made consistent
('empty.ini', {}),
('empty.json', AnyMarkupError),
('empty.json5', AnyMarkupError),
('empty.toml', {}),
('empty.xml', AnyMarkupError),
('empty.yaml', {}),
('example.ini', example_as_dict),
('example.json', example_as_dict),
('example.json5', example_as_dict),
('example.toml', toml_example_as_dict),
('example.xml', example_as_ordered_dict),
('example.yaml', example_as_dict),
])
def test_parse_file_basic(self, file, expected):
f = os.path.join(self.fixtures, file)
if expected == AnyMarkupError:
with pytest.raises(AnyMarkupError):
parse_file(f)
else:
parsed = parse_file(f)
assert parsed == expected
self.assert_unicode(parsed)
def test_parse_file_noextension(self):
parsed = parse_file(os.path.join(self.fixtures, 'without_extension'))
assert parsed == example_as_dict
self.assert_unicode(parsed)
def test_parse_file_fails_on_bad_extension(self):
with pytest.raises(AnyMarkupError):
parse_file(os.path.join(self.fixtures, 'bad_extension.xml'))
def test_parse_file_respects_force_types(self):
f = os.path.join(self.fixtures, 'types.json')
parsed = parse_file(f, force_types=True)
assert parsed == {'a': 1, 'b': 1}
parsed = parse_file(f, force_types=False)
assert parsed == {'a': '1', 'b': '1'}
parsed = parse_file(f, force_types=None)
assert parsed == {'a': 1, 'b': '1'}
anymarkup-core-0.8.1/test/test_serialize.py 0000644 0000765 0000024 00000006124 13515571465 022706 0 ustar slavek.kabrda staff 0000000 0000000 # -*- coding: utf-8 -*-
import io
import os
import pytest
import six
from anymarkup_core import *
from test import *
class TestSerialize(object):
"""Note: testing serialization is a bit tricky, since serializing dicts can result
in different order of values in serialized string in different runs.
That means that we can't just test whether the serialized string equals to expected
string. To solve this, we rather parse the serialized string back and make sure
that it equals the original structure.
"""
fixtures = os.path.join(os.path.dirname(__file__), 'fixtures')
def _read_decode(self, file):
if isinstance(file, six.string_types):
file = open(file, 'rb')
else:
file.seek(0)
return file.read().decode('utf-8')
@pytest.mark.parametrize(('struct', 'format'), [
(example_as_dict, 'ini'),
(example_as_dict, 'json'),
(example_as_dict, 'json5'),
(toml_example_as_dict, 'toml'),
(example_as_ordered_dict, 'xml'),
(example_as_dict, 'yaml'),
(example_as_ordered_dict, 'yaml'),
])
def test_serialize_basic(self, struct, format):
serialized = serialize(struct, format)
parsed_back = parse(serialized, format)
assert parsed_back == struct
assert type(parsed_back) == type(struct)
def test_serialize_works_with_wb_opened_file(self, tmpdir):
f = os.path.join(str(tmpdir), 'foo.xml')
fhandle = open(f, 'wb+')
serialize(example_as_ordered_dict, 'xml', fhandle)
assert self._read_decode(fhandle) == example_xml
def test_serialize_raises_with_unicode_opened_file(self, tmpdir):
# on Python 2, this can only be simulated with io.open
f = os.path.join(str(tmpdir), 'foo.json')
fhandle = io.open(f, 'w+', encoding='utf-8')
with pytest.raises(AnyMarkupError):
serialize(example_as_dict, 'json', fhandle)
@pytest.mark.parametrize(('struct', 'fmt', 'fname'), [
(example_as_dict, None, 'example.ini'),
(example_as_dict, None, 'example.json'),
(example_as_dict, 'json5', 'example.json5'),
(toml_example_as_dict, 'toml', 'example.toml'),
(example_as_ordered_dict, None, 'example.xml'),
(example_as_dict, None, 'example.yaml'),
(example_as_ordered_dict, None, 'example_ordered.yaml'),
])
def test_serialize_file_basic(self, struct, fmt, fname, tmpdir):
f = os.path.join(str(tmpdir), fname)
serialize_file(struct, f)
parsed_back = parse(self._read_decode(f), fmt)
assert parsed_back == struct
assert type(parsed_back) == type(struct)
def test_serialize_file_format_overrides_extension(self, tmpdir):
f = os.path.join(str(tmpdir), 'foo.ini')
serialize_file(example_as_dict, f, 'json')
assert parse(self._read_decode(f)) == example_as_dict
def test_parse_and_serialize_yaml_multiline_string(self):
# https://github.com/bkabrda/anymarkup-core/issues/1
inp = b'foo: |-\n line1\n line2\n line3\n'
assert serialize(parse(inp), 'yaml') == inp