jsonpickle-0.9.5/0000755000175000017500000000000013133025657014073 5ustar daviddavid00000000000000jsonpickle-0.9.5/README.rst0000644000175000017500000000306012770446612015565 0ustar daviddavid00000000000000jsonpickle ========== jsonpickle is a library for the two-way conversion of complex Python objects and `JSON `_. jsonpickle builds upon the existing JSON encoders, such as simplejson, json, and demjson. For complete documentation, please visit the `jsonpickle homepage `_. Bug reports and merge requests are encouraged at the `jsonpickle repository on github `_. Install ======= Install from pip for the latest stable release: :: pip install jsonpickle Install from github for the latest changes: :: pip install git+https://github.com/jsonpickle/jsonpickle.git If you have the files checked out for development: :: git clone https://github.com/jsonpickle/jsonpickle.git cd jsonpickle python setup.py develop Numpy Support ============= jsonpickle includes a built-in numpy extension. If would like to encode sklearn models, numpy arrays, and other numpy-based data then you must enable the numpy extension by registering its handlers:: >>> import jsonpickle.ext.numpy as jsonpickle_numpy >>> jsonpickle_numpy.register_handlers() jsonpickleJS ============ `jsonpickleJS `_ is a javascript implementation of jsonpickle by Michael Scott Cuthbert. jsonpickleJS can be extremely useful for projects that have parallel data structures between Python and Javascript. License ======= Licensed under the BSD License. See COPYING for details. See jsonpickleJS/LICENSE for details about the jsonpickleJS license. jsonpickle-0.9.5/docs/0000755000175000017500000000000013133025657015023 5ustar daviddavid00000000000000jsonpickle-0.9.5/docs/source/0000755000175000017500000000000013133025657016323 5ustar daviddavid00000000000000jsonpickle-0.9.5/docs/source/extensions.rst0000644000175000017500000000057612770447414021271 0ustar daviddavid00000000000000===================== jsonpickle extensions ===================== NumPy ----- jsonpickle includes a built-in numpy extension. If would like to encode sklearn models, numpy arrays, and other numpy-based data then you must enable the numpy extension by registering its handlers:: >>> import jsonpickle.ext.numpy as jsonpickle_numpy >>> jsonpickle_numpy.register_handlers() jsonpickle-0.9.5/docs/source/changelog.rst0000644000175000017500000002264113133025502020776 0ustar daviddavid00000000000000Change Log ========== Version 0.9.5 - July 16, 2017 ----------------------------- * Better support for objects that implement the reduce protocol. (`#170 `_). Version 0.9.4 - January 10, 2017 -------------------------------- * Arbitrary byte streams are now better supported. (`#143 `_). * Better support for NumPy data types. The Python3 NumPy support is especially robust. * Fortran-ordered based NumPy arrays are now properly serialized. Version 0.9.3 - March 9, 2016 ----------------------------- * UUID objects can now be serialized (`#130 `_). * Added `set_decoder_options` method to allow decoder specific options equal to `set_encoder_options`. * Int keys can be encoded directly by e.g. demjson by passing `numeric_keys=True` and setting its backend options via `jsonpickle.set_encoder_options('demjson', strict=False)`. * Newer Numpy versions (v1.10+) are now supported. Version 0.9.2 - March 20, 2015 ------------------------------ * Fixes for serializing objects with custom handlers. * We now properly serialize deque objects constructed with a `maxlen` parameter. * Test suite fixes Version 0.9.1 - March 15, 2015 ------------------------------ * Support datetime objects with FixedOffsets. Version 0.9.0 - January 16, 2015 -------------------------------- * Support for Pickle Protocol v4. * We now support serializing defaultdict subclasses that use `self` as their default factory. * We now have a decorator syntax for registering custom handlers, and allow custom handlers to register themselves for all subclasses. (`#104 `_). * We now support serializing types with metaclasses and their instances (e.g., Python 3 `enum`). * We now support serializing bytestrings in both Python 2 and Python 3. In Python 2, the `str` type is decoded to UTF-8 whenever possible and serialized as a true bytestring elsewise; in Python 3, bytestrings are explicitly encoded/decoded as bytestrings. Unicode strings are always encoded as is in both Python 2 and Python 3. * Added support for serializing numpy arrays, dtypes and scalars (see `jsonpickle.ext.numpy` module). Version 0.8.0 - September 6, 2014 --------------------------------- * We now support serializing objects that contain references to module-level functions (`#77 `_). * Better Pickle Protocol v2 support (`#78 `_). * Support for string __slots__ and iterable __slots__ (`#67 `_) (`#68 `_). * `encode()` now has a `warn` option that makes jsonpickle emit warnings when encountering objects that cannot be pickled. * A Javascript implementation of jsonpickle is now included in the jsonpickleJS directory. Version 0.7.2 - August 6, 2014 ------------------------------ * We now properly serialize classes that inherit from classes that use `__slots__` and add additional slots in the derived class. * jsonpickle can now serialize objects that implement `__getstate__()` but not `__setstate__()`. The result of `__getstate__()` is returned as-is when doing a round-trip from Python objects to jsonpickle and back. * Better support for collections.defaultdict with custom factories. * Added support for `queue.Queue` objects. Version 0.7.1 - May 6, 2014 ------------------------------ * Added support for Python 3.4. * Added support for :class:`posix.stat_result`. Version 0.7.0 - March 15, 2014 ------------------------------ * Added ``handles`` decorator to :class:`jsonpickle.handlers.BaseHandler`, enabling simple declaration of a handler for a class. * `__getstate__()` and `__setstate__()` are now honored when pickling objects that subclass :class:`dict`. * jsonpickle can now serialize :class:`collections.Counter` objects. * Object references are properly handled when using integer keys. * Object references are now supported when using custom handlers. * Decimal objects are supported in Python 3. * jsonpickle's "fallthrough-on-error" behavior can now be disabled. * Simpler API for registering custom handlers. * A new "safe-mode" is provided which avoids eval(). Backwards-compatible deserialization of repr-serialized objects is disabled in this mode. e.g. `decode(string, safe=True)` Version 0.6.1 - August 25, 2013 ------------------------------- * Python 3.2 support, and additional fixes for Python 3. Version 0.6.0 - August 24, 2013 ------------------------------- * Python 3 support! * :class:`time.struct_time` is now serialized using the built-in :class:`jsonpickle.handlers.SimpleReduceHandler`. Version 0.5.0 - August 22, 2013 ------------------------------- * Non-string dictionary keys (e.g. ints, objects) are now supported by passing `keys=True` to :func:`jsonpickle.encode` and :func:`jsonpickle.decode`. * We now support namedtuple, deque, and defaultdict. * Datetimes with timezones are now fully supported. * Better support for complicated structures e.g. datetime inside dicts. * jsonpickle added support for references and cyclical data structures in 0.4.0. This can be disabled by passing `make_refs=False` to :func:`jsonpickle.encode`. Version 0.4.0 - June 21, 2011 ----------------------------- * Switch build from setuptools to distutils * Consistent dictionary key ordering * Fix areas with improper support for unpicklable=False * Added support for cyclical data structures (`#16 `_). * Experimental support for `jsonlib `_ and `py-yajl `_ backends. * New contributers David K. Hess and Alec Thomas .. warning:: To support cyclical data structures (`#16 `_), the storage format has been modified. Efforts have been made to ensure backwards-compatibility. jsonpickle 0.4.0 can read data encoded by jsonpickle 0.3.1, but earlier versions of jsonpickle may be unable to read data encoded by jsonpickle 0.4.0. Version 0.3.1 - December 12, 2009 --------------------------------- * Include tests and docs directories in sdist for distribution packages. Version 0.3.0 - December 11, 2009 --------------------------------- * Officially migrated to git from subversion. Project home now at ``_. Thanks to Michael Jone's `sphinx-to-github `_. * Fortified jsonpickle against common error conditions. * Added support for: * List and set subclasses. * Objects with module references. * Newstyle classes with `__slots__`. * Objects implementing `__setstate__()` and `__getstate__()` (follows the :mod:`pickle` protocol). * Improved support for Zope objects via pre-fetch. * Support for user-defined serialization handlers via the jsonpickle.handlers registry. * Removed cjson support per John Millikin's recommendation. * General improvements to style, including :pep:`257` compliance and refactored project layout. * Steps towards Python 2.3 and Python 3 support. * New contributors Dan Buch and Ian Schenck. * Thanks also to Kieran Darcy, Eoghan Murray, and Antonin Hildebrand for their assistance! Version 0.2.0 - January 10, 2009 -------------------------------- * Support for all major Python JSON backends (including json in Python 2.6, simplejson, cjson, and demjson) * Handle several datetime objects using the repr() of the objects (Thanks to Antonin Hildebrand). * Sphinx documentation * Added support for recursive data structures * Unicode dict-keys support * Support for Google App Engine and Django * Tons of additional testing and bug reports (Antonin Hildebrand, Sorin, Roberto Saccon, Faber Fedor, `FirePython `_, and `Joose `_) Version 0.1.0 - August 21, 2008 ------------------------------- * Added long as basic primitive (thanks Adam Fisk) * Prefer python-cjson to simplejson, if available * Major API change, use python-cjson's decode/encode instead of simplejson's load/loads/dump/dumps * Added benchmark.py to compare simplejson and python-cjson Version 0.0.5 - July 21, 2008 ----------------------------- * Changed prefix of special fields to conform with CouchDB requirements (Thanks Dean Landolt). Break backwards compatibility. * Moved to Google Code subversion * Fixed unit test imports Version 0.0.3 ------------- * Convert back to setup.py from pavement.py (issue found by spidaman) Version 0.0.2 ------------- * Handle feedparser's FeedParserDict * Converted project to Paver * Restructured directories * Increase test coverage Version 0.0.1 ------------- Initial release jsonpickle-0.9.5/docs/source/index.rst0000644000175000017500000000502512770447510020170 0ustar daviddavid00000000000000======================== jsonpickle Documentation ======================== `jsonpickle `_ is a Python library for serialization and deserialization of complex Python objects to and from JSON. The standard Python libraries for encoding Python into JSON, such as the stdlib's json, simplejson, and demjson, can only handle Python primitives that have a direct JSON equivalent (e.g. dicts, lists, strings, ints, etc.). jsonpickle builds on top of these libraries and allows more complex data structures to be serialized to JSON. jsonpickle is highly configurable and extendable--allowing the user to choose the JSON backend and add additional backends. .. contents:: jsonpickle Usage ================ .. automodule:: jsonpickle Download & Install ================== The easiest way to get jsonpickle is via PyPi_ with pip_:: $ pip install -U jsonpickle For Python 2.6+, jsonpickle has no required dependencies (it uses the standard library's :mod:`json` module by default). For Python 2.5 or earlier, you must install a supported JSON backend (including simplejson or demjson). For example:: $ pip install simplejson You can also download or :ref:`checkout ` the latest code and install from source:: $ python setup.py install .. _PyPi: http://pypi.python.org/pypi/jsonpickle .. _pip: http://pypi.python.org/pypi/pip .. _download: http://pypi.python.org/pypi/jsonpickle API Reference ============= .. toctree:: api Extensions ========== .. toctree:: extensions Contributing ============ .. toctree:: contrib Contact ======= Please join our `mailing list `_. You can send email to *jsonpickle@googlegroups.com*. Check http://github.com/jsonpickle/jsonpickle for project updates. Authors ======= * John Paulett - john -at- paulett.org - http://github.com/johnpaulett * David Aguilar - davvid -at- gmail.com - http://github.com/davvid * Dan Buch - http://github.com/meatballhat * Ian Schenck - http://github.com/ianschenck * David K. Hess - http://github.com/davidkhess * Alec Thomas - http://github.com/alecthomas * jaraco - https://github.com/jaraco * Marcin Tustin - https://github.com/marcintustin Change Log ========== .. toctree:: :maxdepth: 2 changelog License ======= jsonpickle is provided under a `New BSD license `_, Copyright (C) 2008-2011 John Paulett (john -at- paulett.org) Copyright (C) 2009-2016 David Aguilar (davvid -at- gmail.com) jsonpickle-0.9.5/docs/source/conf.py0000664000175000017500000001602412205344652017625 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # jsonpickle documentation build configuration file, created by # sphinx-quickstart on Fri Oct 16 23:44:35 2009. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. sys.path.insert(0, os.path.abspath(os.path.join('..', '..'))) import jsonpickle sys.path.insert(1, os.path.abspath(os.path.join('..', '..', 'thirdparty', 'sphinx-to-github'))) try: import sphinxtogithub sphinxtogithub # import side-effects except ImportError, e: raise ImportError('Could not import sphinxtogithub\n' 'Is the git submodule populated at thirdparty/sphinx-to-github?\n' 'At the project root run:\n' '\tgit submodule init\n' '\tgit submodule update') # -- General configuration ----------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinxtogithub'] sphinx_to_github = True # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8' # The master toctree document. master_doc = 'index' # General information about the project. project = u'jsonpickle' copyright = u'2008-2011, John Paulett; 2009-2013, David Aguilar' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = jsonpickle.__version__ # The full version, including alpha/beta/rc tags. release = version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of documents that shouldn't be included in the build. #unused_docs = [] # List of directories, relative to source directory, that shouldn't be searched # for source files. exclude_trees = [] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = filter(os.path.exists, ['_static']) # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_use_modindex = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = '' # Output file base name for HTML help builder. htmlhelp_basename = 'jsonpickledoc' # -- Options for LaTeX output -------------------------------------------------- # The paper size ('letter' or 'a4'). #latex_paper_size = 'letter' # The font size ('10pt', '11pt' or '12pt'). #latex_font_size = '10pt' # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'jsonpickle.tex', u'jsonpickle Documentation', u'John Paulett', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # Additional stuff for the LaTeX preamble. #latex_preamble = '' # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_use_modindex = True # Example configuration for intersphinx: refer to the Python standard library. intersphinx_mapping = {'http://docs.python.org/': None} jsonpickle-0.9.5/docs/source/contrib.rst0000664000175000017500000000541012770445046020523 0ustar daviddavid00000000000000========================== Contributing to jsonpickle ========================== We welcome contributions from everyone. Please fork jsonpickle on `github `_. Get the Code ============ .. _jsonpickle-contrib-checkout: :: git clone git://github.com/jsonpickle/jsonpickle.git Run the Test Suite ================== Before code is pulled into the master jsonpickle branch, all tests should pass. If you are contributing an addition or a change in behavior, we ask that you document the change in the form of test cases. The jsonpickle test suite uses several JSON encoding libraries as well as several libraries for sample objects. To simplify the process of setting up these libraries we recommend creating a virtualenv_ and using a pip_ requirements file to install the dependencies. In the base jsonpickle directory:: # create a virtualenv that is completely isolated from the # site-wide python install virtualenv --no-site-packages env # use pip to install the dependencies listed in the requirements file ./env/bin/pip install --upgrade -r requirements-2.txt # Python2 ./env/bin/pip install --upgrade -r requirements-3.txt # Python3 ./env/bin/pip install --upgrade -r requirements-test.txt To run the suite, simply invoke :file:`tests/runtests.py`:: $ ./env/bin/python tests/runtests.py test_None (util_tests.IsPrimitiveTestCase) ... ok test_bool (util_tests.IsPrimitiveTestCase) ... ok test_dict (util_tests.IsPrimitiveTestCase) ... ok test_float (util_tests.IsPrimitiveTestCase) ... ok ... .. _virtualenv: http://pypi.python.org/pypi/virtualenv .. _pip: http://pypi.python.org/pypi/pip Testing with Tox ================ jsonpickle supports many versions of Python. To make it easy to test mutiple versions of Python you should install the tox_ testing tool, e.g. on Debian:: $ sudo apt-get install tox Once tox_ is installed you can run the test suite against multiple Python interpreters:: $ make tox It is recommended that you install at least one Python2 and one Python3 interpreter for use by tox_. .. _tox: https://tox.readthedocs.io/ Generate Documentation ====================== You do not need to generate the documentation_ when contributing, though, if you are interested, you can generate the docs yourself. The following requires Sphinx_ (present if you follow the virtualenv instructions above):: # pull down the sphinx-to-github project git submodule init git submodule update cd docs make html If you wish to browse the documentation, use Python's :mod:`SimpleHTTPServer` to host them at http://localhost:8000:: cd build/html python -m SimpleHTTPServer .. _documentation: http://jsonpickle.github.com .. _Sphinx: http://sphinx.pocoo.org jsonpickle-0.9.5/docs/source/api.rst0000644000175000017500000000752512616022341017627 0ustar daviddavid00000000000000.. _jsonpickle-api: ============== jsonpickle API ============== .. testsetup:: * import jsonpickle import jsonpickle.pickler import jsonpickle.unpickler import jsonpickle.handlers import jsonpickle.util .. contents:: :mod:`jsonpickle` -- High Level API =================================== .. autofunction:: jsonpickle.encode .. autofunction:: jsonpickle.decode Choosing and Loading Backends ----------------------------- jsonpickle allows the user to specify what JSON backend to use when encoding and decoding. By default, jsonpickle will try to use, in the following order: `simplejson `_, :mod:`json`, and `demjson `_. The prefered backend can be set via :func:`jsonpickle.set_preferred_backend`. Additional JSON backends can be used via :func:`jsonpickle.load_backend`. For example, users of `Django `_ can use the version of simplejson that is bundled in Django:: jsonpickle.load_backend('django.util.simplejson', 'dumps', 'loads', ValueError)) jsonpickle.set_preferred_backend('django.util.simplejson') Supported backends: * :mod:`json` in Python 2.6+ * `simplejson `_ * `demjson `_ Experimental backends: * `jsonlib `_ * yajl via `py-yajl `_ * `ujson `_ .. autofunction:: jsonpickle.set_preferred_backend .. autofunction:: jsonpickle.load_backend .. autofunction:: jsonpickle.remove_backend .. autofunction:: jsonpickle.set_encoder_options .. autofunction:: jsonpickle.set_decoder_options Customizing JSON output ----------------------- jsonpickle supports the standard :mod:`pickle` `__getstate__` and `__setstate__` protocol for representating object instances. .. method:: object.__getstate__() Classes can further influence how their instances are pickled; if the class defines the method :meth:`__getstate__`, it is called and the return state is pickled as the contents for the instance, instead of the contents of the instance's dictionary. If there is no :meth:`__getstate__` method, the instance's :attr:`__dict__` is pickled. .. method:: object.__setstate__(state) Upon unpickling, if the class also defines the method :meth:`__setstate__`, it is called with the unpickled state. If there is no :meth:`__setstate__` method, the pickled state must be a dictionary and its items are assigned to the new instance's dictionary. If a class defines both :meth:`__getstate__` and :meth:`__setstate__`, the state object needn't be a dictionary and these methods can do what they want. :mod:`jsonpickle.handlers` -- Custom Serialization Handlers ----------------------------------------------------------- The `jsonpickle.handlers` module allows plugging in custom serialization handlers at run-time. This feature is useful when jsonpickle is unable to serialize objects that are not under your direct control. .. automodule:: jsonpickle.handlers :members: :undoc-members: Low Level API ============= Typically this low level functionality is not needed by clients. :mod:`jsonpickle.pickler` -- Python to JSON ------------------------------------------- .. automodule:: jsonpickle.pickler :members: :undoc-members: :mod:`jsonpickle.unpickler` -- JSON to Python --------------------------------------------- .. automodule:: jsonpickle.unpickler :members: :undoc-members: :mod:`jsonpickle.backend` -- JSON Backend Management ---------------------------------------------------- .. automodule:: jsonpickle.backend :members: :mod:`jsonpickle.util` -- Helper functions ------------------------------------------ .. automodule:: jsonpickle.util :members: :undoc-members: jsonpickle-0.9.5/COPYING0000644000175000017500000000272512667763165015152 0ustar daviddavid00000000000000Copyright (C) 2008 John Paulett (john -at- paulett.org) Copyright (C) 2009-2016 David Aguilar (davvid -at- gmail.com) All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. jsonpickle-0.9.5/requirements-test.txt0000644000175000017500000000007112770442361020333 0ustar daviddavid00000000000000coverage enum34 feedparser nose numpy pymongo sqlalchemy jsonpickle-0.9.5/requirements-3.txt0000664000175000017500000000005112770443626017523 0ustar daviddavid00000000000000feedparser demjson simplejson ujson yajl jsonpickle-0.9.5/requirements-2.txt0000664000175000017500000000006112770457247017527 0ustar daviddavid00000000000000demjson feedparser jsonlib simplejson ujson yajl jsonpickle-0.9.5/tests/0000755000175000017500000000000013133025657015235 5ustar daviddavid00000000000000jsonpickle-0.9.5/tests/backend_test.py0000644000175000017500000001234712667760616020260 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- import unittest from warnings import warn import jsonpickle from jsonpickle.compat import unicode from jsonpickle.compat import PY2 from jsonpickle.compat import PY3 from jsonpickle.compat import PY32 from helper import SkippableTest class Thing(object): def __init__(self, name): self.name = name self.child = None SAMPLE_DATA = {'things': [Thing('data')]} class BackendBase(SkippableTest): def _is_installed(self, backend): if not jsonpickle.util.is_installed(backend): return self.skip('%s not available; please install' % backend) def set_backend(self, *args): backend = args[0] self._is_installed(backend) jsonpickle.load_backend(*args) jsonpickle.set_preferred_backend(backend) def set_preferred_backend(self, backend): self._is_installed(backend) jsonpickle.set_preferred_backend(backend) def tearDown(self): # always reset to default backend jsonpickle.set_preferred_backend('json') def assertEncodeDecode(self, json_input): expect = SAMPLE_DATA actual = jsonpickle.decode(json_input) self.assertEqual(expect['things'][0].name, actual['things'][0].name) self.assertEqual(expect['things'][0].child, actual['things'][0].child) pickled = jsonpickle.encode(SAMPLE_DATA) actual = jsonpickle.decode(pickled) self.assertEqual(expect['things'][0].name, actual['things'][0].name) self.assertEqual(expect['things'][0].child, actual['things'][0].child) def test_None_dict_key(self): """Ensure that backends produce the same result for None dict keys""" data = {None: None} expect = {'null': None} pickle = jsonpickle.encode(data) actual = jsonpickle.decode(pickle) self.assertEqual(expect, actual) class JsonTestCase(BackendBase): def setUp(self): self.set_preferred_backend('json') def test_backend(self): expected_pickled = ( '{"things": [{' '"py/object": "backend_test.Thing", ' '"name": "data", ' '"child": null} ' ']}') self.assertEncodeDecode(expected_pickled) class SimpleJsonTestCase(BackendBase): def setUp(self): if PY32: return self.set_preferred_backend('simplejson') def test_backend(self): if PY32: return self.skip('no simplejson for python3.2') expected_pickled = ( '{"things": [{' '"py/object": "backend_test.Thing", ' '"name": "data", ' '"child": null}' ']}') self.assertEncodeDecode(expected_pickled) def has_module(module): try: __import__(module) except ImportError: warn(module + ' module not available for testing, ' 'consider installing') return False return True class DemjsonTestCase(BackendBase): def setUp(self): self.set_preferred_backend('demjson') def test_backend(self): expected_pickled = unicode( '{"things":[{' '"child":null,' '"name":"data",' '"py/object":"backend_test.Thing"}' ']}') self.assertEncodeDecode(expected_pickled) def test_int_dict_keys_with_numeric_keys(self): jsonpickle.set_encoder_options('demjson', strict=False) int_dict = {1000: [1, 2]} pickle = jsonpickle.encode(int_dict, numeric_keys=True) actual = jsonpickle.decode(pickle) self.assertEqual(actual[1000], [1, 2]) class JsonlibTestCase(BackendBase): def setUp(self): if PY2: self.set_preferred_backend('jsonlib') def test_backend(self): if PY3: return self.skip('no jsonlib for python3') expected_pickled = ( '{"things":[{' '"py\/object":"backend_test.Thing",' '"name":"data","child":null}' ']}') self.assertEncodeDecode(expected_pickled) class YajlTestCase(BackendBase): def setUp(self): self.set_preferred_backend('yajl') def test_backend(self): expected_pickled = ( '{"things":[{' '"py/object":"backend_test.Thing",' '"name":"data","child":null}' ']}') self.assertEncodeDecode(expected_pickled) class UJsonTestCase(BackendBase): def setUp(self): self.set_preferred_backend('ujson') def test_backend(self): expected_pickled = ( '{"things":[{' '"py\/object":"backend_test.Thing",' '"name":"data","child":null}' ']}') self.assertEncodeDecode(expected_pickled) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(JsonTestCase)) suite.addTest(unittest.makeSuite(UJsonTestCase)) if not PY32: suite.addTest(unittest.makeSuite(SimpleJsonTestCase)) if has_module('demjson'): suite.addTest(unittest.makeSuite(DemjsonTestCase)) if has_module('yajl'): suite.addTest(unittest.makeSuite(YajlTestCase)) if PY2: if has_module('jsonlib'): suite.addTest(unittest.makeSuite(JsonlibTestCase)) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/tests/util_test.py0000664000175000017500000001602412454145546017635 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. import unittest import doctest import time import jsonpickle.util from jsonpickle.compat import unicode, long, PY2 from jsonpickle import util class Thing(object): def __init__(self, name): self.name = name self.child = None class DictSubclass(dict): pass class ListSubclass(list): pass class MethodTestClass(object): @staticmethod def static_method(): pass @classmethod def class_method(cls): pass def bound_method(self): pass variable = None class MethodTestSubclass(MethodTestClass): pass class MethodTestOldStyle: def bound_method(self): pass class UtilTestCase(unittest.TestCase): def test_is_primitive_int(self): self.assertTrue(util.is_primitive(0)) self.assertTrue(util.is_primitive(3)) self.assertTrue(util.is_primitive(-3)) def test_is_primitive_float(self): self.assertTrue(util.is_primitive(0)) self.assertTrue(util.is_primitive(3.5)) self.assertTrue(util.is_primitive(-3.5)) self.assertTrue(util.is_primitive(float(3))) def test_is_primitive_long(self): self.assertTrue(util.is_primitive(long(3))) def test_is_primitive_bool(self): self.assertTrue(util.is_primitive(True)) self.assertTrue(util.is_primitive(False)) def test_is_primitive_None(self): self.assertTrue(util.is_primitive(None)) def test_is_primitive_bytes(self): self.assertFalse(util.is_primitive(b'hello')) if PY2: self.assertFalse(util.is_primitive('foo')) else: self.assertTrue(util.is_primitive('foo')) def test_is_primitive_unicode(self): self.assertTrue(util.is_primitive(unicode('hello'))) self.assertTrue(util.is_primitive(unicode(''))) self.assertTrue(util.is_primitive(unicode('hello'))) def test_is_primitive_list(self): self.assertFalse(util.is_primitive([])) self.assertFalse(util.is_primitive([4, 4])) def test_is_primitive_dict(self): self.assertFalse(util.is_primitive({'key': 'value'})) self.assertFalse(util.is_primitive({})) def test_is_primitive_tuple(self): self.assertFalse(util.is_primitive((1, 3))) self.assertFalse(util.is_primitive((1,))) def test_is_primitive_set(self): self.assertFalse(util.is_primitive(set([1, 3]))) def test_is_primitive_object(self): self.assertFalse(util.is_primitive(Thing('test'))) def test_is_list_list(self): self.assertTrue(util.is_list([1, 2])) def test_is_list_set(self): self.assertTrue(util.is_set(set([1, 2]))) def test_is_list_tuple(self): self.assertTrue(util.is_tuple((1, 2))) def test_is_list_dict(self): self.assertFalse(util.is_list({'key': 'value'})) self.assertFalse(util.is_set({'key': 'value'})) self.assertFalse(util.is_tuple({'key': 'value'})) def test_is_list_other(self): self.assertFalse(util.is_list(1)) self.assertFalse(util.is_set(1)) self.assertFalse(util.is_tuple(1)) def test_is_sequence_various(self): self.assertTrue(util.is_sequence([])) self.assertTrue(util.is_sequence(tuple())) self.assertTrue(util.is_sequence(set())) def test_is_dictionary_dict(self): self.assertTrue(util.is_dictionary({})) def test_is_dicitonary_sequences(self): self.assertFalse(util.is_dictionary([])) self.assertFalse(util.is_dictionary(set())) def test_is_dictionary_tuple(self): self.assertFalse(util.is_dictionary(tuple())) def test_is_dictionary_primitive(self): self.assertFalse(util.is_dictionary(int())) self.assertFalse(util.is_dictionary(None)) self.assertFalse(util.is_dictionary(str())) def test_is_dictionary_subclass_dict(self): self.assertFalse(util.is_dictionary_subclass({})) def test_is_dictionary_subclass_subclass(self): self.assertTrue(util.is_dictionary_subclass(DictSubclass())) def test_is_sequence_subclass_subclass(self): self.assertTrue(util.is_sequence_subclass(ListSubclass())) def test_is_sequence_subclass_list(self): self.assertFalse(util.is_sequence_subclass([])) def test_is_noncomplex_time_struct(self): t = time.struct_time('123456789') self.assertTrue(util.is_noncomplex(t)) def test_is_noncomplex_other(self): self.assertFalse(util.is_noncomplex('a')) def test_is_function_builtins(self): self.assertTrue(util.is_function(globals)) def test_is_function_lambda(self): self.assertTrue(util.is_function(lambda: False)) def test_is_function_instance_method(self): class Foo(object): def method(self): pass @staticmethod def staticmethod(): pass @classmethod def classmethod(cls): pass f = Foo() self.assertTrue(util.is_function(f.method)) self.assertTrue(util.is_function(f.staticmethod)) self.assertTrue(util.is_function(f.classmethod)) def test_itemgetter(self): expect = '0' actual = util.itemgetter((0, 'zero')) self.assertEqual(expect, actual) def test_has_method(self): instance = MethodTestClass() x = 1 has_method = jsonpickle.util.has_method # no attribute self.assertFalse(has_method(instance, 'foo')) # builtin method type self.assertFalse(has_method(int, '__getnewargs__')) self.assertTrue(has_method(x, '__getnewargs__')) # staticmethod self.assertTrue(has_method(instance, 'static_method')) self.assertTrue(has_method(MethodTestClass, 'static_method')) # not a method self.assertFalse(has_method(instance, 'variable')) self.assertFalse(has_method(MethodTestClass, 'variable')) # classmethod self.assertTrue(has_method(instance, 'class_method')) self.assertTrue(has_method(MethodTestClass, 'class_method')) # bound method self.assertTrue(has_method(instance, 'bound_method')) self.assertFalse(has_method(MethodTestClass, 'bound_method')) # subclass bound method sub_instance = MethodTestSubclass() self.assertTrue(has_method(sub_instance, 'bound_method')) self.assertFalse(has_method(MethodTestSubclass, 'bound_method')) # old style object old_instance = MethodTestOldStyle() self.assertTrue(has_method(old_instance, 'bound_method')) self.assertFalse(has_method(MethodTestOldStyle, 'bound_method')) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(UtilTestCase)) suite.addTest(doctest.DocTestSuite(jsonpickle.util)) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/tests/helper.py0000664000175000017500000000026412667750407017103 0ustar daviddavid00000000000000import unittest class SkippableTest(unittest.TestCase): def skip(self, msg): if hasattr(self, 'skipTest'): return self.skipTest(msg) return None jsonpickle-0.9.5/tests/runtests.py0000775000175000017500000000235512562610455017511 0ustar daviddavid00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. import os import sys testdir = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(1, os.path.dirname(testdir)) import unittest import backend_test import datetime_test import document_test import handler_test import jsonpickle_test import object_test import stdlib_test import util_test import feedparser_test import bson_test import numpy_test def suite(): suite = unittest.TestSuite() suite.addTest(util_test.suite()) suite.addTest(handler_test.suite()) suite.addTest(backend_test.suite()) suite.addTest(jsonpickle_test.suite()) suite.addTest(datetime_test.suite()) suite.addTest(document_test.suite()) suite.addTest(object_test.suite()) suite.addTest(stdlib_test.suite()) suite.addTest(feedparser_test.suite()) suite.addTest(numpy_test.suite()) suite.addTest(bson_test.suite()) return suite def main(): return unittest.TextTestRunner(verbosity=2).run(suite()) if __name__ == '__main__': sys.exit(not main().wasSuccessful()) jsonpickle-0.9.5/tests/benchmark.py0000775000175000017500000000153312105131377017543 0ustar daviddavid00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. import sys import timeit IS_25_DOWN = sys.version_info[:2] <= (2, 5) number = 1000 mod = 'json' if IS_25_DOWN: mod = 'simplejson' json = """\ import feedparser import jsonpickle import jsonpickle.tests.thirdparty_tests as test doc = feedparser.parse(test.RSS_DOC) jsonpickle.set_preferred_backend('%s') pickled = jsonpickle.encode(doc) unpickled = jsonpickle.decode(pickled) if doc['feed']['title'] != unpickled['feed']['title']: print 'Not a match' """ % mod print 'Using %s' % mod json_test = timeit.Timer(stmt=json) print "%.9f sec/pass " % (json_test.timeit(number=number) / number) jsonpickle-0.9.5/tests/datetime_test.py0000644000175000017500000002011713034337735020446 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2013 Jason R. Coombs # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. import unittest import datetime import time import jsonpickle from jsonpickle import tags class ObjWithDate(object): def __init__(self): ts = datetime.datetime.now() self.data = dict(a='a', ts=ts) self.data_ref = dict(b='b', ts=ts) # UTC implementation from Python 2.7 docs class UTC(datetime.tzinfo): """UTC""" def utcoffset(self, dt): return datetime.timedelta() def tzname(self, dt): return 'UTC' def dst(self, dt): return datetime.timedelta() utc = UTC() class TimestampedVariable(object): def __init__(self, value=None): self._value = value self._dt_read = datetime.datetime.utcnow() self._dt_write = self._dt_read def get(self, default_value=None): if self._dt_read is None and self._dt_write is None: value = default_value self._value = value self._dt_write = datetime.datetime.utcnow() else: value = self._value self._dt_read = datetime.datetime.utcnow() return value def set(self, new_value): self._dt_write = datetime.datetime.utcnow() self._value = new_value def __repr__(self): dt_now = datetime.datetime.utcnow() td_read = dt_now - self._dt_read td_write = dt_now - self._dt_write s = '\n' s += ' value: ' + str(self._value) + '\n' s += ' dt_read: ' + str(self._dt_read) + ' (%s ago)' % td_read + '\n' s += ' dt_write: ' + str(self._dt_write) + ' (%s ago)' % td_write + '\n' return s def erasable(self, td=datetime.timedelta(seconds=1)): dt_now = datetime.datetime.utcnow() td_read = dt_now - self._dt_read td_write = dt_now - self._dt_write return td_read > td and td_write > td class PersistantVariables(object): def __init__(self): self._data = {} def __getitem__(self, key): if key not in self._data: self._data[key] = TimestampedVariable(None) return self._data[key] def __setitem__(self, key, value): if key not in self._data: self._data[key] = TimestampedVariable(value) return self._data[key] def __repr__(self): return str(self._data) class DateTimeInnerReferenceTestCase(unittest.TestCase): def test_object_with_inner_datetime_refs(self): pvars = PersistantVariables() pvars['z'] = 1 pvars['z2'] = 2 pickled = jsonpickle.encode(pvars) obj = jsonpickle.decode(pickled) # ensure the references are valid self.assertTrue(obj['z']._dt_read is obj['z']._dt_write) self.assertTrue(obj['z2']._dt_read is obj['z2']._dt_write) # ensure the values are valid self.assertEqual(obj['z'].get(), 1) self.assertEqual(obj['z2'].get(), 2) # ensure get() updates _dt_read self.assertTrue(obj['z']._dt_read is not obj['z']._dt_write) self.assertTrue(obj['z2']._dt_read is not obj['z2']._dt_write) class DateTimeSimpleTestCase(unittest.TestCase): def _roundtrip(self, obj): """ pickle and then unpickle object, then assert the new object is the same as the original. """ pickled = jsonpickle.encode(obj) unpickled = jsonpickle.decode(pickled) self.assertEquals(obj, unpickled) def test_datetime(self): """ jsonpickle should pickle a datetime object """ self._roundtrip(datetime.datetime.now()) def test_date(self): """ jsonpickle should pickle a date object """ self._roundtrip(datetime.datetime.today()) def test_time(self): """ jsonpickle should pickle a time object """ self._roundtrip(datetime.datetime.now().time()) def test_timedelta(self): """ jsonpickle should pickle a timedelta object """ self._roundtrip(datetime.timedelta(days=3)) def test_utc(self): """ jsonpickle should be able to encode and decode a datetime with a simple, pickleable UTC tzinfo. """ self._roundtrip(datetime.datetime.utcnow().replace(tzinfo=utc)) def test_unpickleable(self): """ If 'unpickleable' is set on the Pickler, the date objects should be simple, human-readable strings. """ obj = datetime.datetime.now() pickler = jsonpickle.pickler.Pickler(unpicklable=False) flattened = pickler.flatten(obj) self.assertEqual(str(obj), flattened) def test_object_with_datetime(self): test_obj = ObjWithDate() json = jsonpickle.encode(test_obj) test_obj_decoded = jsonpickle.decode(json) self.assertEqual(test_obj_decoded.data['ts'], test_obj_decoded.data_ref['ts']) class DateTimeAdvancedTestCase(unittest.TestCase): def setUp(self): self.pickler = jsonpickle.pickler.Pickler() self.unpickler = jsonpickle.unpickler.Unpickler() def tearDown(self): self.pickler.reset() self.unpickler.reset() def test_struct_time(self): expect = time.struct_time([1, 2, 3, 4, 5, 6, 7, 8, 9]) json = jsonpickle.encode(expect) actual = jsonpickle.decode(json) self.assertEqual(type(actual), time.struct_time) self.assertEqual(expect, actual) def test_struct_time_chars(self): expect = time.struct_time('123456789') flattened = self.pickler.flatten(expect) actual = self.unpickler.restore(flattened) self.assertEqual(expect, actual) def test_datetime_structure(self): obj = datetime.datetime.now() flattened = self.pickler.flatten(obj) self.assertTrue(tags.OBJECT in flattened) self.assertTrue('__reduce__' in flattened) inflated = self.unpickler.restore(flattened) self.assertEqual(obj, inflated) def test_datetime_inside_int_keys_defaults(self): t = datetime.time(hour=10) s = jsonpickle.encode({1: t, 2: t}) d = jsonpickle.decode(s) self.assertEqual(d["1"], d["2"]) self.assertTrue(d["1"] is d["2"]) self.assertTrue(isinstance(d["1"], datetime.time)) def test_datetime_inside_int_keys_with_keys_enabled(self): t = datetime.time(hour=10) s = jsonpickle.encode({1: t, 2: t}, keys=True) d = jsonpickle.decode(s, keys=True) self.assertEqual(d[1], d[2]) self.assertTrue(d[1] is d[2]) self.assertTrue(isinstance(d[1], datetime.time)) def test_datetime_repr_not_unpicklable(self): obj = datetime.datetime.now() pickler = jsonpickle.pickler.Pickler(unpicklable=False) flattened = pickler.flatten(obj) self.assertFalse(tags.REPR in flattened) self.assertFalse(tags.OBJECT in flattened) self.assertEqual(str(obj), flattened) def test_datetime_dict_keys_defaults(self): """Test that we handle datetime objects as keys.""" datetime_dict = {datetime.datetime(2008, 12, 31): True} pickled = jsonpickle.encode(datetime_dict) expect = {'datetime.datetime(2008, 12, 31, 0, 0)': True} actual = jsonpickle.decode(pickled) self.assertEqual(expect, actual) def test_datetime_dict_keys_with_keys_enabled(self): """Test that we handle datetime objects as keys.""" datetime_dict = {datetime.datetime(2008, 12, 31): True} pickled = jsonpickle.encode(datetime_dict, keys=True) expect = datetime_dict actual = jsonpickle.decode(pickled, keys=True) self.assertEqual(expect, actual) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(DateTimeSimpleTestCase)) suite.addTest(unittest.makeSuite(DateTimeAdvancedTestCase)) suite.addTest(unittest.makeSuite(DateTimeInnerReferenceTestCase)) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/tests/jsonpickle_test.py0000644000175000017500000013317613075310375021021 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # Copyright (C) 2009, 2011, 2013 David Aguilar and contributors # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. import doctest import os import unittest import collections import jsonpickle import jsonpickle.backend import jsonpickle.handlers from jsonpickle import tags, util from jsonpickle.compat import unicode from jsonpickle.compat import unichr from jsonpickle.compat import PY32, PY3 from helper import SkippableTest class Thing(object): def __init__(self, name): self.name = name self.child = None def __repr__(self): return 'Thing("%s")' % self.name class Capture(object): def __init__(self, *args, **kwargs): self.args = args self.kwargs = kwargs class ThingWithProps(object): def __init__(self, name='', dogs='reliable', monkies='tricksy'): self.name = name self._critters = (('dogs', dogs), ('monkies', monkies)) def _get_identity(self): keys = [self.dogs, self.monkies, self.name] return hash('-'.join([str(key) for key in keys])) identity = property(_get_identity) def _get_dogs(self): return self._critters[0][1] dogs = property(_get_dogs) def _get_monkies(self): return self._critters[1][1] monkies = property(_get_monkies) def __getstate__(self): out = dict( __identity__=self.identity, nom=self.name, dogs=self.dogs, monkies=self.monkies, ) return out def __setstate__(self, state_dict): self._critters = (('dogs', state_dict.get('dogs')), ('monkies', state_dict.get('monkies'))) self.name = state_dict.get('nom', '') ident = state_dict.get('__identity__') if ident != self.identity: raise ValueError('expanded object does not match originial state!') def __eq__(self, other): return self.identity == other.identity class PicklingTestCase(unittest.TestCase): def setUp(self): self.pickler = jsonpickle.pickler.Pickler() self.unpickler = jsonpickle.unpickler.Unpickler() def tearDown(self): self.pickler.reset() self.unpickler.reset() def test_string(self): self.assertEqual('a string', self.pickler.flatten('a string')) self.assertEqual('a string', self.unpickler.restore('a string')) def test_unicode(self): self.assertEqual(unicode('a string'), self.pickler.flatten('a string')) self.assertEqual(unicode('a string'), self.unpickler.restore('a string')) def test_int(self): self.assertEqual(3, self.pickler.flatten(3)) self.assertEqual(3, self.unpickler.restore(3)) def test_float(self): self.assertEqual(3.5, self.pickler.flatten(3.5)) self.assertEqual(3.5, self.unpickler.restore(3.5)) def test_boolean(self): self.assertTrue(self.pickler.flatten(True)) self.assertFalse(self.pickler.flatten(False)) self.assertTrue(self.unpickler.restore(True)) self.assertFalse(self.unpickler.restore(False)) def test_none(self): self.assertTrue(self.pickler.flatten(None) is None) self.assertTrue(self.unpickler.restore(None) is None) def test_list(self): # multiple types of values listA = [1, 35.0, 'value'] self.assertEqual(listA, self.pickler.flatten(listA)) self.assertEqual(listA, self.unpickler.restore(listA)) # nested list listB = [40, 40, listA, 6] self.assertEqual(listB, self.pickler.flatten(listB)) self.assertEqual(listB, self.unpickler.restore(listB)) # 2D list listC = [[1, 2], [3, 4]] self.assertEqual(listC, self.pickler.flatten(listC)) self.assertEqual(listC, self.unpickler.restore(listC)) # empty list listD = [] self.assertEqual(listD, self.pickler.flatten(listD)) self.assertEqual(listD, self.unpickler.restore(listD)) def test_set(self): setlist = ['orange', 'apple', 'grape'] setA = set(setlist) flattened = self.pickler.flatten(setA) for s in setlist: self.assertTrue(s in flattened[tags.SET]) setA_pickle = {tags.SET: setlist} self.assertEqual(setA, self.unpickler.restore(setA_pickle)) def test_dict(self): dictA = {'key1': 1.0, 'key2': 20, 'key3': 'thirty', tags.JSON_KEY + '6': 6} self.assertEqual(dictA, self.pickler.flatten(dictA)) self.assertEqual(dictA, self.unpickler.restore(dictA)) dictB = {} self.assertEqual(dictB, self.pickler.flatten(dictB)) self.assertEqual(dictB, self.unpickler.restore(dictB)) def test_tuple(self): # currently all collections are converted to lists tupleA = (4, 16, 32) tupleA_pickle = {tags.TUPLE: [4, 16, 32]} self.assertEqual(tupleA_pickle, self.pickler.flatten(tupleA)) self.assertEqual(tupleA, self.unpickler.restore(tupleA_pickle)) tupleB = (4,) tupleB_pickle = {tags.TUPLE: [4]} self.assertEqual(tupleB_pickle, self.pickler.flatten(tupleB)) self.assertEqual(tupleB, self.unpickler.restore(tupleB_pickle)) def test_tuple_roundtrip(self): data = (1, 2, 3) newdata = jsonpickle.decode(jsonpickle.encode(data)) self.assertEqual(data, newdata) def test_set_roundtrip(self): data = set([1, 2, 3]) newdata = jsonpickle.decode(jsonpickle.encode(data)) self.assertEqual(data, newdata) def test_list_roundtrip(self): data = [1, 2, 3] newdata = jsonpickle.decode(jsonpickle.encode(data)) self.assertEqual(data, newdata) def test_class(self): inst = Thing('test name') inst.child = Thing('child name') flattened = self.pickler.flatten(inst) self.assertEqual('test name', flattened['name']) child = flattened['child'] self.assertEqual('child name', child['name']) inflated = self.unpickler.restore(flattened) self.assertEqual('test name', inflated.name) self.assertTrue(type(inflated) is Thing) self.assertEqual('child name', inflated.child.name) self.assertTrue(type(inflated.child) is Thing) def test_classlist(self): array = [Thing('one'), Thing('two'), 'a string'] flattened = self.pickler.flatten(array) self.assertEqual('one', flattened[0]['name']) self.assertEqual('two', flattened[1]['name']) self.assertEqual('a string', flattened[2]) inflated = self.unpickler.restore(flattened) self.assertEqual('one', inflated[0].name) self.assertTrue(type(inflated[0]) is Thing) self.assertEqual('two', inflated[1].name) self.assertTrue(type(inflated[1]) is Thing) self.assertEqual('a string', inflated[2]) def test_classdict(self): dict = {'k1': Thing('one'), 'k2': Thing('two'), 'k3': 3} flattened = self.pickler.flatten(dict) self.assertEqual('one', flattened['k1']['name']) self.assertEqual('two', flattened['k2']['name']) self.assertEqual(3, flattened['k3']) inflated = self.unpickler.restore(flattened) self.assertEqual('one', inflated['k1'].name) self.assertTrue(type(inflated['k1']) is Thing) self.assertEqual('two', inflated['k2'].name) self.assertTrue(type(inflated['k2']) is Thing) self.assertEqual(3, inflated['k3']) def test_recursive(self): """create a recursive structure and test that we can handle it """ parent = Thing('parent') child = Thing('child') child.sibling = Thing('sibling') parent.self = parent parent.child = child parent.child.twin = child parent.child.parent = parent parent.child.sibling.parent = parent cloned = jsonpickle.decode(jsonpickle.encode(parent)) self.assertEqual(parent.name, cloned.name) self.assertEqual(parent.child.name, cloned.child.name) self.assertEqual(parent.child.sibling.name, cloned.child.sibling.name) self.assertEqual(cloned, cloned.child.parent) self.assertEqual(cloned, cloned.child.sibling.parent) self.assertEqual(cloned, cloned.child.twin.parent) self.assertEqual(cloned.child, cloned.child.twin) def test_tuple_notunpicklable(self): self.pickler.unpicklable = False flattened = self.pickler.flatten(('one', 2, 3)) self.assertEqual(flattened, ['one', 2, 3]) def test_set_not_unpicklable(self): self.pickler.unpicklable = False flattened = self.pickler.flatten(set(['one', 2, 3])) self.assertTrue('one' in flattened) self.assertTrue(2 in flattened) self.assertTrue(3 in flattened) self.assertTrue(isinstance(flattened, list)) def test_thing_with_module(self): obj = Thing('with-module') obj.themodule = os flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) self.assertEqual(inflated.themodule, os) def test_thing_with_module_safe(self): obj = Thing('with-module') obj.themodule = os flattened = self.pickler.flatten(obj) self.unpickler.safe = True inflated = self.unpickler.restore(flattened) self.assertEqual(inflated.themodule, None) def test_thing_with_submodule(self): from distutils import sysconfig obj = Thing('with-submodule') obj.submodule = sysconfig flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) self.assertEqual(inflated.submodule, sysconfig) def test_type_reference(self): """This test ensures that users can store references to types. """ obj = Thing('object-with-type-reference') # reference the built-in 'object' type obj.typeref = object flattened = self.pickler.flatten(obj) self.assertEqual(flattened['typeref'], {tags.TYPE: '__builtin__.object'}) inflated = self.unpickler.restore(flattened) self.assertEqual(inflated.typeref, object) def test_class_reference(self): """This test ensures that users can store references to classes. """ obj = Thing('object-with-class-reference') # reference the 'Thing' class (not an instance of the class) obj.classref = Thing flattened = self.pickler.flatten(obj) self.assertEqual(flattened['classref'], {tags.TYPE: 'jsonpickle_test.Thing'}) inflated = self.unpickler.restore(flattened) self.assertEqual(inflated.classref, Thing) def test_supports_getstate_setstate(self): obj = ThingWithProps('object-which-defines-getstate-setstate') flattened = self.pickler.flatten(obj) self.assertTrue(flattened[tags.STATE].get('__identity__')) self.assertTrue(flattened[tags.STATE].get('nom')) inflated = self.unpickler.restore(flattened) self.assertEqual(obj, inflated) def test_references(self): obj_a = Thing('foo') obj_b = Thing('bar') coll = [obj_a, obj_b, obj_b] flattened = self.pickler.flatten(coll) inflated = self.unpickler.restore(flattened) self.assertEqual(len(inflated), len(coll)) for x in range(len(coll)): self.assertEqual(repr(coll[x]), repr(inflated[x])) def test_references_in_number_keyed_dict(self): """ Make sure a dictionary with numbers as keys and objects as values can make the round trip. Because JSON must coerce integers to strings in dict keys, the sort order may have a tendency to change between pickling and unpickling, and this could affect the object references. """ one = Thing('one') two = Thing('two') twelve = Thing('twelve') two.child = twelve obj = { 1: one, 2: two, 12: twelve, } self.assertNotEqual(list(sorted(obj.keys())), list(map(int, sorted(map(str, obj.keys()))))) flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) self.assertEqual(len(inflated), 3) self.assertEqual(inflated['12'].name, 'twelve') def test_builtin_error(self): expect = AssertionError json = jsonpickle.encode(expect) actual = jsonpickle.decode(json) self.assertEqual(expect, actual) self.assertTrue(expect is actual) class JSONPickleTestCase(SkippableTest): def setUp(self): self.obj = Thing('A name') self.expected_json = ( '{"%s": "jsonpickle_test.Thing", "name": "A name", "child": null}' % tags.OBJECT ) def test_encode(self): expect = self.obj pickle = jsonpickle.encode(self.obj) actual = jsonpickle.decode(pickle) self.assertEqual(expect.name, actual.name) self.assertEqual(expect.child, actual.child) def test_encode_notunpicklable(self): expect = {'name': 'A name', 'child': None} pickle = jsonpickle.encode(self.obj, unpicklable=False) actual = jsonpickle.decode(pickle) self.assertEqual(expect['name'], actual['name']) def test_decode(self): actual = jsonpickle.decode(self.expected_json) self.assertEqual(self.obj.name, actual.name) self.assertEqual(type(self.obj), type(actual)) def test_json(self): expect = self.obj pickle = jsonpickle.encode(self.obj) actual = jsonpickle.decode(pickle) self.assertEqual(actual.name, expect.name) self.assertEqual(actual.child, expect.child) actual = jsonpickle.decode(self.expected_json) self.assertEqual(self.obj.name, actual.name) self.assertEqual(type(self.obj), type(actual)) def test_unicode_dict_keys(self): uni = unichr(0x1234) pickle = jsonpickle.encode({uni: uni}) actual = jsonpickle.decode(pickle) self.assertTrue(uni in actual) self.assertEqual(actual[uni], uni) def test_tuple_dict_keys_default(self): """Test that we handle dictionaries with tuples as keys.""" tuple_dict = {(1, 2): 3, (4, 5): {(7, 8): 9}} pickle = jsonpickle.encode(tuple_dict) expect = {'(1, 2)': 3, '(4, 5)': {'(7, 8)': 9}} actual = jsonpickle.decode(pickle) self.assertEqual(expect, actual) tuple_dict = {(1, 2): [1, 2]} pickle = jsonpickle.encode(tuple_dict) actual = jsonpickle.decode(pickle) self.assertEqual(actual['(1, 2)'], [1, 2]) def test_tuple_dict_keys_with_keys_enabled(self): """Test that we handle dictionaries with tuples as keys.""" tuple_dict = {(1, 2): 3, (4, 5): {(7, 8): 9}} pickle = jsonpickle.encode(tuple_dict, keys=True) expect = tuple_dict actual = jsonpickle.decode(pickle, keys=True) self.assertEqual(expect, actual) tuple_dict = {(1, 2): [1, 2]} pickle = jsonpickle.encode(tuple_dict, keys=True) actual = jsonpickle.decode(pickle, keys=True) self.assertEqual(actual[(1, 2)], [1, 2]) def test_None_dict_key_default(self): # We do string coercion for non-string keys so None becomes 'None' expect = {'null': None} obj = {None: None} pickle = jsonpickle.encode(obj) actual = jsonpickle.decode(pickle) self.assertEqual(expect, actual) def test_None_dict_key_with_keys_enabled(self): expect = {None: None} obj = {None: None} pickle = jsonpickle.encode(obj, keys=True) actual = jsonpickle.decode(pickle, keys=True) self.assertEqual(expect, actual) def test_object_dict_keys(self): """Test that we handle random objects as keys. """ thing = Thing('random') pickle = jsonpickle.encode({thing: True}) actual = jsonpickle.decode(pickle) self.assertEqual(actual, {unicode('Thing("random")'): True}) def test_int_dict_keys_defaults(self): int_dict = {1000: [1, 2]} pickle = jsonpickle.encode(int_dict) actual = jsonpickle.decode(pickle) self.assertEqual(actual['1000'], [1, 2]) def test_int_dict_keys_with_keys_enabled(self): int_dict = {1000: [1, 2]} pickle = jsonpickle.encode(int_dict, keys=True) actual = jsonpickle.decode(pickle, keys=True) self.assertEqual(actual[1000], [1, 2]) def test_string_key_requiring_escape_dict_keys_with_keys_enabled(self): json_key_dict = {tags.JSON_KEY + '6': [1, 2]} pickled = jsonpickle.encode(json_key_dict, keys=True) unpickled = jsonpickle.decode(pickled, keys=True) self.assertEqual(unpickled[tags.JSON_KEY + '6'], [1, 2]) def test_string_key_not_requiring_escape_dict_keys_with_keys_enabled(self): """test that string keys that do not require escaping are not escaped""" str_dict = {'name': [1, 2]} pickled = jsonpickle.encode(str_dict, keys=True) unpickled = jsonpickle.decode(pickled) self.assertTrue('name' in unpickled) def test_list_of_objects(self): """Test that objects in lists are referenced correctly""" a = Thing('a') b = Thing('b') pickle = jsonpickle.encode([a, b, b]) actual = jsonpickle.decode(pickle) self.assertEqual(actual[1], actual[2]) self.assertEqual(type(actual[0]), Thing) self.assertEqual(actual[0].name, 'a') self.assertEqual(actual[1].name, 'b') self.assertEqual(actual[2].name, 'b') def test_refs_keys_values(self): """Test that objects in dict keys are referenced correctly """ j = Thing('random') object_dict = {j: j} pickle = jsonpickle.encode(object_dict, keys=True) actual = jsonpickle.decode(pickle, keys=True) self.assertEqual(list(actual.keys()), list(actual.values())) def test_object_keys_to_list(self): """Test that objects in dict values are referenced correctly """ j = Thing('random') object_dict = {j: [j, j]} pickle = jsonpickle.encode(object_dict, keys=True) actual = jsonpickle.decode(pickle, keys=True) obj = list(actual.keys())[0] self.assertEqual(j.name, obj.name) self.assertTrue(obj is actual[obj][0]) self.assertTrue(obj is actual[obj][1]) def test_refs_in_objects(self): """Test that objects in lists are referenced correctly""" a = Thing('a') b = Thing('b') pickle = jsonpickle.encode([a, b, b]) actual = jsonpickle.decode(pickle) self.assertNotEqual(actual[0], actual[1]) self.assertEqual(actual[1], actual[2]) self.assertTrue(actual[1] is actual[2]) def test_refs_recursive(self): """Test that complicated recursive refs work""" a = Thing('a') a.self_list = [Thing('0'), Thing('1'), Thing('2')] a.first = a.self_list[0] a.stuff = {a.first: a.first} a.morestuff = {a.self_list[1]: a.stuff} pickle = jsonpickle.encode(a, keys=True) b = jsonpickle.decode(pickle, keys=True) item = b.self_list[0] self.assertEqual(b.first, item) self.assertEqual(b.stuff[b.first], item) self.assertEqual(b.morestuff[b.self_list[1]][b.first], item) def test_load_backend(self): """Test that we can call jsonpickle.load_backend() """ if PY32: return self.skip('no simplejson for python 3.2') jsonpickle.load_backend('simplejson', 'dumps', 'loads', ValueError) self.assertTrue(True) def test_set_preferred_backend_allows_magic(self): """Tests that we can use the pluggable backends magically """ backend = 'os.path' jsonpickle.load_backend(backend, 'split', 'join', AttributeError) jsonpickle.set_preferred_backend(backend) slash_hello, world = jsonpickle.encode('/hello/world') jsonpickle.remove_backend(backend) self.assertEqual(slash_hello, '/hello') self.assertEqual(world, 'world') def test_load_backend_submodule(self): """Test that we can load a submodule as a backend """ jsonpickle.load_backend('os.path', 'split', 'join', AttributeError) self.assertTrue('os.path' in jsonpickle.json._backend_names and 'os.path' in jsonpickle.json._encoders and 'os.path' in jsonpickle.json._decoders and 'os.path' in jsonpickle.json._encoder_options and 'os.path' in jsonpickle.json._decoder_exceptions) def _backend_is_partially_loaded(self, backend): """Return True if the specified backend is incomplete""" return (backend in jsonpickle.json._backend_names or backend in jsonpickle.json._encoders or backend in jsonpickle.json._decoders or backend in jsonpickle.json._encoder_options or backend in jsonpickle.json._decoder_exceptions) def test_load_backend_handles_bad_encode(self): """Test that we ignore bad encoders""" load_backend = jsonpickle.load_backend self.assertFalse(load_backend('os.path', 'bad!', 'split', AttributeError)) self.failIf(self._backend_is_partially_loaded('os.path')) def test_load_backend_raises_on_bad_decode(self): """Test that we ignore bad decoders""" load_backend = jsonpickle.load_backend self.assertFalse(load_backend('os.path', 'join', 'bad!', AttributeError)) self.failIf(self._backend_is_partially_loaded('os.path')) def test_load_backend_handles_bad_loads_exc(self): """Test that we ignore bad decoder exceptions""" load_backend = jsonpickle.load_backend self.assertFalse(load_backend('os.path', 'join', 'split', 'bad!')) self.failIf(self._backend_is_partially_loaded('os.path')) def test_list_item_reference(self): thing = Thing('parent') thing.child = Thing('child') thing.child.refs = [thing] encoded = jsonpickle.encode(thing) decoded = jsonpickle.decode(encoded) self.assertEqual(id(decoded.child.refs[0]), id(decoded)) def test_reference_to_list(self): thing = Thing('parent') thing.a = [1] thing.b = thing.a thing.b.append(thing.a) thing.b.append([thing.a]) encoded = jsonpickle.encode(thing) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.a[0], 1) self.assertEqual(decoded.b[0], 1) self.assertEqual(id(decoded.a), id(decoded.b)) self.assertEqual(id(decoded.a), id(decoded.a[1])) self.assertEqual(id(decoded.a), id(decoded.a[2][0])) def test_make_refs_disabled_list(self): obj_a = Thing('foo') obj_b = Thing('bar') coll = [obj_a, obj_b, obj_b] encoded = jsonpickle.encode(coll, make_refs=False) decoded = jsonpickle.decode(encoded) self.assertEqual(len(decoded), 3) self.assertTrue(decoded[0] is not decoded[1]) self.assertTrue(decoded[1] is not decoded[2]) def test_make_refs_disabled_reference_to_list(self): thing = Thing('parent') thing.a = [1] thing.b = thing.a thing.b.append(thing.a) thing.b.append([thing.a]) encoded = jsonpickle.encode(thing, make_refs=False) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.a[0], 1) self.assertEqual(decoded.b[0:3], '[1,') self.assertEqual(decoded.a[1][0:3], '[1,') self.assertEqual(decoded.a[2][0][0:3], '[1,') class PicklableNamedTuple(object): """ A picklable namedtuple wrapper, to demonstrate the need for protocol 2 compatibility. Yes, this is contrived in its use of new, but it demonstrates the issue. """ def __new__(cls, propnames, vals): # it's necessary to use the correct class name for class resolution # classes that fake their own names may never be unpicklable ntuple = collections.namedtuple(cls.__name__, propnames) ntuple.__getnewargs__ = (lambda self: (propnames, vals)) instance = ntuple.__new__(ntuple, *vals) return instance class PicklableNamedTupleEx(object): """ A picklable namedtuple wrapper, to demonstrate the need for protocol 4 compatibility. Yes, this is contrived in its use of new, but it demonstrates the issue. """ def __getnewargs__(self): raise NotImplementedError("This class needs __getnewargs_ex__") def __new__(cls, newargs=__getnewargs__, **kwargs): # it's necessary to use the correct class name for class resolution # classes that fake their own names may never be unpicklable ntuple = collections.namedtuple(cls.__name__, sorted(kwargs.keys())) ntuple.__getnewargs_ex__ = (lambda self: ((), kwargs)) ntuple.__getnewargs__ = newargs instance = ntuple.__new__(ntuple, *[b for a, b in sorted(kwargs.items())]) return instance class PickleProtocol2Thing(object): def __init__(self, *args): self.args = args def __getnewargs__(self): return self.args def __eq__(self, other): """ Make PickleProtocol2Thing('slotmagic') == PickleProtocol2Thing('slotmagic') """ if self.__dict__ == other.__dict__ and dir(self) == dir(other): for prop in dir(self): selfprop = getattr(self, prop) if not callable(selfprop) and prop[0] != '_': if selfprop != getattr(other, prop): return False return True else: return False # these two instances are used below and in tests slotmagic = PickleProtocol2Thing('slotmagic') dictmagic = PickleProtocol2Thing('dictmagic') class PickleProtocol2GetState(PickleProtocol2Thing): def __new__(cls, *args): instance = super(PickleProtocol2GetState, cls).__new__(cls) instance.newargs = args return instance def __getstate__(self): return 'I am magic' class PickleProtocol2GetStateDict(PickleProtocol2Thing): def __getstate__(self): return {'magic': True} class PickleProtocol2GetStateSlots(PickleProtocol2Thing): def __getstate__(self): return (None, {'slotmagic': slotmagic}) class PickleProtocol2GetStateSlotsDict(PickleProtocol2Thing): def __getstate__(self): return ({'dictmagic': dictmagic}, {'slotmagic': slotmagic}) class PickleProtocol2GetSetState(PickleProtocol2GetState): def __setstate__(self, state): """ Contrived example, easy to test """ if state == "I am magic": self.magic = True else: self.magic = False class PickleProtocol2ChildThing(object): def __init__(self, child): self.child = child def __getnewargs__(self): return ([self.child],) class PickleProtocol2ReduceString(object): def __reduce__(self): return __name__+'.slotmagic' class PickleProtocol2ReduceExString(object): def __reduce__(self): assert False, "Should not be here" def __reduce_ex__(self, n): return __name__+'.slotmagic' class PickleProtocol2ReduceTuple(object): def __init__(self, argval, optional=None): self.argval = argval self.optional = optional def __reduce__(self): return (PickleProtocol2ReduceTuple, # callable ('yam', 1), # args None, # state iter([]), # listitems iter([]), # dictitems ) def protocol_2_reduce_tuple_func(*args): return PickleProtocol2ReduceTupleFunc(*args) class PickleProtocol2ReduceTupleFunc(object): def __init__(self, argval, optional=None): self.argval = argval self.optional = optional def __reduce__(self): return (protocol_2_reduce_tuple_func, # callable ('yam', 1), # args None, # state iter([]), # listitems iter([]), # dictitems ) def __newobj__(lol, fail): """ newobj is special-cased, such that it is not actually called """ class PickleProtocol2ReduceNewobj(PickleProtocol2ReduceTupleFunc): def __new__(cls, *args): inst = super(cls, cls).__new__(cls) inst.newargs = args return inst def __reduce__(self): return (__newobj__, # callable (PickleProtocol2ReduceNewobj, 'yam', 1), # args None, # state iter([]), # listitems iter([]), # dictitems ) class PickleProtocol2ReduceTupleState(PickleProtocol2ReduceTuple): def __reduce__(self): return (PickleProtocol2ReduceTuple, # callable ('yam', 1), # args {'foo': 1}, # state iter([]), # listitems iter([]), # dictitems ) class PickleProtocol2ReduceTupleSetState(PickleProtocol2ReduceTuple): def __setstate__(self, state): self.bar = state['foo'] def __reduce__(self): return (type(self), # callable ('yam', 1), # args {'foo': 1}, # state iter([]), # listitems iter([]), # dictitems ) class PickleProtocol2ReduceTupleStateSlots(object): __slots__ = ('argval', 'optional', 'foo') def __init__(self, argval, optional=None): self.argval = argval self.optional = optional def __reduce__(self): return (PickleProtocol2ReduceTuple, # callable ('yam', 1), # args {'foo': 1}, # state iter([]), # listitems iter([]), # dictitems ) class PickleProtocol2ReduceListitemsAppend(object): def __init__(self): self.inner = [] def __reduce__(self): return (PickleProtocol2ReduceListitemsAppend, # callable (), # args {}, # state iter(['foo', 'bar']), # listitems iter([]), # dictitems ) def append(self, item): self.inner.append(item) class PickleProtocol2ReduceListitemsExtend(object): def __init__(self): self.inner = [] def __reduce__(self): return (PickleProtocol2ReduceListitemsAppend, # callable (), # args {}, # state iter(['foo', 'bar']), # listitems iter([]), # dictitems ) def extend(self, items): self.inner.exend(items) class PickleProtocol2ReduceDictitems(object): def __init__(self): self.inner = {} def __reduce__(self): return (PickleProtocol2ReduceDictitems, # callable (), # args {}, # state [], # listitems iter(zip(['foo', 'bar'], ['foo', 'bar'])), # dictitems ) def __setitem__(self, k, v): return self.inner.__setitem__(k, v) class PickleProtocol2Classic: def __init__(self, foo): self.foo = foo class PickleProtocol2ClassicInitargs: def __init__(self, foo, bar=None): self.foo = foo if bar: self.bar = bar def __getinitargs__(self): return ('choo', 'choo') class PicklingProtocol4TestCase(unittest.TestCase): def test_pickle_newargs_ex(self): """ Ensure we can pickle and unpickle an object whose class needs arguments to __new__ and get back the same typle """ instance = PicklableNamedTupleEx(**{'a': 'b', 'n': 2}) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(instance, decoded) def test_validate_reconstruct_by_newargs_ex(self): """ Ensure that the exemplar tuple's __getnewargs_ex__ works This is necessary to know whether the breakage exists in jsonpickle or not """ instance = PicklableNamedTupleEx(**{'a': 'b', 'n': 2}) args, kwargs = instance.__getnewargs_ex__() newinstance = PicklableNamedTupleEx.__new__(PicklableNamedTupleEx, *args, **kwargs) self.assertEqual(instance, newinstance) def test_references(self): shared = Thing('shared') instance = PicklableNamedTupleEx(**{'a': shared, 'n': shared}) child = Thing('child') shared.child = child child.child = instance encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded[0], decoded[1]) self.assertTrue(decoded[0] is decoded[1]) self.assertTrue(decoded.a is decoded.n) self.assertEqual(decoded.a.name, 'shared') self.assertEqual(decoded.a.child.name, 'child') self.assertTrue(decoded.a.child.child is decoded) self.assertTrue(decoded.n.child.child is decoded) self.assertTrue(decoded.a.child is decoded.n.child) self.assertEqual(decoded.__class__.__name__, PicklableNamedTupleEx.__name__) # TODO the class itself looks just like the real class, but it's # actually a reconstruction; PicklableNamedTupleEx is not type(decoded). self.assertFalse(decoded.__class__ is PicklableNamedTupleEx) class PicklingProtocol2TestCase(SkippableTest): def test_classic_init_has_args(self): """ Test unpickling a classic instance whose init takes args, has no __getinitargs__ Because classic only exists under 2, skipped if PY3 """ if PY3: return self.skip('No classic classes in PY3') instance = PickleProtocol2Classic(3) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.foo, 3) def test_getinitargs(self): """ Test __getinitargs__ with classic instance Because classic only exists under 2, skipped if PY3 """ if PY3: return self.skip('No classic classes in PY3') instance = PickleProtocol2ClassicInitargs(3) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.bar, 'choo') def test_reduce_complex_num(self): instance = 5j encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded, instance) def test_reduce_complex_zero(self): instance = 0j encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded, instance) def test_reduce_dictitems(self): 'Test reduce with dictitems set (as a generator)' instance = PickleProtocol2ReduceDictitems() encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.inner, {'foo': 'foo', 'bar': 'bar'}) def test_reduce_listitems_extend(self): 'Test reduce with listitems set (as a generator), yielding single items' instance = PickleProtocol2ReduceListitemsExtend() encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.inner, ['foo', 'bar']) def test_reduce_listitems_append(self): 'Test reduce with listitems set (as a generator), yielding single items' instance = PickleProtocol2ReduceListitemsAppend() encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.inner, ['foo', 'bar']) def test_reduce_state_setstate(self): 'Test reduce with the optional state argument set, on an object with '\ 'a __setstate__' instance = PickleProtocol2ReduceTupleSetState(5) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.argval, 'yam') self.assertEqual(decoded.optional, 1) self.assertEqual(decoded.bar, 1) self.assertFalse(hasattr(decoded, 'foo')) def test_reduce_state_no_dict(self): 'Test reduce with the optional state argument set, on an object with '\ 'no __dict__, and no __setstate__' instance = PickleProtocol2ReduceTupleStateSlots(5) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.argval, 'yam') self.assertEqual(decoded.optional, 1) self.assertEqual(decoded.foo, 1) def test_reduce_state_dict(self): 'Test reduce with the optional state argument set, on an object with '\ 'a __dict__, and no __setstate__' instance = PickleProtocol2ReduceTupleState(5) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.argval, 'yam') self.assertEqual(decoded.optional, 1) self.assertEqual(decoded.foo, 1) def test_reduce_basic(self): """Test reduce with only callable and args""" instance = PickleProtocol2ReduceTuple(5) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.argval, 'yam') self.assertEqual(decoded.optional, 1) def test_reduce_basic_func(self): """Test reduce with only callable and args callable is a module-level function """ instance = PickleProtocol2ReduceTupleFunc(5) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.argval, 'yam') self.assertEqual(decoded.optional, 1) def test_reduce_newobj(self): """Test reduce with callable called __newobj__ ensures special-case behaviour """ instance = PickleProtocol2ReduceNewobj(5) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.newargs, ('yam', 1)) def test_reduce_iter(self): instance = iter('123') self.assertTrue(util.is_iterator(instance)) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(next(decoded), '1') self.assertEqual(next(decoded), '2') self.assertEqual(next(decoded), '3') def test_reduce_string(self): """ Ensure json pickle will accept the redirection to another object when __reduce__ returns a string """ instance = PickleProtocol2ReduceString() encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded, slotmagic) def test_reduce_ex_string(self): """ Ensure json pickle will accept the redirection to another object when __reduce_ex__ returns a string ALSO tests that __reduce_ex__ is called in preference to __reduce__ """ instance = PickleProtocol2ReduceExString() encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded, slotmagic) def test_pickle_newargs(self): """ Ensure we can pickle and unpickle an object whose class needs arguments to __new__ and get back the same typle """ instance = PicklableNamedTuple(('a', 'b'), (1, 2)) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(instance, decoded) def test_validate_reconstruct_by_newargs(self): """ Ensure that the exemplar tuple's __getnewargs__ works This is necessary to know whether the breakage exists in jsonpickle or not """ instance = PicklableNamedTuple(('a', 'b'), (1, 2)) newinstance = PicklableNamedTuple.__new__(PicklableNamedTuple, *(instance.__getnewargs__())) self.assertEqual(instance, newinstance) def test_getnewargs_priority(self): """ Ensure newargs are used before py/state when decoding (As per PEP 307, classes are not supposed to implement all three magic methods) """ instance = PickleProtocol2GetState('whatevs') encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.newargs, ('whatevs',)) def test_restore_dict_state(self): """ Ensure that if getstate returns a dict, and there is no custom __setstate__, the dict is used as a source of variables to restore """ instance = PickleProtocol2GetStateDict('whatevs') encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertTrue(decoded.magic) def test_restore_slots_state(self): """ Ensure that if getstate returns a 2-tuple with a dict in the second position, and there is no custom __setstate__, the dict is used as a source of variables to restore """ instance = PickleProtocol2GetStateSlots('whatevs') encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded.slotmagic.__dict__, slotmagic.__dict__) self.assertEqual(decoded.slotmagic, slotmagic) def test_restore_slots_dict_state(self): """ Ensure that if getstate returns a 2-tuple with a dict in both positions, and there is no custom __setstate__, the dicts are used as a source of variables to restore """ instance = PickleProtocol2GetStateSlotsDict('whatevs') encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(PickleProtocol2Thing('slotmagic'), PickleProtocol2Thing('slotmagic')) self.assertEqual(decoded.slotmagic.__dict__, slotmagic.__dict__) self.assertEqual(decoded.slotmagic, slotmagic) self.assertEqual(decoded.dictmagic, dictmagic) def test_setstate(self): """ Ensure output of getstate is passed to setstate """ instance = PickleProtocol2GetSetState('whatevs') encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertTrue(decoded.magic) def test_handles_nested_objects(self): child = PickleProtocol2Thing(None) instance = PickleProtocol2Thing(child, child) encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertEqual(PickleProtocol2Thing, decoded.__class__) self.assertEqual(PickleProtocol2Thing, decoded.args[0].__class__) self.assertEqual(PickleProtocol2Thing, decoded.args[1].__class__) self.assertTrue(decoded.args[0] is decoded.args[1]) def test_handles_cyclical_objects(self): child = Capture(None) instance = Capture(child, child) child.args = (instance,) # create a cycle # TODO we do not properly restore references inside of lists. # Change the above tuple into a list to show the breakage. encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) # Ensure the right objects were constructed self.assertEqual(Capture, decoded.__class__) self.assertEqual(Capture, decoded.args[0].__class__) self.assertEqual(Capture, decoded.args[1].__class__) self.assertEqual(Capture, decoded.args[0].args[0].__class__) self.assertEqual(Capture, decoded.args[1].args[0].__class__) # It's turtles all the way down self.assertEqual(Capture, decoded.args[0].args[0] .args[0].args[0] .args[0].args[0] .args[0].args[0] .args[0].args[0] .args[0].args[0] .args[0].args[0] .args[0].__class__) # Ensure that references are properly constructed self.assertTrue(decoded.args[0] is decoded.args[1]) self.assertTrue(decoded is decoded.args[0].args[0]) self.assertTrue(decoded is decoded.args[1].args[0]) self.assertTrue(decoded.args[0] is decoded.args[0].args[0].args[0]) self.assertTrue(decoded.args[0] is decoded.args[1].args[0].args[0]) def test_handles_cyclical_objects_in_lists(self): child = PickleProtocol2ChildThing(None) instance = PickleProtocol2ChildThing([child, child]) child.child = instance # create a cycle encoded = jsonpickle.encode(instance) decoded = jsonpickle.decode(encoded) self.assertTrue(decoded is decoded.child[0].child) self.assertTrue(decoded is decoded.child[1].child) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(JSONPickleTestCase)) suite.addTest(unittest.makeSuite(PicklingTestCase)) suite.addTest(unittest.makeSuite(PicklingProtocol2TestCase)) suite.addTest(unittest.makeSuite(PicklingProtocol4TestCase)) suite.addTest(doctest.DocTestSuite(jsonpickle)) suite.addTest(doctest.DocTestSuite(jsonpickle.pickler)) suite.addTest(doctest.DocTestSuite(jsonpickle.unpickler)) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/tests/bson_test.py0000664000175000017500000000441112502235440017601 0ustar daviddavid00000000000000"""Test serializing pymongo bson structures""" import datetime import pickle import unittest import jsonpickle from helper import SkippableTest bson = None class Object(object): def __init__(self, offset): self.offset = datetime.timedelta(offset) def __getinitargs__(self): return self.offset, class BSONTestCase(SkippableTest): def setUp(self): global bson try: bson = __import__('bson.tz_util') self.should_skip = False except ImportError: self.should_skip = True def test_FixedOffsetSerializable(self): if self.should_skip: return self.skip('bson is not installed') fo = bson.tz_util.FixedOffset(-60*5, 'EST') serialized = jsonpickle.dumps(fo) restored = jsonpickle.loads(serialized) self.assertEqual(vars(restored), vars(fo)) def test_timedelta(self): if self.should_skip: return self.skip('bson is not installed') td = datetime.timedelta(-1, 68400) serialized = jsonpickle.dumps(td) restored = jsonpickle.loads(serialized) self.assertEqual(restored, td) def test_stdlib_pickle(self): if self.should_skip: return self.skip('bson is not installed') fo = bson.tz_util.FixedOffset(-60*5, 'EST') serialized = pickle.dumps(fo) restored = pickle.loads(serialized) self.assertEqual(vars(restored), vars(fo)) def test_nested_objects(self): if self.should_skip: return self.skip('bson is not installed') o = Object(99) serialized = jsonpickle.dumps(o) restored = jsonpickle.loads(serialized) self.assertEqual(restored.offset, datetime.timedelta(99)) def test_datetime_with_fixed_offset(self): if self.should_skip: return self.skip('bson is not installed') fo = bson.tz_util.FixedOffset(-60*5, 'EST') dt = datetime.datetime.now().replace(tzinfo=fo) serialized = jsonpickle.dumps(dt) restored = jsonpickle.loads(serialized) self.assertEqual(restored, dt) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(BSONTestCase, 'test')) return suite if __name__ == '__main__': unittest.main() jsonpickle-0.9.5/tests/numpy_test.py0000644000175000017500000002452613035346227020027 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- from __future__ import absolute_import, division, unicode_literals import unittest import datetime import warnings import jsonpickle from jsonpickle.compat import PY2, PY3, PY_MINOR from helper import SkippableTest try: import numpy as np import numpy.testing as npt from numpy.compat import asbytes from numpy.testing import assert_equal except ImportError: np = None class NumpyTestCase(SkippableTest): def setUp(self): if np is None: self.should_skip = True return self.should_skip = False import jsonpickle.ext.numpy jsonpickle.ext.numpy.register_handlers() def tearDown(self): if self.should_skip: return import jsonpickle.ext.numpy jsonpickle.ext.numpy.unregister_handlers() def roundtrip(self, obj): return jsonpickle.decode(jsonpickle.encode(obj)) def test_dtype_roundtrip(self): if self.should_skip: return self.skip('numpy is not importable') dtypes = [ np.int, np.float, np.complex, np.int32, np.str, np.object, np.unicode, np.dtype('f4,i4,f2,i1'), np.dtype(('f4', 'i4'), ('f2', 'i1')), np.dtype('1i4', align=True), np.dtype('M8[7D]'), np.dtype({'names': ['f0', 'f1', 'f2'], 'formats': ['f4', (64, 64)), (1,)), ('rtile', '>f4', (64, 36))], (3,)), ('bottom', [('bleft', ('>f4', (8, 64)), (1,)), ('bright', '>f4', (8, 36))])]), ]) for dtype in dtypes: self.assertEqual(self.roundtrip(dtype), dtype) def test_generic_roundtrip(self): if self.should_skip: return self.skip('numpy is not importable') values = [ np.int_(1), np.int32(-2), np.float_(2.5), np.nan, -np.inf, np.inf, np.datetime64('2014-01-01'), np.str_('foo'), np.unicode_('bar'), np.object_({'a': 'b'}), np.complex_(1 - 2j) ] for value in values: decoded = self.roundtrip(value) assert_equal(decoded, value) self.assertTrue(isinstance(decoded, type(value))) def test_ndarray_roundtrip(self): if self.should_skip: return self.skip('numpy is not importable') arrays = [ np.random.random((10, 20)), np.array([[True, False, True]]), np.array(['foo', 'bar']), np.array(['baz'.encode('utf-8')]), np.array(['2010', 'NaT', '2030']).astype('M'), np.rec.array(asbytes('abcdefg') * 100, formats='i2,a3,i4', shape=3), np.rec.array([(1, 11, 'a'), (2, 22, 'b'), (3, 33, 'c'), (4, 44, 'd'), (5, 55, 'ex'), (6, 66, 'f'), (7, 77, 'g')], formats='u1,f4,a1'), np.array(['1960-03-12', datetime.date(1960, 3, 12)], dtype='M8[D]'), np.array([0, 1, -1, np.inf, -np.inf, np.nan], dtype='f2'), ] if not PY2: arrays.extend([ np.rec.array([('NGC1001', 11), ('NGC1002', 1.), ('NGC1003', 1.)], dtype=[('target', 'S20'), ('V_mag', 'f4')]) ]) for array in arrays: decoded = self.roundtrip(array) assert_equal(decoded, array) self.assertEqual(decoded.dtype, array.dtype) def test_shape(self): """test that shapes containing zeros, which cannot be represented as nested lists, are deserialized correctly""" a = np.eye(3)[3:] _a = self.roundtrip(a) npt.assert_array_equal(a, _a) def test_accuracy(self): """test if the string representation maintains accuracy""" rand = np.random.randn(3, 3) _rand = self.roundtrip(rand) npt.assert_array_equal(rand, _rand) def test_b64(self): """test that binary encoding works""" a = np.random.rand(10, 10) # array of substantial size is stored as b64 _a = self.roundtrip(a) npt.assert_array_equal(a, _a) def test_views(self): """Test that views are maintained under serialization""" rng = np.arange(20) # a range of an array view = rng[10:] # a view referencing a portion of an array data = [rng, view] _data = self.roundtrip(data) _data[0][15] = -1 self.assertEqual(_data[1][5], -1) def test_strides(self): """test that cases with non-standard strides and offsets work correctly""" arr = np.eye(3) view = arr[1:, 1:] self.assertTrue(view.base is arr) data = [arr, view] _data = self.roundtrip(data) # test that the deserialized arrays indeed view the same memory _arr, _view = _data _arr[1, 2] = -1 self.assertEqual(_view[0, 1], -1) self.assertTrue(_view.base is _arr) def test_weird_arrays(self): """test that we disallow serialization of references to arrays that do not effectively own their memory""" a = np.arange(9) b = a[5:] a.strides = 1 # this is kinda fishy; a has overlapping memory, _a does not if PY2 or (PY3 and PY_MINOR <= 3): warn_count = 0 else: warn_count = 1 with warnings.catch_warnings(record=True) as w: _a = self.roundtrip(a) self.assertEqual(len(w), warn_count) npt.assert_array_equal(a, _a) # this also requires a deepcopy to work with warnings.catch_warnings(record=True) as w: _a, _b = self.roundtrip([a, b]) self.assertEqual(len(w), warn_count) npt.assert_array_equal(a, _a) npt.assert_array_equal(b, _b) def test_transpose(self): """test handling of non-c-contiguous memory layout""" # simple case; view a c-contiguous array a = np.arange(9).reshape(3, 3) b = a[1:, 1:] self.assertTrue(b.base is a.base) _a, _b = self.roundtrip([a, b]) self.assertTrue(_b.base is _a.base) npt.assert_array_equal(a, _a) npt.assert_array_equal(b, _b) # a and b both view the same contiguous array a = np.arange(9).reshape(3, 3).T b = a[1:, 1:] self.assertTrue(b.base is a.base) _a, _b = self.roundtrip([a, b]) self.assertTrue(_b.base is _a.base) npt.assert_array_equal(a, _a) npt.assert_array_equal(b, _b) # view an f-contiguous array a = a.copy() a.strides = a.strides[::-1] b = a[1:, 1:] self.assertTrue(b.base is a) _a, _b = self.roundtrip([a, b]) self.assertTrue(_b.base is _a) npt.assert_array_equal(a, _a) npt.assert_array_equal(b, _b) # now a.data.contiguous is False; we have to make a deepcopy to make this work # note that this is a pretty contrived example though! a = np.arange(8).reshape(2, 2, 2).copy() a.strides = a.strides[0], a.strides[2], a.strides[1] b = a[1:, 1:] self.assertTrue(b.base is a) if PY2 or (PY3 and PY_MINOR <= 3): warn_count = 0 else: warn_count = 1 with warnings.catch_warnings(record=True) as w: _a, _b = self.roundtrip([a, b]) self.assertEqual(len(w), warn_count) npt.assert_array_equal(a, _a) npt.assert_array_equal(b, _b) def test_fortran_base(self): """Test a base array in fortran order""" a = np.asfortranarray(np.arange(100).reshape((10, 10))) _a = self.roundtrip(a) npt.assert_array_equal(a, _a) def test_buffer(self): """test behavior with memoryviews which are not ndarrays""" bstring = 'abcdefgh'.encode('utf-8') a = np.frombuffer(bstring, dtype=np.byte) if PY2 or (PY3 and PY_MINOR <= 3): warn_count = 0 else: warn_count = 1 with warnings.catch_warnings(record=True) as w: _a = self.roundtrip(a) npt.assert_array_equal(a, _a) self.assertEqual(len(w), warn_count) def test_as_strided(self): """test object with array interface which isnt an ndarray, like the result of as_strided""" a = np.arange(10) b = np.lib.stride_tricks.as_strided(a, shape=(5,), strides=(a.dtype.itemsize * 2,)) data = [a, b] with warnings.catch_warnings(record=True) as w: # as_strided returns a DummyArray object, which we can not currently serialize correctly # FIXME: would be neat to add support for all objects implementing the __array_interface__ _data = self.roundtrip(data) self.assertEqual(len(w), 1) # as we were warned, deserialized result is no longer a view _data[0][0] = -1 self.assertEqual(_data[1][0], 0) def test_immutable(self): """test that immutability flag is copied correctly""" a = np.arange(10) a.flags.writeable = False _a = self.roundtrip(a) try: _a[0] = 0 self.assertTrue(False, 'item assignment must raise') except ValueError: self.assertTrue(True) def test_byteorder(self): """test that byteorder is properly conserved across views, for text and binary encoding""" # small arr is stored as text a = np.arange(10).newbyteorder() b = a[:].newbyteorder() _a, _b = self.roundtrip([a, b]) npt.assert_array_equal(a, _a) npt.assert_array_equal(b, _b) # bigger arr is stored as binary a = np.arange(100).newbyteorder() b = a[:].newbyteorder() _a, _b = self.roundtrip([a, b]) npt.assert_array_equal(a, _a) npt.assert_array_equal(b, _b) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(NumpyTestCase, 'test')) return suite if __name__ == '__main__': unittest.main() jsonpickle-0.9.5/tests/document_test.py0000664000175000017500000000506312502235440020462 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- import unittest import jsonpickle class Node(object): def __init__(self, name): self._name = name self._children = [] self._parent = None def add_child(self, child, index=-1): if index == -1: index = len(self._children) self._children.insert(index, child) child._parent = self class Document(Node): def __init__(self, name): Node.__init__(self, name) def __str__(self): ret_str = 'Document "%s"\n' % self._name for c in self._children: ret_str += repr(c) return ret_str def __repr__(self): return str(self) class Question(Node): def __init__(self, name): Node.__init__(self, name) def __str__(self): return ('Question "%s", parent: "%s"\n' % (self._name, self._parent._name)) def __repr__(self): return self.__str__() class Section(Node): def __init__(self, name): Node.__init__(self, name) def __str__(self): ret_str = ('Section "%s", parent: "%s"\n' % (self._name, self._parent._name)) for c in self._children: ret_str += repr(c) return ret_str def __repr__(self): return self.__str__() class DocumentTestCase(unittest.TestCase): def test_cyclical(self): """Test that we can pickle cyclical data structure This test is ensures that we can reference objects which first appear within a list (in other words, not a top-level object or attribute). Later children will reference that object through its "_parent" field. This makes sure that we handle this case correctly. """ document = Document('My Document') section1 = Section('Section 1') section2 = Section('Section 2') question1 = Question('Question 1') question2 = Question('Question 2') question3 = Question('Question 3') question4 = Question('Question 4') document.add_child(section1) document.add_child(section2) section1.add_child(question1) section1.add_child(question2) section2.add_child(question3) section2.add_child(question4) pickled = jsonpickle.encode(document) unpickled = jsonpickle.decode(pickled) self.assertEqual(str(document), str(unpickled)) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(DocumentTestCase)) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/tests/object_test.py0000644000175000017500000006674313075310375020133 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- import base64 import collections import decimal import re import sys import unittest import datetime import jsonpickle from jsonpickle import handlers from jsonpickle import tags from jsonpickle.compat import queue, set, unicode, bytes, PY2, PY_MINOR import jsonpickle.util as util from helper import SkippableTest class Thing(object): def __init__(self, name): self.name = name self.child = None class DictSubclass(dict): name = 'Test' class ListSubclass(list): pass class BrokenReprThing(Thing): def __repr__(self): raise Exception('%s has a broken repr' % self.name) def __str__(self): return '' % self.name class GetstateDict(dict): def __init__(self, name, **kwargs): dict.__init__(self, **kwargs) self.name = name self.active = False def __getstate__(self): return (self.name, dict(self.items())) def __setstate__(self, state): self.name, vals = state self.update(vals) self.active = True class GetstateOnly(object): def __init__(self, a=1, b=2): self.a = a self.b = b def __getstate__(self): return [self.a, self.b] class GetstateReturnsList(object): def __init__(self, x, y): self.x = x self.y = y def __getstate__(self): return [self.x, self.y] def __setstate__(self, state): self.x, self.y = state[0], state[1] class ListSubclassWithInit(list): def __init__(self, attr): self.attr = attr super(ListSubclassWithInit, self).__init__() NamedTuple = collections.namedtuple('NamedTuple', 'a, b, c') class ObjWithJsonPickleRepr(object): def __init__(self): self.data = {'a': self} def __repr__(self): return jsonpickle.encode(self) class OldStyleClass: pass class SetSubclass(set): pass class ThingWithFunctionRefs(object): def __init__(self): self.fn = func def func(x): return x class ThingWithQueue(object): def __init__(self): self.child_1 = queue.Queue() self.child_2 = queue.Queue() self.childref_1 = self.child_1 self.childref_2 = self.child_2 class ThingWithSlots(object): __slots__ = ('a', 'b') def __init__(self, a, b): self.a = a self.b = b class ThingWithInheritedSlots(ThingWithSlots): __slots__ = ('c',) def __init__(self, a, b, c): ThingWithSlots.__init__(self, a, b) self.c = c class ThingWithIterableSlots(object): __slots__ = iter('ab') def __init__(self, a, b): self.a = a self.b = b class ThingWithStringSlots(object): __slots__ = 'ab' def __init__(self, a, b): self.ab = a + b class ThingWithSelfAsDefaultFactory(collections.defaultdict): """defaultdict subclass that uses itself as its default factory""" def __init__(self): self.default_factory = self def __call__(self): return self.__class__() class ThingWithClassAsDefaultFactory(collections.defaultdict): """defaultdict subclass that uses its class as its default factory""" def __init__(self): self.default_factory = self.__class__ def __call__(self): return self.__class__() try: import enum class IntEnumTest(enum.IntEnum): X = 1 Y = 2 class StringEnumTest(enum.Enum): A = 'a' B = 'b' class SubEnum(enum.Enum): a = 1 b = 2 class EnumClass(object): def __init__(self): self.enum_a = SubEnum.a self.enum_b = SubEnum.b except ImportError: IntEnumTest = None StringEnumTest = None EnumClass = None class ThingWithTimedeltaAttribute(object): def __init__(self, offset): self.offset = datetime.timedelta(offset) def __getinitargs__(self): return self.offset, class AdvancedObjectsTestCase(SkippableTest): def setUp(self): self.pickler = jsonpickle.pickler.Pickler() self.unpickler = jsonpickle.unpickler.Unpickler() def tearDown(self): self.pickler.reset() self.unpickler.reset() def test_defaultdict_roundtrip(self): """Make sure we can handle collections.defaultdict(list)""" # setup defaultdict = collections.defaultdict defdict = defaultdict(list) defdict['a'] = 1 defdict['b'].append(2) defdict['c'] = defaultdict(dict) # jsonpickle work your magic encoded = jsonpickle.encode(defdict) newdefdict = jsonpickle.decode(encoded) # jsonpickle never fails self.assertEqual(newdefdict['a'], 1) self.assertEqual(newdefdict['b'], [2]) self.assertEqual(type(newdefdict['c']), defaultdict) self.assertEqual(defdict.default_factory, list) self.assertEqual(newdefdict.default_factory, list) def test_defaultdict_roundtrip_simple_lambda(self): """Make sure we can handle defaultdict(lambda: defaultdict(int))""" # setup a sparse collections.defaultdict with simple lambdas defaultdict = collections.defaultdict defdict = defaultdict(lambda: defaultdict(int)) defdict[0] = 'zero' defdict[1] = defaultdict(lambda: defaultdict(dict)) defdict[1][0] = 'zero' # roundtrip encoded = jsonpickle.encode(defdict, keys=True) newdefdict = jsonpickle.decode(encoded, keys=True) self.assertEqual(newdefdict[0], 'zero') self.assertEqual(type(newdefdict[1]), defaultdict) self.assertEqual(newdefdict[1][0], 'zero') self.assertEqual(newdefdict[1][1], {}) # inner defaultdict self.assertEqual(newdefdict[2][0], 0) # outer defaultdict self.assertEqual(type(newdefdict[3]), defaultdict) # outer-most defaultdict self.assertEqual(newdefdict[3].default_factory, int) def test_defaultdict_subclass_with_self_as_default_factory(self): cls = ThingWithSelfAsDefaultFactory tree = cls() newtree = self._test_defaultdict_tree(tree, cls) self.assertEqual(type(newtree['A'].default_factory), cls) self.assertTrue(newtree.default_factory is newtree) self.assertTrue(newtree['A'].default_factory is newtree['A']) self.assertTrue(newtree['Z'].default_factory is newtree['Z']) def test_defaultdict_subclass_with_class_as_default_factory(self): cls = ThingWithClassAsDefaultFactory tree = cls() newtree = self._test_defaultdict_tree(tree, cls) self.assertTrue(newtree.default_factory is cls) self.assertTrue(newtree['A'].default_factory is cls) self.assertTrue(newtree['Z'].default_factory is cls) def _test_defaultdict_tree(self, tree, cls): tree['A']['B'] = 1 tree['A']['C'] = 2 # roundtrip encoded = jsonpickle.encode(tree) newtree = jsonpickle.decode(encoded) # make sure we didn't lose anything self.assertEqual(type(newtree), cls) self.assertEqual(type(newtree['A']), cls) self.assertEqual(newtree['A']['B'], 1) self.assertEqual(newtree['A']['C'], 2) # ensure that the resulting default_factory is callable and creates # a new instance of cls. self.assertEqual(type(newtree['A'].default_factory()), cls) # we've never seen 'D' before so the reconstructed defaultdict tree # should create an instance of cls. self.assertEqual(type(newtree['A']['D']), cls) # ensure that proxies do not escape into user code self.assertNotEqual(type(newtree.default_factory), jsonpickle.unpickler._Proxy) self.assertNotEqual(type(newtree['A'].default_factory), jsonpickle.unpickler._Proxy) self.assertNotEqual(type(newtree['A']['Z'].default_factory), jsonpickle.unpickler._Proxy) return newtree def test_deque_roundtrip(self): """Make sure we can handle collections.deque""" py26 = PY2 and PY_MINOR < 7 if py26: old_deque = collections.deque([0, 1, 2]) else: old_deque = collections.deque([0, 1, 2], maxlen=5) encoded = jsonpickle.encode(old_deque) new_deque = jsonpickle.decode(encoded) self.assertNotEqual(encoded, 'nil') self.assertEqual(old_deque[0], 0) self.assertEqual(new_deque[0], 0) self.assertEqual(old_deque[1], 1) self.assertEqual(new_deque[1], 1) self.assertEqual(old_deque[2], 2) self.assertEqual(new_deque[2], 2) if not py26: self.assertEqual(old_deque.maxlen, 5) self.assertEqual(new_deque.maxlen, 5) def test_namedtuple_roundtrip(self): old_nt = NamedTuple(0, 1, 2) encoded = jsonpickle.encode(old_nt) new_nt = jsonpickle.decode(encoded) self.assertEqual(type(old_nt), type(new_nt)) self.assertTrue(old_nt is not new_nt) self.assertEqual(old_nt.a, new_nt.a) self.assertEqual(old_nt.b, new_nt.b) self.assertEqual(old_nt.c, new_nt.c) self.assertEqual(old_nt[0], new_nt[0]) self.assertEqual(old_nt[1], new_nt[1]) self.assertEqual(old_nt[2], new_nt[2]) def test_counter_roundtrip(self): if sys.version_info < (2, 7): # collections.Counter was introduced in Python 2.7 return counter = collections.Counter({1: 2}) encoded = jsonpickle.encode(counter) decoded = jsonpickle.decode(encoded) self.assertTrue(type(decoded) is collections.Counter) # the integer key becomes a string when keys=False self.assertEqual(decoded.get('1'), 2) def test_counter_roundtrip_with_keys(self): if sys.version_info < (2, 7): # collections.Counter was introduced in Python 2.7 return counter = collections.Counter({1: 2}) encoded = jsonpickle.encode(counter, keys=True) decoded = jsonpickle.decode(encoded, keys=True) self.assertTrue(type(decoded) is collections.Counter) self.assertEqual(decoded.get(1), 2) def test_list_with_fd(self): fd = open(__file__, 'r') fd.close() obj = [fd] jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertEqual([None], newobj) def test_thing_with_fd(self): fd = open(__file__, 'r') fd.close() obj = Thing(fd) jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertEqual(None, newobj.name) def test_dict_with_fd(self): fd = open(__file__, 'r') fd.close() obj = {'fd': fd} jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertEqual(None, newobj['fd']) def test_thing_with_lamda(self): obj = Thing(lambda: True) jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertFalse(hasattr(newobj, 'name')) def test_newstyleslots(self): obj = ThingWithSlots(True, False) jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertTrue(newobj.a) self.assertFalse(newobj.b) def test_newstyleslots_inherited(self): obj = ThingWithInheritedSlots(True, False, None) jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertTrue(newobj.a) self.assertFalse(newobj.b) self.assertEqual(newobj.c, None) def test_newstyleslots_inherited_deleted_attr(self): obj = ThingWithInheritedSlots(True, False, None) del obj.c jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertTrue(newobj.a) self.assertFalse(newobj.b) self.assertFalse(hasattr(newobj, 'c')) def test_newstyleslots_with_children(self): obj = ThingWithSlots(Thing('a'), Thing('b')) jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertEqual(newobj.a.name, 'a') self.assertEqual(newobj.b.name, 'b') def test_newstyleslots_with_children_inherited(self): obj = ThingWithInheritedSlots(Thing('a'), Thing('b'), Thing('c')) jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertEqual(newobj.a.name, 'a') self.assertEqual(newobj.b.name, 'b') self.assertEqual(newobj.c.name, 'c') def test_newstyleslots_iterable(self): obj = ThingWithIterableSlots('alpha', 'bravo') jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertEqual(newobj.a, 'alpha') self.assertEqual(newobj.b, 'bravo') def test_newstyleslots_string_slot(self): obj = ThingWithStringSlots('a', 'b') jsonstr = jsonpickle.encode(obj) newobj = jsonpickle.decode(jsonstr) self.assertEqual(newobj.ab, 'ab') def test_list_subclass(self): obj = ListSubclass() obj.extend([1, 2, 3]) flattened = self.pickler.flatten(obj) self.assertTrue(tags.OBJECT in flattened) self.assertTrue(tags.SEQ in flattened) self.assertEqual(len(flattened[tags.SEQ]), 3) for v in obj: self.assertTrue(v in flattened[tags.SEQ]) restored = self.unpickler.restore(flattened) self.assertEqual(type(restored), ListSubclass) self.assertEqual(restored, obj) def test_list_subclass_with_init(self): obj = ListSubclassWithInit('foo') self.assertEqual(obj.attr, 'foo') flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) self.assertEqual(type(inflated), ListSubclassWithInit) def test_list_subclass_with_data(self): obj = ListSubclass() obj.extend([1, 2, 3]) data = SetSubclass([1, 2, 3]) obj.data = data flattened = self.pickler.flatten(obj) restored = self.unpickler.restore(flattened) self.assertEqual(restored, obj) self.assertEqual(type(restored.data), SetSubclass) self.assertEqual(restored.data, data) def test_set_subclass(self): obj = SetSubclass([1, 2, 3]) flattened = self.pickler.flatten(obj) self.assertTrue(tags.OBJECT in flattened) self.assertTrue(tags.SEQ in flattened) self.assertEqual(len(flattened[tags.SEQ]), 3) for v in obj: self.assertTrue(v in flattened[tags.SEQ]) restored = self.unpickler.restore(flattened) self.assertEqual(type(restored), SetSubclass) self.assertEqual(restored, obj) def test_set_subclass_with_data(self): obj = SetSubclass([1, 2, 3]) data = ListSubclass() data.extend([1, 2, 3]) obj.data = data flattened = self.pickler.flatten(obj) restored = self.unpickler.restore(flattened) self.assertEqual(restored.data.__class__, ListSubclass) self.assertEqual(restored.data, data) def test_decimal(self): obj = decimal.Decimal(1) flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) self.assertEqual(type(inflated), decimal.Decimal) def test_repr_using_jsonpickle(self): thing = ObjWithJsonPickleRepr() thing.child = ObjWithJsonPickleRepr() thing.child.parent = thing encoded = jsonpickle.encode(thing) decoded = jsonpickle.decode(encoded) self.assertEqual(id(decoded), id(decoded.child.parent)) def test_broken_repr_dict_key(self): """Tests that we can pickle dictionaries with keys that have broken __repr__ implementations. """ br = BrokenReprThing('test') obj = {br: True} pickler = jsonpickle.pickler.Pickler() flattened = pickler.flatten(obj) self.assertTrue('' in flattened) self.assertTrue(flattened['']) def test_ordered_dict_reduces(self): if sys.version_info < (2, 7): return d = collections.OrderedDict() d.update(c=3) d.update(a=1) d.update(b=2) has_reduce, has_reduce_ex = util.has_reduce(d) self.assertTrue(util.is_reducible(d)) self.assertTrue(has_reduce or has_reduce_ex) def test_ordered_dict(self): if sys.version_info < (2, 7): return d = collections.OrderedDict() d.update(c=3) d.update(a=1) d.update(b=2) encoded = jsonpickle.encode(d) decoded = jsonpickle.decode(encoded) self.assertEqual(d, decoded) def test_ordered_dict_int_keys(self): if sys.version_info < (2, 7): return d = { 1: collections.OrderedDict([(2, -2), (3, -3)]), 4: collections.OrderedDict([(5, -5), (6, -6)]), } encoded = jsonpickle.encode(d, keys=True) decoded = jsonpickle.decode(encoded, keys=True) self.assertEqual(collections.OrderedDict, type(decoded[1])) self.assertEqual(collections.OrderedDict, type(decoded[4])) self.assertEqual(-2, decoded[1][2]) self.assertEqual(-3, decoded[1][3]) self.assertEqual(-5, decoded[4][5]) self.assertEqual(-6, decoded[4][6]) self.assertEqual(d, decoded) def test_posix_stat_result(self): try: import posix except ImportError: return expect = posix.stat(__file__) encoded = jsonpickle.encode(expect) actual = jsonpickle.decode(encoded) self.assertEqual(expect, actual) def test_oldstyleclass(self): obj = OldStyleClass() obj.value = 1234 flattened = self.pickler.flatten(obj) self.assertEqual(1234, flattened['value']) inflated = self.unpickler.restore(flattened) self.assertEqual(1234, inflated.value) def test_dictsubclass(self): obj = DictSubclass() obj['key1'] = 1 flattened = self.pickler.flatten(obj) self.assertEqual({'key1': 1, tags.OBJECT: 'object_test.DictSubclass'}, flattened) self.assertEqual(flattened[tags.OBJECT], 'object_test.DictSubclass') inflated = self.unpickler.restore(flattened) self.assertEqual(1, inflated['key1']) self.assertEqual(inflated.name, 'Test') def test_dictsubclass_notunpickable(self): self.pickler.unpicklable = False obj = DictSubclass() obj['key1'] = 1 flattened = self.pickler.flatten(obj) self.assertEqual(1, flattened['key1']) self.assertFalse(tags.OBJECT in flattened) inflated = self.unpickler.restore(flattened) self.assertEqual(1, inflated['key1']) def test_getstate_dict_subclass_structure(self): obj = GetstateDict('test') obj['key1'] = 1 flattened = self.pickler.flatten(obj) self.assertTrue(tags.OBJECT in flattened) self.assertEqual('object_test.GetstateDict', flattened[tags.OBJECT]) self.assertTrue(tags.STATE in flattened) self.assertTrue(tags.TUPLE in flattened[tags.STATE]) self.assertEqual(['test', {'key1': 1}], flattened[tags.STATE][tags.TUPLE]) def test_getstate_dict_subclass_roundtrip_simple(self): obj = GetstateDict('test') obj['key1'] = 1 flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) self.assertEqual(1, inflated['key1']) self.assertEqual(inflated.name, 'test') def test_getstate_dict_subclass_roundtrip_cyclical(self): obj = GetstateDict('test') obj['key1'] = 1 # The "name" field of obj2 points to obj (reference) obj2 = GetstateDict(obj) # The "obj2" key in obj points to obj2 (cyclical reference) obj['obj2'] = obj2 flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) # The dict must be preserved self.assertEqual(1, inflated['key1']) # __getstate__/__setstate__ must have been run self.assertEqual(inflated.name, 'test') self.assertEqual(inflated.active, True) self.assertEqual(inflated['obj2'].active, True) # The reference must be preserved self.assertTrue(inflated is inflated['obj2'].name) def test_getstate_list_simple(self): obj = GetstateReturnsList(1, 2) flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) self.assertEqual(inflated.x, 1) self.assertEqual(inflated.y, 2) def test_getstate_list_inside_list(self): obj1 = GetstateReturnsList(1, 2) obj2 = GetstateReturnsList(3, 4) obj = [obj1, obj2] flattened = self.pickler.flatten(obj) inflated = self.unpickler.restore(flattened) self.assertEqual(inflated[0].x, 1) self.assertEqual(inflated[0].y, 2) self.assertEqual(inflated[1].x, 3) self.assertEqual(inflated[1].y, 4) def test_getstate_with_getstate_only(self): obj = GetstateOnly() a = obj.a = 'this object implements' b = obj.b = '__getstate__ but not __setstate__' expect = [a, b] flat = self.pickler.flatten(obj) actual = flat[tags.STATE] self.assertEqual(expect, actual) restored = self.unpickler.restore(flat) self.assertEqual(expect, restored) def test_thing_with_queue(self): obj = ThingWithQueue() flattened = self.pickler.flatten(obj) restored = self.unpickler.restore(flattened) self.assertEqual(type(restored.child_1), type(queue.Queue())) self.assertEqual(type(restored.child_2), type(queue.Queue())) # Check references self.assertTrue(restored.child_1 is restored.childref_1) self.assertTrue(restored.child_2 is restored.childref_2) def test_thing_with_func(self): obj = ThingWithFunctionRefs() obj.ref = obj flattened = self.pickler.flatten(obj) restored = self.unpickler.restore(flattened) self.assertTrue(restored.fn is obj.fn) expect = 'success' actual1 = restored.fn(expect) self.assertEqual(expect, actual1) self.assertTrue(restored is restored.ref) def test_thing_with_compiled_regex(self): rgx = re.compile(r'(.*)(cat)') obj = Thing(rgx) flattened = self.pickler.flatten(obj) restored = self.unpickler.restore(flattened) match = restored.name.match('fatcat') self.assertEqual('fat', match.group(1)) self.assertEqual('cat', match.group(2)) def test_base_object_roundrip(self): roundtrip = self.unpickler.restore(self.pickler.flatten(object())) self.assertEqual(type(roundtrip), object) def test_enum34(self): if IntEnumTest is None or StringEnumTest is None: return self.skip('enum34 module is not installed') restore = self.unpickler.restore flatten = self.pickler.flatten roundtrip = lambda obj: restore(flatten(obj)) self.assertTrue(roundtrip(IntEnumTest.X) is IntEnumTest.X) self.assertTrue(roundtrip(IntEnumTest) is IntEnumTest) self.assertTrue(roundtrip(StringEnumTest.A) is StringEnumTest.A) self.assertTrue(roundtrip(StringEnumTest) is StringEnumTest) def test_enum34_nested(self): if EnumClass is None: return self.skip('enum34 module is not installed') ec = EnumClass() encoded = jsonpickle.encode(ec) decoded = jsonpickle.decode(encoded) self.assertEqual(ec.enum_a, decoded.enum_a) self.assertEqual(ec.enum_b, decoded.enum_b) def test_enum_references(self): if IntEnumTest is None: return self.skip('enum34 module is not installed') a = IntEnumTest.X b = IntEnumTest.X enums_list = [a, b] encoded = jsonpickle.encode(enums_list) decoded = jsonpickle.decode(encoded) self.assertEqual(enums_list, decoded) def test_bytes_unicode(self): b1 = b'foo' b2 = b'foo\xff' u1 = unicode('foo') # unicode strings get encoded/decoded as is encoded = self.pickler.flatten(u1) self.assertTrue(encoded == u1) self.assertTrue(type(encoded) is unicode) decoded = self.unpickler.restore(encoded) self.assertTrue(decoded == u1) self.assertTrue(type(decoded) is unicode) # bytestrings are wrapped in PY3 but in PY2 we try to decode first encoded = self.pickler.flatten(b1) if PY2: self.assertEqual(encoded, u1) self.assertEqual(type(encoded), unicode) else: self.assertNotEqual(encoded, u1) b64ustr= base64.encodestring(b'foo').decode('utf-8') self.assertEqual({tags.B64: b64ustr}, encoded) self.assertEqual(type(encoded[tags.B64]), unicode) decoded = self.unpickler.restore(encoded) self.assertTrue(decoded == b1) if PY2: self.assertTrue(type(decoded) is unicode) else: self.assertTrue(type(decoded) is bytes) # bytestrings that we can't decode to UTF-8 will always be wrapped encoded = self.pickler.flatten(b2) self.assertNotEqual(encoded, b2) b64ustr= base64.encodestring(b'foo\xff').decode('utf-8') self.assertEqual({tags.B64: b64ustr}, encoded) self.assertEqual(type(encoded[tags.B64]), unicode) decoded = self.unpickler.restore(encoded) self.assertEqual(decoded, b2) self.assertTrue(type(decoded) is bytes) def test_backcompat_bytes_quoted_printable(self): """Test decoding bytes objects from older jsonpickle versions""" b1 = b'foo' b2 = b'foo\xff' # older versions of jsonpickle used a quoted-printable encoding expect = b1 actual = self.unpickler.restore({tags.BYTES: unicode('foo')}) self.assertEqual(expect, actual) expect = b2 actual = self.unpickler.restore({tags.BYTES: unicode('foo=FF')}) self.assertEqual(expect, actual) def test_nested_objects(self): obj = ThingWithTimedeltaAttribute(99) flattened = self.pickler.flatten(obj) restored = self.unpickler.restore(flattened) self.assertEqual(restored.offset, datetime.timedelta(99)) # Test classes for ExternalHandlerTestCase class Mixin(object): def ok(self): return True class UnicodeMixin(unicode, Mixin): def __add__(self, rhs): obj = super(UnicodeMixin, self).__add__(rhs) return UnicodeMixin(obj) class UnicodeMixinHandler(handlers.BaseHandler): def flatten(self, obj, data): data['value'] = obj return data def restore(self, obj): return UnicodeMixin(obj['value']) handlers.register(UnicodeMixin, UnicodeMixinHandler) class ExternalHandlerTestCase(unittest.TestCase): def test_unicode_mixin(self): obj = UnicodeMixin('test') self.assertEqual(type(obj), UnicodeMixin) self.assertEqual(unicode(obj), unicode('test')) # Encode into JSON content = jsonpickle.encode(obj) # Resurrect from JSON new_obj = jsonpickle.decode(content) new_obj += ' passed' self.assertEqual(unicode(new_obj), unicode('test passed')) self.assertEqual(type(new_obj), UnicodeMixin) self.assertTrue(new_obj.ok()) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(AdvancedObjectsTestCase)) suite.addTest(unittest.makeSuite(ExternalHandlerTestCase)) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/tests/sqlalchemy_test.py0000644000175000017500000000572712770463604021027 0ustar daviddavid00000000000000"""Test serializing sqlalchemy models""" import unittest import jsonpickle from helper import SkippableTest Table = None class SQLAlchemyTestCase(SkippableTest): def setUp(self): global Table try: import sqlalchemy as sqa self.should_skip = False except ImportError: self.should_skip = True return from sqlalchemy.ext import declarative from sqlalchemy.orm import Session Base = declarative.declarative_base() class Table(Base): __tablename__ = 'table' id = sqa.Column(sqa.Integer, primary_key=True) name = sqa.Column(sqa.Text) value = sqa.Column(sqa.Float) url = 'sqlite:///:memory:' self.engine = engine = sqa.create_engine(url) Base.metadata.drop_all(engine) Base.metadata.create_all(engine) self.Base = Base self.Session = Session self.sqa = sqa def test_sqlalchemy_roundtrip_with_detached_session(self): """Test cloned SQLAlchemy objects detached from any session""" if self.should_skip: return self.skip('sqlalchemy is not installed') expect = Table(name='coolness', value=11.0) session = self.Session(bind=self.engine, expire_on_commit=False) session.add(expect) session.commit() jsonstr = jsonpickle.dumps(expect) actual = jsonpickle.loads(jsonstr) # actual is a shadow object; it cannot be added to the same # session otherwise sqlalchemy will detect an identity conflict. # To make this work we use expire_on_commit=True so that sqlalchemy # allows us to do read-only operations detached from any session. self.assertEqual(expect.id, actual.id) self.assertEqual(expect.name, actual.name) self.assertEqual(expect.value, actual.value) def test_sqlalchemy_roundtrip_with_two_sessions(self): """Test cloned SQLAlchemy objects attached to a secondary session""" if self.should_skip: return self.skip('sqlalchemy is not installed') expect = Table(name='coolness', value=11.0) session = self.Session(bind=self.engine, expire_on_commit=False) session.add(expect) session.commit() jsonstr = jsonpickle.dumps(expect) actual = jsonpickle.loads(jsonstr) # actual is a shadow object; it cannot be added to the same # session otherwise sqlalchemy will detect an identity conflict. # To make this work we use expire_on_commit=True so that sqlalchemy # allows us to do read-only operations detached from any session. self.assertEqual(expect.id, actual.id) self.assertEqual(expect.name, actual.name) self.assertEqual(expect.value, actual.value) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(SQLAlchemyTestCase, 'test')) return suite if __name__ == '__main__': unittest.main() jsonpickle-0.9.5/tests/feedparser_test.py0000664000175000017500000000612012502235440020757 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. import unittest import jsonpickle from helper import SkippableTest RSS_DOC = """ Sample Feed For documentation <em>only</em> <p>Copyright 2005, Mark Pilgrim</p>< Sample Toolkit tag:feedparser.org,2005-11-09:/docs/examples/atom10.xml 2005-11-09T11:56:34Z First entry title tag:feedparser.org,2005-11-09:/docs/examples/atom10.xml:3 2005-11-09T00:23:47Z 2005-11-09T11:56:34Z Mark Pilgrim http://diveintomark.org/ mark@example.org Joe http://example.org/joe/ joe@example.org Sam http://example.org/sam/ sam@example.org Watch out for nasty tricks
Watch out for nasty tricks
""" class FeedParserTestCase(SkippableTest): def setUp(self): try: import feedparser self.should_skip = False self.doc = feedparser.parse(RSS_DOC) except ImportError: self.should_skip = True return def test(self): if self.should_skip: return self.skip('feedparser module not available, please install') pickled = jsonpickle.encode(self.doc) unpickled = jsonpickle.decode(pickled) self.assertEqual(self.doc['feed']['title'], unpickled['feed']['title']) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(FeedParserTestCase, 'test')) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/tests/handler_test.py0000664000175000017500000000761412502235440020265 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2013 Jason R. Coombs # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. import doctest import unittest import jsonpickle import jsonpickle.handlers class CustomObject(object): "A class to be serialized by a custom handler" def __init__(self, name=None, creator=None): self.name = name self.creator = creator def __eq__(self, other): return self.name == other.name class CustomA(CustomObject): pass class CustomB(CustomA): pass class NullHandler(jsonpickle.handlers.BaseHandler): def flatten(self, obj, data): data['name'] = obj.name return data def restore(self, obj): return CustomObject(obj['name'], creator=type(self)) class DecoratedBase(CustomObject): pass class DecoratedChild(DecoratedBase): pass @jsonpickle.handlers.register(DecoratedBase, base=True) class DecoratedHandler(NullHandler): pass class HandlerTestCase(unittest.TestCase): def setUp(self): jsonpickle.handlers.register(CustomObject, NullHandler) def tearDown(self): jsonpickle.handlers.unregister(CustomObject) def roundtrip(self, ob): encoded = jsonpickle.encode(ob) decoded = jsonpickle.decode(encoded) self.assertEqual(decoded, ob) return decoded def test_custom_handler(self): """Ensure that the custom handler is indeed used""" expect = CustomObject('hello') encoded = jsonpickle.encode(expect) actual = jsonpickle.decode(encoded) self.assertEqual(expect.name, actual.name) self.assertTrue(expect.creator is None) self.assertTrue(actual.creator is NullHandler) def test_references(self): """ Ensure objects handled by a custom handler are properly dereferenced. """ ob = CustomObject() # create a dictionary which contains several references to ob subject = dict(a=ob, b=ob, c=ob) # ensure the subject can be roundtripped new_subject = self.roundtrip(subject) self.assertEqual(new_subject['a'], new_subject['b']) self.assertEqual(new_subject['b'], new_subject['c']) self.assertTrue(new_subject['a'] is new_subject['b']) self.assertTrue(new_subject['b'] is new_subject['c']) def test_invalid_class(self): self.assertRaises(TypeError, jsonpickle.handlers.register, 'foo', NullHandler) def test_base_handler(self): a = CustomA('a') self.assertTrue(a.creator is None) self.assertTrue(jsonpickle.decode(jsonpickle.encode(a)).creator is None) b = CustomB('b') self.assertTrue(b.creator is None) self.assertTrue(jsonpickle.decode(jsonpickle.encode(b)).creator is None) OtherHandler = type('OtherHandler', (NullHandler,), {}) jsonpickle.handlers.register(CustomA, OtherHandler, base=True) self.assertTrue(self.roundtrip(a).creator is OtherHandler) self.assertTrue(self.roundtrip(b).creator is OtherHandler) SpecializedHandler = type('SpecializedHandler', (NullHandler,), {}) jsonpickle.handlers.register(CustomB, SpecializedHandler) self.assertTrue(self.roundtrip(a).creator is OtherHandler) self.assertTrue(self.roundtrip(b).creator is SpecializedHandler) def test_decorated_register(self): db = DecoratedBase('db') dc = DecoratedChild('dc') self.assertTrue(self.roundtrip(db).creator is DecoratedHandler) self.assertTrue(self.roundtrip(dc).creator is DecoratedHandler) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(HandlerTestCase)) suite.addTest(doctest.DocTestSuite(jsonpickle.handlers)) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/tests/stdlib_test.py0000644000175000017500000000322512673353257020141 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- """Test miscellaneous objects from the standard library""" import uuid import unittest import jsonpickle class UUIDTestCase(unittest.TestCase): def test_random_uuid(self): u = uuid.uuid4() encoded = jsonpickle.encode(u) decoded = jsonpickle.decode(encoded) expect = u.hex actual = decoded.hex self.assertEqual(expect, actual) def test_known_uuid(self): expect = '28b56adbd18f44e2a5556bba2f23e6f6' exemplar = uuid.UUID(expect) encoded = jsonpickle.encode(exemplar) decoded = jsonpickle.decode(encoded) actual = decoded.hex self.assertEqual(expect, actual) class BytesTestCase(unittest.TestCase): def test_bytestream(self): expect = (b'\x89HDF\r\n\x1a\n\x00\x00\x00\x00\x00\x08\x08\x00' b'\x04\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00' b'\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xffh' b'\x848\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff' b'\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00`\x00\x00' b'\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00' b'\x00\x88\x00\x00\x00\x00\x00\x00\x00\xa8\x02\x00' b'\x00\x00\x00\x00\x00\x01\x00\x01\x00') encoded = jsonpickle.encode(expect) actual = jsonpickle.decode(encoded) self.assertEqual(expect, actual) def suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(UUIDTestCase)) suite.addTest(unittest.makeSuite(BytesTestCase)) return suite if __name__ == '__main__': unittest.main(defaultTest='suite') jsonpickle-0.9.5/MANIFEST.in0000664000175000017500000000024112402535726015631 0ustar daviddavid00000000000000include COPYING README.rst include Rakefile include *.py include jsonpickle/*.py include tests/*.py include docs/source/** include jsonpickleJS/** include *.txt jsonpickle-0.9.5/setup.cfg0000644000175000017500000000004613133025657015714 0ustar daviddavid00000000000000[egg_info] tag_build = tag_date = 0 jsonpickle-0.9.5/setup.py0000644000175000017500000000350613035361075015607 0ustar daviddavid00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # Copyright (C) 2009-2017 David Aguilar (davvid -at- gmail.com) # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. import os try: import setuptools as setup_mod except ImportError: import distutils.core as setup_mod here = os.path.dirname(__file__) version = os.path.join(here, 'jsonpickle', 'version.py') scope = {} exec(open(version).read(), scope) SETUP_ARGS = dict( name='jsonpickle', version=scope['VERSION'], description='Python library for serializing any arbitrary object graph into JSON', long_description='jsonpickle converts complex Python objects to and from JSON.', author='David Aguilar', author_email='davvid@gmail.com', url='http://jsonpickle.github.io/', license='BSD', platforms=['POSIX', 'Windows'], keywords=['json pickle', 'json', 'pickle', 'marshal', 'serialization', 'JavaScript Object Notation'], classifiers=[ 'License :: OSI Approved :: BSD License', 'Operating System :: OS Independent', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Topic :: Software Development :: Libraries :: Python Modules', 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'Programming Language :: Python', 'Programming Language :: JavaScript', ], options={'clean': {'all': 1}}, packages=['jsonpickle', 'jsonpickle.ext'], ) if __name__ == '__main__': setup_mod.setup(**SETUP_ARGS) jsonpickle-0.9.5/jsonpickleJS/0000755000175000017500000000000013133025657016471 5ustar daviddavid00000000000000jsonpickle-0.9.5/jsonpickleJS/util.js0000664000175000017500000000416412402535671020013 0ustar daviddavid00000000000000define(['./tags'], function(tags) { var util = {}; util.merge = function(destination, source) { if (source !== undefined) { for (var property in source) { destination[property] = source[property]; } } return destination; }; util.PRIMITIVES = ['string', 'number', 'boolean']; /* from jsonpickle.util */ // to_do... util.is_type = function (obj) { return false; }; // same as dictionary in JS... util.is_object = function (obj) { return util.is_dictionary(obj); }; util.is_primitive = function (obj) { if (obj === undefined || obj == null) { return true; } if (util.PRIMITIVES.indexOf(typeof obj) != -1) { return true; } return false; }; util.is_dictionary = function (obj) { return ((typeof obj == 'object') && (obj !== null)); }; util.is_sequence = function (obj) { return (util.is_list(obj) || util.is_set(obj) || util.is_tuple(obj)); }; util.is_list = function (obj) { return (obj instanceof Array); }; // to_do... util.is_set = function (obj) { return false; }; util.is_tuple = function (obj) { return false; }; util.is_dictionary_subclass = function (obj) { return false; }; util.is_sequence_subclass = function (obj) { return false; }; util.is_noncomplex = function (obj) { return false; }; util.is_function = function (obj) { return (typeof obj == 'function'); }; util.is_module = function (obj) { return false; }; util.is_picklable = function(name, value) { if (tags.RESERVED.indexOf(name) != -1) { return false; } if (util.is_function(value)) { return false; } else { return true; } }; util.is_installed = function (module) { return true; // ??? }; util.is_list_like = function (obj) { return util.is_list(obj); }; if (typeof jsonpickle != "undefined") { jsonpickle.util = util; } return util; });jsonpickle-0.9.5/jsonpickleJS/main.js0000664000175000017500000000262312402535671017760 0ustar daviddavid00000000000000/** * jsonpickleJS -- interpretation of python jsonpickle in Javascript * main.js -- main loader -- this should be the only file that most users care about. * * Copyright (c) 2014 Michael Scott Cuthbert and cuthbertLab */ if (typeof jsonpickle == 'undefined') { jsonpickle = {}; // global for now... } define(['./unpickler', './pickler', './util', './tags', './handlers'], function(unpickler, pickler, util, tags, handlers) { jsonpickle.pickler = pickler; jsonpickle.unpickler = unpickler; jsonpickle.util = util; jsonpickle.tags = tags; jsonpickle.handlers = handlers; jsonpickle.encode = function (value, unpicklable, make_refs, keys, max_depth, backend) { if (unpicklable === undefined) { unpicklable = true; } if (make_refs === undefined) { make_refs = true; } if (keys === undefined) { keys = false; } var options = { unpicklable: unpicklable, make_refs: make_refs, keys: keys, max_depth: max_depth, backend: backend, }; return pickler.encode(value, options); }; jsonpickle.decode = function (string, backend, keys) { if (keys === undefined) { keys = false; } return unpickler.decode(string, backend, keys); }; return jsonpickle; });jsonpickle-0.9.5/jsonpickleJS/unpickler.js0000664000175000017500000002706412402535671021036 0ustar daviddavid00000000000000/** * jsonPickle/javascript/unpickler -- Conversion from music21p jsonpickle streams * * Copyright (c) 2013-14, Michael Scott Cuthbert and cuthbertLab * Based on music21 (=music21p), Copyright (c) 2006–14, Michael Scott Cuthbert and cuthbertLab * * usage: * * js_obj = unpickler.decode(json_string); * * */ define(['./util', './handlers', './tags'], function(util, handlers, tags) { var unpickler = {}; unpickler.decode = function (string, user_handlers, options) { var params = { keys: false, safe: false, reset: true, backend: JSON, }; util.merge(params, options); var use_handlers = {}; util.merge(use_handlers, handlers); // don't screw up default handlers... util.merge(use_handlers, user_handlers); // backend does not do anything -- it is there for // compat with py-JSON-Pickle if (params.context === undefined) { var unpickler_options = { keys: params.keys, backend: params.backend, safe: params.safe, }; context = new unpickler.Unpickler(unpickler_options, use_handlers); } var jsonObj = params.backend.parse(string); return context.restore(jsonObj, params.reset); }; unpickler.Unpickler = function (options, handlers) { var params = { keys: false, safe: false, }; util.merge(params, options); this.keys = params.keys; this.safe = params.safe; this.handlers = handlers; //obsolete... //this._namedict = {}; // The namestack grows whenever we recurse into a child object this._namestack = []; // Maps objects to their index in the _objs list this._obj_to_idx = {}; this._objs = []; }; unpickler.Unpickler.prototype.reset = function () { //this._namedict = {}; this._namestack = []; this._obj_to_idx = {}; this._objs = []; }; /** * Restores a flattened object to a JavaScript representation * as close to the original python object as possible. * * Requires that javascript */ unpickler.Unpickler.prototype.restore = function (obj, reset) { if (reset) { this.reset(); } return this._restore(obj); }; unpickler.Unpickler.prototype._restore = function (obj) { var has_tag = unpickler.has_tag; var restore = undefined; if (has_tag(obj, tags.ID)) { restore = this._restore_id.bind(this); } else if (has_tag(obj, tags.REF)) { // backwards compat. not supported } else if (has_tag(obj, tags.TYPE)) { restore = this._restore_type.bind(this); } else if (has_tag(obj, tags.REPR)) { // backwards compat. not supported } else if (has_tag(obj, tags.OBJECT)) { restore = this._restore_object.bind(this); } else if (has_tag(obj, tags.TUPLE)) { restore = this._restore_tuple.bind(this); } else if (has_tag(obj, tags.SET)) { restore = this._restore_set.bind(this); } else if (util.is_list(obj)) { restore = this._restore_list.bind(this); } else if (util.is_dictionary(obj)) { restore = this._restore_dict.bind(this); } else { restore = function (obj) { return obj; }; } return restore(obj); }; unpickler.Unpickler.prototype._restore_id = function (obj) { return this._objs[obj[tags.ID]]; }; unpickler.Unpickler.prototype._restore_type = function (obj) { var typeref = unpickler.loadclass(obj[tags.TYPE]); if (typeref === undefined) { return obj; } else { return typeref; } }; unpickler.Unpickler.prototype._restore_object = function (obj) { var class_name = obj[tags.OBJECT]; var handler = this.handlers[class_name]; if (handler !== undefined && handler.restore !== undefined) { var instance = handler.restore(obj); instance[tags.PY_CLASS] = class_name; return this._mkref(instance); } else { var cls = unpickler.loadclass(class_name); if (cls === undefined) { obj[tags.PY_CLASS] = class_name; return this._mkref(obj); } var instance = this._restore_object_instance(obj, cls); instance[tags.PY_CLASS] = class_name; if (handler !== undefined && handler.post_restore !== undefined) { return handler.post_restore(instance); } else { return instance; } } }; unpickler.Unpickler.prototype._loadfactory = function (obj) { var default_factory = obj['default_factory']; if (default_factory === undefined) { return undefined; } else { obj['default_factory'] = undefined; return this._restore(default_factory); } }; unpickler.Unpickler.prototype._restore_object_instance = function (obj, cls) { //var factory = this._loadfactory(obj); var args = unpickler.getargs(obj); if (args.length > 0) { args = this._restore(args); } // not using factory... does not seem to apply to JS var instance = unpickler.construct(cls, args); this._mkref(instance); return this._restore_object_instance_variables(obj, instance); }; unpickler.Unpickler.prototype._restore_object_instance_variables = function (obj, instance) { var has_tag = unpickler.has_tag; var restore_key = this._restore_key_fn(); var keys = []; for (var k in obj) { if (obj.hasOwnProperty(k)) { keys.push(k); } } keys.sort(); for (var i = 0; i < keys.length; i++) { var k = keys[i]; if (tags.RESERVED.indexOf(k) != -1) { continue; } var v = obj[k]; this._namestack.push(k); k = restore_key(k); var value = undefined; if (v !== undefined && v !== null) { value = this._restore(v); } // no setattr checks... instance[k] = value; this._namestack.pop(); } if (has_tag(obj, tags.SEQ)) { if (instance.push !== undefined) { for (var v in obj[tags.SEQ]) { instance.push(this._restore(v)); } } // no .add ... } if (has_tag(obj, tags.STATE)) { instance = this._restore_state(obj, instance); } return instance; }; unpickler.Unpickler.prototype._restore_state = function (obj, instance) { // only if the JS object implements __setstate__ if (instance.__setstate__ !== undefined) { var state = this._restore(obj[tags.STATE]); instance.__setstate__(state); } else { instance = this._restore(obj[tags.STATE]); } return instance; }; unpickler.Unpickler.prototype._restore_list = function (obj) { var parent = []; this._mkref(parent); var children = []; for (var i = 0; i < obj.length; i++) { var v = obj[i]; var rest = this._restore(v); children.push(rest); } parent.push.apply(parent, children); return parent; }; unpickler.Unpickler.prototype._restore_tuple = function (obj) { // JS having no difference between list, tuple, set -- returns Array var children = []; var tupleContents = obj[tags.TUPLE]; for (var i = 0; i < tupleContents.length; i++) { children.push(this._restore(tupleContents[i])); } return children; }; unpickler.Unpickler.prototype._restore_set = function (obj) { // JS having no difference between list, tuple, set -- returns Array var children = []; var setContents = obj[tags.SET]; for (var i = 0; i < setContents.length; i++) { children.push(this._restore(setContents[i])); } return children; }; unpickler.Unpickler.prototype._restore_dict = function (obj) { var data = {}; //var restore_key = this._restore_key_fn(); var keys = []; for (var k in obj) { if (obj.hasOwnProperty(k)) { keys.push(k); } } keys.sort(); for (var i = 0; i < keys.length; i++) { var k = keys[i]; var v = obj[k]; this._namestack.push(k); data[k] = this._restore(v); // no setattr checks... this._namestack.pop(); } return data; }; unpickler.Unpickler.prototype._restore_key_fn = function () { if (this.keys) { return function (key) { if (key.indexOf(tags.JSON_KEY) == 0) { key = unpickler.decode(key.slice(tags.JSON_KEY.length), this.handlers, {context: this, keys: this.keys, reset: false} ); return key; } }; } else { return function (key) { return key; }; } }; // _refname not needed... unpickler.Unpickler.prototype._mkref = function (obj) { // does not use id(obj) in javascript this._objs.push(obj); return obj; }; unpickler.getargs = function (obj) { var seq_list = obj[tags.SEQ]; var obj_dict = obj[tags.OBJECT]; if (seq_list === undefined || obj_dict === undefined) { return []; } var typeref = unpickler.loadclass(obj_dict); if (typeref === undefined) { return []; } if (typeref['_fields'] !== undefined) { if (typeref['_fields'].length == seq_list.length) { return seq_list; } } return []; }; unpickler.loadclass = function (module_and_name) { var main_check = '__main__.'; if (module_and_name.indexOf(main_check) == 0) { module_and_name = module_and_name.slice(main_check.length); } var parent = window; var module_class_split = module_and_name.split('.'); for (var i = 0; i < module_class_split.length; i++) { var this_module_or_class = module_class_split[i]; parent = parent[this_module_or_class]; if (parent === undefined) { return parent; } } return parent; }; unpickler.has_tag = function (obj, tag) { if ((typeof obj == 'object') && (obj[tag] !== undefined)) { return true; } else { return false; } }; // http://stackoverflow.com/questions/1606797/use-of-apply-with-new-operator-is-this-possible unpickler.construct = function (constructor, args) { function F() { return constructor.apply(this, args); } F.prototype = constructor.prototype; return new F(); }; if (jsonpickle !== undefined) { jsonpickle.unpickler = unpickler; } return unpickler; });jsonpickle-0.9.5/jsonpickleJS/tags.js0000664000175000017500000000133412402535671017770 0ustar daviddavid00000000000000define(function () { var tags = { ID: 'py/id', OBJECT: 'py/object', TYPE: 'py/type', REPR: 'py/repr', REF: 'py/ref', TUPLE: 'py/tuple', SET: 'py/set', SEQ: 'py/seq', STATE: 'py/state', JSON_KEY: 'json://', }; tags.RESERVED = [tags.ID, tags.OBJECT, tags.TYPE, tags.REPR, tags.REF, tags.TUPLE, tags.SET, tags.SEQ, tags.STATE, tags.JSON_KEY]; tags.PY_CLASS = '_py_class'; if (typeof jsonpickle != 'undefined') { jsonpickle.tags = tags; } return tags; });jsonpickle-0.9.5/jsonpickleJS/testPickle.html0000664000175000017500000000154312402535671021473 0ustar daviddavid00000000000000 See Console for output... jsonpickle-0.9.5/jsonpickleJS/.gitignore0000664000175000017500000000054212402535671020464 0ustar daviddavid00000000000000*.py[cod] # OSX .DS_Store # C extensions *.so # Packages *.egg *.egg-info dist build eggs parts bin var sdist develop-eggs .installed.cfg lib lib64 __pycache__ # Installer logs pip-log.txt # Unit test / coverage reports .coverage .tox nosetests.xml # Translations *.mo # Mr Developer .mr.developer.cfg # Eclipse .project .pydevproject /.settings jsonpickle-0.9.5/jsonpickleJS/LICENSE0000664000175000017500000000301612402535671017500 0ustar daviddavid00000000000000Copyright (c) 2013-14, Michael Scott Cuthbert and cuthbertLab All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Michael Scott Cuthbert/cuthbertLab nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL Cuthbert OR cuthbertLab BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. jsonpickle-0.9.5/jsonpickleJS/pickler.js0000664000175000017500000002144412402535671020467 0ustar daviddavid00000000000000define(['./util', './handlers', './tags'], function(util, handlers, tags) { var pickler = {}; pickler.encode = function(value, options) { var params = { unpicklable: false, make_refs: true, keys: false, max_depth: undefined, reset: true, backend: undefined, // does nothing; python compat context: undefined, }; util.merge(params, options); if (params.context === undefined) { outparams = { unpicklable: params.unpicklable, make_refs: params.make_refs, keys: params.keys, backend: params.backend, max_depth: params.max_depth, }; params.context = new pickler.Pickler(outparams); var fixed_obj = params.context.flatten(value, params.reset); return JSON.stringify(fixed_obj); } }; pickler.Pickler = function (options) { var params = { unpicklable: true, make_refs: true, max_depth: undefined, backend: undefined, keys: false, }; util.merge(params, options); this.unpicklable = params.unpicklable; this.make_refs = params.make_refs; this.backend = params.backend; this.keys = params.keys; this._depth = -1; this._max_depth = params.max_depth; this._objs = []; this._seen = []; }; pickler.Pickler.prototype.reset = function () { this._objs = []; this._depth = -1; this._seen = []; }; pickler.Pickler.prototype._push = function () { this._depth += 1; }; pickler.Pickler.prototype._pop = function (value) { this._depth -= 1; if (this._depth == -1) { this.reset(); } return value; }; pickler.Pickler.prototype._mkref = function (obj) { var found_id = this._get_id_in_objs(obj); //console.log(found_id); if (found_id != -1) { return false; // if (this.unpicklable == false || this.make_refs == false) { // return true; // } else { // return false; // } } this._objs.push(obj); return true; }; pickler.Pickler.prototype._get_id_in_objs = function (obj) { var objLength = this._objs.length; // console.log('sought obj', obj); // console.log('stored objs: ', this._objs); for (var i = 0; i < objLength; i++) { if (obj === this._objs[i]) { return i; } } return -1; }; pickler.Pickler.prototype._getref = function (obj) { var wrap_obj = {}; wrap_obj[tags.ID] = this._get_id_in_objs(obj); return wrap_obj; }; pickler.Pickler.prototype.flatten = function (obj, reset) { if (reset === undefined) { reset = true; } if (reset == true) { this.reset(); } var flatOut = this._flatten(obj); console.log(this._objs); return flatOut; }; pickler.Pickler.prototype._flatten = function (obj) { this._push(); return this._pop(this._flatten_obj(obj)); }; pickler.Pickler.prototype._flatten_obj = function (obj) { this._seen.push(obj); max_reached = (this._depth == this._max_depth) ? true : false; if (max_reached || (this.make_refs == false && this._get_id_in_objs(obj) != -1)) { // no repr equivalent, use str; return toString(obj); } else { var flattener = this._get_flattener(obj); //console.log(flattener); return flattener.call(this, obj); } }; pickler.Pickler.prototype._list_recurse = function (obj) { var l = []; for (var i = 0; i < obj.length; i ++) { l.push(this._flatten(obj[i])); } return l; }; pickler.Pickler.prototype._get_flattener = function (obj) { if (util.is_primitive(obj)) { return function (obj) { return obj; }; } if (util.is_list(obj)) { if (this._mkref(obj)) { return this._list_recurse; } else { this._push(); return this._getref; } } if (util.is_tuple(obj)) { if (this.unpicklable == false) { return this._list_recurse; } else { return function (obj) { var obj_wrap = {}; obj_wrap[tags.TUPLE] = this._list_recurse(obj); }; } } if (util.is_set(obj)) { if (this.unpicklable == false) { return this._list_recurse; } else { return function (obj) { var obj_wrap = {}; obj_wrap[tags.SET] = this._list_recurse(obj); }; } } // better -- translate as object... //if (util.is_dictionary(obj)) { // return this._flatten_dict_obj; //} //if (util.is_type(obj)) { // return _mktyperef; //} if (util.is_object(obj)) { return this._ref_obj_instance; } console.log('no flattener for ', obj, ' of type ', typeof obj); return undefined; }; pickler.Pickler.prototype._ref_obj_instance = function (obj) { if (this._mkref(obj)) { return this._flatten_obj_instance(obj); } return this._getref(obj); }; pickler.Pickler.prototype._flatten_obj_instance = function (obj) { var data = {}; has_class = (obj[tags.PY_CLASS] !== undefined); // ? or ... has_dict = true; has_slots = false; has_getstate = (obj.__getstate__ !== undefined); if (has_class && util.is_module(obj) == false) { var fullModuleName = pickler._getclassdetail(obj); if (this.unpicklable) { data[tags.OBJECT] = fullModuleName; console.log(data); } handler = handlers[fullModuleName]; if (handler !== undefined) { handler.flatten(obj, data); } } if (util.is_module(obj)) { // todo if possible to have this happen... } if (util.is_dictionary_subclass(obj)) { // todo if possible to have this happen... } if (has_dict) { // where every object currently ends up if (util.is_sequence_subclass(obj)) { // todo if possible to be... } if (has_getstate) { return this._getstate(obj, data); } return this._flatten_dict_obj(obj, data); } // todo: is_sequence_subclass // todo: is_noncomplex // todo: has_slots... }; pickler.Pickler.prototype._flatten_dict_obj = function (obj, data) { if (data === undefined) { data = new obj.prototype.constructor(); } var key_index = []; for (var key in obj) { if (obj.hasOwnProperty(key)) { key_index.push(key); } } for (var i = 0; i < key_index.length; i++) { var key = key_index[i]; var value = obj[key]; if (key === tags.PY_CLASS) { continue; } this._flatten_key_value_pair(key, value, data); } // default_factory... return data; }; // _flatten_newstyle_with_slots pickler.Pickler.prototype._flatten_key_value_pair = function (k, v, data) { if (util.is_picklable(k, v) == false) { return data; } // assume all keys are strings -- Javascript; data[k] = this._flatten(v); return data; }; //pickler.Pickler.prototype._flatten_sequence_obj = function () {}; pickler.Pickler.prototype._getstate = function (obj, data) { var state = this._flatten_obj(obj.__getstate__()); if (this.unpicklable) { data[tags.STATE] = state; } else { data = state; } return data; }; //pickler._mktyperef = function (obj) {}; pickler._getclassdetail = function(obj) { // just returns the Python class name return obj[tags.PY_CLASS]; }; if (jsonpickle !== undefined) { jsonpickle.pickler = pickler; } return pickler; });jsonpickle-0.9.5/jsonpickleJS/README.md0000664000175000017500000001110212402535671017745 0ustar daviddavid00000000000000jsonpickleJS ============ Javascript reinterpretation of Python jsonpickle to allow reading and (to a lesser extent) writing JSON objects Copyright (c) 2014 Michael Scott Cuthbert and cuthbertLab. Released under the BSD (3-clause) license. See LICENSE. Python to Javascript and Back ============================== Python has a remarkable number of ways (for a language that believes there's only one way to do it) to transfer data between itself and Javascript, most obviously with the ``json.dump()``/``json.dumps()`` calls, which work well on specifically created data, especially of primitive objects. It also has a good number of ways to store the contents of an object so that it could be reloaded later into the same Python interpreter/session. The most well known of these is the ``pickle`` module, which stores the Python data as binary data. The external ``jsonpickle`` module, which you'll need to have installed to make this module have any sense, combines the features of ``json`` and ``pickle``, storing Python data in a JSON (Javascript Object Notation) string, via ``jsonpickle.encode()`` that can be parsed back by ``jsonpickle.decode()`` to get nearly all data types restored to their previous state. (Highly recommended: jsonpickle v1.7 or higher. Older versions will not serialize recursive data structures as necessary for a lot of applications.) Since JSON is one of the most easily parsed formats in Javascript (the name should be a giveaway), this project, jsonpickleJS, exists to parse the output of Python's jsonpickle into objects in Javascript. The constructors of the objects need to have the same names and exist in the global namespace, such as ``window``, in the Javascript. For instance, if you have a class called ``myobject.Thing`` in Python, you'll need to have a Prototype constructor function called ``window.myobject.Thing`` in Javascript. The object, and any subobjects, will be created as closely as possible in Javascript. The reverse is also possible, with some caveats. Since Javascript doesn't (until ECMAScript 6) have the concept of named classes, each object will need to have a marker somewhere on it saying what Python object it should convert back to. The marker is ``o[jsonpickle.tags.PY_CLASS] = 'fully.qualified.ClassName'``. It may be possible in the future to use ``instanceof`` through the entire Global namespace to figure out what something is, but that seems rather dangerous and inefficient (A project for later). Limitations =========== Remember that Javascript does not have tuples, so all tuple objects are changed to lists. Namedtuples behave the same way, I believe. Dicts and Objects are identical in Javascript (both a blessing and a curse). Security ======== Pickle, jsonpickle, and jsonpickleJS all raise important security considerations you must be aware of. You will be loading data directly into Python or Javascript with no checks on what the data contains. Only load data you have personally produced if you want to be safe. In Javascript, malicious data may compromise your browser, browser history, functioning of the current web page, etc. That's pretty bad, but nothing compared to what can happen if you load jsonpickle data from Javascript into Python: a maliciously written set of jsonpickle data may be able to send the contents of any file back to the net or manipulate or delete the hard drive of the server. It may be that the parsed Python object have to be called in some way, but it may even be possible to have malicious code executed just on parsing; assume the worst. Your .html/.js may only produce safe data, but anyone with JS programming experience can inject other data into your server scripts. Be safe: be cautious going from Python to Javascript and NEVER accept Javascript-produced jsonpickle data from the net into your Python program in any sort of open environment. Usage ===== See the source code of: testUnpickle.html to see how to use it. JsonpickleJS follows the AMD moduleloader standard, so set the ``src=""`` attribute in the `` See Console for output... jsonpickle-0.9.5/PKG-INFO0000644000175000017500000000201713133025657015170 0ustar daviddavid00000000000000Metadata-Version: 1.1 Name: jsonpickle Version: 0.9.5 Summary: Python library for serializing any arbitrary object graph into JSON Home-page: http://jsonpickle.github.io/ Author: David Aguilar Author-email: davvid@gmail.com License: BSD Description: jsonpickle converts complex Python objects to and from JSON. Keywords: json pickle,json,pickle,marshal,serialization,JavaScript Object Notation Platform: POSIX Platform: Windows Classifier: License :: OSI Approved :: BSD License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: Programming Language :: Python Classifier: Programming Language :: JavaScript jsonpickle-0.9.5/jsonpickle.egg-info/0000755000175000017500000000000013133025657017726 5ustar daviddavid00000000000000jsonpickle-0.9.5/jsonpickle.egg-info/top_level.txt0000644000175000017500000000001313133025657022452 0ustar daviddavid00000000000000jsonpickle jsonpickle-0.9.5/jsonpickle.egg-info/dependency_links.txt0000644000175000017500000000000113133025657023774 0ustar daviddavid00000000000000 jsonpickle-0.9.5/jsonpickle.egg-info/SOURCES.txt0000644000175000017500000000226413133025657021616 0ustar daviddavid00000000000000COPYING MANIFEST.in README.rst requirements-2.txt requirements-3.txt requirements-test.txt setup.py docs/source/api.rst docs/source/changelog.rst docs/source/conf.py docs/source/contrib.rst docs/source/extensions.rst docs/source/index.rst jsonpickle/__init__.py jsonpickle/backend.py jsonpickle/compat.py jsonpickle/handlers.py jsonpickle/pickler.py jsonpickle/tags.py jsonpickle/unpickler.py jsonpickle/util.py jsonpickle/version.py jsonpickle.egg-info/PKG-INFO jsonpickle.egg-info/SOURCES.txt jsonpickle.egg-info/dependency_links.txt jsonpickle.egg-info/top_level.txt jsonpickle/ext/__init__.py jsonpickle/ext/numpy.py jsonpickleJS/.gitignore jsonpickleJS/LICENSE jsonpickleJS/README.md jsonpickleJS/handlers.js jsonpickleJS/main.js jsonpickleJS/pickler.js jsonpickleJS/tags.js jsonpickleJS/testPickle.html jsonpickleJS/testUnpickle.html jsonpickleJS/unpickler.js jsonpickleJS/util.js tests/backend_test.py tests/benchmark.py tests/bson_test.py tests/datetime_test.py tests/document_test.py tests/feedparser_test.py tests/handler_test.py tests/helper.py tests/jsonpickle_test.py tests/numpy_test.py tests/object_test.py tests/runtests.py tests/sqlalchemy_test.py tests/stdlib_test.py tests/util_test.pyjsonpickle-0.9.5/jsonpickle.egg-info/PKG-INFO0000644000175000017500000000201713133025657021023 0ustar daviddavid00000000000000Metadata-Version: 1.1 Name: jsonpickle Version: 0.9.5 Summary: Python library for serializing any arbitrary object graph into JSON Home-page: http://jsonpickle.github.io/ Author: David Aguilar Author-email: davvid@gmail.com License: BSD Description: jsonpickle converts complex Python objects to and from JSON. Keywords: json pickle,json,pickle,marshal,serialization,JavaScript Object Notation Platform: POSIX Platform: Windows Classifier: License :: OSI Approved :: BSD License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: Programming Language :: Python Classifier: Programming Language :: JavaScript jsonpickle-0.9.5/jsonpickle/0000755000175000017500000000000013133025657016234 5ustar daviddavid00000000000000jsonpickle-0.9.5/jsonpickle/handlers.py0000644000175000017500000001636213075310375020415 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- """ Custom handlers may be created to handle other objects. Each custom handler must derive from :class:`jsonpickle.handlers.BaseHandler` and implement ``flatten`` and ``restore``. A handler can be bound to other types by calling :func:`jsonpickle.handlers.register`. :class:`jsonpickle.customhandlers.SimpleReduceHandler` is suitable for handling objects that implement the reduce protocol:: from jsonpickle import handlers class MyCustomObject(handlers.BaseHandler): ... def __reduce__(self): return MyCustomObject, self._get_args() handlers.register(MyCustomObject, handlers.SimpleReduceHandler) """ from __future__ import absolute_import, division, unicode_literals import collections import copy import datetime import decimal import re import sys import time import uuid from . import util from .compat import queue from .compat import unicode class Registry(object): def __init__(self): self._handlers = {} self._base_handlers = {} def get(self, cls_or_name, default=None): """ :param cls_or_name: the type or its fully qualified name :param default: default value, if a matching handler is not found Looks up a handler by type reference or its fully qualified name. If a direct match is not found, the search is performed over all handlers registered with base=True. """ handler = self._handlers.get(cls_or_name) if handler is None and util.is_type(cls_or_name): # attempt to find a base class for cls, base_handler in self._base_handlers.items(): if issubclass(cls_or_name, cls): return base_handler return default if handler is None else handler def register(self, cls, handler=None, base=False): """Register the a custom handler for a class :param cls: The custom object class to handle :param handler: The custom handler class (if None, a decorator wrapper is returned) :param base: Indicates whether the handler should be registered for all subclasses This function can be also used as a decorator by omitting the `handler` argument:: @jsonpickle.handlers.register(Foo, base=True) class FooHandler(jsonpickle.handlers.BaseHandler): pass """ if handler is None: def _register(handler_cls): self.register(cls, handler=handler_cls, base=base) return handler_cls return _register if not util.is_type(cls): raise TypeError('{0!r} is not a class/type'.format(cls)) # store both the name and the actual type for the ugly cases like # _sre.SRE_Pattern that cannot be loaded back directly self._handlers[util.importable_name(cls)] = self._handlers[cls] = handler if base: # only store the actual type for subclass checking self._base_handlers[cls] = handler def unregister(self, cls): self._handlers.pop(cls, None) self._handlers.pop(util.importable_name(cls), None) self._base_handlers.pop(cls, None) registry = Registry() register = registry.register unregister = registry.unregister get = registry.get class BaseHandler(object): def __init__(self, context): """ Initialize a new handler to handle a registered type. :Parameters: - `context`: reference to pickler/unpickler """ self.context = context def __call__(self, context): """This permits registering either Handler instances or classes :Parameters: - `context`: reference to pickler/unpickler """ self.context = context return self def flatten(self, obj, data): """ Flatten `obj` into a json-friendly form and write result to `data`. :param object obj: The object to be serialized. :param dict data: A partially filled dictionary which will contain the json-friendly representation of `obj` once this method has finished. """ raise NotImplementedError('You must implement flatten() in %s' % self.__class__) def restore(self, obj): """ Restore an object of the registered type from the json-friendly representation `obj` and return it. """ raise NotImplementedError('You must implement restore() in %s' % self.__class__) @classmethod def handles(self, cls): """ Register this handler for the given class. Suitable as a decorator, e.g.:: @SimpleReduceHandler.handles class MyCustomClass: def __reduce__(self): ... """ registry.register(cls, self) return cls class DatetimeHandler(BaseHandler): """Custom handler for datetime objects Datetime objects use __reduce__, and they generate binary strings encoding the payload. This handler encodes that payload to reconstruct the object. """ def flatten(self, obj, data): pickler = self.context if not pickler.unpicklable: return unicode(obj) cls, args = obj.__reduce__() flatten = pickler.flatten payload = util.b64encode(args[0]) args = [payload] + [flatten(i, reset=False) for i in args[1:]] data['__reduce__'] = (flatten(cls, reset=False), args) return data def restore(self, data): cls, args = data['__reduce__'] unpickler = self.context restore = unpickler.restore cls = restore(cls, reset=False) value = util.b64decode(args[0]) params = (value,) + tuple([restore(i, reset=False) for i in args[1:]]) return cls.__new__(cls, *params) DatetimeHandler.handles(datetime.datetime) DatetimeHandler.handles(datetime.date) DatetimeHandler.handles(datetime.time) class RegexHandler(BaseHandler): """Flatten _sre.SRE_Pattern (compiled regex) objects""" def flatten(self, obj, data): data['pattern'] = obj.pattern return data def restore(self, data): return re.compile(data['pattern']) RegexHandler.handles(type(re.compile(''))) class QueueHandler(BaseHandler): """Opaquely serializes Queue objects Queues contains mutex and condition variables which cannot be serialized. Construct a new Queue instance when restoring. """ def flatten(self, obj, data): return data def restore(self, data): return queue.Queue() QueueHandler.handles(queue.Queue) class CloneFactory(object): """Serialization proxy for collections.defaultdict's default_factory""" def __init__(self, exemplar): self.exemplar = exemplar def __call__(self, clone=copy.copy): """Create new instances by making copies of the provided exemplar""" return clone(self.exemplar) def __repr__(self): return ('' % (id(self), self.exemplar)) class UUIDHandler(BaseHandler): """Serialize uuid.UUID objects""" def flatten(self, obj, data): data['hex'] = obj.hex return data def restore(self, data): return uuid.UUID(data['hex']) UUIDHandler.handles(uuid.UUID) jsonpickle-0.9.5/jsonpickle/pickler.py0000644000175000017500000004340313075310375020242 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # Copyright (C) 2009, 2011, 2013 David Aguilar (davvid -at- gmail.com) and contributors # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. from __future__ import absolute_import, division, unicode_literals import base64 import warnings import sys from itertools import chain, islice from . import util from . import tags from . import handlers from .backend import JSONBackend from .compat import numeric_types, unicode, PY3, PY2 def encode(value, unpicklable=False, make_refs=True, keys=False, max_depth=None, reset=True, backend=None, warn=False, context=None, max_iter=None, numeric_keys=False): backend = _make_backend(backend) if context is None: context = Pickler(unpicklable=unpicklable, make_refs=make_refs, keys=keys, backend=backend, max_depth=max_depth, warn=warn, max_iter=max_iter, numeric_keys=numeric_keys) return backend.encode(context.flatten(value, reset=reset)) def _make_backend(backend): if backend is None: return JSONBackend() else: return backend class Pickler(object): def __init__(self, unpicklable=True, make_refs=True, max_depth=None, backend=None, keys=False, warn=False, max_iter=None, numeric_keys=False): self.unpicklable = unpicklable self.make_refs = make_refs self.backend = _make_backend(backend) self.keys = keys self.warn = warn self.numeric_keys = numeric_keys # The current recursion depth self._depth = -1 # The maximal recursion depth self._max_depth = max_depth # Maps id(obj) to reference IDs self._objs = {} # Avoids garbage collection self._seen = [] # maximum amount of items to take from a pickled iterator self._max_iter = max_iter def reset(self): self._objs = {} self._depth = -1 self._seen = [] def _push(self): """Steps down one level in the namespace. """ self._depth += 1 def _pop(self, value): """Step up one level in the namespace and return the value. If we're at the root, reset the pickler's state. """ self._depth -= 1 if self._depth == -1: self.reset() return value def _log_ref(self, obj): """ Log a reference to an in-memory object. Return True if this object is new and was assigned a new ID. Otherwise return False. """ objid = id(obj) is_new = objid not in self._objs if is_new: new_id = len(self._objs) self._objs[objid] = new_id return is_new def _mkref(self, obj): """ Log a reference to an in-memory object, and return if that object should be considered newly logged. """ is_new = self._log_ref(obj) # Pretend the object is new pretend_new = not self.unpicklable or not self.make_refs return pretend_new or is_new def _getref(self, obj): return {tags.ID: self._objs.get(id(obj))} def flatten(self, obj, reset=True): """Takes an object and returns a JSON-safe representation of it. Simply returns any of the basic builtin datatypes >>> p = Pickler() >>> p.flatten('hello world') == 'hello world' True >>> p.flatten(49) 49 >>> p.flatten(350.0) 350.0 >>> p.flatten(True) True >>> p.flatten(False) False >>> r = p.flatten(None) >>> r is None True >>> p.flatten(False) False >>> p.flatten([1, 2, 3, 4]) [1, 2, 3, 4] >>> p.flatten((1,2,))[tags.TUPLE] [1, 2] >>> p.flatten({'key': 'value'}) == {'key': 'value'} True """ if reset: self.reset() return self._flatten(obj) def _flatten(self, obj): self._push() return self._pop(self._flatten_obj(obj)) def _flatten_obj(self, obj): self._seen.append(obj) max_reached = self._depth == self._max_depth if max_reached or (not self.make_refs and id(obj) in self._objs): # break the cycle flatten_func = repr else: flatten_func = self._get_flattener(obj) if flatten_func is None: self._pickle_warning(obj) return None return flatten_func(obj) def _list_recurse(self, obj): return [self._flatten(v) for v in obj] def _get_flattener(self, obj): if PY2 and isinstance(obj, file): return self._flatten_file if util.is_primitive(obj): return lambda obj: obj if util.is_bytes(obj): return self._flatten_bytestring list_recurse = self._list_recurse if util.is_list(obj): if self._mkref(obj): return list_recurse else: self._push() return self._getref # We handle tuples and sets by encoding them in a "(tuple|set)dict" if util.is_tuple(obj): if not self.unpicklable: return list_recurse return lambda obj: {tags.TUPLE: [self._flatten(v) for v in obj]} if util.is_set(obj): if not self.unpicklable: return list_recurse return lambda obj: {tags.SET: [self._flatten(v) for v in obj]} if util.is_dictionary(obj): return self._flatten_dict_obj if util.is_type(obj): return _mktyperef if util.is_object(obj): return self._ref_obj_instance if util.is_module_function(obj): return self._flatten_function # instance methods, lambdas, old style classes... self._pickle_warning(obj) return None def _ref_obj_instance(self, obj): """Reference an existing object or flatten if new """ if self._mkref(obj): # We've never seen this object so return its # json representation. return self._flatten_obj_instance(obj) # We've seen this object before so place an object # reference tag in the data. This avoids infinite recursion # when processing cyclical objects. return self._getref(obj) def _flatten_file(self, obj): """ Special case file objects """ assert not PY3 and isinstance(obj, file) return None def _flatten_bytestring(self, obj): if PY2: try: return obj.decode('utf-8') except: pass return {tags.B64: base64.encodestring(obj).decode('utf-8')} def _flatten_obj_instance(self, obj): """Recursively flatten an instance and return a json-friendly dict """ data = {} has_class = hasattr(obj, '__class__') has_dict = hasattr(obj, '__dict__') has_slots = not has_dict and hasattr(obj, '__slots__') has_getnewargs = util.has_method(obj, '__getnewargs__') has_getnewargs_ex = util.has_method(obj, '__getnewargs_ex__') has_getinitargs = util.has_method(obj, '__getinitargs__') has_reduce, has_reduce_ex = util.has_reduce(obj) # Support objects with __getstate__(); this ensures that # both __setstate__() and __getstate__() are implemented has_getstate = hasattr(obj, '__getstate__') # not using has_method since __getstate__() is handled separately below if has_class: cls = obj.__class__ else: cls = type(obj) # Check for a custom handler class_name = util.importable_name(cls) handler = handlers.get(cls, handlers.get(class_name)) if handler is not None: if self.unpicklable: data[tags.OBJECT] = class_name return handler(self).flatten(obj, data) reduce_val = None if self.unpicklable: if has_reduce and not has_reduce_ex: try: reduce_val = obj.__reduce__() except TypeError: # A lot of builtin types have a reduce which # just raises a TypeError # we ignore those pass # test for a reduce implementation, and redirect before # doing anything else if that is what reduce requests elif has_reduce_ex: try: # we're implementing protocol 2 reduce_val = obj.__reduce_ex__(2) except TypeError: # A lot of builtin types have a reduce which # just raises a TypeError # we ignore those pass if reduce_val and isinstance(reduce_val, (str, unicode)): try: varpath = iter(reduce_val.split('.')) # curmod will be transformed by the # loop into the value to pickle curmod = sys.modules[next(varpath)] for modname in varpath: curmod = getattr(curmod, modname) # replace obj with value retrieved return self._flatten(curmod) except KeyError: # well, we can't do anything with that, so we ignore it pass elif reduce_val: # at this point, reduce_val should be some kind of iterable # pad out to len 5 rv_as_list = list(reduce_val) insufficiency = 5 - len(rv_as_list) if insufficiency: rv_as_list += [None] * insufficiency if rv_as_list[0].__name__ == '__newobj__': rv_as_list[0] = tags.NEWOBJ f, args, state, listitems, dictitems = rv_as_list # check that getstate/setstate is sane if not (state and hasattr(obj, '__getstate__') and not hasattr(obj, '__setstate__') and not isinstance(obj, dict)): # turn iterators to iterables for convenient serialization if rv_as_list[3]: rv_as_list[3] = tuple(rv_as_list[3]) if rv_as_list[4]: rv_as_list[4] = tuple(rv_as_list[4]) data[tags.REDUCE] = list(map(self._flatten, rv_as_list)) return data if has_class and not util.is_module(obj): if self.unpicklable: class_name = util.importable_name(cls) data[tags.OBJECT] = class_name if has_getnewargs_ex: data[tags.NEWARGSEX] = list( map(self._flatten, obj.__getnewargs_ex__())) if has_getnewargs and not has_getnewargs_ex: data[tags.NEWARGS] = self._flatten(obj.__getnewargs__()) if has_getinitargs: data[tags.INITARGS] = self._flatten(obj.__getinitargs__()) if has_getstate: try: state = obj.__getstate__() except TypeError: # Has getstate but it cannot be called, e.g. file descriptors # in Python3 self._pickle_warning(obj) return None else: return self._getstate(state, data) if util.is_module(obj): if self.unpicklable: data[tags.REPR] = '%s/%s' % (obj.__name__, obj.__name__) else: data = unicode(obj) return data if util.is_dictionary_subclass(obj): self._flatten_dict_obj(obj, data) return data if util.is_sequence_subclass(obj): return self._flatten_sequence_obj(obj, data) if util.is_iterator(obj): # force list in python 3 data[tags.ITERATOR] = list(map(self._flatten, islice(obj, self._max_iter))) return data if has_dict: # Support objects that subclasses list and set if util.is_sequence_subclass(obj): return self._flatten_sequence_obj(obj, data) # hack for zope persistent objects; this unghostifies the object getattr(obj, '_', None) return self._flatten_dict_obj(obj.__dict__, data) if has_slots: return self._flatten_newstyle_with_slots(obj, data) # catchall return for data created above without a return # (e.g. __getnewargs__ is not supposed to be the end of the story) if data: return data self._pickle_warning(obj) return None def _flatten_function(self, obj): if self.unpicklable: data = {tags.FUNCTION: util.importable_name(obj)} else: data = None return data def _flatten_dict_obj(self, obj, data=None): """Recursively call flatten() and return json-friendly dict """ if data is None: data = obj.__class__() flatten = self._flatten_key_value_pair for k, v in sorted(obj.items(), key=util.itemgetter): flatten(k, v, data) # the collections.defaultdict protocol if hasattr(obj, 'default_factory') and callable(obj.default_factory): factory = obj.default_factory if util.is_type(factory): # Reference the class/type value = _mktyperef(factory) else: # The factory is not a type and could reference e.g. functions # or even the object instance itself, which creates a cycle. if self._mkref(factory): # We've never seen this object before so pickle it in-place. # Create an instance from the factory and assume that the # resulting instance is a suitable examplar. value = self._flatten(handlers.CloneFactory(factory())) else: # We've seen this object before. # Break the cycle by emitting a reference. value = self._getref(factory) data['default_factory'] = value return data def _flatten_obj_attrs(self, obj, attrs, data): flatten = self._flatten_key_value_pair ok = False for k in attrs: try: value = getattr(obj, k) flatten(k, value, data) except AttributeError: # The attribute may have been deleted continue ok = True return ok def _flatten_newstyle_with_slots(self, obj, data): """Return a json-friendly dict for new-style objects with __slots__. """ allslots = [_wrap_string_slot(getattr(cls, '__slots__', tuple())) for cls in obj.__class__.mro()] if not self._flatten_obj_attrs(obj, chain(*allslots), data): attrs = [x for x in dir(obj) if not x.startswith('__') and not x.endswith('__')] self._flatten_obj_attrs(obj, attrs, data) return data def _flatten_key_value_pair(self, k, v, data): """Flatten a key/value pair into the passed-in dictionary.""" if not util.is_picklable(k, v): return data if self.keys: if not isinstance(k, (str, unicode)) or k.startswith(tags.JSON_KEY): k = self._escape_key(k) else: if k is None: k = 'null' # for compatibility with common json encoders if self.numeric_keys and isinstance(k, numeric_types): pass elif not isinstance(k, (str, unicode)): try: k = repr(k) except: k = unicode(k) data[k] = self._flatten(v) return data def _flatten_sequence_obj(self, obj, data): """Return a json-friendly dict for a sequence subclass.""" if hasattr(obj, '__dict__'): self._flatten_dict_obj(obj.__dict__, data) value = [self._flatten(v) for v in obj] if self.unpicklable: data[tags.SEQ] = value else: return value return data def _escape_key(self, k): return tags.JSON_KEY + encode(k, reset=False, keys=True, context=self, backend=self.backend, make_refs=self.make_refs) def _getstate(self, obj, data): state = self._flatten_obj(obj) if self.unpicklable: data[tags.STATE] = state else: data = state return data def _pickle_warning(self, obj): if self.warn: msg = 'jsonpickle cannot pickle %r: replaced with None' % obj warnings.warn(msg) def _mktyperef(obj): """Return a typeref dictionary >>> _mktyperef(AssertionError) == {'py/type': '__builtin__.AssertionError'} True """ return {tags.TYPE: util.importable_name(obj)} def _wrap_string_slot(string): """Converts __slots__ = 'a' into __slots__ = ('a',) """ if isinstance(string, (str, unicode)): return (string,) return string jsonpickle-0.9.5/jsonpickle/unpickler.py0000644000175000017500000005007313075310375020606 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # Copyright (C) 2009, 2011, 2013 David Aguilar (davvid -at- gmail.com) and contributors # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. from __future__ import absolute_import, division, unicode_literals import base64 import quopri import sys from . import util from . import tags from . import handlers from .compat import numeric_types, set, unicode from .backend import JSONBackend def decode(string, backend=None, context=None, keys=False, reset=True, safe=False): backend = _make_backend(backend) if context is None: context = Unpickler(keys=keys, backend=backend, safe=safe) return context.restore(backend.decode(string), reset=reset) def _make_backend(backend): if backend is None: return JSONBackend() else: return backend class _Proxy(object): """Proxies are dummy objects that are later replaced by real instances The `restore()` function has to solve a tricky problem when pickling objects with cyclical references -- the parent instance does not yet exist. The problem is that `__getnewargs__()`, `__getstate__()`, custom handlers, and cyclical objects graphs are allowed to reference the yet-to-be-created object via the referencing machinery. In other words, objects are allowed to depend on themselves for construction! We solve this problem by placing dummy Proxy objects into the referencing machinery so that we can construct the child objects before constructing the parent. Objects are initially created with Proxy attribute values instead of real references. We collect all objects that contain references to proxies and run a final sweep over them to swap in the real instance. This is done at the very end of the top-level `restore()`. The `instance` attribute below is replaced with the real instance after `__new__()` has been used to construct the object and is used when swapping proxies with real instances. """ def __init__(self): self.instance = None def get(self): return self.instance def reset(self, instance): self.instance = instance class _IDProxy(_Proxy): def __init__(self, objs, index): self._index = index self._objs = objs def get(self): return self._objs[self._index] def _obj_setattr(obj, attr, proxy): setattr(obj, attr, proxy.get()) def _obj_setvalue(obj, idx, proxy): obj[idx] = proxy.get() class Unpickler(object): def __init__(self, backend=None, keys=False, safe=False): self.backend = _make_backend(backend) self.keys = keys self.safe = safe self.reset() def reset(self): """Resets the object's internal state. """ # Map reference names to object instances self._namedict = {} # The stack of names traversed for child objects self._namestack = [] # Map of objects to their index in the _objs list self._obj_to_idx = {} self._objs = [] self._proxies = [] def restore(self, obj, reset=True): """Restores a flattened object to its original python state. Simply returns any of the basic builtin types >>> u = Unpickler() >>> u.restore('hello world') == 'hello world' True >>> u.restore({'key': 'value'}) == {'key': 'value'} True """ if reset: self.reset() value = self._restore(obj) if reset: self._swap_proxies() return value def _swap_proxies(self): """Replace proxies with their corresponding instances""" for (obj, attr, proxy, method) in self._proxies: method(obj, attr, proxy) self._proxies = [] def _restore(self, obj): if has_tag(obj, tags.B64): restore = self._restore_base64 elif has_tag(obj, tags.BYTES): # Backwards compatibility restore = self._restore_quopri elif has_tag(obj, tags.ID): restore = self._restore_id elif has_tag(obj, tags.REF): # Backwards compatibility restore = self._restore_ref elif has_tag(obj, tags.ITERATOR): restore = self._restore_iterator elif has_tag(obj, tags.TYPE): restore = self._restore_type elif has_tag(obj, tags.REPR): # Backwards compatibility restore = self._restore_repr elif has_tag(obj, tags.REDUCE): restore = self._restore_reduce elif has_tag(obj, tags.OBJECT): restore = self._restore_object elif has_tag(obj, tags.FUNCTION): restore = self._restore_function elif util.is_list(obj): restore = self._restore_list elif has_tag(obj, tags.TUPLE): restore = self._restore_tuple elif has_tag(obj, tags.SET): restore = self._restore_set elif util.is_dictionary(obj): restore = self._restore_dict else: restore = lambda x: x return restore(obj) def _restore_base64(self, obj): return base64.decodestring(obj[tags.B64].encode('utf-8')) #: For backwards compatibility with bytes data produced by older versions def _restore_quopri(self, obj): return quopri.decodestring(obj[tags.BYTES].encode('utf-8')) def _restore_iterator(self, obj): return iter(self._restore_list(obj[tags.ITERATOR])) def _restore_reduce(self, obj): """ Supports restoring with all elements of __reduce__ as per pep 307. Assumes that iterator items (the last two) are represented as lists as per pickler implementation. """ proxy = _Proxy() self._mkref(proxy) reduce_val = obj[tags.REDUCE] reduce_tuple = f, args, state, listitems, dictitems = map(self._restore, reduce_val) if f == tags.NEWOBJ or f.__name__ == '__newobj__': # mandated special case cls = args[0] if not isinstance(cls, type): cls = self._restore(cls) stage1 = cls.__new__(cls, *args[1:]) else: stage1 = f(*args) if state: try: stage1.__setstate__(state) except AttributeError: # it's fine - we'll try the prescribed default methods try: # we can't do a straight update here because we # need object identity of the state dict to be # preserved so that _swap_proxies works out for k in stage1.__dict__.keys(): if k not in state: state[k] = stage1.__dict__[k] stage1.__dict__ = state except AttributeError: # next prescribed default try: for k, v in state.items(): setattr(stage1, k, v) except: dict_state, slots_state = state if dict_state: stage1.__dict__.update(dict_state) if slots_state: for k, v in slots_state.items(): setattr(stage1, k, v) if listitems: # should be lists if not None try: stage1.extend(listitems) except AttributeError: for x in listitems: stage1.append(x) if dictitems: for k, v in dictitems: stage1.__setitem__(k, v) proxy.reset(stage1) self._swapref(proxy, stage1) return stage1 def _restore_id(self, obj): try: idx = obj[tags.ID] return self._objs[idx] except IndexError: return _IDProxy(self._objs, idx) def _restore_ref(self, obj): return self._namedict.get(obj[tags.REF]) def _restore_type(self, obj): typeref = loadclass(obj[tags.TYPE]) if typeref is None: return obj return typeref def _restore_repr(self, obj): if self.safe: # eval() is not allowed in safe mode return None obj = loadrepr(obj[tags.REPR]) return self._mkref(obj) def _restore_object(self, obj): class_name = obj[tags.OBJECT] cls = loadclass(class_name) handler = handlers.get(cls, handlers.get(class_name)) if handler is not None: # custom handler proxy = _Proxy() self._mkref(proxy) instance = handler(self).restore(obj) proxy.reset(instance) self._swapref(proxy, instance) return instance if cls is None: return self._mkref(obj) return self._restore_object_instance(obj, cls) def _restore_function(self, obj): return loadclass(obj[tags.FUNCTION]) def _loadfactory(self, obj): try: default_factory = obj['default_factory'] except KeyError: return None del obj['default_factory'] return self._restore(default_factory) def _restore_object_instance(self, obj, cls): # This is a placeholder proxy object which allows child objects to # reference the parent object before it has been instantiated. proxy = _Proxy() self._mkref(proxy) # An object can install itself as its own factory, so load the factory # after the instance is available for referencing. factory = self._loadfactory(obj) if has_tag(obj, tags.NEWARGSEX): args, kwargs = obj[tags.NEWARGSEX] else: args = getargs(obj) kwargs = {} if args: args = self._restore(args) if kwargs: kwargs = self._restore(kwargs) is_oldstyle = not (isinstance(cls, type) or getattr(cls, '__meta__', None)) try: if (not is_oldstyle) and hasattr(cls, '__new__'): # new style classes if factory: instance = cls.__new__(cls, factory, *args, **kwargs) instance.default_factory = factory else: instance = cls.__new__(cls, *args, **kwargs) else: instance = object.__new__(cls) except TypeError: # old-style classes is_oldstyle = True if is_oldstyle: try: instance = cls(*args) except TypeError: # fail gracefully try: instance = make_blank_classic(cls) except: # fail gracefully return self._mkref(obj) proxy.reset(instance) self._swapref(proxy, instance) if isinstance(instance, tuple): return instance if (hasattr(instance, 'default_factory') and isinstance(instance.default_factory, _Proxy)): instance.default_factory = instance.default_factory.get() return self._restore_object_instance_variables(obj, instance) def _restore_from_dict(self, obj, instance, ignorereserved=True): restore_key = self._restore_key_fn() method = _obj_setattr for k, v in sorted(obj.items(), key=util.itemgetter): # ignore the reserved attribute if ignorereserved and k in tags.RESERVED: continue if isinstance(k, numeric_types): str_k = unicode(k) else: str_k = k self._namestack.append(str_k) k = restore_key(k) # step into the namespace value = self._restore(v) if (util.is_noncomplex(instance) or util.is_dictionary_subclass(instance)): instance[k] = value else: setattr(instance, k, value) # This instance has an instance variable named `k` that is # currently a proxy and must be replaced if isinstance(value, _Proxy): self._proxies.append((instance, k, value, method)) # step out self._namestack.pop() def _restore_object_instance_variables(self, obj, instance): self._restore_from_dict(obj, instance) # Handle list and set subclasses if has_tag(obj, tags.SEQ): if hasattr(instance, 'append'): for v in obj[tags.SEQ]: instance.append(self._restore(v)) if hasattr(instance, 'add'): for v in obj[tags.SEQ]: instance.add(self._restore(v)) if has_tag(obj, tags.STATE): instance = self._restore_state(obj, instance) return instance def _restore_state(self, obj, instance): state = self._restore(obj[tags.STATE]) has_slots = (isinstance(state, tuple) and len(state) == 2 and isinstance(state[1], dict)) has_slots_and_dict = has_slots and isinstance(state[0], dict) if hasattr(instance, '__setstate__'): instance.__setstate__(state) elif isinstance(state, dict): # implements described default handling # of state for object with instance dict # and no slots self._restore_from_dict(state, instance, ignorereserved=False) elif has_slots: self._restore_from_dict(state[1], instance, ignorereserved=False) if has_slots_and_dict: self._restore_from_dict(state[0], instance, ignorereserved=False) elif (not hasattr(instance, '__getnewargs__') and not hasattr(instance, '__getnewargs_ex__')): # __setstate__ is not implemented so that means that the best # we can do is return the result of __getstate__() rather than # return an empty shell of an object. # However, if there were newargs, it's not an empty shell instance = state return instance def _restore_list(self, obj): parent = [] self._mkref(parent) children = [self._restore(v) for v in obj] parent.extend(children) method = _obj_setvalue proxies = [(parent, idx, value, method) for idx, value in enumerate(parent) if isinstance(value, _Proxy)] self._proxies.extend(proxies) return parent def _restore_tuple(self, obj): return tuple([self._restore(v) for v in obj[tags.TUPLE]]) def _restore_set(self, obj): return set([self._restore(v) for v in obj[tags.SET]]) def _restore_dict(self, obj): data = {} restore_key = self._restore_key_fn() for k, v in sorted(obj.items(), key=util.itemgetter): if isinstance(k, numeric_types): str_k = unicode(k) else: str_k = k self._namestack.append(str_k) k = restore_key(k) data[k] = self._restore(v) # k is currently a proxy and must be replaced if isinstance(data[k], _Proxy): self._proxies.append((data, k, data[k], _obj_setvalue)) self._namestack.pop() return data def _restore_key_fn(self): """Return a callable that restores keys This function is responsible for restoring non-string keys when we are decoding with `keys=True`. """ # This function is called before entering a tight loop # where the returned function will be called. # We return a specific function after checking self.keys # instead of doing so in the body of the function to # avoid conditional branching inside a tight loop. if self.keys: restore_key = self._restore_pickled_key else: restore_key = lambda key: key return restore_key def _restore_pickled_key(self, key): if key.startswith(tags.JSON_KEY): key = decode(key[len(tags.JSON_KEY):], backend=self.backend, context=self, keys=True, reset=False) return key def _refname(self): """Calculates the name of the current location in the JSON stack. This is called as jsonpickle traverses the object structure to create references to previously-traversed objects. This allows cyclical data structures such as doubly-linked lists. jsonpickle ensures that duplicate python references to the same object results in only a single JSON object definition and special reference tags to represent each reference. >>> u = Unpickler() >>> u._namestack = [] >>> u._refname() == '/' True >>> u._namestack = ['a'] >>> u._refname() == '/a' True >>> u._namestack = ['a', 'b'] >>> u._refname() == '/a/b' True """ return '/' + '/'.join(self._namestack) def _mkref(self, obj): obj_id = id(obj) try: self._obj_to_idx[obj_id] except KeyError: self._obj_to_idx[obj_id] = len(self._objs) self._objs.append(obj) # Backwards compatibility: old versions of jsonpickle # produced "py/ref" references. self._namedict[self._refname()] = obj return obj def _swapref(self, proxy, instance): proxy_id = id(proxy) instance_id = id(instance) instance_index = self._obj_to_idx[proxy_id] self._obj_to_idx[instance_id] = instance_index del self._obj_to_idx[proxy_id] self._objs[instance_index] = instance self._namedict[self._refname()] = instance def loadclass(module_and_name): """Loads the module and returns the class. >>> cls = loadclass('datetime.datetime') >>> cls.__name__ 'datetime' >>> loadclass('does.not.exist') >>> loadclass('__builtin__.int')() 0 """ try: module, name = module_and_name.rsplit('.', 1) module = util.untranslate_module_name(module) __import__(module) return getattr(sys.modules[module], name) except: return None def getargs(obj): """Return arguments suitable for __new__()""" # Let saved newargs take precedence over everything if has_tag(obj, tags.NEWARGSEX): raise ValueError("__newargs_ex__ returns both args and kwargs") if has_tag(obj, tags.NEWARGS): return obj[tags.NEWARGS] if has_tag(obj, tags.INITARGS): return obj[tags.INITARGS] try: seq_list = obj[tags.SEQ] obj_dict = obj[tags.OBJECT] except KeyError: return [] typeref = loadclass(obj_dict) if not typeref: return [] if hasattr(typeref, '_fields'): if len(typeref._fields) == len(seq_list): return seq_list return [] class _trivialclassic: """ A trivial class that can be instantiated with no args """ def make_blank_classic(cls): """ Implement the mandated strategy for dealing with classic classes which cannot be instantiated without __getinitargs__ because they take parameters """ instance = _trivialclassic() instance.__class__ = cls return instance def loadrepr(reprstr): """Returns an instance of the object from the object's repr() string. It involves the dynamic specification of code. >>> obj = loadrepr('datetime/datetime.datetime.now()') >>> obj.__class__.__name__ 'datetime' """ module, evalstr = reprstr.split('/') mylocals = locals() localname = module if '.' in localname: localname = module.split('.', 1)[0] mylocals[localname] = __import__(module) return eval(evalstr) def has_tag(obj, tag): """Helper class that tests to see if the obj is a dictionary and contains a particular key/tag. >>> obj = {'test': 1} >>> has_tag(obj, 'test') True >>> has_tag(obj, 'fail') False >>> has_tag(42, 'fail') False """ return type(obj) is dict and tag in obj jsonpickle-0.9.5/jsonpickle/tags.py0000644000175000017500000000200013067700544017535 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- """The jsonpickle.tags module provides the custom tags used for pickling and unpickling Python objects. These tags are keys into the flattened dictionaries created by the Pickler class. The Unpickler uses these custom key names to identify dictionaries that need to be specially handled. """ from __future__ import absolute_import, division, unicode_literals from .compat import set BYTES = 'py/bytes' B64 = 'py/b64' FUNCTION = 'py/function' ID = 'py/id' INITARGS = 'py/initargs' ITERATOR = 'py/iterator' JSON_KEY = 'json://' NEWARGS = 'py/newargs' NEWARGSEX = 'py/newargsex' NEWOBJ = 'py/newobj' OBJECT = 'py/object' REDUCE = 'py/reduce' REF = 'py/ref' REPR = 'py/repr' SEQ = 'py/seq' SET = 'py/set' STATE = 'py/state' TUPLE = 'py/tuple' TYPE = 'py/type' # All reserved tag names RESERVED = set([ BYTES, FUNCTION, ID, INITARGS, ITERATOR, NEWARGS, NEWARGSEX, NEWOBJ, OBJECT, REDUCE, REF, REPR, SEQ, SET, STATE, TUPLE, TYPE, ]) jsonpickle-0.9.5/jsonpickle/__init__.py0000644000175000017500000001175313070273753020356 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # Copyright (C) 2009, 2011, 2013 David Aguilar (davvid -at- gmail.com) # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. """Python library for serializing any arbitrary object graph into JSON. jsonpickle can take almost any Python object and turn the object into JSON. Additionally, it can reconstitute the object back into Python. The object must be accessible globally via a module and must inherit from object (AKA new-style classes). Create an object:: class Thing(object): def __init__(self, name): self.name = name obj = Thing('Awesome') Use jsonpickle to transform the object into a JSON string:: import jsonpickle frozen = jsonpickle.encode(obj) Use jsonpickle to recreate a Python object from a JSON string:: thawed = jsonpickle.decode(frozen) .. warning:: Loading a JSON string from an untrusted source represents a potential security vulnerability. jsonpickle makes no attempt to sanitize the input. The new object has the same type and data, but essentially is now a copy of the original. .. code-block:: python assert obj.name == thawed.name If you will never need to load (regenerate the Python class from JSON), you can pass in the keyword unpicklable=False to prevent extra information from being added to JSON:: oneway = jsonpickle.encode(obj, unpicklable=False) result = jsonpickle.decode(oneway) assert obj.name == result['name'] == 'Awesome' """ from __future__ import absolute_import, division, unicode_literals from . import pickler from . import unpickler from .backend import JSONBackend from .version import VERSION # ensure built-in handlers are loaded from . import handlers as _ __all__ = ('encode', 'decode') __version__ = VERSION json = JSONBackend() # Export specific JSONPluginMgr methods into the jsonpickle namespace set_preferred_backend = json.set_preferred_backend set_decoder_options = json.set_decoder_options set_encoder_options = json.set_encoder_options load_backend = json.load_backend remove_backend = json.remove_backend enable_fallthrough = json.enable_fallthrough def encode(value, unpicklable=True, make_refs=True, keys=False, max_depth=None, backend=None, warn=False, max_iter=None, numeric_keys=False): """Return a JSON formatted representation of value, a Python object. :param unpicklable: If set to False then the output will not contain the information necessary to turn the JSON data back into Python objects, but a simpler JSON stream is produced. :param max_depth: If set to a non-negative integer then jsonpickle will not recurse deeper than 'max_depth' steps into the object. Anything deeper than 'max_depth' is represented using a Python repr() of the object. :param make_refs: If set to False jsonpickle's referencing support is disabled. Objects that are id()-identical won't be preserved across encode()/decode(), but the resulting JSON stream will be conceptually simpler. jsonpickle detects cyclical objects and will break the cycle by calling repr() instead of recursing when make_refs is set False. :param keys: If set to True then jsonpickle will encode non-string dictionary keys instead of coercing them into strings via `repr()`. :param warn: If set to True then jsonpickle will warn when it returns None for an object which it cannot pickle (e.g. file descriptors). :param max_iter: If set to a non-negative integer then jsonpickle will consume at most `max_iter` items when pickling iterators. >>> encode('my string') == '"my string"' True >>> encode(36) == '36' True >>> encode({'foo': True}) == '{"foo": true}' True >>> encode({'foo': True}, max_depth=1) == '{"foo": "True"}' True """ if backend is None: backend = json return pickler.encode(value, backend=backend, unpicklable=unpicklable, make_refs=make_refs, keys=keys, max_depth=max_depth, warn=warn, max_iter=max_iter, numeric_keys=numeric_keys) def decode(string, backend=None, keys=False): """Convert a JSON string into a Python object. The keyword argument 'keys' defaults to False. If set to True then jsonpickle will decode non-string dictionary keys into python objects via the jsonpickle protocol. >>> str(decode('"my string"')) 'my string' >>> decode('36') 36 """ if backend is None: backend = json return unpickler.decode(string, backend=backend, keys=keys) # json.load(),loads(), dump(), dumps() compatibility dumps = encode loads = decode jsonpickle-0.9.5/jsonpickle/compat.py0000644000175000017500000000141413067700415020067 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- from __future__ import absolute_import, division, unicode_literals import sys __all__ = ('bytes', 'set', 'unicode', 'long', 'unichr', 'queue') PY_MAJOR = sys.version_info[0] PY_MINOR = sys.version_info[1] PY2 = PY_MAJOR == 2 PY3 = PY_MAJOR == 3 PY32 = PY3 and PY_MINOR == 2 try: bytes = bytes except NameError: bytes = str try: set = set except NameError: from sets import Set as set set = set try: unicode = unicode except NameError: unicode = str try: long = long numeric_types = (int, long) except NameError: long = int numeric_types = (int,) try: unichr = unichr except NameError: unichr = chr try: # Python3 import queue except ImportError: # Python2 import Queue as queue jsonpickle-0.9.5/jsonpickle/util.py0000644000175000017500000003224113075310375017564 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- # # Copyright (C) 2008 John Paulett (john -at- paulett.org) # All rights reserved. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. """Helper functions for pickling and unpickling. Most functions assist in determining the type of an object. """ from __future__ import absolute_import, division, unicode_literals import base64 import collections import io import operator import time import types import inspect from . import tags from .compat import set, unicode, long, bytes, PY3 if not PY3: import __builtin__ SEQUENCES = (list, set, tuple) SEQUENCES_SET = set(SEQUENCES) PRIMITIVES = set((unicode, bool, float, int, long)) def is_type(obj): """Returns True is obj is a reference to a type. >>> is_type(1) False >>> is_type(object) True >>> class Klass: pass >>> is_type(Klass) True """ # use "isinstance" and not "is" to allow for metaclasses if PY3: return isinstance(obj, type) else: return isinstance(obj, (type, types.ClassType)) def has_method(obj, name): # false if attribute doesn't exist if not hasattr(obj, name): return False func = getattr(obj, name) # builtin descriptors like __getnewargs__ if isinstance(func, types.BuiltinMethodType): return True # note that FunctionType has a different meaning in py2/py3 if not isinstance(func, (types.MethodType, types.FunctionType)): return False # need to go through __dict__'s since in py3 methods are essentially descriptors base_type = obj if is_type(obj) else obj.__class__ # __class__ for old-style classes original = None for subtype in inspect.getmro(base_type): # there is no .mro() for old-style classes original = vars(subtype).get(name) if original is not None: break # name not found in the mro if original is None: return False # static methods are always fine if isinstance(original, staticmethod): return True # at this point, the method has to be an instancemthod or a classmethod self_attr = '__self__' if PY3 else 'im_self' if not hasattr(func, self_attr): return False bound_to = getattr(func, self_attr) # class methods if isinstance(original, classmethod): return issubclass(base_type, bound_to) # bound methods return isinstance(obj, type(bound_to)) def is_object(obj): """Returns True is obj is a reference to an object instance. >>> is_object(1) True >>> is_object(object()) True >>> is_object(lambda x: 1) False """ return (isinstance(obj, object) and not isinstance(obj, (type, types.FunctionType))) def is_primitive(obj): """Helper method to see if the object is a basic data type. Unicode strings, integers, longs, floats, booleans, and None are considered primitive and will return True when passed into *is_primitive()* >>> is_primitive(3) True >>> is_primitive([4,4]) False """ if obj is None: return True elif type(obj) in PRIMITIVES: return True return False def is_dictionary(obj): """Helper method for testing if the object is a dictionary. >>> is_dictionary({'key':'value'}) True """ return type(obj) is dict def is_sequence(obj): """Helper method to see if the object is a sequence (list, set, or tuple). >>> is_sequence([4]) True """ return type(obj) in SEQUENCES_SET def is_list(obj): """Helper method to see if the object is a Python list. >>> is_list([4]) True """ return type(obj) is list def is_set(obj): """Helper method to see if the object is a Python set. >>> is_set(set()) True """ return type(obj) is set def is_bytes(obj): """Helper method to see if the object is a bytestring. >>> is_bytes(b'foo') True """ return type(obj) is bytes def is_unicode(obj): """Helper method to see if the object is a unicode string""" return type(obj) is unicode def is_tuple(obj): """Helper method to see if the object is a Python tuple. >>> is_tuple((1,)) True """ return type(obj) is tuple def is_dictionary_subclass(obj): """Returns True if *obj* is a subclass of the dict type. *obj* must be a subclass and not the actual builtin dict. >>> class Temp(dict): pass >>> is_dictionary_subclass(Temp()) True """ # TODO: add UserDict return (hasattr(obj, '__class__') and issubclass(obj.__class__, dict) and not is_dictionary(obj)) def is_sequence_subclass(obj): """Returns True if *obj* is a subclass of list, set or tuple. *obj* must be a subclass and not the actual builtin, such as list, set, tuple, etc.. >>> class Temp(list): pass >>> is_sequence_subclass(Temp()) True """ return (hasattr(obj, '__class__') and (issubclass(obj.__class__, SEQUENCES) or is_list_like(obj)) and not is_sequence(obj)) def is_noncomplex(obj): """Returns True if *obj* is a special (weird) class, that is more complex than primitive data types, but is not a full object. Including: * :class:`~time.struct_time` """ if type(obj) is time.struct_time: return True return False def is_function(obj): """Returns true if passed a function >>> is_function(lambda x: 1) True >>> is_function(locals) True >>> def method(): pass >>> is_function(method) True >>> is_function(1) False """ if type(obj) in (types.FunctionType, types.MethodType, types.LambdaType, types.BuiltinFunctionType, types.BuiltinMethodType): return True if not hasattr(obj, '__class__'): return False module = translate_module_name(obj.__class__.__module__) name = obj.__class__.__name__ return (module == '__builtin__' and name in ('function', 'builtin_function_or_method', 'instancemethod', 'method-wrapper')) def is_module_function(obj): """Return True if `obj` is a module-global function >>> import os >>> is_module_function(os.path.exists) True >>> is_module_function(lambda: None) False """ return (hasattr(obj, '__class__') and isinstance(obj, types.FunctionType) and hasattr(obj, '__module__') and hasattr(obj, '__name__') and obj.__name__ != '') def is_module(obj): """Returns True if passed a module >>> import os >>> is_module(os) True """ return isinstance(obj, types.ModuleType) def is_picklable(name, value): """Return True if an object can be pickled >>> import os >>> is_picklable('os', os) True >>> def foo(): pass >>> is_picklable('foo', foo) True >>> is_picklable('foo', lambda: None) False """ if name in tags.RESERVED: return False return is_module_function(value) or not is_function(value) def is_installed(module): """Tests to see if ``module`` is available on the sys.path >>> is_installed('sys') True >>> is_installed('hopefullythisisnotarealmodule') False """ try: __import__(module) return True except ImportError: return False def is_list_like(obj): return hasattr(obj, '__getitem__') and hasattr(obj, 'append') def is_iterator(obj): is_file = False if not PY3: is_file = isinstance(obj, __builtin__.file) return (isinstance(obj, collections.Iterator) and not isinstance(obj, io.IOBase) and not is_file) def is_collections(obj): try: return type(obj).__module__ == 'collections' except: return False IteratorType = type(iter('')) def is_reducible(obj): """ Returns false if of a type which have special casing, and should not have their __reduce__ methods used """ # defaultdicts may contain functions which we cannot serialise if is_collections(obj) and not isinstance(obj, collections.defaultdict): return True return (not (is_list(obj) or is_list_like(obj) or is_primitive(obj) or is_bytes(obj) or is_unicode(obj) or is_dictionary(obj) or is_sequence(obj) or is_set(obj) or is_tuple(obj) or is_dictionary_subclass(obj) or is_sequence_subclass(obj) or is_function(obj) or is_module(obj) or is_iterator(obj) or type(getattr(obj, '__slots__', None)) is IteratorType or type(obj) is object or obj is object or (is_type(obj) and obj.__module__ == 'datetime') )) def in_dict(obj, key, default=False): """ Returns true if key exists in obj.__dict__; false if not in. If obj.__dict__ is absent, return default """ return (key in obj.__dict__) if getattr(obj, '__dict__', None) else default def in_slots(obj, key, default=False): """ Returns true if key exists in obj.__slots__; false if not in. If obj.__slots__ is absent, return default """ return (key in obj.__slots__) if getattr(obj, '__slots__', None) else default def has_reduce(obj): """ Tests if __reduce__ or __reduce_ex__ exists in the object dict or in the class dicts of every class in the MRO *except object*. Returns a tuple of booleans (has_reduce, has_reduce_ex) """ if not is_reducible(obj) or is_type(obj): return (False, False) # in this case, reduce works and is desired # notwithstanding depending on default object # reduce if is_noncomplex(obj): return (False, True) has_reduce = False has_reduce_ex = False REDUCE = '__reduce__' REDUCE_EX = '__reduce_ex__' # For object instance has_reduce = in_dict(obj, REDUCE) or in_slots(obj, REDUCE) has_reduce_ex = in_dict(obj, REDUCE_EX) or in_slots(obj, REDUCE_EX) # turn to the MRO for base in type(obj).__mro__: if is_reducible(base): has_reduce = has_reduce or in_dict(base, REDUCE) has_reduce_ex = has_reduce_ex or in_dict(base, REDUCE_EX) if has_reduce and has_reduce_ex: return (has_reduce, has_reduce_ex) # for things that don't have a proper dict but can be getattred (rare, but includes some # builtins) cls = type(obj) object_reduce = getattr(object, REDUCE) object_reduce_ex = getattr(object, REDUCE_EX) if not has_reduce: has_reduce_cls = getattr(cls, REDUCE, False) if not has_reduce_cls is object_reduce: has_reduce = has_reduce_cls if not has_reduce_ex: has_reduce_ex_cls = getattr(cls, REDUCE_EX, False) if not has_reduce_ex_cls is object_reduce_ex: has_reduce_ex = has_reduce_ex_cls return (has_reduce, has_reduce_ex) def translate_module_name(module): """Rename builtin modules to a consistent (Python2) module name This is used so that references to Python's `builtins` module can be loaded in both Python 2 and 3. We remap to the "__builtin__" name and unmap it when importing. See untranslate_module_name() for the reverse operation. """ if (PY3 and module == 'builtins') or module == 'exceptions': # We map the Python2 `exceptions` module to `__builtin__` because # `__builtin__` is a superset and contains everything that is # available in `exceptions`, which makes the translation simpler. return '__builtin__' else: return module def untranslate_module_name(module): """Rename module names mention in JSON to names that we can import This reverses the translation applied by translate_module_name() to a module name available to the current version of Python. """ if PY3: # remap `__builtin__` and `exceptions` to the `builtins` module if module == '__builtin__': module = 'builtins' elif module == 'exceptions': module = 'builtins' return module def importable_name(cls): """ >>> class Example(object): ... pass >>> ex = Example() >>> importable_name(ex.__class__) == 'jsonpickle.util.Example' True >>> importable_name(type(25)) == '__builtin__.int' True >>> importable_name(None.__class__) == '__builtin__.NoneType' True >>> importable_name(False.__class__) == '__builtin__.bool' True >>> importable_name(AttributeError) == '__builtin__.AttributeError' True """ name = cls.__name__ module = translate_module_name(cls.__module__) return '%s.%s' % (module, name) def b64encode(data): payload = base64.b64encode(data) if PY3 and type(payload) is bytes: payload = payload.decode('ascii') return payload def b64decode(payload): if PY3 and type(payload) is not bytes: payload = bytes(payload, 'ascii') return base64.b64decode(payload) def itemgetter(obj, getter=operator.itemgetter(0)): return unicode(getter(obj)) jsonpickle-0.9.5/jsonpickle/version.py0000644000175000017500000000002213133025420020251 0ustar daviddavid00000000000000VERSION = '0.9.5' jsonpickle-0.9.5/jsonpickle/backend.py0000644000175000017500000002307113067700337020201 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- from __future__ import absolute_import, division, unicode_literals from .compat import PY32 from .compat import unicode class JSONBackend(object): """Manages encoding and decoding using various backends. It tries these modules in this order: simplejson, json, demjson simplejson is a fast and popular backend and is tried first. json comes with python2.6 and is tried second. demjson is the most permissive backend and is tried last. """ def __init__(self, fallthrough=True): # Whether we should fallthrough to the next backend self._fallthrough = fallthrough # The names of backends that have been successfully imported self._backend_names = [] # A dictionary mapping backend names to encode/decode functions self._encoders = {} self._decoders = {} # Options to pass to specific encoders json_opts = ((), {'sort_keys': True}) self._encoder_options = { 'json': json_opts, 'simplejson': json_opts, 'django.util.simplejson': json_opts, } # Options to pass to specific encoders self._decoder_options = {} # The exception class that is thrown when a decoding error occurs self._decoder_exceptions = {} # Whether we've loaded any backends successfully self._verified = False if not PY32: self.load_backend('simplejson') self.load_backend('json') self.load_backend('demjson', 'encode', 'decode', 'JSONDecodeError') self.load_backend('jsonlib', 'write', 'read', 'ReadError') self.load_backend('yajl') self.load_backend('ujson') def _verify(self): """Ensures that we've loaded at least one JSON backend.""" if self._verified: return raise AssertionError('jsonpickle requires at least one of the ' 'following:\n' ' python2.6, simplejson, or demjson') def enable_fallthrough(self, enable): """ Disable jsonpickle's fallthrough-on-error behavior By default, jsonpickle tries the next backend when decoding or encoding using a backend fails. This can make it difficult to force jsonpickle to use a specific backend, and catch errors, because the error will be suppressed and may not be raised by the subsequent backend. Calling `enable_backend(False)` will make jsonpickle immediately re-raise any exceptions raised by the backends. """ self._fallthrough = enable def load_backend(self, name, dumps='dumps', loads='loads', loads_exc=ValueError): """Load a JSON backend by name. This method loads a backend and sets up references to that backend's loads/dumps functions and exception classes. :param dumps: is the name of the backend's encode method. The method should take an object and return a string. Defaults to 'dumps'. :param loads: names the backend's method for the reverse operation -- returning a Python object from a string. :param loads_exc: can be either the name of the exception class used to denote decoding errors, or it can be a direct reference to the appropriate exception class itself. If it is a name, then the assumption is that an exception class of that name can be found in the backend module's namespace. :param load: names the backend's 'load' method. :param dump: names the backend's 'dump' method. :rtype bool: True on success, False if the backend could not be loaded. """ try: # Load the JSON backend mod = __import__(name) except ImportError: return False # Handle submodules, e.g. django.utils.simplejson try: for attr in name.split('.')[1:]: mod = getattr(mod, attr) except AttributeError: return False if (not self._store(self._encoders, name, mod, dumps) or not self._store(self._decoders, name, mod, loads)): return False if isinstance(loads_exc, (str, unicode)): # This backend's decoder exception is part of the backend if not self._store(self._decoder_exceptions, name, mod, loads_exc): return False else: # simplejson uses ValueError self._decoder_exceptions[name] = loads_exc # Setup the default args and kwargs for this encoder/decoder self._encoder_options[name] = ([], {}) self._decoder_options[name] = ([], {}) # Add this backend to the list of candidate backends self._backend_names.append(name) # Indicate that we successfully loaded a JSON backend self._verified = True return True def remove_backend(self, name): """Remove all entries for a particular backend.""" self._encoders.pop(name, None) self._decoders.pop(name, None) self._decoder_exceptions.pop(name, None) self._decoder_options.pop(name, None) self._encoder_options.pop(name, None) if name in self._backend_names: self._backend_names.remove(name) self._verified = bool(self._backend_names) def encode(self, obj): """ Attempt to encode an object into JSON. This tries the loaded backends in order and passes along the last exception if no backend is able to encode the object. """ self._verify() if not self._fallthrough: name = self._backend_names[0] return self.backend_encode(name, obj) for idx, name in enumerate(self._backend_names): try: return self.backend_encode(name, obj) except Exception as e: if idx == len(self._backend_names) - 1: raise e # def dumps dumps = encode def backend_encode(self, name, obj): optargs, optkwargs = self._encoder_options[name] encoder_kwargs = optkwargs.copy() encoder_args = (obj,) + tuple(optargs) return self._encoders[name](*encoder_args, **encoder_kwargs) def decode(self, string): """ Attempt to decode an object from a JSON string. This tries the loaded backends in order and passes along the last exception if no backends are able to decode the string. """ self._verify() if not self._fallthrough: name = self._backend_names[0] return self.backend_decode(name, string) for idx, name in enumerate(self._backend_names): try: return self.backend_decode(name, string) except self._decoder_exceptions[name] as e: if idx == len(self._backend_names) - 1: raise e else: pass # and try a more forgiving encoder, e.g. demjson # def loads loads = decode def backend_decode(self, name, string): optargs, optkwargs = self._decoder_options.get(name, ((), {})) decoder_kwargs = optkwargs.copy() return self._decoders[name](string, *optargs, **decoder_kwargs) def set_preferred_backend(self, name): """ Set the preferred json backend. If a preferred backend is set then jsonpickle tries to use it before any other backend. For example:: set_preferred_backend('simplejson') If the backend is not one of the built-in jsonpickle backends (json/simplejson, or demjson) then you must load the backend prior to calling set_preferred_backend. AssertionError is raised if the backend has not been loaded. """ if name in self._backend_names: self._backend_names.remove(name) self._backend_names.insert(0, name) else: errmsg = 'The "%s" backend has not been loaded.' % name raise AssertionError(errmsg) def set_encoder_options(self, name, *args, **kwargs): """ Associate encoder-specific options with an encoder. After calling set_encoder_options, any calls to jsonpickle's encode method will pass the supplied args and kwargs along to the appropriate backend's encode method. For example:: set_encoder_options('simplejson', sort_keys=True, indent=4) set_encoder_options('demjson', compactly=False) See the appropriate encoder's documentation for details about the supported arguments and keyword arguments. """ self._encoder_options[name] = (args, kwargs) def set_decoder_options(self, name, *args, **kwargs): """ Associate decoder-specific options with a decoder. After calling set_decoder_options, any calls to jsonpickle's decode method will pass the supplied args and kwargs along to the appropriate backend's decode method. For example:: set_decoder_options('simplejson', encoding='utf8', cls=JSONDecoder) set_decoder_options('demjson', strict=True) See the appropriate decoder's documentation for details about the supported arguments and keyword arguments. """ self._decoder_options[name] = (args, kwargs) def _store(self, dct, backend, obj, name): try: dct[backend] = getattr(obj, name) except AttributeError: self.remove_backend(backend) return False return True jsonpickle-0.9.5/jsonpickle/ext/0000755000175000017500000000000013133025657017034 5ustar daviddavid00000000000000jsonpickle-0.9.5/jsonpickle/ext/__init__.py0000664000175000017500000000000012454145546021142 0ustar daviddavid00000000000000jsonpickle-0.9.5/jsonpickle/ext/numpy.py0000644000175000017500000002424013067700157020561 0ustar daviddavid00000000000000# -*- coding: utf-8 -*- from __future__ import absolute_import import sys import zlib import warnings import numpy as np import ast from ..handlers import BaseHandler, register, unregister from ..compat import unicode from ..util import b64decode, b64encode __all__ = ['register_handlers', 'unregister_handlers'] native_byteorder = '<' if sys.byteorder == 'little' else '>' def get_byteorder(arr): """translate equals sign to native order""" byteorder = arr.dtype.byteorder return native_byteorder if byteorder == '=' else byteorder class NumpyBaseHandler(BaseHandler): def flatten_dtype(self, dtype, data): if hasattr(dtype, 'tostring'): data['dtype'] = dtype.tostring() else: dtype = unicode(dtype) prefix = '(numpy.record, ' if dtype.startswith(prefix): dtype = dtype[len(prefix):-1] data['dtype'] = dtype def restore_dtype(self, data): dtype = data['dtype'] if dtype.startswith(('{', '[')): dtype = ast.literal_eval(dtype) return np.dtype(dtype) class NumpyDTypeHandler(NumpyBaseHandler): def flatten(self, obj, data): self.flatten_dtype(obj, data) return data def restore(self, data): return self.restore_dtype(data) class NumpyGenericHandler(NumpyBaseHandler): def flatten(self, obj, data): self.flatten_dtype(obj.dtype.newbyteorder('N'), data) data['value'] = self.context.flatten(obj.tolist(), reset=False) return data def restore(self, data): value = self.context.restore(data['value'], reset=False) return self.restore_dtype(data).type(value) class NumpyNDArrayHandler(NumpyBaseHandler): """Stores arrays as text representation, without regard for views """ def flatten_flags(self, obj, data): if obj.flags.writeable is False: data['writeable'] = False def restore_flags(self, data, arr): if not data.get('writeable', True): arr.flags.writeable = False def flatten(self, obj, data): self.flatten_dtype(obj.dtype.newbyteorder('N'), data) self.flatten_flags(obj, data) data['values'] = self.context.flatten(obj.tolist(), reset=False) if 0 in obj.shape: # add shape information explicitly as it cannot be inferred from an empty list data['shape'] = obj.shape return data def restore(self, data): values = self.context.restore(data['values'], reset=False) arr = np.array( values, dtype=self.restore_dtype(data), order=data.get('order', 'C') ) shape = data.get('shape', None) if shape is not None: arr = arr.reshape(shape) self.restore_flags(data, arr) return arr class NumpyNDArrayHandlerBinary(NumpyNDArrayHandler): """stores arrays with size greater than 'size_treshold' as (optionally) compressed base64 Notes ----- This would be easier to implement using np.save/np.load, but that would be less language-agnostic """ def __init__(self, size_treshold=16, compression=zlib): """ :param size_treshold: nonnegative int or None valid values for 'size_treshold' are all nonnegative integers and None if size_treshold is None, values are always stored as nested lists :param compression: a compression module or None valid values for 'compression' are {zlib, bz2, None} if compresion is None, no compression is applied """ self.size_treshold = size_treshold self.compression = compression def flatten_byteorder(self, obj, data): byteorder = obj.dtype.byteorder if byteorder != '|': data['byteorder'] = get_byteorder(obj) def restore_byteorder(self, data, arr): byteorder = data.get('byteorder', None) if byteorder: arr.dtype = arr.dtype.newbyteorder(byteorder) def flatten(self, obj, data): """encode numpy to json""" if self.size_treshold >= obj.size or self.size_treshold is None: # encode as text data = super(NumpyNDArrayHandlerBinary, self).flatten(obj, data) else: # encode as binary buffer = obj.tobytes(order='a') # numpy docstring is lacking as of 1.11.2, but this is the option we need if self.compression: buffer = self.compression.compress(buffer) data['values'] = b64encode(buffer) data['shape'] = obj.shape self.flatten_dtype(obj.dtype.newbyteorder('N'), data) self.flatten_byteorder(obj, data) self.flatten_flags(obj, data) if not obj.flags.c_contiguous: data['order'] = 'F' return data def restore(self, data): """decode numpy from json""" values = data['values'] if isinstance(values, list): # decode text representation arr = super(NumpyNDArrayHandlerBinary, self).restore(data) else: # decode binary representation buffer = b64decode(values) if self.compression: buffer = self.compression.decompress(buffer) arr = np.ndarray( buffer=buffer, dtype=self.restore_dtype(data), shape=data.get('shape'), order=data.get('order', 'C') ).copy() # make a copy, to force the result to own the data self.restore_byteorder(data, arr) self.restore_flags(data, arr) return arr class NumpyNDArrayHandlerView(NumpyNDArrayHandlerBinary): """Pickles references inside ndarrays, or array-views Notes ----- The current implementation has some restrictions. 'base' arrays, or arrays which are viewed by other arrays, must be f-or-c-contiguous. This is not such a large restriction in practice, because all numpy array creation is c-contiguous by default. Relaxing this restriction would be nice though; especially if it can be done without bloating the design too much. Furthermore, ndarrays which are views of array-like objects implementing __array_interface__, but which are not themselves nd-arrays, are deepcopied with a warning (by default), as we cannot guarantee whatever custom logic such classes implement is correctly reproduced. """ def __init__(self, mode='warn', size_treshold=16, compression=zlib): """ :param mode: {'warn', 'raise', 'ignore'} How to react when encountering array-like objects whos references we cannot safely serialize :param size_treshold: nonnegative int or None valid values for 'size_treshold' are all nonnegative integers and None if size_treshold is None, values are always stored as nested lists :param compression: a compression module or None valid values for 'compression' are {zlib, bz2, None} if compresion is None, no compression is applied """ super(NumpyNDArrayHandlerView, self).__init__(size_treshold, compression) self.mode = mode def flatten(self, obj, data): """encode numpy to json""" base = obj.base if base is None and obj.flags.forc: # store by value data = super(NumpyNDArrayHandlerView, self).flatten(obj, data) # ensure that views on arrays stored as text are interpreted correctly if not obj.flags.c_contiguous: data['order'] = 'F' elif isinstance(base, np.ndarray) and base.flags.forc: # store by reference data['base'] = self.context.flatten(base, reset=False) offset = obj.ctypes.data - base.ctypes.data if offset: data['offset'] = offset if not obj.flags.c_contiguous: data['strides'] = obj.strides data['shape'] = obj.shape self.flatten_dtype(obj.dtype.newbyteorder('N'), data) self.flatten_flags(obj, data) if get_byteorder(obj) != '|': byteorder = 'S' if get_byteorder(obj) != get_byteorder(base) else None if byteorder: data['byteorder'] = byteorder if self.size_treshold >= obj.size: # not used in restore since base is present, but include values for human-readability super(NumpyNDArrayHandlerBinary, self).flatten(obj, data) else: # store a deepcopy or fail if self.mode == 'warn': msg = "ndarray is defined by reference to an object we do not know how to serialize. " \ "A deep copy is serialized instead, breaking memory aliasing." warnings.warn(msg) elif self.mode == 'raise': msg = "ndarray is defined by reference to an object we do not know how to serialize." raise ValueError(msg) data = super(NumpyNDArrayHandlerView, self).flatten(obj.copy(), data) return data def restore(self, data): """decode numpy from json""" base = data.get('base', None) if base is None: # decode array with owndata=True arr = super(NumpyNDArrayHandlerView, self).restore(data) else: # decode array view, which references the data of another array base = self.context.restore(base, reset=False) assert base.flags.forc, \ "Current implementation assumes base is C or F contiguous" arr = np.ndarray( buffer=base.data, dtype=self.restore_dtype(data).newbyteorder(data.get('byteorder', '|')), shape=data.get('shape'), offset=data.get('offset', 0), strides=data.get('strides', None) ) self.restore_flags(data, arr) return arr def register_handlers(): register(np.dtype, NumpyDTypeHandler, base=True) register(np.generic, NumpyGenericHandler, base=True) register(np.ndarray, NumpyNDArrayHandlerView(), base=True) def unregister_handlers(): unregister(np.dtype) unregister(np.generic) unregister(np.ndarray)