pax_global_header00006660000000000000000000000064144701276030014516gustar00rootroot0000000000000052 comment=c0ffee7cb53081b1d430e58f070bca3423c15fbf pyjson_tricks-3.17.3/000077500000000000000000000000001447012760300145125ustar00rootroot00000000000000pyjson_tricks-3.17.3/.github/000077500000000000000000000000001447012760300160525ustar00rootroot00000000000000pyjson_tricks-3.17.3/.github/workflows/000077500000000000000000000000001447012760300201075ustar00rootroot00000000000000pyjson_tricks-3.17.3/.github/workflows/tests.yml000066400000000000000000000043141447012760300217760ustar00rootroot00000000000000 name: 'pyjson-tricks' on: push: jobs: build: name: tests runs-on: ubuntu-latest strategy: max-parallel: 8 fail-fast: false matrix: libraries: [ 'vanilla', 'tz', 'path', 'numpy', 'pandas', 'all' ] python-version: [ '3.7', '3.8', '3.9', '3.10', '3.11' ] steps: - uses: actions/checkout@v1 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - name: Install dependencies run: | python -m pip install --upgrade pip pip install pytest if [ "${{ matrix.python-version }}" == "2.7" ] ; then pip install enum34 fi export LIBS="${{ matrix.libraries }}" if [ "$LIBS" == "tz" ] || [ "$LIBS" == "all" ] ; then pip install pytz fi if [ "$LIBS" == "path" ] || [ "$LIBS" == "all" ] ; then pip install pathlib fi if [ "$LIBS" == "numpy" ] || [ "$LIBS" == "all" ] ; then pip install numpy fi if [ "$LIBS" == "pandas" ] || [ "$LIBS" == "all" ] ; then pip install pandas fi - name: Run tests run: | python --version PYTEST_ARGS='-v --strict tests/test_bare.py tests/test_class.py tests/test_meta.py tests/test_enum.py' export LIBS="${{ matrix.libraries }}" if [ "$LIBS" == "vanilla" ] ; then py.test $PYTEST_ARGS elif [ "$LIBS" == "tz" ] ; then py.test $PYTEST_ARGS tests/test_tz.py elif [ "$LIBS" == "path" ] ; then py.test $PYTEST_ARGS tests/test_pathlib.py elif [ "$LIBS" == "numpy" ] ; then py.test $PYTEST_ARGS tests/test_np.py elif [ "$LIBS" == "pandas" ] ; then py.test $PYTEST_ARGS tests/test_pandas.py elif [ "$LIBS" == "all" ] ; then py.test -v --strict else echo "UNKNOWN LIBRARY '$LIBS'" exit 1 fi pyjson_tricks-3.17.3/.gitignore000066400000000000000000000001571447012760300165050ustar00rootroot00000000000000*~ *.pyc *.egg-info *.swp /.idea/ *.sqlite3 /.pypirc /dist/ /build/ /MANIFEST /.cache/ /try.py .pytest_cache/ pyjson_tricks-3.17.3/CODE_OF_CONDUCT.md000066400000000000000000000004231447012760300173100ustar00rootroot00000000000000# Code of conduct **Just be nice.** Saturday Morning Breakfast Serial summarized it well: [![image](https://www.smbc-comics.com/comics/20090727.gif)](https://www.smbc-comics.com/comic/2009-07-27) I do not expect that the type of people that behaves badly will read this. pyjson_tricks-3.17.3/CONTRIBUTING.md000066400000000000000000000023041447012760300167420ustar00rootroot00000000000000# Contributing Contributions are very welcome! Bug reports, feature suggestions and code contributions help this project become more useful for everyone! Contributing can be done through: * Reporting bugs ([issues](https://github.com/mverleg/pyjson_tricks/issues)) * Suggesting features ([issues](https://github.com/mverleg/pyjson_tricks/issues)) * Fixing bugs ([pull request](https://github.com/mverleg/pyjson_tricks/pulls)) * Implementing features ([pull request](https://github.com/mverleg/pyjson_tricks/pulls)) * Reviewing a [pull request](https://github.com/mverleg/pyjson_tricks/pulls) * Telling a friend :) There are only few things to keep in mind: * Your contributions become [BSD-licensed](https://github.com/mverleg/pyjson_tricks/blob/master/LICENSE.txt) * Discuss features in an issue before contributing code. * Automated tests are required to go live, not necessarily to commit. * Try to follow the [code conventions](https://www.python.org/dev/peps/pep-0008/). * [Be nice](https://github.com/mverleg/pyjson_tricks/blob/master/CODE_OF_CONDUCT.rst). Feel free to add yourself in the README if you send a pull request! Either at the specific feature ("thanks to NAME"), or in "Usage & contributions". pyjson_tricks-3.17.3/LICENSE.txt000066400000000000000000000027351447012760300163440ustar00rootroot00000000000000LICENSE: BSD-3-Clause Copyright (c) 2023 Mark V. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. pyjson_tricks-3.17.3/README.md000066400000000000000000000360621447012760300160000ustar00rootroot00000000000000# JSON tricks (python) The [pyjson-tricks] package brings several pieces of functionality to python handling of json files: 1. **Store and load numpy arrays** in human-readable format. 2. **Store and load class instances** both generic and customized. 3. **Store and load date/times** as a dictionary (including timezone). 4. **Preserve map order** `{}` using `OrderedDict`. 5. **Allow for comments** in json files by starting lines with `#`. 6. Sets, complex numbers, Decimal, Fraction, enums, compression, duplicate keys, pathlib Paths, bytes ... As well as compression and disallowing duplicate keys. * Code: * Documentation: * PIP: Several keys of the format `__keyname__` have special meanings, and more might be added in future releases. If you're considering JSON-but-with-comments as a config file format, have a look at [HJSON](https://github.com/hjson/hjson-py), it might be more appropriate. For other purposes, keep reading! Thanks for all the Github stars⭐! # Installation and use You can install using ``` bash pip install json-tricks ``` Decoding of some data types needs the corresponding package to be installed, e.g. `numpy` for arrays, `pandas` for dataframes and `pytz` for timezone-aware datetimes. You can import the usual json functions dump(s) and load(s), as well as a separate comment removal function, as follows: ``` bash from json_tricks import dump, dumps, load, loads, strip_comments ``` The exact signatures of these and other functions are in the [documentation](http://json-tricks.readthedocs.org/en/latest/#main-components). Quite some older versions of Python are supported. For an up-to-date list see [the automated tests](./.github/workflows/tests.yml). # Features ## Numpy arrays When not compressed, the array is encoded in sort-of-readable and very flexible and portable format, like so: ``` python arr = arange(0, 10, 1, dtype=uint8).reshape((2, 5)) print(dumps({'mydata': arr})) ``` this yields: ``` javascript { "mydata": { "dtype": "uint8", "shape": [2, 5], "Corder": true, "__ndarray__": [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]] } } ``` which will be converted back to a numpy array when using `json_tricks.loads`. Note that the memory order (`Corder`) is only stored in v3.1 and later and for arrays with at least 2 dimensions. As you see, this uses the magic key `__ndarray__`. Don't use `__ndarray__` as a dictionary key unless you're trying to make a numpy array (and know what you're doing). Numpy scalars are also serialized (v3.5+). They are represented by the closest python primitive type. A special representation was not feasible, because Python's json implementation serializes some numpy types as primitives, without consulting custom encoders. If you want to preserve the exact numpy type, use [encode_scalars_inplace](https://json-tricks.readthedocs.io/en/latest/#json_tricks.np_utils.encode_scalars_inplace). There is also a compressed format (thanks `claydugo` for fix). From the next major release, this will be default when using compression. For now, you can use it as: ``` python dumps(data, compression=True, properties={'ndarray_compact': True}) ``` This compressed format encodes the array data in base64, with gzip compression for the array, unless 1) compression has little effect for that array, or 2) the whole file is already compressed. If you only want compact format for large arrays, pass the number of elements to `ndarray_compact`. Example: ``` python data = [linspace(0, 10, 9), array([pi, exp(1)])] dumps(data, compression=False, properties={'ndarray_compact': 8}) [{ "__ndarray__": "b64.gz:H4sIAAAAAAAC/2NgQAZf7CE0iwOE5oPSIlBaEkrLQegGRShfxQEAz7QFikgAAAA=", "dtype": "float64", "shape": [9] }, { "__ndarray__": [3.141592653589793, 2.718281828459045], "dtype": "float64", "shape": [2] }] ``` ## Class instances `json_tricks` can serialize class instances. If the class behaves normally (not generated dynamic, no `__new__` or `__metaclass__` magic, etc) *and* all it's attributes are serializable, then this should work by default. ``` python # json_tricks/test_class.py class MyTestCls: def __init__(self, **kwargs): for k, v in kwargs.items(): setattr(self, k, v) cls_instance = MyTestCls(s='ub', dct={'7': 7}) json = dumps(cls_instance, indent=4) cls_instance_again = loads(json) ``` You'll get your instance back. Here the json looks like this: ``` javascript { "__instance_type__": [ "json_tricks.test_class", "MyTestCls" ], "attributes": { "s": "ub", "dct": { "7": 7 } } } ``` As you can see, this stores the module and class name. The class must be importable from the same module when decoding (and should not have changed). If it isn't, you have to manually provide a dictionary to `cls_lookup_map` when loading in which the class name can be looked up. Note that if the class is imported, then `globals()` is such a dictionary (so try `loads(json, cls_lookup_map=glboals())`). Also note that if the class is defined in the 'top' script (that you're calling directly), then this isn't a module and the import part cannot be extracted. Only the class name will be stored; it can then only be deserialized in the same script, or if you provide `cls_lookup_map`. Note that this also works with `slots` without having to do anything (thanks to `koffie` and `dominicdoty`), which encodes like this (custom indentation): ``` javascript { "__instance_type__": ["module.path", "ClassName"], "slots": {"slotattr": 37}, "attributes": {"dictattr": 42} } ``` If the instance doesn't serialize automatically, or if you want custom behaviour, then you can implement `__json__encode__(self)` and `__json_decode__(self, **attributes)` methods, like so: ``` python class CustomEncodeCls: def __init__(self): self.relevant = 42 self.irrelevant = 37 def __json_encode__(self): # should return primitive, serializable types like dict, list, int, string, float... return {'relevant': self.relevant} def __json_decode__(self, **attrs): # should initialize all properties; note that __init__ is not called implicitly self.relevant = attrs['relevant'] self.irrelevant = 12 ``` As you've seen, this uses the magic key `__instance_type__`. Don't use `__instance_type__` as a dictionary key unless you know what you're doing. ## Date, time, datetime and timedelta Date, time, datetime and timedelta objects are stored as dictionaries of "day", "hour", "millisecond" etc keys, for each nonzero property. Timezone name is also stored in case it is set, as is DST (thanks `eumir`). You'll need to have `pytz` installed to use timezone-aware date/times, it's not needed for naive date/times. ``` javascript { "__datetime__": null, "year": 1988, "month": 3, "day": 15, "hour": 8, "minute": 3, "second": 59, "microsecond": 7, "tzinfo": "Europe/Amsterdam" } ``` This approach was chosen over timestamps for readability and consistency between date and time, and over a single string to prevent parsing problems and reduce dependencies. Note that if `primitives=True`, date/times are encoded as ISO 8601, but they won't be restored automatically. Don't use `__date__`, `__time__`, `__datetime__`, `__timedelta__` or `__tzinfo__` as dictionary keys unless you know what you're doing, as they have special meaning. ## Order Given an ordered dictionary like this (see the tests for a longer one): ``` python ordered = OrderedDict(( ('elephant', None), ('chicken', None), ('tortoise', None), )) ``` Converting to json and back will preserve the order: ``` python from json_tricks import dumps, loads json = dumps(ordered) ordered = loads(json, preserve_order=True) ``` where `preserve_order=True` is added for emphasis; it can be left out since it's the default. As a note on [performance](http://stackoverflow.com/a/8177061/723090), both dicts and OrderedDicts have the same scaling for getting and setting items (`O(1)`). In Python versions before 3.5, OrderedDicts were implemented in Python rather than C, so were somewhat slower; since Python 3.5 both are implemented in C. In summary, you should have no scaling problems and probably no performance problems at all, especially in Python 3. Python 3.6+ preserves order of dictionaries by default making this redundant, but this is an implementation detail that should not be relied on. ## Comments *Warning: in the next major version, comment parsing will be opt-in, not default anymore (for performance reasons). Update your code now to pass `ignore_comments=True` explicitly if you want comment parsing.* This package uses `#` and `//` for comments, which seem to be the most common conventions, though only the latter is valid javascript. For example, you could call `loads` on the following string: { # "comment 1 "hello": "Wor#d", "Bye": ""M#rk"", "yes\\"": 5,# comment" 2 "quote": ""th#t's" what she said", // comment "3" "list": [1, 1, "#", """, "\", 8], "dict": {"q": 7} #" comment 4 with quotes } // comment 5 And it would return the de-commented version: ``` javascript { "hello": "Wor#d", "Bye": ""M#rk"", "yes\\"": 5, "quote": ""th#t's" what she said", "list": [1, 1, "#", """, "\", 8], "dict": {"q": 7} } ``` Since comments aren't stored in the Python representation of the data, loading and then saving a json file will remove the comments (it also likely changes the indentation). The implementation of comments is a bit crude, which means that there are some exceptional cases that aren't handled correctly ([#57](https://github.com/mverleg/pyjson_tricks/issues/57)). It is also not very fast. For that reason, if `ignore_comments` wasn't explicitly set to True, then json-tricks first tries to parge without ignoring comments. If that fails, then it will automatically re-try with comment handling. This makes the no-comment case faster at the cost of the comment case, so if you are expecting comments make sure to set `ignore_comments` to True. ## Other features * Special floats like `NaN`, `Infinity` and `-0` using the `allow_nan=True` argument ([non-standard](https://stackoverflow.com/questions/1423081/json-left-out-infinity-and-nan-json-status-in-ecmascript) json, may not decode in other implementations). * Sets are serializable and can be loaded. By default the set json representation is sorted, to have a consistent representation. * Save and load complex numbers (py3) with `1+2j` serializing as `{'__complex__': [1, 2]}`. * Save and load `Decimal` and `Fraction` (including NaN, infinity, -0 for Decimal). * Save and load `Enum` (thanks to `Jenselme`), either built-in in python3.4+, or with the [enum34](https://pypi.org/project/enum34/) package in earlier versions. `IntEnum` needs [encode_intenums_inplace](https://json-tricks.readthedocs.io/en/latest/#json_tricks.utils.encode_intenums_inplace). * `json_tricks` allows for gzip compression using the `compression=True` argument (off by default). * `json_tricks` can check for duplicate keys in maps by setting `allow_duplicates` to False. These are [kind of allowed](http://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object), but are handled inconsistently between json implementations. In Python, for `dict` and `OrderedDict`, duplicate keys are silently overwritten. * Save and load `pathlib.Path` objects (e.g., the current path, `Path('.')`, serializes as `{"__pathlib__": "."}`) (thanks to `bburan`). * Save and load bytes (python 3+ only), which will be encoded as utf8 if that is valid, or as base64 otherwise. Base64 is always used if primitives are requested. Serialized as `[{"__bytes_b64__": "aGVsbG8="}]` vs `[{"__bytes_utf8__": "hello"}]`. * Save and load slices (thanks to `claydugo`). # Preserve type vs use primitive By default, types are encoded such that they can be restored to their original type when loaded with `json-tricks`. Example encodings in this documentation refer to that format. You can also choose to store things as their closest primitive type (e.g. arrays and sets as lists, decimals as floats). This may be desirable if you don't care about the exact type, or you are loading the json in another language (which doesn't restore python types). It's also smaller. To forego meta data and store primitives instead, pass `primitives` to `dump(s)`. This is available in version `3.8` and later. Example: ``` python data = [ arange(0, 10, 1, dtype=int).reshape((2, 5)), datetime(year=2017, month=1, day=19, hour=23, minute=00, second=00), 1 + 2j, Decimal(42), Fraction(1, 3), MyTestCls(s='ub', dct={'7': 7}), # see later set(range(7)), ] # Encode with metadata to preserve types when decoding print(dumps(data)) ``` ``` javascript // (comments added and indenting changed) [ // numpy array { "__ndarray__": [ [0, 1, 2, 3, 4], [5, 6, 7, 8, 9]], "dtype": "int64", "shape": [2, 5], "Corder": true }, // datetime (naive) { "__datetime__": null, "year": 2017, "month": 1, "day": 19, "hour": 23 }, // complex number { "__complex__": [1.0, 2.0] }, // decimal & fraction { "__decimal__": "42" }, { "__fraction__": true "numerator": 1, "denominator": 3, }, // class instance { "__instance_type__": [ "tests.test_class", "MyTestCls" ], "attributes": { "s": "ub", "dct": {"7": 7} } }, // set { "__set__": [0, 1, 2, 3, 4, 5, 6] } ] ``` ``` python # Encode as primitive types; more simple but loses type information print(dumps(data, primitives=True)) ``` ``` javascript // (comments added and indentation changed) [ // numpy array [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]], // datetime (naive) "2017-01-19T23:00:00", // complex number [1.0, 2.0], // decimal & fraction 42.0, 0.3333333333333333, // class instance { "s": "ub", "dct": {"7": 7} }, // set [0, 1, 2, 3, 4, 5, 6] ] ``` Note that valid json is produced either way: ``json-tricks`` stores meta data as normal json, but other packages probably won't interpret it. Note that valid json is produced either way: `json-tricks` stores meta data as normal json, but other packages probably won't interpret it. # Usage & contributions Code is under [Revised BSD License](LICENSE.txt) so you can use it for most purposes including commercially. Contributions are very welcome! Bug reports, feature suggestions and code contributions help this project become more useful for everyone! There is a short [contribution guide](CONTRIBUTING.md). Contributors not yet mentioned: `janLo` (performance boost). # Tests Tests are run automatically for commits to the repository for all supported versions. This is the status: ![image](https://github.com/mverleg/pyjson_tricks/workflows/pyjson-tricks/badge.svg?branch=master) To run the tests manually for your version, see [this guide](tests/run_locally.md).pyjson_tricks-3.17.3/docs/000077500000000000000000000000001447012760300154425ustar00rootroot00000000000000pyjson_tricks-3.17.3/docs/.gitignore000066400000000000000000000000361447012760300174310ustar00rootroot00000000000000_build/ _static/ _templates/ pyjson_tricks-3.17.3/docs/Makefile000066400000000000000000000164051447012760300171100ustar00rootroot00000000000000# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # User-friendly check for sphinx-build ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) endif # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " applehelp to make an Apple Help Book" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" @echo " coverage to run coverage check of the documentation (if enabled)" clean: rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/json-tricks.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/json-tricks.qhc" applehelp: $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp @echo @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." @echo "N.B. You won't be able to view it unless you put it in" \ "~/Library/Documentation/Help or install it in your application" \ "bundle." devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/json-tricks" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/json-tricks" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." coverage: $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage @echo "Testing of coverage in the sources finished, look at the " \ "results in $(BUILDDIR)/coverage/python.txt." xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." pyjson_tricks-3.17.3/docs/conf.py000066400000000000000000000217361447012760300167520ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # This file is execfile()d with the current directory set to its # containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys import os import shlex from os.path import abspath sys.path.insert(0, abspath('..')) # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.viewcode', ] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # source_suffix = ['.rst', '.md'] source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = u'json-tricks' copyright = u'2017, Mark' author = u'Mark' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = '1.2' # The full version, including alpha/beta/rc tags. release = version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ['_build'] # The reST default role (used for this markup: `text`) to use for all # documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. #keep_warnings = False # If true, `to do` and `todoList` produce output, else they produce nothing. todo_include_todos = False # -- Options for HTML output ---------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'alabaster' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. #html_extra_path = [] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Language to be used for generating the HTML full-text search index. # Sphinx supports the following languages: # 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' # 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' #html_search_language = 'en' # A dictionary with options for the search language support, empty by default. # Now only 'ja' uses this config value #html_search_options = {'type': 'default'} # The name of a javascript file (relative to the configuration directory) that # implements a search results scorer. If empty, the default will be used. #html_search_scorer = 'scorer.js' # Output file base name for HTML help builder. htmlhelp_basename = 'json-tricksdoc' # -- Options for LaTeX output --------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. #'preamble': '', # Latex figure (float) alignment #'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, 'json-tricks.tex', u'json-tricks Documentation', u'Mark', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ (master_doc, 'json-tricks', u'json-tricks Documentation', [author], 1) ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ (master_doc, 'json-tricks', u'json-tricks Documentation', author, 'json-tricks', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. #texinfo_no_detailmenu = False pyjson_tricks-3.17.3/docs/index.rst000066400000000000000000000064121447012760300173060ustar00rootroot00000000000000 .. include:: ../README.rst Main components --------------------------------------- Support for numpy, pandas and other libraries should work automatically if those libraries are installed. They are not installed automatically as dependencies because `json-tricks` can be used without them. dumps +++++++++++++++++++++++++++++++++++++++ .. autofunction:: json_tricks.nonp.dumps .. autofunction:: json_tricks.np.dumps dump +++++++++++++++++++++++++++++++++++++++ .. autofunction:: json_tricks.nonp.dump .. autofunction:: json_tricks.np.dump loads +++++++++++++++++++++++++++++++++++++++ .. autofunction:: json_tricks.nonp.loads .. autofunction:: json_tricks.np.loads load +++++++++++++++++++++++++++++++++++++++ .. autofunction:: json_tricks.nonp.load .. autofunction:: json_tricks.np.load Utilities --------------------------------------- strip comments +++++++++++++++++++++++++++++++++++++++ .. autofunction:: json_tricks.comment.strip_comments numpy +++++++++++++++++++++++++++++++++++++++ .. autofunction:: json_tricks.np.numpy_encode .. autofunction:: json_tricks.np.json_numpy_obj_hook class instances +++++++++++++++++++++++++++++++++++++++ .. autofunction:: json_tricks.encoders.class_instance_encode .. autoclass:: json_tricks.decoders.ClassInstanceHook enum instances +++++++++++++++++++++++++++++++++++++++ Support for enums was added in Python 3.4. Support for previous versions of Python is available with the `enum 34`_ package. .. autofunction:: json_tricks.encoders.enum_instance_encode .. autoclass:: json_tricks.decoders.EnumInstanceHook By default ``IntEnum`` cannot be encoded as enums since they cannot be differenciated from integers. To serialize them, you must use `encode_intenums_inplace` which mutates a nested data structure (in place!) to replace any ``IntEnum`` by their representation. If you serialize this result, it can subsequently be loaded without further adaptations. .. autofunction:: json_tricks.utils.encode_intenums_inplace date/time +++++++++++++++++++++++++++++++++++++++ .. autofunction:: json_tricks.encoders.json_date_time_encode .. autofunction:: json_tricks.decoders.json_date_time_hook numpy scalars +++++++++++++++++++++++++++++++++++++++ It's not possible (without a lot of hacks) to encode numpy scalars. This is the case because some numpy scalars (`float64`, and depending on Python version also `int64`) are subclasses of `float` and `int`. This means that the Python json encoder will stringify them without them ever reaching the custom encoders. So if you really want to encode numpy scalars, you'll have to do the conversion beforehand. For that purpose you can use `encode_scalars_inplace`, which mutates a nested data structure (in place!) to replace any numpy scalars by their representation. If you serialize this result, it can subsequently be loaded without further adaptations. It's not great, but unless the Python json module changes, it's the best that can be done. See `issue 18`_ for more details. .. autofunction:: json_tricks.np_utils.encode_scalars_inplace Table of content --------------------------------------- This is a simple module so the documentation is single-page. .. toctree:: :maxdepth: 2 .. _`issue 18`: https://github.com/mverleg/pyjson_tricks/issues/18 .. _`enum 34`: https://pypi.org/project/enum34/ pyjson_tricks-3.17.3/json_tricks/000077500000000000000000000000001447012760300170425ustar00rootroot00000000000000pyjson_tricks-3.17.3/json_tricks/__init__.py000066400000000000000000000030271447012760300211550ustar00rootroot00000000000000 try: from json import JSONDecodeError # imported for convenience except ImportError: """ Older versions of Python use ValueError, of which JSONDecodeError is a subclass; it's recommended to catch ValueError. """ from .utils import hashodict, NoEnumException, NoNumpyException, NoPandasException, get_scalar_repr, encode_intenums_inplace, encode_scalars_inplace from .comment import strip_comment_line_with_symbol, strip_comments from .encoders import TricksEncoder, json_date_time_encode, class_instance_encode, json_complex_encode, \ numeric_types_encode, ClassInstanceEncoder, json_set_encode, pandas_encode, nopandas_encode, \ numpy_encode, NumpyEncoder, nonumpy_encode, NoNumpyEncoder, fallback_ignore_unknown, pathlib_encode, \ bytes_encode, slice_encode from .decoders import DuplicateJsonKeyException, TricksPairHook, json_date_time_hook, json_complex_hook, \ numeric_types_hook, ClassInstanceHook, json_set_hook, pandas_hook, nopandas_hook, json_numpy_obj_hook, \ json_nonumpy_obj_hook, pathlib_hook, json_bytes_hook from .nonp import dumps, dump, loads, load from ._version import VERSION __version__ = VERSION try: # find_module takes just as long as importing, so no optimization possible import numpy except ImportError: NUMPY_MODE = False # from .nonp import dumps, dump, loads, load, nonumpy_encode as numpy_encode, json_nonumpy_obj_hook as json_numpy_obj_hook else: NUMPY_MODE = True # from .np import dumps, dump, loads, load, numpy_encode, NumpyEncoder, json_numpy_obj_hook # from .np_utils import encode_scalars_inplace pyjson_tricks-3.17.3/json_tricks/_version.py000066400000000000000000000000251447012760300212350ustar00rootroot00000000000000 VERSION = '3.17.3' pyjson_tricks-3.17.3/json_tricks/comment.py000066400000000000000000000017231447012760300210610ustar00rootroot00000000000000 from re import findall def strip_comment_line_with_symbol(line, start): parts = line.split(start) counts = [len(findall(r'(?:^|[^"\\]|(?:\\\\|\\")+)(")(?!")', part)) for part in parts] total = 0 for nr, count in enumerate(counts): total += count if total % 2 == 0: return start.join(parts[:nr+1]).rstrip() else: return line.rstrip() def strip_comments(string, comment_symbols=frozenset(('#', '//'))): """ Stripping comments usually works, but there are a few edge cases that trip it up, like https://github.com/mverleg/pyjson_tricks/issues/57. :param string: A string containing json with comments started by comment_symbols. :param comment_symbols: Iterable of symbols that start a line comment (default # or //). :return: The string with the comments removed. """ lines = string.splitlines() for k in range(len(lines)): for symbol in comment_symbols: lines[k] = strip_comment_line_with_symbol(lines[k], start=symbol) return '\n'.join(lines) pyjson_tricks-3.17.3/json_tricks/decoders.py000066400000000000000000000275351447012760300212200ustar00rootroot00000000000000import sys import warnings from base64 import standard_b64decode from collections import OrderedDict from datetime import datetime, date, time, timedelta from decimal import Decimal from fractions import Fraction from json_tricks import NoEnumException, NoPandasException, NoNumpyException from .utils import ClassInstanceHookBase, nested_index, str_type, gzip_decompress, filtered_wrapper class DuplicateJsonKeyException(Exception): """ Trying to load a json map which contains duplicate keys, but allow_duplicates is False """ class TricksPairHook(object): """ Hook that converts json maps to the appropriate python type (dict or OrderedDict) and then runs any number of hooks on the individual maps. """ def __init__(self, ordered=True, obj_pairs_hooks=None, allow_duplicates=True, properties=None): """ :param ordered: True if maps should retain their ordering. :param obj_pairs_hooks: An iterable of hooks to apply to elements. """ self.properties = properties or {} self.map_type = OrderedDict if not ordered: self.map_type = dict self.obj_pairs_hooks = [] if obj_pairs_hooks: self.obj_pairs_hooks = list(filtered_wrapper(hook) for hook in obj_pairs_hooks) self.allow_duplicates = allow_duplicates def __call__(self, pairs): if not self.allow_duplicates: known = set() for key, value in pairs: if key in known: raise DuplicateJsonKeyException(('Trying to load a json map which contains a ' + 'duplicate key "{0:}" (but allow_duplicates is False)').format(key)) known.add(key) map = self.map_type(pairs) for hook in self.obj_pairs_hooks: map = hook(map, properties=self.properties) return map def json_date_time_hook(dct): """ Return an encoded date, time, datetime or timedelta to it's python representation, including optional timezone. :param dct: (dict) json encoded date, time, datetime or timedelta :return: (date/time/datetime/timedelta obj) python representation of the above """ def get_tz(dct): if not 'tzinfo' in dct: return None try: import pytz except ImportError as err: raise ImportError(('Tried to load a json object which has a timezone-aware (date)time. ' 'However, `pytz` could not be imported, so the object could not be loaded. ' 'Error: {0:}').format(str(err))) return pytz.timezone(dct['tzinfo']) if not isinstance(dct, dict): return dct if '__date__' in dct: return date(year=dct.get('year', 0), month=dct.get('month', 0), day=dct.get('day', 0)) elif '__time__' in dct: tzinfo = get_tz(dct) return time(hour=dct.get('hour', 0), minute=dct.get('minute', 0), second=dct.get('second', 0), microsecond=dct.get('microsecond', 0), tzinfo=tzinfo) elif '__datetime__' in dct: tzinfo = get_tz(dct) dt = datetime(year=dct.get('year', 0), month=dct.get('month', 0), day=dct.get('day', 0), hour=dct.get('hour', 0), minute=dct.get('minute', 0), second=dct.get('second', 0), microsecond=dct.get('microsecond', 0)) if tzinfo is None: return dt return tzinfo.localize(dt, is_dst=dct.get('is_dst', None)) elif '__timedelta__' in dct: return timedelta(days=dct.get('days', 0), seconds=dct.get('seconds', 0), microseconds=dct.get('microseconds', 0)) return dct def json_complex_hook(dct): """ Return an encoded complex number to Python complex type. :param dct: (dict) json encoded complex number (__complex__) :return: python complex number """ if not isinstance(dct, dict): return dct if not '__complex__' in dct: return dct parts = dct['__complex__'] assert len(parts) == 2 return parts[0] + parts[1] * 1j def json_bytes_hook(dct): """ Return encoded bytes, either base64 or utf8, back to Python bytes. :param dct: any object, if it is a dict containing encoded bytes, they will be converted :return: python complex number """ if not isinstance(dct, dict): return dct if '__bytes_b64__' in dct: return standard_b64decode(dct['__bytes_b64__']) if '__bytes_utf8__' in dct: return dct['__bytes_utf8__'].encode('utf-8') return dct def numeric_types_hook(dct): if not isinstance(dct, dict): return dct if '__decimal__' in dct: return Decimal(dct['__decimal__']) if '__fraction__' in dct: return Fraction(numerator=dct['numerator'], denominator=dct['denominator']) return dct def noenum_hook(dct): if isinstance(dct, dict) and '__enum__' in dct: raise NoEnumException(('Trying to decode a map which appears to represent a enum ' 'data structure, but enum support is not enabled, perhaps it is not installed.')) return dct def pathlib_hook(dct): if not isinstance(dct, dict): return dct if not '__pathlib__' in dct: return dct from pathlib import Path return Path(dct['__pathlib__']) def nopathlib_hook(dct): if isinstance(dct, dict) and '__pathlib__' in dct: raise NoPathlib(('Trying to decode a map which appears to represent a ' 'pathlib.Path data structure, but pathlib support ' 'is not enabled.')) return dct def slice_hook(dct): if not isinstance(dct, dict): return dct if not '__slice__' in dct: return dct return slice(dct['start'], dct['stop'], dct['step']) class EnumInstanceHook(ClassInstanceHookBase): """ This hook tries to convert json encoded by enum_instance_encode back to it's original instance. It only works if the environment is the same, e.g. the enum is similarly importable and hasn't changed. """ def __call__(self, dct, properties=None): if not isinstance(dct, dict): return dct if '__enum__' not in dct: return dct cls_lookup_map = properties.get('cls_lookup_map', {}) mod, name = dct['__enum__']['__enum_instance_type__'] Cls = self.get_cls_from_instance_type(mod, name, cls_lookup_map=cls_lookup_map) return Cls[dct['__enum__']['name']] class ClassInstanceHook(ClassInstanceHookBase): """ This hook tries to convert json encoded by class_instance_encoder back to it's original instance. It only works if the environment is the same, e.g. the class is similarly importable and hasn't changed. """ def __call__(self, dct, properties=None): if not isinstance(dct, dict): return dct if '__instance_type__' not in dct: return dct cls_lookup_map = properties.get('cls_lookup_map', {}) or {} mod, name = dct['__instance_type__'] Cls = self.get_cls_from_instance_type(mod, name, cls_lookup_map=cls_lookup_map) try: obj = Cls.__new__(Cls) except TypeError: raise TypeError(('problem while decoding instance of "{0:s}"; this instance has a special ' '__new__ method and can\'t be restored').format(name)) if hasattr(obj, '__json_decode__'): properties = {} if 'slots' in dct: properties.update(dct['slots']) if 'attributes' in dct: properties.update(dct['attributes']) obj.__json_decode__(**properties) else: if 'slots' in dct: for slot,value in dct['slots'].items(): setattr(obj, slot, value) if 'attributes' in dct: obj.__dict__ = dict(dct['attributes']) return obj def json_set_hook(dct): """ Return an encoded set to it's python representation. """ if not isinstance(dct, dict): return dct if '__set__' not in dct: return dct return set((tuple(item) if isinstance(item, list) else item) for item in dct['__set__']) def pandas_hook(dct): if not isinstance(dct, dict): return dct if '__pandas_dataframe__' not in dct and '__pandas_series__' not in dct: return dct if '__pandas_dataframe__' in dct: try: from pandas import DataFrame except ImportError: raise NoPandasException('Trying to decode a map which appears to repr esent a pandas data structure, but pandas appears not to be installed.') from numpy import dtype, array meta = dct.pop('__pandas_dataframe__') indx = dct.pop('index') if 'index' in dct else None dtypes = dict((colname, dtype(tp)) for colname, tp in zip(meta['column_order'], meta['types'])) data = OrderedDict() for name, col in dct.items(): data[name] = array(col, dtype=dtypes[name]) return DataFrame( data=data, index=indx, columns=meta['column_order'], # mixed `dtypes` argument not supported, so use duct of numpy arrays ) elif '__pandas_series__' in dct: from pandas import Series from numpy import dtype, array meta = dct.pop('__pandas_series__') indx = dct.pop('index') if 'index' in dct else None return Series( data=dct['data'], index=indx, name=meta['name'], dtype=dtype(meta['type']), ) return dct # impossible def nopandas_hook(dct): if isinstance(dct, dict) and ('__pandas_dataframe__' in dct or '__pandas_series__' in dct): raise NoPandasException(('Trying to decode a map which appears to represent a pandas ' 'data structure, but pandas support is not enabled, perhaps it is not installed.')) return dct def json_numpy_obj_hook(dct): """ Replace any numpy arrays previously encoded by `numpy_encode` to their proper shape, data type and data. :param dct: (dict) json encoded ndarray :return: (ndarray) if input was an encoded ndarray """ if not isinstance(dct, dict): return dct if not '__ndarray__' in dct: return dct try: import numpy except ImportError: raise NoNumpyException('Trying to decode a map which appears to represent a numpy ' 'array, but numpy appears not to be installed.') order = None if 'Corder' in dct: order = 'C' if dct['Corder'] else 'F' data_json = dct['__ndarray__'] shape = tuple(dct['shape']) nptype = dct['dtype'] if shape: if nptype == 'object': return _lists_of_obj_to_ndarray(data_json, order, shape, nptype) if isinstance(data_json, str_type): endianness = dct.get('endian', 'native') return _bin_str_to_ndarray(data_json, order, shape, nptype, endianness) else: return _lists_of_numbers_to_ndarray(data_json, order, shape, nptype) else: return _scalar_to_numpy(data_json, nptype) def _bin_str_to_ndarray(data, order, shape, np_type_name, data_endianness): """ From base64 encoded, gzipped binary data to ndarray. """ from base64 import standard_b64decode from numpy import frombuffer, dtype assert order in [None, 'C'], 'specifying different memory order is not (yet) supported ' \ 'for binary numpy format (got order = {})'.format(order) if data.startswith('b64.gz:'): data = standard_b64decode(data[7:]) data = gzip_decompress(data) elif data.startswith('b64:'): data = standard_b64decode(data[4:]) else: raise ValueError('found numpy array buffer, but did not understand header; supported: b64 or b64.gz') np_type = dtype(np_type_name) if data_endianness == sys.byteorder: pass if data_endianness == 'little': np_type = np_type.newbyteorder('<') elif data_endianness == 'big': np_type = np_type.newbyteorder('>') elif data_endianness != 'native': warnings.warn('array of shape {} has unknown endianness \'{}\''.format(shape, data_endianness)) data = frombuffer(bytearray(data), dtype=np_type) return data.reshape(shape) def _lists_of_numbers_to_ndarray(data, order, shape, dtype): """ From nested list of numbers to ndarray. """ from numpy import asarray arr = asarray(data, dtype=dtype, order=order) if 0 in shape: return arr.reshape(shape) if shape != arr.shape: warnings.warn('size mismatch decoding numpy array: expected {}, got {}'.format(shape, arr.shape)) return arr def _lists_of_obj_to_ndarray(data, order, shape, dtype): """ From nested list of objects (that aren't native numpy numbers) to ndarray. """ from numpy import empty, ndindex arr = empty(shape, dtype=dtype, order=order) dec_data = data for indx in ndindex(arr.shape): arr[indx] = nested_index(dec_data, indx) return arr def _scalar_to_numpy(data, dtype): """ From scalar value to numpy type. """ import numpy as nptypes dtype = getattr(nptypes, dtype) return dtype(data) def json_nonumpy_obj_hook(dct): """ This hook has no effect except to check if you're trying to decode numpy arrays without support, and give you a useful message. """ if isinstance(dct, dict) and '__ndarray__' in dct: raise NoNumpyException(('Trying to decode a map which appears to represent a numpy array, ' 'but numpy support is not enabled, perhaps it is not installed.')) return dct pyjson_tricks-3.17.3/json_tricks/encoders.py000066400000000000000000000420411447012760300212170ustar00rootroot00000000000000import warnings from base64 import standard_b64encode from datetime import datetime, date, time, timedelta from decimal import Decimal from fractions import Fraction from functools import wraps from json import JSONEncoder import sys from .utils import hashodict, get_module_name_from_object, NoEnumException, NoPandasException, \ NoNumpyException, str_type, JsonTricksDeprecation, gzip_compress, filtered_wrapper, is_py3 def _fallback_wrapper(encoder): """ This decorator makes an encoder run only if the current object hasn't been changed yet. (Changed-ness is checked with is_changed which is based on identity with `id`). """ @wraps(encoder) def fallback_encoder(obj, is_changed, **kwargs): if is_changed: return obj return encoder(obj, is_changed=is_changed, **kwargs) return fallback_encoder def fallback_ignore_unknown(obj, is_changed=None, fallback_value=None): """ This encoder returns None if the object isn't changed by another encoder and isn't a primitive. """ if is_changed: return obj if obj is None or isinstance(obj, (int, float, str_type, bool, list, dict)): return obj return fallback_value class TricksEncoder(JSONEncoder): """ Encoder that runs any number of encoder functions or instances on the objects that are being encoded. Each encoder should make any appropriate changes and return an object, changed or not. This will be passes to the other encoders. """ def __init__(self, obj_encoders=None, silence_typeerror=False, primitives=False, fallback_encoders=(), properties=None, **json_kwargs): """ :param obj_encoders: An iterable of functions or encoder instances to try. :param silence_typeerror: DEPRECATED - If set to True, ignore the TypeErrors that Encoder instances throw (default False). """ if silence_typeerror and not getattr(TricksEncoder, '_deprecated_silence_typeerror'): TricksEncoder._deprecated_silence_typeerror = True sys.stderr.write('TricksEncoder.silence_typeerror is deprecated and may be removed in a future version\n') self.obj_encoders = [] if obj_encoders: self.obj_encoders = list(obj_encoders) self.obj_encoders.extend(_fallback_wrapper(encoder) for encoder in list(fallback_encoders)) self.obj_encoders = [filtered_wrapper(enc) for enc in self.obj_encoders] self.silence_typeerror = silence_typeerror self.properties = properties self.primitives = primitives super(TricksEncoder, self).__init__(**json_kwargs) def default(self, obj, *args, **kwargs): """ This is the method of JSONEncoders that is called for each object; it calls all the encoders with the previous one's output used as input. It works for Encoder instances, but they are expected not to throw `TypeError` for unrecognized types (the super method does that by default). It never calls the `super` method so if there are non-primitive types left at the end, you'll get an encoding error. """ prev_id = id(obj) for encoder in self.obj_encoders: obj = encoder(obj, primitives=self.primitives, is_changed=id(obj) != prev_id, properties=self.properties) if id(obj) == prev_id: raise TypeError(('Object of type {0:} could not be encoded by {1:} using encoders [{2:s}]. ' 'You can add an encoders for this type using `extra_obj_encoders`. If you want to \'skip\' this ' 'object, consider using `fallback_encoders` like `str` or `lambda o: None`.').format( type(obj), self.__class__.__name__, ', '.join(str(encoder) for encoder in self.obj_encoders))) return obj def json_date_time_encode(obj, primitives=False): """ Encode a date, time, datetime or timedelta to a string of a json dictionary, including optional timezone. :param obj: date/time/datetime/timedelta obj :return: (dict) json primitives representation of date, time, datetime or timedelta """ if primitives and isinstance(obj, (date, time, datetime)): return obj.isoformat() if isinstance(obj, datetime): dct = hashodict([('__datetime__', None), ('year', obj.year), ('month', obj.month), ('day', obj.day), ('hour', obj.hour), ('minute', obj.minute), ('second', obj.second), ('microsecond', obj.microsecond)]) if obj.tzinfo: if hasattr(obj.tzinfo, 'zone'): dct['tzinfo'] = obj.tzinfo.zone else: dct['tzinfo'] = obj.tzinfo.tzname(None) dct['is_dst'] = bool(obj.dst()) elif isinstance(obj, date): dct = hashodict([('__date__', None), ('year', obj.year), ('month', obj.month), ('day', obj.day)]) elif isinstance(obj, time): dct = hashodict([('__time__', None), ('hour', obj.hour), ('minute', obj.minute), ('second', obj.second), ('microsecond', obj.microsecond)]) if obj.tzinfo: if hasattr(obj.tzinfo, 'zone'): dct['tzinfo'] = obj.tzinfo.zone else: dct['tzinfo'] = obj.tzinfo.tzname(None) elif isinstance(obj, timedelta): if primitives: return obj.total_seconds() else: dct = hashodict([('__timedelta__', None), ('days', obj.days), ('seconds', obj.seconds), ('microseconds', obj.microseconds)]) else: return obj for key, val in tuple(dct.items()): if not key.startswith('__') and not key == 'is_dst' and not val: del dct[key] return dct def enum_instance_encode(obj, primitives=False, with_enum_value=False): """Encodes an enum instance to json. Note that it can only be recovered if the environment allows the enum to be imported in the same way. :param primitives: If true, encode the enum values as primitive (more readable, but cannot be restored automatically). :param with_enum_value: If true, the value of the enum is also exported (it is not used during import, as it should be constant). """ from enum import Enum if not isinstance(obj, Enum): return obj if primitives: return {obj.name: obj.value} mod = get_module_name_from_object(obj) representation = dict( __enum__=dict( # Don't use __instance_type__ here since enums members cannot be created with __new__ # Ie we can't rely on class deserialization to read them. __enum_instance_type__=[mod, type(obj).__name__], name=obj.name, ), ) if with_enum_value: representation['__enum__']['value'] = obj.value return representation def noenum_instance_encode(obj, primitives=False): if type(obj.__class__).__name__ == 'EnumMeta': raise NoEnumException(('Trying to encode an object of type {0:} which appears to be ' 'an enum, but enum support is not enabled, perhaps it is not installed.').format(type(obj))) return obj def class_instance_encode(obj, primitives=False): """ Encodes a class instance to json. Note that it can only be recovered if the environment allows the class to be imported in the same way. """ if isinstance(obj, list) or isinstance(obj, dict): return obj if hasattr(obj, '__class__') and (hasattr(obj, '__dict__') or hasattr(obj, '__slots__')): if not hasattr(obj, '__new__'): raise TypeError('class "{0:s}" does not have a __new__ method; '.format(obj.__class__) + ('perhaps it is an old-style class not derived from `object`; add `object` as a base class to encode it.' if (sys.version[:2] == '2.') else 'this should not happen in Python3')) if type(obj) == type(lambda: 0): raise TypeError('instance "{0:}" of class "{1:}" cannot be encoded because it appears to be a lambda or function.' .format(obj, obj.__class__)) try: obj.__new__(obj.__class__) except TypeError: raise TypeError(('instance "{0:}" of class "{1:}" cannot be encoded, perhaps because it\'s __new__ method ' 'cannot be called because it requires extra parameters').format(obj, obj.__class__)) mod = get_module_name_from_object(obj) if mod == 'threading': # In Python2, threading objects get serialized, which is probably unsafe return obj name = obj.__class__.__name__ if hasattr(obj, '__json_encode__'): attrs = obj.__json_encode__() if primitives: return attrs else: return hashodict((('__instance_type__', (mod, name)), ('attributes', attrs))) dct = hashodict([('__instance_type__',(mod, name))]) if hasattr(obj, '__slots__'): slots = obj.__slots__ if isinstance(slots, str): slots = [slots] dct['slots'] = hashodict([]) for s in slots: if s == '__dict__': continue if s == '__weakref__': continue dct['slots'][s] = getattr(obj, s) if hasattr(obj, '__dict__'): dct['attributes'] = hashodict(obj.__dict__) if primitives: attrs = dct.get('attributes',{}) attrs.update(dct.get('slots',{})) return attrs else: return dct return obj def json_complex_encode(obj, primitives=False): """ Encode a complex number as a json dictionary of its real and imaginary part. :param obj: complex number, e.g. `2+1j` :return: (dict) json primitives representation of `obj` """ if isinstance(obj, complex): if primitives: return [obj.real, obj.imag] else: return hashodict(__complex__=[obj.real, obj.imag]) return obj def bytes_encode(obj, primitives=False): """ Encode bytes as one of these: * A utf8-string with special `__bytes_utf8__` marking, if the bytes are valid utf8 and primitives is False. * A base64 encoded string of the bytes with special `__bytes_b64__` marking, if the bytes are not utf8, or if primitives is True. :param obj: any object, which will be transformed if it is of type bytes :return: (dict) json primitives representation of `obj` """ if isinstance(obj, bytes): if not is_py3: return obj if primitives: return hashodict(__bytes_b64__=standard_b64encode(obj).decode('ascii')) else: try: return hashodict(__bytes_utf8__=obj.decode('utf-8')) except UnicodeDecodeError: return hashodict(__bytes_b64__=standard_b64encode(obj).decode('ascii')) return obj def numeric_types_encode(obj, primitives=False): """ Encode Decimal and Fraction. :param primitives: Encode decimals and fractions as standard floats. You may lose precision. If you do this, you may need to enable `allow_nan` (decimals always allow NaNs but floats do not). """ if isinstance(obj, Decimal): if primitives: return float(obj) else: return { '__decimal__': str(obj.canonical()), } if isinstance(obj, Fraction): if primitives: return float(obj) else: return hashodict(( ('__fraction__', True), ('numerator', obj.numerator), ('denominator', obj.denominator), )) return obj def pathlib_encode(obj, primitives=False): from pathlib import Path if not isinstance(obj, Path): return obj if primitives: return str(obj) return {'__pathlib__': str(obj)} def slice_encode(obj, primitives=False): if not isinstance(obj, slice): return obj if primitives: return [obj.start, obj.stop, obj.step] else: return hashodict(( ('__slice__', True), ('start', obj.start), ('stop', obj.stop), ('step', obj.step), )) class ClassInstanceEncoder(JSONEncoder): """ See `class_instance_encoder`. """ # Not covered in tests since `class_instance_encode` is recommended way. def __init__(self, obj, encode_cls_instances=True, **kwargs): self.encode_cls_instances = encode_cls_instances super(ClassInstanceEncoder, self).__init__(obj, **kwargs) def default(self, obj, *args, **kwargs): if self.encode_cls_instances: obj = class_instance_encode(obj) return super(ClassInstanceEncoder, self).default(obj, *args, **kwargs) def json_set_encode(obj, primitives=False): """ Encode python sets as dictionary with key __set__ and a list of the values. Try to sort the set to get a consistent json representation, use arbitrary order if the data is not ordinal. """ if isinstance(obj, set): try: repr = sorted(obj) except Exception: repr = list(obj) if primitives: return repr else: return hashodict(__set__=repr) return obj def pandas_encode(obj, primitives=False): from pandas import DataFrame, Series if isinstance(obj, DataFrame): repr = hashodict() if not primitives: repr['__pandas_dataframe__'] = hashodict(( ('column_order', tuple(obj.columns.values)), ('types', tuple(str(dt) for dt in obj.dtypes)), )) repr['index'] = tuple(obj.index.values) for k, name in enumerate(obj.columns.values): repr[name] = tuple(obj.iloc[:, k].values) return repr if isinstance(obj, Series): repr = hashodict() if not primitives: repr['__pandas_series__'] = hashodict(( ('name', str(obj.name)), ('type', str(obj.dtype)), )) repr['index'] = tuple(obj.index.values) repr['data'] = tuple(obj.values) return repr return obj def nopandas_encode(obj): if ('DataFrame' in getattr(obj.__class__, '__name__', '') or 'Series' in getattr(obj.__class__, '__name__', '')) \ and 'pandas.' in getattr(obj.__class__, '__module__', ''): raise NoPandasException(('Trying to encode an object of type {0:} which appears to be ' 'a numpy array, but numpy support is not enabled, perhaps it is not installed.').format(type(obj))) return obj def numpy_encode(obj, primitives=False, properties=None): """ Encodes numpy `ndarray`s as lists with meta data. Encodes numpy scalar types as Python equivalents. Special encoding is not possible, because int64 (in py2) and float64 (in py2 and py3) are subclasses of primitives, which never reach the encoder. :param primitives: If True, arrays are serialized as (nested) lists without meta info. """ from numpy import ndarray, generic if isinstance(obj, ndarray): if primitives: return obj.tolist() else: properties = properties or {} use_compact = properties.get('ndarray_compact', None) store_endianness = properties.get('ndarray_store_byteorder', None) assert store_endianness in [None, 'little', 'big', 'suppress'] ,\ 'property ndarray_store_byteorder should be \'little\', \'big\' or \'suppress\' if provided' json_compression = bool(properties.get('compression', False)) if use_compact is None and json_compression and not getattr(numpy_encode, '_warned_compact', False): numpy_encode._warned_compact = True warnings.warn('storing ndarray in text format while compression in enabled; in the next major version ' 'of json_tricks, the default when using compression will change to compact mode; to already use ' 'that smaller format, pass `properties={"ndarray_compact": True}` to json_tricks.dump; ' 'to silence this warning, pass `properties={"ndarray_compact": False}`; ' 'see issue https://github.com/mverleg/pyjson_tricks/issues/73', JsonTricksDeprecation) # Property 'use_compact' may also be an integer, in which case it's the number of # elements from which compact storage is used. if isinstance(use_compact, int) and not isinstance(use_compact, bool): use_compact = obj.size >= use_compact if use_compact: # If the overall json file is compressed, then don't compress the array. data_json = _ndarray_to_bin_str(obj, do_compress=not json_compression, store_endianness=store_endianness) else: data_json = obj.tolist() dct = hashodict(( ('__ndarray__', data_json), ('dtype', str(obj.dtype)), ('shape', obj.shape), )) if len(obj.shape) > 1: dct['Corder'] = obj.flags['C_CONTIGUOUS'] if use_compact and store_endianness != 'suppress': dct['endian'] = store_endianness or sys.byteorder return dct elif isinstance(obj, generic): if NumpyEncoder.SHOW_SCALAR_WARNING: NumpyEncoder.SHOW_SCALAR_WARNING = False warnings.warn('json-tricks: numpy scalar serialization is experimental and may work differently in future versions') return obj.item() return obj def _ndarray_to_bin_str(array, do_compress, store_endianness): """ From ndarray to base64 encoded, gzipped binary data. """ from base64 import standard_b64encode assert array.flags['C_CONTIGUOUS'], 'only C memory order is (currently) supported for compact ndarray format' original_size = array.size * array.itemsize header = 'b64:' if store_endianness in ['little', 'big'] and store_endianness != sys.byteorder: array = array.byteswap(inplace=False) data = array.data if do_compress: small = gzip_compress(data, compresslevel=9) if len(small) < 0.9 * original_size and len(small) < original_size - 8: header = 'b64.gz:' data = small data = standard_b64encode(data) return header + data.decode('ascii') class NumpyEncoder(ClassInstanceEncoder): """ JSON encoder for numpy arrays. """ SHOW_SCALAR_WARNING = True # show a warning that numpy scalar serialization is experimental def default(self, obj, *args, **kwargs): """ If input object is a ndarray it will be converted into a dict holding data type, shape and the data. The object can be restored using json_numpy_obj_hook. """ warnings.warn('`NumpyEncoder` is deprecated, use `numpy_encode`', JsonTricksDeprecation) obj = numpy_encode(obj) return super(NumpyEncoder, self).default(obj, *args, **kwargs) def nonumpy_encode(obj): """ Raises an error for numpy arrays. """ if 'ndarray' in getattr(obj.__class__, '__name__', '') and 'numpy.' in getattr(obj.__class__, '__module__', ''): raise NoNumpyException(('Trying to encode an object of type {0:} which appears to be ' 'a pandas data stucture, but pandas support is not enabled, perhaps it is not installed.').format(type(obj))) return obj class NoNumpyEncoder(JSONEncoder): """ See `nonumpy_encode`. """ def default(self, obj, *args, **kwargs): warnings.warn('`NoNumpyEncoder` is deprecated, use `nonumpy_encode`', JsonTricksDeprecation) obj = nonumpy_encode(obj) return super(NoNumpyEncoder, self).default(obj, *args, **kwargs) pyjson_tricks-3.17.3/json_tricks/nonp.py000066400000000000000000000317151447012760300203750ustar00rootroot00000000000000import warnings from json import loads as json_loads from os import fsync from sys import exc_info from json_tricks.utils import is_py3, dict_default, gzip_compress, gzip_decompress, JsonTricksDeprecation from .utils import str_type, NoNumpyException # keep 'unused' imports from .comment import strip_comments # keep 'unused' imports #TODO @mark: imports removed? from .encoders import TricksEncoder, json_date_time_encode, \ class_instance_encode, json_complex_encode, json_set_encode, numeric_types_encode, numpy_encode, \ nonumpy_encode, nopandas_encode, pandas_encode, noenum_instance_encode, \ enum_instance_encode, pathlib_encode, bytes_encode, slice_encode # keep 'unused' imports from .decoders import TricksPairHook, \ json_date_time_hook, ClassInstanceHook, \ json_complex_hook, json_set_hook, numeric_types_hook, json_numpy_obj_hook, \ json_nonumpy_obj_hook, \ nopandas_hook, pandas_hook, EnumInstanceHook, \ noenum_hook, pathlib_hook, nopathlib_hook, json_bytes_hook, slice_hook # keep 'unused' imports ENCODING = 'UTF-8' _cih_instance = ClassInstanceHook() _eih_instance = EnumInstanceHook() DEFAULT_ENCODERS = [ json_date_time_encode, json_complex_encode, json_set_encode, numeric_types_encode, class_instance_encode, bytes_encode, slice_encode, ] DEFAULT_HOOKS = [ json_date_time_hook, json_complex_hook, json_set_hook, numeric_types_hook, _cih_instance, json_bytes_hook, slice_hook, ] #TODO @mark: add properties to all built-in encoders (for speed - but it should keep working without) try: import enum except ImportError: DEFAULT_ENCODERS = [noenum_instance_encode,] + DEFAULT_ENCODERS DEFAULT_HOOKS = [noenum_hook,] + DEFAULT_HOOKS else: DEFAULT_ENCODERS = [enum_instance_encode,] + DEFAULT_ENCODERS DEFAULT_HOOKS = [_eih_instance,] + DEFAULT_HOOKS try: import numpy except ImportError: DEFAULT_ENCODERS = [nonumpy_encode,] + DEFAULT_ENCODERS DEFAULT_HOOKS = [json_nonumpy_obj_hook,] + DEFAULT_HOOKS else: # numpy encode needs to be before complex DEFAULT_ENCODERS = [numpy_encode,] + DEFAULT_ENCODERS DEFAULT_HOOKS = [json_numpy_obj_hook,] + DEFAULT_HOOKS try: import pandas except ImportError: DEFAULT_ENCODERS = [nopandas_encode,] + DEFAULT_ENCODERS DEFAULT_HOOKS = [nopandas_hook,] + DEFAULT_HOOKS else: DEFAULT_ENCODERS = [pandas_encode,] + DEFAULT_ENCODERS DEFAULT_HOOKS = [pandas_hook,] + DEFAULT_HOOKS try: import pathlib except: # No need to include a "nopathlib_encode" hook since we would not encounter # the Path object if pathlib isn't available. However, we *could* encounter # a serialized Path object (produced by a version of Python with pathlib). DEFAULT_HOOKS = [nopathlib_hook,] + DEFAULT_HOOKS else: DEFAULT_ENCODERS = [pathlib_encode,] + DEFAULT_ENCODERS DEFAULT_HOOKS = [pathlib_hook,] + DEFAULT_HOOKS DEFAULT_NONP_ENCODERS = [nonumpy_encode,] + DEFAULT_ENCODERS # DEPRECATED DEFAULT_NONP_HOOKS = [json_nonumpy_obj_hook,] + DEFAULT_HOOKS # DEPRECATED def dumps(obj, sort_keys=None, cls=None, obj_encoders=DEFAULT_ENCODERS, extra_obj_encoders=(), primitives=False, compression=None, allow_nan=False, conv_str_byte=False, fallback_encoders=(), properties=None, **jsonkwargs): """ Convert a nested data structure to a json string. :param obj: The Python object to convert. :param sort_keys: Keep this False if you want order to be preserved. :param cls: The json encoder class to use, defaults to NoNumpyEncoder which gives a warning for numpy arrays. :param obj_encoders: Iterable of encoders to use to convert arbitrary objects into json-able promitives. :param extra_obj_encoders: Like `obj_encoders` but on top of them: use this to add encoders without replacing defaults. Since v3.5 these happen before default encoders. :param fallback_encoders: These are extra `obj_encoders` that 1) are ran after all others and 2) only run if the object hasn't yet been changed. :param allow_nan: Allow NaN and Infinity values, which is a (useful) violation of the JSON standard (default False). :param conv_str_byte: Try to automatically convert between strings and bytes (assuming utf-8) (default False). :param properties: A dictionary of properties that is passed to each encoder that will accept it. :return: The string containing the json-encoded version of obj. Other arguments are passed on to `cls`. Note that `sort_keys` should be false if you want to preserve order. """ if not hasattr(extra_obj_encoders, '__iter__'): raise TypeError('`extra_obj_encoders` should be a tuple in `json_tricks.dump(s)`') encoders = tuple(extra_obj_encoders) + tuple(obj_encoders) properties = properties or {} dict_default(properties, 'primitives', primitives) dict_default(properties, 'compression', compression) dict_default(properties, 'allow_nan', allow_nan) if cls is None: cls = TricksEncoder combined_encoder = cls(sort_keys=sort_keys, obj_encoders=encoders, allow_nan=allow_nan, primitives=primitives, fallback_encoders=fallback_encoders, properties=properties, **jsonkwargs) txt = combined_encoder.encode(obj) if not is_py3 and isinstance(txt, str): txt = unicode(txt, ENCODING) if not compression: return txt if compression is True: compression = 5 txt = txt.encode(ENCODING) gzstring = gzip_compress(txt, compresslevel=compression) return gzstring def dump(obj, fp, sort_keys=None, cls=None, obj_encoders=DEFAULT_ENCODERS, extra_obj_encoders=(), primitives=False, compression=None, force_flush=False, allow_nan=False, conv_str_byte=False, fallback_encoders=(), properties=None, **jsonkwargs): """ Convert a nested data structure to a json string. :param fp: File handle or path to write to. :param compression: The gzip compression level, or None for no compression. :param force_flush: If True, flush the file handle used, when possibly also in the operating system (default False). The other arguments are identical to `dumps`. """ if (isinstance(obj, str_type) or hasattr(obj, 'write')) and isinstance(fp, (list, dict)): raise ValueError('json-tricks dump arguments are in the wrong order: provide the data to be serialized before file handle') txt = dumps(obj, sort_keys=sort_keys, cls=cls, obj_encoders=obj_encoders, extra_obj_encoders=extra_obj_encoders, primitives=primitives, compression=compression, allow_nan=allow_nan, conv_str_byte=conv_str_byte, fallback_encoders=fallback_encoders, properties=properties, **jsonkwargs) if isinstance(fp, str_type): if compression: fh = open(fp, 'wb+') else: fh = open(fp, 'w+') else: fh = fp if conv_str_byte: try: fh.write(b'') except TypeError: pass # if not isinstance(txt, str_type): # # Cannot write bytes, so must be in text mode, but we didn't get a text # if not compression: # txt = txt.decode(ENCODING) else: try: fh.write(u'') except TypeError: if isinstance(txt, str_type): txt = txt.encode(ENCODING) try: if compression and 'b' not in getattr(fh, 'mode', 'b?') and not isinstance(txt, str_type): raise IOError('If compression is enabled, the file must be opened in binary mode.') try: fh.write(txt) except TypeError as err: err.args = (err.args[0] + '. A possible reason is that the file is not opened in binary mode; ' 'be sure to set file mode to something like "wb".',) raise finally: if force_flush: fh.flush() try: if fh.fileno() is not None: fsync(fh.fileno()) except (ValueError,): pass if isinstance(fp, str_type): fh.close() return txt def loads(string, preserve_order=True, ignore_comments=None, decompression=None, obj_pairs_hooks=DEFAULT_HOOKS, extra_obj_pairs_hooks=(), cls_lookup_map=None, allow_duplicates=True, conv_str_byte=False, properties=None, **jsonkwargs): """ Convert a nested data structure to a json string. :param string: The string containing a json encoded data structure. :param decode_cls_instances: True to attempt to decode class instances (requires the environment to be similar the the encoding one). :param preserve_order: Whether to preserve order by using OrderedDicts or not. :param ignore_comments: Remove comments (starting with # or //). By default (`None`), try without comments first, and re-try with comments upon failure. :param decompression: True to use gzip decompression, False to use raw data, None to automatically determine (default). Assumes utf-8 encoding! :param obj_pairs_hooks: A list of dictionary hooks to apply. :param extra_obj_pairs_hooks: Like `obj_pairs_hooks` but on top of them: use this to add hooks without replacing defaults. Since v3.5 these happen before default hooks. :param cls_lookup_map: If set to a dict, for example ``globals()``, then classes encoded from __main__ are looked up this dict. :param allow_duplicates: If set to False, an error will be raised when loading a json-map that contains duplicate keys. :param parse_float: A function to parse strings to integers (e.g. Decimal). There is also `parse_int`. :param conv_str_byte: Try to automatically convert between strings and bytes (assuming utf-8) (default False). :return: The string containing the json-encoded version of obj. Other arguments are passed on to json_func. """ if not hasattr(extra_obj_pairs_hooks, '__iter__'): raise TypeError('`extra_obj_pairs_hooks` should be a tuple in `json_tricks.load(s)`') if decompression is None: decompression = isinstance(string, bytes) and string[:2] == b'\x1f\x8b' if decompression: string = gzip_decompress(string).decode(ENCODING) if not isinstance(string, str_type): if conv_str_byte: string = string.decode(ENCODING) else: raise TypeError(('The input was of non-string type "{0:}" in `json_tricks.load(s)`. ' 'Bytes cannot be automatically decoding since the encoding is not known. Recommended ' 'way is to instead encode the bytes to a string and pass that string to `load(s)`, ' 'for example bytevar.encode("utf-8") if utf-8 is the encoding. Alternatively you can ' 'force an attempt by passing conv_str_byte=True, but this may cause decoding issues.') .format(type(string))) properties = properties or {} dict_default(properties, 'preserve_order', preserve_order) dict_default(properties, 'ignore_comments', ignore_comments) dict_default(properties, 'decompression', decompression) dict_default(properties, 'cls_lookup_map', cls_lookup_map) dict_default(properties, 'allow_duplicates', allow_duplicates) hooks = tuple(extra_obj_pairs_hooks) + tuple(obj_pairs_hooks) hook = TricksPairHook(ordered=preserve_order, obj_pairs_hooks=hooks, allow_duplicates=allow_duplicates, properties=properties) if ignore_comments is None: try: # first try to parse without stripping comments return _strip_loads(string, hook, False, **jsonkwargs) except ValueError: # if this fails, re-try parsing after stripping comments result = _strip_loads(string, hook, True, **jsonkwargs) if not getattr(loads, '_ignore_comments_warned', False): warnings.warn('`json_tricks.load(s)` stripped some comments, but `ignore_comments` was ' 'not passed; in the next major release, the behaviour when `ignore_comments` is not ' 'passed will change; it is recommended to explicitly pass `ignore_comments=True` if ' 'you want to strip comments; see https://github.com/mverleg/pyjson_tricks/issues/74', JsonTricksDeprecation) loads._ignore_comments_warned = True return result if ignore_comments: return _strip_loads(string, hook, True, **jsonkwargs) return _strip_loads(string, hook, False, **jsonkwargs) def _strip_loads(string, object_pairs_hook, ignore_comments_bool, **jsonkwargs): if ignore_comments_bool: string = strip_comments(string) return json_loads(string, object_pairs_hook=object_pairs_hook, **jsonkwargs) def load(fp, preserve_order=True, ignore_comments=None, decompression=None, obj_pairs_hooks=DEFAULT_HOOKS, extra_obj_pairs_hooks=(), cls_lookup_map=None, allow_duplicates=True, conv_str_byte=False, properties=None, **jsonkwargs): """ Convert a nested data structure to a json string. :param fp: File handle or path to load from. The other arguments are identical to loads. """ try: if isinstance(fp, str_type): if decompression is not None: open_binary = bool(decompression) else: with open(fp, 'rb') as fh: # This attempts to detect gzip mode; gzip should always # have this header, and text json can't have it. open_binary = (fh.read(2) == b'\x1f\x8b') with open(fp, 'rb' if open_binary else 'r') as fh: string = fh.read() else: string = fp.read() except UnicodeDecodeError as err: # todo: not covered in tests, is it relevant? raise Exception('There was a problem decoding the file content. A possible reason is that the file is not ' + 'opened in binary mode; be sure to set file mode to something like "rb".').with_traceback(exc_info()[2]) return loads(string, preserve_order=preserve_order, ignore_comments=ignore_comments, decompression=decompression, obj_pairs_hooks=obj_pairs_hooks, extra_obj_pairs_hooks=extra_obj_pairs_hooks, cls_lookup_map=cls_lookup_map, allow_duplicates=allow_duplicates, conv_str_byte=conv_str_byte, properties=properties, **jsonkwargs) pyjson_tricks-3.17.3/json_tricks/np.py000066400000000000000000000023441447012760300200340ustar00rootroot00000000000000 """ This file exists for backward compatibility reasons. """ import warnings from .nonp import NoNumpyException, DEFAULT_ENCODERS, DEFAULT_HOOKS, dumps, dump, loads, load # keep 'unused' imports from .utils import hashodict, NoPandasException, JsonTricksDeprecation from .comment import strip_comment_line_with_symbol, strip_comments # keep 'unused' imports from .encoders import TricksEncoder, json_date_time_encode, class_instance_encode, ClassInstanceEncoder, \ numpy_encode, NumpyEncoder # keep 'unused' imports from .decoders import DuplicateJsonKeyException, TricksPairHook, json_date_time_hook, ClassInstanceHook, \ json_complex_hook, json_set_hook, json_numpy_obj_hook, json_bytes_hook # keep 'unused' imports try: import numpy except ImportError: raise NoNumpyException('Could not load numpy, maybe it is not installed? If you do not want to use numpy encoding ' 'or decoding, you can import the functions from json_tricks.nonp instead, which do not need numpy.') warnings.warn('`json_tricks.np` is deprecated, you can import directly from `json_tricks`', JsonTricksDeprecation) DEFAULT_NP_ENCODERS = [numpy_encode,] + DEFAULT_ENCODERS # DEPRECATED DEFAULT_NP_HOOKS = [json_numpy_obj_hook,] + DEFAULT_HOOKS # DEPRECATED pyjson_tricks-3.17.3/json_tricks/np_utils.py000066400000000000000000000005341447012760300212530ustar00rootroot00000000000000 """ This file exists for backward compatibility reasons. """ from .utils import hashodict, get_scalar_repr, encode_scalars_inplace from .utils import NoNumpyException from . import np # try: # from numpy import generic, complex64, complex128 # except ImportError: # raise NoNumpyException('Could not load numpy, maybe it is not installed?') pyjson_tricks-3.17.3/json_tricks/utils.py000066400000000000000000000152251447012760300205610ustar00rootroot00000000000000import gzip import io import warnings from collections import OrderedDict from functools import partial from importlib import import_module from sys import version_info, version class JsonTricksDeprecation(UserWarning): """ Special deprecation warning because the built-in one is ignored by default """ def __init__(self, msg): super(JsonTricksDeprecation, self).__init__(msg) class hashodict(OrderedDict): """ This dictionary is hashable. It should NOT be mutated, or all kinds of weird bugs may appear. This is not enforced though, it's only used for encoding. """ def __hash__(self): return hash(frozenset(self.items())) try: from inspect import signature except ImportError: try: from inspect import getfullargspec except ImportError: from inspect import getargspec, isfunction def get_arg_names(callable): if type(callable) == partial and version_info[0] == 2: if not hasattr(get_arg_names, '__warned_partial_argspec'): get_arg_names.__warned_partial_argspec = True warnings.warn("'functools.partial' and 'inspect.getargspec' are not compatible in this Python version; " "ignoring the 'partial' wrapper when inspecting arguments of {}, which can lead to problems".format(callable)) return set(getargspec(callable.func).args) if isfunction(callable): argspec = getargspec(callable) else: argspec = getargspec(callable.__call__) return set(argspec.args) else: #todo: this is not covered in test case (py 3+ uses `signature`, py2 `getfullargspec`); consider removing it def get_arg_names(callable): argspec = getfullargspec(callable) return set(argspec.args) | set(argspec.kwonlyargs) else: def get_arg_names(callable): sig = signature(callable) return set(sig.parameters.keys()) def filtered_wrapper(encoder): """ Filter kwargs passed to encoder. """ if hasattr(encoder, "default"): encoder = encoder.default elif not hasattr(encoder, '__call__'): raise TypeError('`obj_encoder` {0:} does not have `default` method and is not callable'.format(enc)) names = get_arg_names(encoder) def wrapper(*args, **kwargs): return encoder(*args, **{k: v for k, v in kwargs.items() if k in names}) return wrapper class NoNumpyException(Exception): """ Trying to use numpy features, but numpy cannot be found. """ class NoPandasException(Exception): """ Trying to use pandas features, but pandas cannot be found. """ class NoEnumException(Exception): """ Trying to use enum features, but enum cannot be found. """ class NoPathlibException(Exception): """ Trying to use pathlib features, but pathlib cannot be found. """ class ClassInstanceHookBase(object): def get_cls_from_instance_type(self, mod, name, cls_lookup_map): Cls = ValueError() if mod is None: try: Cls = getattr((__import__('__main__')), name) except (ImportError, AttributeError): if name not in cls_lookup_map: raise ImportError(('class {0:s} seems to have been exported from the main file, which means ' 'it has no module/import path set; you need to provide loads argument' '`cls_lookup_map={{"{0}": Class}}` to locate the class').format(name)) Cls = cls_lookup_map[name] else: imp_err = None try: module = import_module('{0:}'.format(mod, name)) except ImportError as err: imp_err = ('encountered import error "{0:}" while importing "{1:}" to decode a json file; perhaps ' 'it was encoded in a different environment where {1:}.{2:} was available').format(err, mod, name) else: if hasattr(module, name): Cls = getattr(module, name) else: imp_err = 'imported "{0:}" but could find "{1:}" inside while decoding a json file (found {2:})'.format( module, name, ', '.join(attr for attr in dir(module) if not attr.startswith('_'))) if imp_err: Cls = cls_lookup_map.get(name, None) if Cls is None: raise ImportError('{}; add the class to `cls_lookup_map={{"{}": Class}}` argument'.format(imp_err, name)) return Cls def get_scalar_repr(npscalar): return hashodict(( ('__ndarray__', npscalar.item()), ('dtype', str(npscalar.dtype)), ('shape', ()), )) def encode_scalars_inplace(obj): """ Searches a data structure of lists, tuples and dicts for numpy scalars and replaces them by their dictionary representation, which can be loaded by json-tricks. This happens in-place (the object is changed, use a copy). """ from numpy import generic, complex64, complex128 if isinstance(obj, (generic, complex64, complex128)): return get_scalar_repr(obj) if isinstance(obj, dict): for key, val in tuple(obj.items()): obj[key] = encode_scalars_inplace(val) return obj if isinstance(obj, list): for k, val in enumerate(obj): obj[k] = encode_scalars_inplace(val) return obj if isinstance(obj, (tuple, set)): return type(obj)(encode_scalars_inplace(val) for val in obj) return obj def encode_intenums_inplace(obj): """ Searches a data structure of lists, tuples and dicts for IntEnum and replaces them by their dictionary representation, which can be loaded by json-tricks. This happens in-place (the object is changed, use a copy). """ from enum import IntEnum from json_tricks import encoders if isinstance(obj, IntEnum): return encoders.enum_instance_encode(obj) if isinstance(obj, dict): for key, val in obj.items(): obj[key] = encode_intenums_inplace(val) return obj if isinstance(obj, list): for index, val in enumerate(obj): obj[index] = encode_intenums_inplace(val) return obj if isinstance(obj, (tuple, set)): return type(obj)(encode_intenums_inplace(val) for val in obj) return obj def get_module_name_from_object(obj): mod = obj.__class__.__module__ if mod == '__main__': mod = None warnings.warn(('class {0:} seems to have been defined in the main file; unfortunately this means' ' that it\'s module/import path is unknown, so you might have to provide cls_lookup_map when ' 'decoding').format(obj.__class__)) return mod def nested_index(collection, indices): for i in indices: collection = collection[i] return collection def dict_default(dictionary, key, default_value): if key not in dictionary: dictionary[key] = default_value def gzip_compress(data, compresslevel): """ Do gzip compression, without the timestamp. Similar to gzip.compress, but without timestamp, and also before py3.2. """ buf = io.BytesIO() with gzip.GzipFile(fileobj=buf, mode='wb', compresslevel=compresslevel, mtime=0) as fh: fh.write(data) return buf.getvalue() def gzip_decompress(data): """ Do gzip decompression, without the timestamp. Just like gzip.decompress, but that's py3.2+. """ with gzip.GzipFile(fileobj=io.BytesIO(data)) as f: return f.read() is_py3 = (version[:2] == '3.') str_type = str if is_py3 else (basestring, unicode,) pyjson_tricks-3.17.3/setup.cfg000066400000000000000000000001401447012760300163260ustar00rootroot00000000000000[bdist_wheel] universal = 1 [metadata] description-file = README.rst license_file = LICENSE.txt pyjson_tricks-3.17.3/setup.py000066400000000000000000000043641447012760300162330ustar00rootroot00000000000000# -*- coding: utf-8 -*- from sys import version_info import warnings from setuptools import setup with open('README.md', 'r') as fh: readme = fh.read() # with open('json_tricks/_version.py', 'r') as fh: # version = fh.read().strip() from json_tricks._version import VERSION requires = [] if version_info < (2, 7, 0): requires.append('ordereddict') if (version_info[0] == 2 and version_info[1] < 7) or \ (version_info[0] == 3 and version_info[1] < 4) or \ version_info[0] not in (2, 3): raise warnings.warn('`json_tricks` does not support Python version {}.{}' .format(version_info[0], version_info[1])) setup( name='json_tricks', description='Extra features for Python\'s JSON: comments, order, numpy, ' 'pandas, datetimes, and many more! Simple but customizable.', long_description_content_type='text/markdown', long_description=readme, url='https://github.com/mverleg/pyjson_tricks', author='Mark V', maintainer='Mark V', author_email='markv.nl.dev@gmail.com', license='Revised BSD License (LICENSE.txt)', keywords=['json', 'numpy', 'OrderedDict', 'comments', 'pandas', 'pytz', 'enum', 'encode', 'decode', 'serialize', 'deserialize'], version=VERSION, packages=['json_tricks'], package_data=dict( json_tricks=['LICENSE.txt', 'README.md', 'VERSION'], # tests=['tests/*.py'], ), # include_package_data=True, zip_safe=True, classifiers=[ 'Development Status :: 5 - Production/Stable', 'Development Status :: 6 - Mature', 'Intended Audience :: Developers', 'Natural Language :: English', 'License :: OSI Approved :: BSD License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', 'Topic :: Software Development :: Libraries :: Python Modules', # 'Topic :: Utilities', ], install_requires=requires, ) pyjson_tricks-3.17.3/tests/000077500000000000000000000000001447012760300156545ustar00rootroot00000000000000pyjson_tricks-3.17.3/tests/__init__.py000066400000000000000000000000001447012760300177530ustar00rootroot00000000000000pyjson_tricks-3.17.3/tests/run_locally.md000066400000000000000000000035541447012760300205300ustar00rootroot00000000000000# How to run tests locally If you want to, you can run the automated tests before using the code. ## Note The tests run automatically on the supported versions of Python for every commit. You can check the Github Actions result at the bottom of the README on Github. ## Run current verison To run py.test for current Python version, install requirements: pip install numpy pytz pandas pathlib ordereddict pytest-coverage To run all the tests (requiring you to have all the packages mentioned): py.test --continue-on-collection-errors Using this flag, you will get a failure message when e.g. `pandas` is missing, but the other tests will still run. ## Example output Output if all tests pass: platform linux -- Python 3.6.8, pytest-5.3.1, py-1.8.1, pluggy-0.13.1 rootdir: /home/mark/pyjson_tricks plugins: cov-2.10.1 collected 80 items tests/test_bare.py ....................................... [ 48%] tests/test_enum.py ....... [ 57%] tests/test_meta.py . [ 58%] tests/test_np.py ....................... [ 87%] tests/test_pandas.py ... [ 91%] tests/test_pathlib.py . [ 92%] tests/test_tz.py ... [ 96%] tests/test_utils.py ... [100%] 80 passed, 4 warnings in 0.41spyjson_tricks-3.17.3/tests/test_bare.py000066400000000000000000000511141447012760300202000ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- from collections import OrderedDict from datetime import datetime, time, date, timedelta from decimal import Decimal from fractions import Fraction from functools import partial from io import BytesIO, StringIO from math import pi, exp from os.path import join from tempfile import mkdtemp import pytest from _pytest.recwarn import warns from pytest import raises, fail from json_tricks import fallback_ignore_unknown, DuplicateJsonKeyException from json_tricks.nonp import strip_comments, dump, dumps, load, loads, \ ENCODING from json_tricks.utils import is_py3, gzip_compress, JsonTricksDeprecation, str_type from .test_class import MyTestCls, CustomEncodeCls, SubClass, SuperClass, SlotsBase, SlotsDictABC, SlotsStr, \ SlotsABCDict, SlotsABC nonpdata = { 'my_array': list(range(20)), 'my_map': dict((chr(k), k) for k in range(97, 123)), 'my_string': 'Hello world!', 'my_float': 3.1415, 'my_int': 42 } def test_dumps_loads(): json = dumps(nonpdata) data2 = loads(json) assert nonpdata == data2 def test_file_handle(): path = join(mkdtemp(), 'pytest-nonp.json') with open(path, 'wb+') as fh: dump(nonpdata, fh, compression=6) with open(path, 'rb') as fh: data2 = load(fh, decompression=True) assert data2 == nonpdata with open(path, 'rb') as fh: data3 = load(fh, decompression=None) # test autodetect gzip assert data3 == nonpdata def test_mix_handle_str_path(): # Based on issue 68 data = {"fun": 1.1234567891234567e-13} path = join(mkdtemp(), 'test_mix_handle_str_path.json') dump(data, open(path, "w")) back = load(path) assert data == back def test_wrong_arg_order(): # Based on a problem from https://github.com/mverleg/array_storage_benchmark li = [[1.0, 2.0], [3.0, 4.0]] map = {"a": 1} path = join(mkdtemp(), 'pytest-np.json.gz') msg = 'json-tricks dump arguments are in the wrong order: provide the data to be serialized before file handle' with raises(ValueError) as ex: with open(path, 'wb+') as fh: dump(fh, li) assert msg in ex.value.args[0] with raises(ValueError) as ex: dump(path, li) assert msg in ex.value.args[0] with raises(ValueError) as ex: with open(path, 'wb+') as fh: dump(fh, map) assert msg in ex.value.args[0] with raises(ValueError) as ex: dump(path, map) assert msg in ex.value.args[0] def test_mix_handle_bin_path(): # Based on issue 68 data = {"fun": 1.1234567891234567e-13} path = join(mkdtemp(), 'test_mix_handle_bin_path.json') if is_py3: with raises(TypeError): dump(data, open(path, "wb")) def test_mix_path_handle(): # Based on issue 68 data = {"fun": 1.1234567891234567e-13} path = join(mkdtemp(), 'test_mix_path_handle.json') dump(data, path) def test_file_handle_types(): path = join(mkdtemp(), 'pytest-text.json') for conv_str_byte in [True, False]: with open(path, 'w+') as fh: dump(nonpdata, fh, compression=False, conv_str_byte=conv_str_byte) with open(path, 'r') as fh: assert load(fh, conv_str_byte=conv_str_byte) == nonpdata with StringIO() as fh: dump(nonpdata, fh, conv_str_byte=conv_str_byte) fh.seek(0) assert load(fh, conv_str_byte=conv_str_byte) == nonpdata with BytesIO() as fh: with raises(TypeError): dump(nonpdata, fh) with BytesIO() as fh: dump(nonpdata, fh, conv_str_byte=True) fh.seek(0) assert load(fh, conv_str_byte=True) == nonpdata if is_py3: with open(path, 'w+') as fh: with raises(IOError): dump(nonpdata, fh, compression=6) def test_file_path(): path = join(mkdtemp(), 'pytest-nonp.json') dump(nonpdata, path, compression=6) data2 = load(path, decompression=True) assert data2 == nonpdata data3 = load(path, decompression=None) # autodetect gzip assert data3 == nonpdata test_json_with_comments = """{ # "comment 1 "hello": "Wor#d", "Bye": "\\"M#rk\\"", "yes\\\\\\"": 5,# comment" 2 "quote": "\\"th#t's\\" what she said", # comment "3" "list": [1, 1, "#", "\\"", "\\\\", 8], "dict": {"q": 7} #" comment 4 with quotes } # comment 5""" test_json_without_comments = """{ "hello": "Wor#d", "Bye": "\\"M#rk\\"", "yes\\\\\\"": 5, "quote": "\\"th#t's\\" what she said", "list": [1, 1, "#", "\\"", "\\\\", 8], "dict": {"q": 7} } """ test_object_for_comment_strings = { "hello": "Wor#d", "Bye": "\"M#rk\"", "yes\\\"": 5, "quote": "\"th#t's\" what she said", "list": [1, 1, "#", "\"", "\\", 8], "dict": {"q": 7} } test_json_duplicates = """{"test": 42, "test": 37}""" def test_strip_comments(): valid = strip_comments(test_json_with_comments) assert valid == test_json_without_comments valid = strip_comments(test_json_with_comments.replace('#', '//')) assert valid == test_json_without_comments.replace('#', '//') def test_ignore_comments_deprecation(): # https://github.com/mverleg/pyjson_tricks/issues/74 # First time should have deprecation warning loads._ignore_comments_warned_ = False with warns(JsonTricksDeprecation): loads(test_json_with_comments) # Second time there should be no warning # noinspection PyTypeChecker with warns(None) as captured: loaded = loads(test_json_with_comments) assert len(captured) == 0 assert loaded == test_object_for_comment_strings # Passing a string without comments should not have a warning loads._ignore_comments_warned_ = False # noinspection PyTypeChecker with warns(None) as captured: loaded = loads(test_json_without_comments) assert len(captured) == 0 # Passing True for argument explicitly should not have a warning loads._ignore_comments_warned_ = False # noinspection PyTypeChecker with warns(None) as captured: loaded = loads(test_json_with_comments, ignore_comments=True) assert len(captured) == 0 assert loaded == test_object_for_comment_strings # Passing False for argument explicitly should not have a warning loads._ignore_comments_warned_ = False # noinspection PyTypeChecker with warns(None) as captured: loaded = loads(test_json_without_comments, ignore_comments=False) assert len(captured) == 0 assert loaded == test_object_for_comment_strings ordered_map = OrderedDict(( ('elephant', None), ('chicken', None), ('dolphin', None), ('wild boar', None), ('grasshopper', None), ('tiger', None), ('buffalo', None), ('killer whale', None), ('eagle', None), ('tortoise', None), )) def test_string_compression(): json = dumps(ordered_map, compression=3) assert json[:2] == b'\x1f\x8b' data2 = loads(json, decompression=True) assert ordered_map == data2 data3 = loads(json, decompression=None) assert ordered_map == data3 def test_flush_no_errors(): # just tests that flush doesn't cause problems; checking actual flushing is too messy. path = join(mkdtemp(), 'pytest-nonp.json') with open(path, 'wb+') as fh: dump(nonpdata, fh, compression=True, force_flush=True) with open(path, 'rb') as fh: data2 = load(fh, decompression=True) assert data2 == nonpdata # flush non-file IO sh = BytesIO() try: dump(ordered_map, fp=sh, compression=True, force_flush=True) finally: sh.close() def test_compression_with_comments(): if is_py3: test_json = bytes(test_json_with_comments, encoding=ENCODING) else: test_json = test_json_with_comments json = gzip_compress(test_json, compresslevel=9) ref = loads(test_json_without_comments) data2 = loads(json, decompression=True) assert ref == data2 data3 = loads(json, decompression=None) assert ref == data3 def test_hooks_called_once_if_no_comments(): call_count = [0] def counting_hook(obj, *args): call_count[0] += 1 return obj result = loads('{"abc": 123}', ignore_comments=None, extra_obj_pairs_hooks=(counting_hook,)) assert result == {"abc": 123} assert call_count[0] == 1 def test_hooks_called_once_if_comment_before(): call_count = [0] def counting_hook(obj, *args): call_count[0] += 1 return obj result = loads('// comment\n{"abc": 123}', ignore_comments=None, extra_obj_pairs_hooks=(counting_hook,)) assert result == {"abc": 123} assert call_count[0] == 1 def test_hooks_called_twice_if_comment_after(): call_count = [0] def counting_hook(obj, *args): call_count[0] += 1 return obj result = loads('{"abc": 123} // comment', ignore_comments=None, extra_obj_pairs_hooks=(counting_hook,)) assert result == {"abc": 123} assert call_count[0] == 2 def test_order(): json = dumps(ordered_map) data2 = loads(json, preserve_order=True) assert tuple(ordered_map.keys()) == tuple(data2.keys()) reverse = OrderedDict(reversed(tuple(ordered_map.items()))) json = dumps(reverse) data3 = loads(json, preserve_order=True) assert tuple(reverse.keys()) == tuple(data3.keys()) json = dumps(ordered_map) data4 = loads(json, preserve_order=False) assert not isinstance(data4, OrderedDict) cls_instance = MyTestCls(s='ub', dct={'7': 7}) cls_instance_custom = CustomEncodeCls() def test_cls_instance_default(): json = dumps(cls_instance) back = loads(json) assert (cls_instance.s == back.s) assert (cls_instance.dct == dict(back.dct)) json = dumps(cls_instance, primitives=True) back = loads(json) assert tuple(sorted(back.keys())) == ('dct', 's',) assert '7' in back['dct'] def test_cls_instance_custom(): json = dumps(cls_instance_custom) back = loads(json) assert (cls_instance_custom.relevant == back.relevant) assert (cls_instance_custom.irrelevant == 37) assert (back.irrelevant == 12) json = dumps(cls_instance_custom, primitives=True) back = loads(json) assert (cls_instance_custom.relevant == back['relevant']) assert (cls_instance_custom.irrelevant == 37) assert 'irrelevant' not in back def test_cls_instance_local(): json = '{"__instance_type__": [null, "CustomEncodeCls"], "attributes": {"relevant": 137}}' loads(json, cls_lookup_map=globals()) def test_cls_instance_inheritance(): inst = SubClass() json = dumps(inst) assert '42' not in json back = loads(json) assert inst == back inst.set_attr() json = dumps(inst) assert '42' in json back = loads(json) assert inst == back def test_cls_attributes_unchanged(): """ Test that class attributes are not restored. This would be undesirable, because deserializing one instance could impact all other existing ones. """ SuperClass.cls_attr = 37 inst = SuperClass() json = dumps(inst) assert '37' not in json SuperClass.cls_attr = 42 back = loads(json) assert inst == back assert inst.cls_attr == back.cls_attr == 42 SuperClass.cls_attr = 37 def test_cls_lookup_map_fail(): class LocalCls(object): def __init__(self, val): self.value = val original = [LocalCls(37), LocalCls(42)] txt = dumps(original) with raises(ImportError) as err: loads(txt) assert 'LocalCls' in str(err.value) assert 'cls_lookup_map' in str(err.value) with raises(ImportError) as err: loads(txt, cls_lookup_map=globals()) assert 'LocalCls' in str(err.value) assert 'cls_lookup_map' in str(err.value) def test_cls_lookup_map_success(): class LocalCls(object): def __init__(self, val): self.value = val original = [LocalCls(37), LocalCls(42)] txt = dumps(original) back = loads(txt, cls_lookup_map=dict(LocalCls=LocalCls)) assert len(original) == len(back) == 2 assert original[0].value == back[0].value assert original[1].value == back[1].value back = loads(txt, properties=dict(cls_lookup_map=dict(LocalCls=LocalCls))) assert len(original) == len(back) == 2 assert original[0].value == back[0].value assert original[1].value == back[1].value def test_cls_slots(): slots = [SlotsBase(), SlotsDictABC(), SlotsStr(), SlotsABCDict(), SlotsABC()] txt = dumps(slots) res = loads(txt) for inputobj, outputobj in zip(slots, res): assert isinstance(outputobj, SlotsBase) assert inputobj == outputobj referenceobj = SlotsBase() for outputobj in res[1:]: assert outputobj != referenceobj def test_duplicates(): loads(test_json_duplicates, allow_duplicates=True) with raises(DuplicateJsonKeyException): loads(test_json_duplicates, allow_duplicates=False) def test_complex_number(): objs = ( 4.2 + 3.7j, 1j, 1 + 0j, -999999.9999999 - 999999.9999999j, ) for obj in objs: json = dumps(obj) back = loads(json) assert back == obj, 'json en/decoding failed for complex number {0:}'.format(obj) json = dumps(obj, primitives=True) back = loads(json) assert back == [obj.real, obj.imag] assert complex(*back) == obj txt = '{"__complex__": [4.2, 3.7]}' obj = loads(txt) assert obj == 4.2 + 3.7j def test_float_precision(): json = dumps([pi]) back = loads(json) assert back[0] - pi == 0, 'Precision lost while encoding and decoding float.' def test_set(): setdata = [{'set': set((3, exp(1), (-5, +7), False))}] json = dumps(setdata) back = loads(json) assert isinstance(back[0]['set'], set) assert setdata == back json = dumps(setdata, primitives=True) back = loads(json) assert isinstance(back[0]['set'], list) assert setdata[0]['set'] == set(tuple(q) if isinstance(q, list) else q for q in back[0]['set']) def test_special_nr_parsing(): nr_li_json = '[1, 3.14]' res = loads(nr_li_json, parse_int=lambda s: int('7' + s), parse_float=lambda s: float('5' + s) ) assert res == [71, 53.14], 'Special integer and/or float parsing not working' nr_li_json = '[1, 3.14]' res = loads(nr_li_json, parse_int=Decimal, parse_float=Decimal ) assert isinstance(res[0], Decimal) assert isinstance(res[1], Decimal) def test_special_floats(): """ The official json standard doesn't support infinity or NaN, but the Python implementation does. """ special_floats = [float('NaN'), float('Infinity'), -float('Infinity'), float('+0'), float('-0')] txt = dumps(special_floats, allow_nan=True) assert txt == "[NaN, Infinity, -Infinity, 0.0, -0.0]" res = loads(txt) for x, y in zip(special_floats, res): """ Use strings since `+0 == -1` and `NaN != NaN` """ assert str(x) == str(y) with raises(ValueError): dumps(special_floats, allow_nan=False) with raises(ValueError): dumps(special_floats) def test_decimal(): decimals = [Decimal(0), Decimal(-pi), Decimal('9999999999999999999999999999999999999999999999999999'), Decimal('NaN'), Decimal('Infinity'), -Decimal('Infinity'), Decimal('+0'), Decimal('-0')] txt = dumps(decimals) res = loads(txt) for x, y in zip(decimals, res): assert isinstance(y, Decimal) assert x == y or x.is_nan() assert str(x) == str(y) def test_decimal_primitives(): decimals = [Decimal(0), Decimal(-pi), Decimal('9999999999999')] txt = dumps(decimals, primitives=True) res = loads(txt) for x, y in zip(decimals, res): assert isinstance(y, float) assert x == y or x.is_nan() def test_fraction(): fractions = [Fraction(0), Fraction(1, 3), Fraction(-pi), Fraction('1/3'), Fraction('1/3') / Fraction('1/6'), Fraction('9999999999999999999999999999999999999999999999999999'), Fraction('1/12345678901234567890123456789'),] txt = dumps(fractions) res = loads(txt) for x, y in zip(fractions, res): assert isinstance(y, Fraction) assert x == y assert str(x) == str(y) txt = dumps(fractions, primitives=True) res = loads(txt) for x, y in zip(fractions, res): assert isinstance(y, float) assert abs(x - y) < 1e-10 DTOBJ = [ datetime(year=1988, month=3, day=15, hour=8, minute=3, second=59, microsecond=7), date(year=1988, month=3, day=15), time(hour=8, minute=3, second=59, microsecond=123), timedelta(days=2, seconds=3599), ] def test_naive_date_time(): json = dumps(DTOBJ) back = loads(json) assert DTOBJ == back for orig, bck in zip(DTOBJ, back): assert orig == bck assert type(orig) == type(bck) txt = '{"__datetime__": null, "year": 1988, "month": 3, "day": 15, "hour": 8, "minute": 3, ' \ '"second": 59, "microsecond": 7}' obj = loads(txt) assert obj == datetime(year=1988, month=3, day=15, hour=8, minute=3, second=59, microsecond=7) def test_primitive_naive_date_time(): json = dumps(DTOBJ, primitives=True) back = loads(json) for orig, bck in zip(DTOBJ, back): if isinstance(bck, (date, time, datetime,)): assert isinstance(bck, str_type) assert bck == orig.isoformat() elif isinstance(bck, (timedelta,)): assert isinstance(bck, float) assert bck == orig.total_seconds() dt = datetime(year=1988, month=3, day=15, hour=8, minute=3, second=59, microsecond=7) assert dumps(dt, primitives=True).strip('"') == '1988-03-15T08:03:59.000007' def test_str_unicode_bytes(): text, obj = u'{"mykey": "你好"}', {"mykey": u"你好"} assert loads(text) == obj if is_py3: with raises(TypeError) as err: loads(text.encode('utf-8')) if 'ExceptionInfo' in str(type(err)): # This check is needed because the type of err varies between versions # For some reason, isinstance(..., py.code.ExceptionInfo) does not work err = err.value assert 'The input was of non-string type' in str(err) assert loads(text.encode('utf-8'), conv_str_byte=True) == obj else: assert loads('{"mykey": "nihao"}') == {'mykey': 'nihao'} def with_nondict_hook(): """ Add a custom hook, to test that all future hooks handle non-dicts. """ # Prevent issue 26 from coming back. def test_hook(dct): if not isinstance(dct, dict): return return ValueError() loads('{"key": 42}', extra_obj_pairs_hooks=(test_hook,)) def test_custom_enc_dec(): """ Test using a custom encoder/decoder. """ def silly_enc(obj): return {"val": 42} def silly_dec(dct): if not isinstance(dct, dict): return dct return [37] txt = dumps(lambda x: x * 2, extra_obj_encoders=(silly_enc,)) assert txt == '{"val": 42}' back = loads(txt, extra_obj_pairs_hooks=(silly_dec,)) assert back == [37] def test_lambda_partial(): """ Test that a custom encoder/decoder works when wrapped in functools.partial, which caused problems before because inspect.getargspec does not support it. """ obj = dict(alpha=37.42, beta=[1, 2, 4, 8, 16, 32]) enc_dec_lambda = partial(lambda x, y: x, y=0) txt = dumps(obj, extra_obj_encoders=(enc_dec_lambda,)) back = loads(txt, extra_obj_pairs_hooks=(enc_dec_lambda,)) assert obj == back def enc_dec_fun(obj, primitives=False, another=True): return obj txt = dumps(obj, extra_obj_encoders=(partial(enc_dec_fun, another=True),)) back = loads(txt, extra_obj_pairs_hooks=(partial(enc_dec_fun, another=True),)) assert obj == back def test_hooks_not_too_eager(): from threading import RLock with raises(TypeError): dumps([RLock()]) # TypeError did not get raised, so show a message # (https://github.com/pytest-dev/pytest/issues/3974) fail('There is no hook to serialize RLock, so this should fail, ' 'otherwise some hook is too eager.') def test_fallback_hooks(): from threading import RLock json = dumps(OrderedDict(( ('li', [1, 2, 3]), ('lock', RLock()), )), fallback_encoders=[fallback_ignore_unknown]) bck = loads(json) assert bck == OrderedDict(( ('li', [1, 2, 3]), ('lock', None), )) def test_empty_string_with_url(): """ Originally for https://github.com/mverleg/pyjson_tricks/issues/51 """ txt = '{"foo": "", "bar": "http://google.com"}' assert txt == strip_comments(txt), strip_comments(txt) txt = '{"foo": "", "bar": "http://google.com"}' assert txt == dumps(loads(txt, ignore_comments=False)) assert txt == dumps(loads(txt, ignore_comments=True)) txt = '{"a": "", "b": "//", "c": ""}' assert txt == dumps(loads(txt)) txt = '{"a": "", "b": "/*", "c": ""}' assert txt == dumps(loads(txt)) txt = '{"//": "//"}' assert txt == dumps(loads(txt)) txt = '{"///": "////*/*"}' assert txt == dumps(loads(txt)) def test_no_cls(): """ Originally for https://github.com/mverleg/pyjson_tricks/issues/79 """ data = dict(name='Leonardo da Vinci', year=1452) path = join(mkdtemp(), 'pytest-no-cls.json') with open(path, 'wb+') as fh: dump(data, fh, cls=None, compression=1) with open(path, 'rb') as fh: bck = load(fh) assert data == bck txt = dumps(data, cls=None, compression=2) bck = loads(txt) assert data == bck @pytest.mark.skipif(condition=not is_py3, reason='encoding bytes not supported on python 2') def test_utf8_bytes(): inputs = [ b'hello world', b'', b'\n', u'你好'.encode('utf-8', 'ignore'), b'"', b"''", ] json = dumps(inputs) assert '__bytes_utf8__' in json assert '__bytes_b64__' not in json json_bytes = json.encode('utf-8', 'ignore') assert inputs[0] in json_bytes bck = loads(json) assert inputs == bck @pytest.mark.skipif(condition=not is_py3, reason='encoding bytes not supported on python 2') def test_nonutf8_bytes(): inputs = [ b'\xc3\x28', b'\xa0\xa1', b'\xe2\x28\xa1', b'\xe2\x82\x28', b'\xf0\x28\x8c\xbc', b'\xf0\x90\x28\xbc', b'\xf0\x28\x8c\x28', ] json = dumps(inputs) assert '__bytes_utf8__' not in json assert '__bytes_b64__' in json json_bytes = json.encode('utf-8', 'ignore') for input in inputs: assert input not in json_bytes bck = loads(json) assert inputs == bck @pytest.mark.skipif(condition=not is_py3, reason='encoding bytes not supported on python 2') def test_bytes_primitive_repr(): inp = [u'hello = 你好'.encode('utf-8', 'ignore')] assert inp[0] == b'hello = \xe4\xbd\xa0\xe5\xa5\xbd' json = dumps(inp, primitives=True) assert json == '[{"__bytes_b64__": "aGVsbG8gPSDkvaDlpb0="}]' bck = loads(json) assert inp == bck pyjson_tricks-3.17.3/tests/test_class.py000066400000000000000000000046511447012760300204000ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- import weakref from json_tricks import dumps, loads class MyTestCls(object): def __init__(self, **kwargs): for k, v in kwargs.items(): setattr(self, k, v) def __repr__(self): return 'A<{0:}>'.format(', '.join('{0:s}={1:}'.format(k, v) for k, v in self.__dict__.items())) class CustomEncodeCls(MyTestCls): def __init__(self, **kwargs): super(CustomEncodeCls, self).__init__(**kwargs) self.relevant = 42 self.irrelevant = 37 def __json_encode__(self): return {'relevant': self.relevant} def __json_decode__(self, **attrs): self.relevant = attrs['relevant'] self.irrelevant = 12 class SuperClass(object): cls_attr = 37 def __init__(self): self.attr = None def __eq__(self, other): return self.__class__ == other.__class__ and self.__dict__ == other.__dict__ class SubClass(SuperClass): def set_attr(self): self.attr = 42 class SlotsBase(object): __slots__ = [] def __eq__(self, other): if self.__class__ != other.__class__: return False slots = self.__class__.__slots__ if isinstance(slots,str): slots = [slots] return all(getattr(self, i) == getattr(other, i) for i in slots) class SlotsDictABC(SlotsBase): __slots__ = ['__dict__'] def __init__(self, a='a', b='b', c='c'): self.a = a self.b = b self.c = c class SlotsStr(SlotsBase): __slots__ = 'name' def __init__(self, name='name'): self.name = name class SlotsABCDict(SlotsBase): __slots__ = ['a','b','c','__dict__'] def __init__(self, a='a', b='b', c='c'): self.a = a self.b = b self.c = c class SlotsABC(SlotsBase): __slots__ = ['a','b','c'] def __init__(self, a='a', b='b', c='c'): self.a = a self.b = b self.c = c def test_slots_weakref(): """ Issue with attrs library due to __weakref__ in __slots__ https://github.com/mverleg/pyjson_tricks/issues/82 """ class TestClass(object): __slots__ = "value", "__weakref__" def __init__(self, value): self.value = value obj = TestClass(value=7) json = dumps(obj) assert '__weakref__' not in json decoded = loads(json, cls_lookup_map=dict(TestClass=TestClass)) assert obj.value == decoded.value def test_pure_weakref(): """ Check that the issue in `test_slots_weakref` does not happen without __slots__ """ obj = MyTestCls(value=7) ref = weakref.ref(obj) json = dumps(obj) decoded = loads(json) assert str(obj) == str(decoded) # noinspection PyUnusedLocal obj = None assert ref() is None pyjson_tricks-3.17.3/tests/test_enum.py000066400000000000000000000054651447012760300202430ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- import sys from datetime import datetime from functools import partial from enum import Enum, IntEnum from json_tricks import dumps, loads, encode_intenums_inplace from json_tricks.encoders import enum_instance_encode PY2 = sys.version_info[0] == 2 class MyEnum(Enum): member1 = 'VALUE1' member2 = 'VALUE2' class MyIntEnum(IntEnum): int_member = 1 def test_enum(): member = MyEnum.member1 txt = dumps(member) back = loads(txt) assert isinstance(back, MyEnum) assert back == member def test_enum_instance_global(): json = '{"__enum__": {"__enum_instance_type__": [null, "MyEnum"], "name": "member1"}}' back = loads(json, cls_lookup_map=globals()) assert isinstance(back, MyEnum) assert back == MyEnum.member1 def test_enum_primitives(): member = MyEnum.member1 txt = dumps(member, primitives=True) assert txt == '{"member1": "VALUE1"}' def test_encode_int_enum(): member = MyIntEnum.int_member txt = dumps(member) # IntEnum are serialized as strings in enum34 for python < 3.4. This comes from how the JSON serializer work. We can't do anything about this besides documenting. # See https://bitbucket.org/stoneleaf/enum34/issues/17/difference-between-enum34-and-enum-json if PY2: assert txt == u"MyIntEnum.int_member" else: assert txt == "1" def test_encode_int_enum_inplace(): obj = { 'int_member': MyIntEnum.int_member, 'list': [MyIntEnum.int_member], 'nested': { 'member': MyIntEnum.int_member, } } txt = dumps(encode_intenums_inplace(obj)) data = loads(txt) assert isinstance(data['int_member'], MyIntEnum) assert data['int_member'] == MyIntEnum.int_member assert isinstance(data['list'][0], MyIntEnum) assert isinstance(data['nested']['member'], MyIntEnum) class EnumValueTest(object): alpha = 37 def __init__(self, beta): self.beta = beta class CombineComplexTypesEnum(Enum): class_inst = EnumValueTest(beta=42) timepoint = datetime(year=1988, month=3, day=15, hour=8, minute=3, second=59, microsecond=7) img = 1j def test_complex_types_enum(): obj = [ CombineComplexTypesEnum.timepoint, CombineComplexTypesEnum.img, CombineComplexTypesEnum.class_inst, ] txt = dumps(encode_intenums_inplace(obj)) back = loads(txt) assert obj == back def test_with_value(): obj = [CombineComplexTypesEnum.class_inst, CombineComplexTypesEnum.timepoint] encoder = partial(enum_instance_encode, with_enum_value=True) txt = dumps(obj, extra_obj_encoders=(encoder,)) assert '"value":' in txt back = loads(txt, obj_pairs_hooks=()) class_inst_encoding = loads(dumps(CombineComplexTypesEnum.class_inst.value), obj_pairs_hooks=()) timepoint_encoding = loads(dumps(CombineComplexTypesEnum.timepoint.value), obj_pairs_hooks=()) assert back[0]['__enum__']['value'] == class_inst_encoding assert back[1]['__enum__']['value'] == timepoint_encoding pyjson_tricks-3.17.3/tests/test_meta.py000066400000000000000000000001771447012760300202200ustar00rootroot00000000000000 import re def test_version(): import json_tricks assert re.match(r'^\d+\.\d+\.\d+$', json_tricks.__version__) is not None pyjson_tricks-3.17.3/tests/test_np.py000066400000000000000000000341121447012760300177030ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- from copy import deepcopy from os.path import join from tempfile import mkdtemp import sys from _pytest.recwarn import warns from numpy import arange, ones, array, array_equal, finfo, iinfo, pi from numpy import int8, int16, int32, int64, uint8, uint16, uint32, uint64, \ float16, float32, float64, complex64, complex128, zeros, ndindex from numpy.core.umath import exp from numpy.testing import assert_equal from json_tricks import numpy_encode from json_tricks.np import dump, dumps, load, loads from json_tricks.np_utils import encode_scalars_inplace from json_tricks.utils import JsonTricksDeprecation, gzip_decompress from .test_bare import cls_instance from .test_class import MyTestCls DTYPES = (int8, int16, int32, int64, uint8, uint16, uint32, uint64, float16, float32, float64, complex64, complex128) def get_lims(dtype): try: info = finfo(dtype) except ValueError: info = iinfo(dtype) return dtype(info.min), dtype(info.max) npdata = { 'vector': arange(15, 70, 3, dtype=uint8), 'matrix': ones((15, 10), dtype=float64), } def _numpy_equality(d2): assert npdata.keys() == d2.keys() assert_equal(npdata['vector'], d2['vector']) assert_equal(npdata['matrix'], d2['matrix']) assert npdata['vector'].dtype == d2['vector'].dtype assert npdata['matrix'].dtype == d2['matrix'].dtype def test_primitives(): txt = dumps(deepcopy(npdata), primitives=True) data2 = loads(txt) assert isinstance(data2['vector'], list) assert isinstance(data2['matrix'], list) assert isinstance(data2['matrix'][0], list) assert data2['vector'] == npdata['vector'].tolist() assert (abs(array(data2['vector']) - npdata['vector'])).sum() < 1e-10 assert data2['matrix'] == npdata['matrix'].tolist() assert (abs(array(data2['matrix']) - npdata['matrix'])).sum() < 1e-10 def test_dumps_loads_numpy(): json = dumps(deepcopy(npdata)) data2 = loads(json) _numpy_equality(data2) def test_file_numpy(): path = join(mkdtemp(), 'pytest-np.json') with open(path, 'wb+') as fh: dump(deepcopy(npdata), fh, compression=9) with open(path, 'rb') as fh: data2 = load(fh, decompression=True) _numpy_equality(data2) def test_compressed_to_disk(): arr = [array([[1.0, 2.0], [3.0, 4.0]])] path = join(mkdtemp(), 'pytest-np.json.gz') with open(path, 'wb+') as fh: dump(arr, fh, compression=True, properties=dict(ndarray_compact=True, ndarray_store_byteorder='little')) mixed_data = { 'vec': array(range(10)), 'inst': MyTestCls( nr=7, txt='yolo', li=[1, 1, 2, 3, 5, 8, 12], vec=array(range(7, 16, 2)), inst=cls_instance ), } def test_mixed_cls_arr(): json = dumps(mixed_data) back = dict(loads(json)) assert mixed_data.keys() == back.keys() assert (mixed_data['vec'] == back['vec']).all() assert (mixed_data['inst'].vec == back['inst'].vec).all() assert (mixed_data['inst'].nr == back['inst'].nr) assert (mixed_data['inst'].li == back['inst'].li) assert (mixed_data['inst'].inst.s == back['inst'].inst.s) assert (mixed_data['inst'].inst.dct == dict(back['inst'].inst.dct)) def test_memory_order(): arrC = array([[1., 2.], [3., 4.]], order='C') json = dumps(arrC) arr = loads(json) assert array_equal(arrC, arr) assert arrC.flags['C_CONTIGUOUS'] == arr.flags['C_CONTIGUOUS'] and \ arrC.flags['F_CONTIGUOUS'] == arr.flags['F_CONTIGUOUS'] arrF = array([[1., 2.], [3., 4.]], order='F') json = dumps(arrF) arr = loads(json) assert array_equal(arrF, arr) assert arrF.flags['C_CONTIGUOUS'] == arr.flags['C_CONTIGUOUS'] and \ arrF.flags['F_CONTIGUOUS'] == arr.flags['F_CONTIGUOUS'] def test_scalars_types(): # from: https://docs.scipy.org/doc/numpy/user/basics.types.html encme = [] for dtype in DTYPES: for val in (dtype(0),) + get_lims(dtype): assert isinstance(val, dtype) encme.append(val) json = dumps(encme, indent=2) rec = loads(json) assert encme == rec def test_array_types(): # from: https://docs.scipy.org/doc/numpy/user/basics.types.html # see also `test_scalars_types` for dtype in DTYPES: vec = [array((dtype(0), dtype(exp(1))) + get_lims(dtype), dtype=dtype)] json = dumps(vec) assert dtype.__name__ in json rec = loads(json) assert rec[0].dtype == dtype assert array_equal(vec, rec) def test_encode_scalar(): encd = encode_scalars_inplace([complex128(1+2j)]) assert isinstance(encd[0], dict) assert encd[0]['__ndarray__'] == 1+2j assert encd[0]['shape'] == () assert encd[0]['dtype'] == complex128.__name__ def test_dump_np_scalars(): data = [ int8(-27), complex64(exp(1)+37j), ( { 'alpha': float64(-exp(10)), 'str-only': complex64(-1-1j), }, uint32(123456789), float16(exp(-1)), set(( int64(37), uint64(-0), )), ), ] replaced = encode_scalars_inplace(deepcopy(data)) json = dumps(replaced) rec = loads(json) assert data[0] == rec[0] assert data[1] == rec[1] assert data[2][0] == rec[2][0] assert data[2][1] == rec[2][1] assert data[2][2] == rec[2][2] assert data[2][3] == rec[2][3] assert data[2] == tuple(rec[2]) def test_ndarray_object_nesting(): # Based on issue 53 # With nested ndarrays before = zeros((2, 2,), dtype=object) for i in ndindex(before.shape): before[i] = array([1, 2, 3]) after = loads(dumps(before)) assert before.shape == after.shape, \ 'shape of array changed for nested ndarrays:\n{}'.format(dumps(before, indent=2)) assert before.dtype == before.dtype assert array_equal(before[0, 0], after[0, 0]) # With nested lists before = zeros((2, 2,), dtype=object) for i in ndindex(before.shape): before[i] = [1, 2, 3] after = loads(dumps(before)) assert before.shape == after.shape, \ 'shape of array changed for nested ndarrays:\n{}'.format(dumps(before, indent=2)) assert before.dtype == before.dtype assert array_equal(before[0, 0], after[0, 0]) def test_dtype_object(): # Based on issue 64 arr = array(['a', 'b', 'c'], dtype=object) json = dumps(arr) back = loads(json) assert array_equal(back, arr) def test_compact_mode_unspecified(): # Other tests may have raised deprecation warning, so reset the cache here numpy_encode._warned_compact = False data = [array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]), array([pi, exp(1)])] with warns(JsonTricksDeprecation): gz_json_1 = dumps(data, compression=True) # noinspection PyTypeChecker with warns(None) as captured: gz_json_2 = dumps(data, compression=True) assert len(captured) == 0 assert gz_json_1 == gz_json_2 json = gzip_decompress(gz_json_1).decode('ascii') assert json == '[{"__ndarray__": [[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]], "dtype": "float64", "shape": [2, 4], "Corder": true}, ' \ '{"__ndarray__": [3.141592653589793, 2.718281828459045], "dtype": "float64", "shape": [2]}]' def test_compact(): data = [array(list(2**(x + 0.5) for x in range(-30, +31)))] json = dumps(data, compression=True, properties=dict(ndarray_compact=True, ndarray_store_byteorder='little')) back = loads(json) assert_equal(data, back) def test_encode_disable_compact(): data = [array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]), array([pi, exp(1)])] gz_json = dumps(data, compression=True, properties=dict(ndarray_compact=False)) json = gzip_decompress(gz_json).decode('ascii') assert json == '[{"__ndarray__": [[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]], "dtype": "float64", "shape": [2, 4], "Corder": true}, ' \ '{"__ndarray__": [3.141592653589793, 2.718281828459045], "dtype": "float64", "shape": [2]}]' def test_encode_enable_compact_little_endian(): data = [array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]), array([pi, exp(1)])] gz_json = dumps(data, compression=True, properties=dict(ndarray_compact=True, ndarray_store_byteorder='little')) json = gzip_decompress(gz_json).decode('ascii') assert json == '[{"__ndarray__": "b64:AAAAAAAA8D8AAAAAAAAAQAAAAAAAAAhAAAAAAAAAEEAAAAAAAAA' \ 'UQAAAAAAAABhAAAAAAAAAHEAAAAAAAAAgQA==", "dtype": "float64", "shape": [2, 4], "Corder": ' \ 'true, "endian": "little"}, {"__ndarray__": "b64:GC1EVPshCUBpVxSLCr8FQA==", "dtype": "float64", ' \ '"shape": [2], "endian": "little"}]' def test_encode_enable_compact_big_endian(): data = array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]) gz_json = dumps(data, compression=True, properties=dict(ndarray_compact=True, ndarray_store_byteorder='big')) json = gzip_decompress(gz_json).decode('ascii') assert json == '{"__ndarray__": "b64:P/AAAAAAAABAAAAAAAAAAEAIAAAAAAAAQBAAAAAAAABAFAAAAAAAAEAYAA' \ 'AAAAAAQBwAAAAAAABAIAAAAAAAAA==", "dtype": "float64", "shape": [2, 4], "Corder": ' \ 'true, "endian": "big"}' def test_encode_enable_compact_native_endian(): data = array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]) gz_json = dumps(data, compression=True, properties=dict(ndarray_compact=True)) json = gzip_decompress(gz_json).decode('ascii') if sys.byteorder == 'little': assert json == '{"__ndarray__": "b64:AAAAAAAA8D8AAAAAAAAAQAAAAAAAAAhAAAAAAAAAEEAAAAAAAAA' \ 'UQAAAAAAAABhAAAAAAAAAHEAAAAAAAAAgQA==", "dtype": "float64", "shape": [2, 4], "Corder": ' \ 'true, "endian": "little"}' elif sys.byteorder == 'big': assert json == '{"__ndarray__": "b64:P/AAAAAAAABAAAAAAAAAAEAIAAAAAAAAQBAAAAAAAABAFAAAAAAAAEAYAA' \ 'AAAAAAQBwAAAAAAABAIAAAAAAAAA==", "dtype": "float64", "shape": [2, 4], "Corder": ' \ 'true, "endian": "big"}' else: raise Exception("unknown system endianness '{}'".format(sys.byteorder)) def test_encode_enable_compact_suppress_endianness(): data = array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]) gz_json = dumps(data, compression=True, properties=dict(ndarray_compact=True, ndarray_store_byteorder='suppress')) json = gzip_decompress(gz_json).decode('ascii') assert "endian" not in json def test_encode_compact_cutoff(): data = [array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]), array([pi, exp(1)])] gz_json = dumps(data, compression=True, properties=dict(ndarray_compact=5, ndarray_store_byteorder='little')) json = gzip_decompress(gz_json).decode('ascii') assert json == '[{"__ndarray__": "b64:AAAAAAAA8D8AAAAAAAAAQAAAAAAAAAhAAAAAAAAAEEAAAAAAAAA' \ 'UQAAAAAAAABhAAAAAAAAAHEAAAAAAAAAgQA==", "dtype": "float64", "shape": [2, 4], "Corder": ' \ 'true, "endian": "little"}, {"__ndarray__": [3.141592653589793, 2.718281828459045], "dtype": "float64", ' \ '"shape": [2]}]' def test_encode_compact_inline_compression(): data = [array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [9.0, 10.0, 11.0, 12.0], [13.0, 14.0, 15.0, 16.0]])] json = dumps(data, compression=False, properties=dict(ndarray_compact=True, ndarray_store_byteorder='little')) assert 'b64.gz:' in json, 'If the overall file is not compressed and there are significant savings, then do inline gzip compression.' assert json == '[{"__ndarray__": "b64.gz:H4sIAAAAAAAC/2NgAIEP9gwQ4AChOKC0AJQWgdISUFoGSitAaSUorQKl1aC0BpTWgtI6UFoPShs4AABmfqWAgAAAAA==", ' \ '"dtype": "float64", "shape": [4, 4], "Corder": true, "endian": "little"}]' def test_encode_compact_no_inline_compression(): data = [array([[1.0, 2.0], [3.0, 4.0]])] json = dumps(data, compression=False, properties=dict(ndarray_compact=True, ndarray_store_byteorder='little')) assert 'b64.gz:' not in json, 'If the overall file is not compressed, but there are no significant savings, then do not do inline compression.' assert json == '[{"__ndarray__": "b64:AAAAAAAA8D8AAAAAAAAAQAAAAAAAAAhAAAAAAAAAEEA=", ' \ '"dtype": "float64", "shape": [2, 2], "Corder": true, "endian": "little"}]' def test_decode_compact_mixed_compactness(): json = '[{"__ndarray__": "b64:AAAAAAAA8D8AAAAAAAAAQAAAAAAAAAhAAAAAAAAAEEAAAAAAAAA' \ 'UQAAAAAAAABhAAAAAAAAAHEAAAAAAAAAgQA==", "dtype": "float64", "shape": [2, 4], "Corder": ' \ 'true}, {"__ndarray__": [3.141592653589793, 2.718281828459045], "dtype": "float64", "shape": [2]}]' data = loads(json) assert_equal(data[0], array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]), array([pi, exp(1)])) def test_decode_big_endian(): json = '{"__ndarray__": "b64:P/AAAAAAAABAAAAAAAAAAEAIAAAAAAAAQBAAAAAAAABAFAAAAAAAAEAYAA' \ 'AAAAAAQBwAAAAAAABAIAAAAAAAAA==", "dtype": "float64", "shape": [2, 4], "Corder": ' \ 'true, "endian": "big"}' data = loads(json) assert_equal(data, array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]])) def test_decode_little_endian(): json = '{"__ndarray__": "b64:AAAAAAAA8D8AAAAAAAAAQAAAAAAAAAhAAAAAAAAAEEAAAAAAAAA' \ 'UQAAAAAAAABhAAAAAAAAAHEAAAAAAAAAgQA==", "dtype": "float64", "shape": [2, 4], "Corder": ' \ 'true, "endian": "little"}' data = loads(json) assert_equal(data, array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]])) def test_decode_without_endianness(): json = '[{"__ndarray__": "b64:AAAAAAAA8D8AAAAAAAAAQAAAAAAAAAhAAAAAAAAAEEAAAAAAAAA' \ 'UQAAAAAAAABhAAAAAAAAAHEAAAAAAAAAgQA==", "dtype": "float64", "shape": [2, 4], "Corder": true}]' data = loads(json) if sys.byteorder == 'big': import pytest pytest.skip('skip for big endian systems') assert_equal(data[0], array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]])) def test_decode_compact_inline_compression(): json = '[{"__ndarray__": "b64.gz:H4sIAAAAAAAC/2NgAIEP9gwQ4AChOKC0AJQWgdISUFoGSitAaSUorQKl1aC0BpTWgtI6UFoPShs4AABmfqWAgAAAAA==", "dtype": "float64", "shape": [4, 4], "Corder": true}]' data = loads(json) assert_equal(data[0], array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [9.0, 10.0, 11.0, 12.0], [13.0, 14.0, 15.0, 16.0]])) def test_decode_compact_no_inline_compression(): json = '[{"__ndarray__": "b64:AAAAAAAA8D8AAAAAAAAAQAAAAAAAAAhAAAAAAAAAEEA=", ' \ '"dtype": "float64", "shape": [2, 2], "Corder": true}]' data = loads(json) assert_equal(data[0], array([[1.0, 2.0], [3.0, 4.0]])) def test_empty(): # issue https://github.com/mverleg/pyjson_tricks/issues/76 datas = [ zeros(shape=(1, 0)), zeros(shape=(0, 1)), zeros(shape=(0, 0)), ] for data in datas: json = dumps(data) assert_equal(loads(json), data, 'shape = {} ; json = {}'.format(data.shape, json)) def test_decode_writeable(): # issue https://github.com/mverleg/pyjson_tricks/issues/90 data = zeros((2, 2)) data_uncompressed = dumps(data) data_compressed = dumps(data, properties={'ndarray_compact': True}) reloaded_uncompressed = loads(data_uncompressed) reloaded_compressed = loads(data_compressed) assert array_equal(data, reloaded_uncompressed) assert array_equal(data, reloaded_compressed) assert reloaded_uncompressed.flags.writeable assert reloaded_compressed.flags.writeable pyjson_tricks-3.17.3/tests/test_pandas.py000066400000000000000000000040611447012760300205340ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- from collections import OrderedDict from numpy import linspace, isnan from numpy.testing import assert_equal from pandas import DataFrame, Series from json_tricks import dumps, loads from tests.test_bare import nonpdata COLUMNS = OrderedDict(( ('name', ('Alfa', 'Bravo', 'Charlie', 'Delta', 'Echo', 'Foxtrot', 'Golf', 'Hotel', 'India', 'Juliett',)), ('count', linspace(0, 10, 10, dtype=int)), ('real', linspace(0, 7.5, 10, dtype=float)), ('special', (float('NaN'), float('+inf'), float('-inf'), float('+0'), float('-0'), 1, 2, 3, 4, 5)), #todo: other types? )) def test_pandas_dataframe(): df = DataFrame(COLUMNS, columns=tuple(COLUMNS.keys())) txt = dumps(df, allow_nan=True) back = loads(txt) assert isnan(back.iloc[0, -1]) assert (df.equals(back)) assert (df.dtypes == back.dtypes).all() df = DataFrame(COLUMNS, columns=tuple(COLUMNS.keys())) txt = dumps(df, primitives=True, allow_nan=True) back = loads(txt) assert isinstance(back, dict) assert isnan(back['special'][0]) assert all(df.index.values == tuple(back.pop('index'))) for name, col in back.items(): assert name in COLUMNS assert_equal(list(COLUMNS[name]), col) def test_pandas_series(): for name, col in COLUMNS.items(): ds = Series(data=col, name=name) txt = dumps(ds, allow_nan=True) back = loads(txt) assert (ds.equals(back)) assert ds.dtype == back.dtype for name, col in COLUMNS.items(): ds = Series(data=col, name=name) txt = dumps(ds, primitives=True, allow_nan=True) back = loads(txt) assert isinstance(back, dict) assert_equal(ds.index.values, back['index']) assert_equal(ds.values, back['data']) def test_pandas_mixed_with_other_types(): df = DataFrame(COLUMNS, columns=tuple(COLUMNS.keys())) mixed = dict( complex=1+42j, frames=[df, df], **nonpdata ) txt = dumps(mixed, allow_nan=True) back = loads(txt) assert mixed['frames'][0].equals(back['frames'][0]) and mixed['frames'][1].equals(back['frames'][1]) del mixed['frames'], back['frames'] # cannot compare dataframes with '==' assert mixed == back pyjson_tricks-3.17.3/tests/test_pathlib.py000066400000000000000000000013461447012760300207140ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """ This tests Paths, which need pathlib. """ from pathlib import Path from json_tricks import dumps, loads # These paths are not necessarily actual paths that exist, but are sufficient # for testing to ensure that we can properly serialize/deserialize them. PATHS = [ Path(), Path('c:/users/pyjson_tricks'), Path('/home/users/pyjson_tricks'), Path('../'), Path('..'), Path('./'), Path('.'), Path('test_pathlib.py'), Path('/home/users/pyjson_tricks/test_pathlib.py'), ] def test_path(): json = dumps(PATHS) back = loads(json) assert PATHS == back for orig, bck in zip(PATHS, back): assert orig == bck txt = '{"__pathlib__": "."}' obj = loads(txt) assert obj == Path() pyjson_tricks-3.17.3/tests/test_slice.py000066400000000000000000000007151447012760300203670ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- from pathlib import Path from json_tricks import dumps, loads def test_slice(): original_slice = slice(0, 10, 2) json_slice = dumps(original_slice) loaded_slice = loads(json_slice) assert original_slice == loaded_slice def test_slice_no_step(): original_slice = slice(0, 5) json_slice = dumps(original_slice) loaded_slice = loads(json_slice) assert original_slice == loaded_slice pyjson_tricks-3.17.3/tests/test_tz.py000066400000000000000000000107501447012760300177250ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """ This tests timezone-aware date/time objects, which need pytz. Naive date/times should work with just Python code functionality, and are tested in `nonp`. """ from datetime import datetime, date, time, timedelta, timezone from json_tricks import dumps, loads from json_tricks.utils import is_py3 import pytz DTOBJ = [ datetime(year=1988, month=3, day=15, hour=8, minute=3, second=59, microsecond=7), datetime.now(timezone.utc), pytz.UTC.localize(datetime(year=1988, month=3, day=15, minute=3, second=59, microsecond=7)), pytz.timezone('Europe/Amsterdam').localize(datetime(year=1988, month=3, day=15, microsecond=7)), date(year=1988, month=3, day=15), time(hour=8, minute=3, second=59, microsecond=123), time(hour=8, second=59, microsecond=123, tzinfo=pytz.timezone('Europe/Amsterdam')), time(hour=8, second=59, microsecond=123, tzinfo=timezone.utc), timedelta(days=2, seconds=3599), timedelta(days=0, seconds=-42, microseconds=123), [{'obj': [pytz.timezone('Europe/Amsterdam').localize(datetime(year=1988, month=3, day=15, microsecond=7))]}], ] def test_tzaware_date_time_without_dst(): json = dumps(DTOBJ) back = loads(json) assert DTOBJ == back for orig, bck in zip(DTOBJ, back): assert orig == bck assert type(orig) == type(bck) txt = '{"__datetime__": null, "year": 1988, "month": 3, "day": 15, "hour": 8, "minute": 3, ' \ '"second": 59, "microsecond": 7, "tzinfo": "Europe/Amsterdam"}' obj = loads(txt) assert obj == pytz.timezone('Europe/Amsterdam').localize(datetime(year=1988, month=3, day=15, hour=8, minute=3, second=59, microsecond=7)) def test_tzaware_date_time_with_dst(): json = dumps(DTOBJ) back = loads(json) assert DTOBJ == back for orig, bck in zip(DTOBJ, back): assert orig == bck assert type(orig) == type(bck) txt = '{"__datetime__": null, "year": 1988, "month": 3, "day": 15, "hour": 8, "minute": 3, ' \ '"second": 59, "microsecond": 7, "tzinfo": "Europe/Amsterdam", "is_dst": true}' obj = loads(txt) assert obj == pytz.timezone('Europe/Amsterdam').localize(datetime(year=1988, month=3, day=15, hour=8, minute=3, second=59, microsecond=7)) def test_tzaware_naive_date_time(): json = dumps(DTOBJ, primitives=True) back = loads(json) for orig, bck in zip(DTOBJ, back): if isinstance(bck, (date, time, datetime,)): assert isinstance(bck, str if is_py3 else (str, unicode)) assert bck == orig.isoformat() elif isinstance(bck, (timedelta,)): assert isinstance(bck, float) assert bck == orig.total_seconds() dt = pytz.timezone('Europe/Amsterdam').localize(datetime(year=1988, month=3, day=15, hour=8, minute=3, second=59, microsecond=7)) assert dumps(dt, primitives=True).strip('"') == '1988-03-15T08:03:59.000007+01:00' def test_avoiding_tz_datettime_problem(): """ There's a weird problem (bug? feature?) when passing timezone object to datetime constructor. This tests checks that json_tricks doesn't suffer from this problem. https://github.com/mverleg/pyjson_tricks/issues/41 / https://stackoverflow.com/a/25390097/723090 """ tzdt = datetime(2007, 12, 5, 6, 30, 0, 1) tzdt = pytz.timezone('US/Pacific').localize(tzdt) back = loads(dumps([tzdt]))[0] assert pytz.utc.normalize(tzdt) == pytz.utc.normalize(back), \ "Mismatch due to pytz localizing error {} != {}".format( pytz.utc.normalize(tzdt), pytz.utc.normalize(back)) def test_serialization_remains_unchanged(): json = dumps(datetime(2023, 10, 29, 1, 30, 0, 0, pytz.UTC) \ .astimezone(pytz.timezone("Europe/Paris"))) assert json == '{"__datetime__": null, "year": 2023, "month": 10, "day": 29, ' \ '"hour": 2, "minute": 30, "tzinfo": "Europe/Paris", "is_dst": false}' def test_before_dst_fold(): # issue #89 before_dst = datetime(2023, 10, 29, 0, 30, 0, 0, pytz.UTC) \ .astimezone(pytz.timezone("Europe/Paris")) back = loads(dumps(before_dst)) assert back == before_dst assert back.tzinfo.zone == before_dst.tzinfo.zone assert back.utcoffset() == before_dst.utcoffset() def test_after_dst_fold(): after_dst = datetime(2023, 10, 29, 1, 30, 0, 0, pytz.UTC) \ .astimezone(pytz.timezone("Europe/Paris")) back = loads(dumps(after_dst)) assert back == after_dst assert back.tzinfo.zone == after_dst.tzinfo.zone assert back.utcoffset() == after_dst.utcoffset() pyjson_tricks-3.17.3/tests/test_utils.py000066400000000000000000000034101447012760300204230ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- from json_tricks.utils import hashodict, get_arg_names, nested_index def test_hashodict(): data = hashodict((('alpha', 37), ('beta', 42), ('gamma', -99))) assert tuple(data.keys()) == ('alpha', 'beta', 'gamma',) assert isinstance(hash(data), int) def test_get_args(): def get_my_args(hello, world=7): pass argnames = get_arg_names(get_my_args) assert argnames == set(('hello', 'world')) def test_nested_index(): arr = [[[1, 2], [1, 2]], [[1, 2], [3, 3]]] assert 1 == nested_index(arr, (0, 0, 0,)) assert 2 == nested_index(arr, (1, 0, 1,)) assert [1, 2] == nested_index(arr, (1, 0,)) assert [3, 3] == nested_index(arr, (1, 1,)) assert [[1, 2], [1, 2]] == nested_index(arr, (0,)) assert [[[1, 2], [1, 2]], [[1, 2], [3, 3]]] == nested_index(arr, ()) try: nested_index(arr, (0, 0, 0, 0,)) except TypeError: pass else: raise AssertionError('indexing more than nesting level should yield IndexError') def base85_vsbase64_performance(): from base64 import b85encode, standard_b64encode, urlsafe_b64encode from random import getrandbits test_data = bytearray(getrandbits(8) for _ in range(10000000)) from timeit import default_timer print('') start = default_timer() for _ in range(20): standard_b64encode(test_data) end = default_timer() print('standard_b64encode took {} s'.format(end - start)) start = default_timer() for _ in range(20): urlsafe_b64encode(test_data) end = default_timer() print('urlsafe_b64encode took {} s'.format(end - start)) start = default_timer() for _ in range(20): b85encode(test_data) end = default_timer() print('b85encode took {} s'.format(end - start)) # Result on local PC in 2020: base84 is 53x slower to encode # (urlsafe also costs a bit of performance, about 2x)