pytest-arraydiff-0.2/0000755000077000000240000000000013233572322014570 5ustar tomstaff00000000000000pytest-arraydiff-0.2/PKG-INFO0000644000077000000240000002253713233572322015676 0ustar tomstaff00000000000000Metadata-Version: 1.1 Name: pytest-arraydiff Version: 0.2 Summary: pytest plugin to help with comparing array output from tests Home-page: https://github.com/astrofrog/pytest-arraydiff Author: Thomas Robitaille Author-email: thomas.robitaille@gmail.com License: BSD Description: |Travis Build Status| |AppVeyor Build status| |Coveralls coverage| About ----- This is a `py.test `__ plugin to facilitate the generation and comparison of data arrays produced during tests (this is a spin-off from `pytest-arraydiff `__). The basic idea is that you can write a test that generates a Numpy array (or other related objects depending on the format). You can then either run the tests in a mode to **generate** reference files from the arrays, or you can run the tests in **comparison** mode, which will compare the results of the tests to the reference ones within some tolerance. At the moment, the supported file formats for the reference files are: - A plain text-based format (baed on Numpy ``loadtxt`` output) - The FITS format (requires `astropy `__). With this format, tests can return either a Numpy array for a FITS HDU object. For more information on how to write tests to do this, see the **Using** section below. Installing ---------- This plugin is compatible with Python 2.7, and 3.5 and later, and requires `pytest `__ and `numpy `__ to be installed. To install, you can do: :: pip install pytest-arraydiff You can check that the plugin is registered with pytest by doing: :: py.test --version which will show a list of plugins: :: This is pytest version 2.7.1, imported from ... setuptools registered plugins: pytest-arraydiff-0.1 at ... Using ----- To use, you simply need to mark the function where you want to compare arrays using ``@pytest.mark.array_compare``, and make sure that the function returns a plain Numpy array: :: python import pytest import numpy as np @pytest.mark.array_compare def test_succeeds(): return np.arange(3 * 5 * 4).reshape((3, 5, 4)) To generate the reference data files, run the tests with the ``--arraydiff-generate-path`` option with the name of the directory where the generated files should be placed: :: py.test --arraydiff-generate-path=reference If the directory does not exist, it will be created. The directory will be interpreted as being relative to where you are running ``py.test``. Make sure you manually check the reference arrays to ensure they are correct. Once you are happy with the generated data files, you should move them to a sub-directory called ``reference`` relative to the test files (this name is configurable, see below). You can also generate the baseline arrays directly in the right directory. You can then run the tests simply with: :: py.test --arraydiff and the tests will pass if the arrays are the same. If you omit the ``--arraydiff`` option, the tests will run but will only check that the code runs without checking the output arrays. Options ------- The ``@pytest.mark.array_compare`` marker take an argument to specify the format to use for the reference files: .. code:: python @pytest.mark.array_compare(file_format='text') def test_array(): ... The default file format can also be specified using the ``--arraydiff-default-format=`` flag when running ``py.test``, and ```` should be either ``fits`` or ``text``. The supported formats at this time are ``text`` and ``fits``, and contributions for other formats are welcome. The default format is ``text``. Another argument is the relative tolerance for floating point values (which defaults to 1e-7): .. code:: python @pytest.mark.array_compare(rtol=20) def test_array(): ... You can also pass keyword arguments to the writers using the ``write_kwargs``. For the ``text`` format, these arguments are passed to ``savetxt`` while for the ``fits`` format they are passed to Astropy's ``fits.writeto`` function. .. code:: python @pytest.mark.array_compare(file_format='fits', write_kwargs={'output_verify': 'silentfix'}) def test_array(): ... Other options include the name of the reference directory (which defaults to ``reference`` ) and the filename for the reference file (which defaults to the name of the test with a format-dependent extension). .. code:: python @pytest.mark.array_compare(reference_dir='baseline_arrays', filename='other_name.fits') def test_array(): ... The reference directory in the decorator above will be interpreted as being relative to the test file. Note that the baseline directory can also be a URL (which should start with ``http://`` or ``https://`` and end in a slash). Finally, you can also set a custom baseline directory globally when running tests by running ``py.test`` with: :: py.test --arraydiff --arraydiff-reference-path=baseline_arrays This directory will be interpreted as being relative to where the tests are run. In addition, if both this option and the ``reference_dir`` option in the ``array_compare`` decorator are used, the one in the decorator takes precedence. Test failure example -------------------- If the arrays produced by the tests are correct, then the test will pass, but if they are not, the test will fail with a message similar to the following: :: E AssertionError: E E a: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/test_to_mask_rect-mode_subpixels-subpixels_18.txt E b: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/reference-test_to_mask_rect-mode_subpixels-subpixels_18.txt E E Not equal to tolerance rtol=1e-07, atol=0 E E (mismatch 47.22222222222222%) E x: array([[ 0. , 0. , 0. , 0. , 0.404012, 0.55 , E 0.023765, 0. , 0. ], E [ 0. , 0. , 0. , 0.112037, 1.028704, 1.1 ,... E y: array([[ 0. , 0. , 0. , 0. , 0.367284, 0.5 , E 0.021605, 0. , 0. ], E [ 0. , 0. , 0. , 0.101852, 0.935185, 1. ,... The file paths included in the exception are then available for inspection. Running the tests for pytest-arraydiff -------------------------------------- If you are contributing some changes and want to run the tests, first install the latest version of the plugin then do: :: cd tests py.test --arraydiff The reason for having to install the plugin first is to ensure that the plugin is correctly loaded as part of the test suite. .. |Travis Build Status| image:: https://travis-ci.org/astrofrog/pytest-arraydiff.svg?branch=master :target: https://travis-ci.org/astrofrog/pytest-arraydiff .. |AppVeyor Build status| image:: https://ci.appveyor.com/api/projects/status/4hfoe954rxhgslsl/branch/master?svg=true :target: https://ci.appveyor.com/project/astrofrog/pytest-arraydiff .. |Coveralls coverage| image:: https://coveralls.io/repos/matplotlib/pytest-arraydiff/badge.svg :target: https://coveralls.io/r/matplotlib/pytest-arraydiff Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Framework :: Pytest Classifier: Intended Audience :: Developers Classifier: Topic :: Software Development :: Testing Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 3 Classifier: Operating System :: OS Independent Classifier: License :: OSI Approved :: BSD License pytest-arraydiff-0.2/pytest_arraydiff/0000755000077000000240000000000013233572322020147 5ustar tomstaff00000000000000pytest-arraydiff-0.2/pytest_arraydiff/__init__.py0000755000077000000240000000002413233572312022256 0ustar tomstaff00000000000000__version__ = '0.2' pytest-arraydiff-0.2/pytest_arraydiff/plugin.py0000755000077000000240000002647013233557651022043 0ustar tomstaff00000000000000# Copyright (c) 2016, Thomas P. Robitaille # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: # # 1. Redistributions of source code must retain the above copyright notice, # this list of conditions and the following disclaimer. # # 2. Redistributions in binary form must reproduce the above copyright notice, # this list of conditions and the following disclaimer in the documentation # and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. # # This package was derived from pytest-mpl, which is released under a BSD # license and can be found here: # # https://github.com/astrofrog/pytest-mpl from functools import wraps import os import abc import shutil import tempfile import warnings import six from six.moves.urllib.request import urlopen import pytest import numpy as np if six.PY2: def abstractstaticmethod(func): return func def abstractclassmethod(func): return func else: abstractstaticmethod = abc.abstractstaticmethod abstractclassmethod = abc.abstractclassmethod @six.add_metaclass(abc.ABCMeta) class BaseDiff(object): @abstractstaticmethod def read(filename): """ Given a filename, return a data object. """ raise NotImplementedError() @abstractstaticmethod def write(filename, data, **kwargs): """ Given a filename and a data object (and optional keyword arguments), write the data to a file. """ raise NotImplementedError() @abstractclassmethod def compare(self, reference_file, test_file, atol=None, rtol=None): """ Given a reference and test filename, compare the data to the specified absolute (``atol``) and relative (``rtol``) tolerances. Should return two arguments: a boolean indicating whether the data are identical, and a string giving the full error message if not. """ raise NotImplementedError() class SimpleArrayDiff(BaseDiff): @classmethod def compare(cls, reference_file, test_file, atol=None, rtol=None): array_ref = cls.read(reference_file) array_new = cls.read(test_file) try: np.testing.assert_allclose(array_ref, array_new, atol=atol, rtol=rtol) except AssertionError as exc: message = "\n\na: {0}".format(test_file) + '\n' message += "b: {0}".format(reference_file) + '\n' message += exc.args[0] return False, message else: return True, "" class FITSDiff(BaseDiff): extension = 'fits' @staticmethod def read(filename): from astropy.io import fits return fits.getdata(filename) @staticmethod def write(filename, data, **kwargs): from astropy.io import fits if isinstance(data, np.ndarray): data = fits.PrimaryHDU(data) return data.writeto(filename, **kwargs) @classmethod def compare(cls, reference_file, test_file, atol=None, rtol=None): import astropy from astropy.io.fits.diff import FITSDiff from astropy.utils.introspection import minversion if minversion(astropy, '2.0'): diff = FITSDiff(reference_file, test_file, rtol=rtol) else: diff = FITSDiff(reference_file, test_file, tolerance=rtol) return diff.identical, diff.report() class TextDiff(SimpleArrayDiff): extension = 'txt' @staticmethod def read(filename): return np.loadtxt(filename) @staticmethod def write(filename, data, **kwargs): fmt = kwargs.get('fmt', '%g') # Workaround for a known issue in `numpy.savetxt` for the `fmt` argument: # https://github.com/numpy/numpy/pull/4053#issuecomment-263808221 # Convert `unicode` to `str` (i.e. bytes) on Python 2 if six.PY2 and isinstance(fmt, six.text_type): fmt = fmt.encode('ascii') kwargs['fmt'] = fmt return np.savetxt(filename, data, **kwargs) FORMATS = {} FORMATS['fits'] = FITSDiff FORMATS['text'] = TextDiff def _download_file(url): u = urlopen(url) result_dir = tempfile.mkdtemp() filename = os.path.join(result_dir, 'downloaded') with open(filename, 'wb') as tmpfile: tmpfile.write(u.read()) return filename def pytest_addoption(parser): group = parser.getgroup("general") group.addoption('--arraydiff', action='store_true', help="Enable comparison of arrays to reference arrays stored in files") group.addoption('--arraydiff-generate-path', help="directory to generate reference files in, relative to location where py.test is run", action='store') group.addoption('--arraydiff-reference-path', help="directory containing reference files, relative to location where py.test is run", action='store') group.addoption('--arraydiff-default-format', help="Default format for the reference arrays (can be 'fits' or 'text' currently)") def pytest_configure(config): if config.getoption("--arraydiff") or config.getoption("--arraydiff-generate-path") is not None: reference_dir = config.getoption("--arraydiff-reference-path") generate_dir = config.getoption("--arraydiff-generate-path") if reference_dir is not None and generate_dir is not None: warnings.warn("Ignoring --arraydiff-reference-path since --arraydiff-generate-path is set") if reference_dir is not None: reference_dir = os.path.abspath(reference_dir) if generate_dir is not None: reference_dir = os.path.abspath(generate_dir) default_format = config.getoption("--arraydiff-default-format") or 'text' config.pluginmanager.register(ArrayComparison(config, reference_dir=reference_dir, generate_dir=generate_dir, default_format=default_format)) class ArrayComparison(object): def __init__(self, config, reference_dir=None, generate_dir=None, default_format='text'): self.config = config self.reference_dir = reference_dir self.generate_dir = generate_dir self.default_format = default_format def pytest_runtest_setup(self, item): compare = item.keywords.get('array_compare') if compare is None: return file_format = compare.kwargs.get('file_format', self.default_format) if file_format not in FORMATS: raise ValueError("Unknown format: {0}".format(file_format)) if 'extension' in compare.kwargs: extension = compare.kwargs['extension'] else: extension = FORMATS[file_format].extension atol = compare.kwargs.get('atol', 0.) rtol = compare.kwargs.get('rtol', 1e-7) single_reference = compare.kwargs.get('single_reference', False) write_kwargs = compare.kwargs.get('write_kwargs', {}) original = item.function @wraps(item.function) def item_function_wrapper(*args, **kwargs): reference_dir = compare.kwargs.get('reference_dir', None) if reference_dir is None: if self.reference_dir is None: reference_dir = os.path.join(os.path.dirname(item.fspath.strpath), 'reference') else: reference_dir = self.reference_dir else: if not reference_dir.startswith(('http://', 'https://')): reference_dir = os.path.join(os.path.dirname(item.fspath.strpath), reference_dir) baseline_remote = reference_dir.startswith('http') # Run test and get figure object import inspect if inspect.ismethod(original): # method array = original(*args[1:], **kwargs) else: # function array = original(*args, **kwargs) # Find test name to use as plot name filename = compare.kwargs.get('filename', None) if filename is None: if single_reference: filename = original.__name__ + '.' + extension else: filename = item.name + '.' + extension filename = filename.replace('[', '_').replace(']', '_') filename = filename.replace('_.' + extension, '.' + extension) # What we do now depends on whether we are generating the reference # files or simply running the test. if self.generate_dir is None: # Save the figure result_dir = tempfile.mkdtemp() test_array = os.path.abspath(os.path.join(result_dir, filename)) FORMATS[file_format].write(test_array, array, **write_kwargs) # Find path to baseline array if baseline_remote: baseline_file_ref = _download_file(reference_dir + filename) else: baseline_file_ref = os.path.abspath(os.path.join(os.path.dirname(item.fspath.strpath), reference_dir, filename)) if not os.path.exists(baseline_file_ref): raise Exception("""File not found for comparison test Generated file: \t{test} This is expected for new tests.""".format( test=test_array)) # distutils may put the baseline arrays in non-accessible places, # copy to our tmpdir to be sure to keep them in case of failure baseline_file = os.path.abspath(os.path.join(result_dir, 'reference-' + filename)) shutil.copyfile(baseline_file_ref, baseline_file) identical, msg = FORMATS[file_format].compare(baseline_file, test_array, atol=atol, rtol=rtol) if identical: shutil.rmtree(result_dir) else: raise Exception(msg) else: if not os.path.exists(self.generate_dir): os.makedirs(self.generate_dir) FORMATS[file_format].write(os.path.abspath(os.path.join(self.generate_dir, filename)), array, **write_kwargs) pytest.skip("Skipping test, since generating data") if item.cls is not None: setattr(item.cls, item.function.__name__, item_function_wrapper) else: item.obj = item_function_wrapper pytest-arraydiff-0.2/LICENSE0000644000077000000240000000265013233571576015612 0ustar tomstaff00000000000000Copyright (c) 2016, Thomas P. Robitaille All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. This package was adapted from pytest-mpl, which is released under a BSD license and can be found here: https://github.com/astrofrog/pytest-mpl pytest-arraydiff-0.2/tests/0000755000077000000240000000000013233572322015732 5ustar tomstaff00000000000000pytest-arraydiff-0.2/tests/baseline/0000755000077000000240000000000013233572322017514 5ustar tomstaff00000000000000pytest-arraydiff-0.2/tests/baseline/test_succeeds_func_default.txt0000644000077000000240000000004313016275531025627 0ustar tomstaff000000000000000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 pytest-arraydiff-0.2/tests/baseline/test_tolerance.fits0000644000077000000240000001320013016275531023413 0ustar tomstaff00000000000000SIMPLE = T / conforms to FITS standard BITPIX = -64 / array data type NAXIS = 2 / number of array dimensions NAXIS1 = 4 NAXIS2 = 3 EXTEND = T END ?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™š?ù™™™™™špytest-arraydiff-0.2/tests/baseline/test_succeeds_func_text.txt0000644000077000000240000000004313016275531025167 0ustar tomstaff000000000000000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 pytest-arraydiff-0.2/tests/baseline/test_succeeds_func_fits_hdu.fits0000644000077000000240000001320013233557651026144 0ustar tomstaff00000000000000SIMPLE = T / conforms to FITS standard BITPIX = 64 / array data type NAXIS = 2 / number of array dimensions NAXIS1 = 5 NAXIS2 = 3 EXTEND = T END  pytest-arraydiff-0.2/tests/baseline/test_succeeds_func_fits.fits0000644000077000000240000001320013016275531025275 0ustar tomstaff00000000000000SIMPLE = T / conforms to FITS standard BITPIX = 64 / array data type NAXIS = 2 / number of array dimensions NAXIS1 = 5 NAXIS2 = 3 EXTEND = T END  pytest-arraydiff-0.2/tests/baseline/test_succeeds_class.fits0000644000077000000240000001320013016275531024422 0ustar tomstaff00000000000000SIMPLE = T / conforms to FITS standard BITPIX = 64 / array data type NAXIS = 3 / number of array dimensions NAXIS1 = 3 NAXIS2 = 4 NAXIS3 = 2 EXTEND = T END  pytest-arraydiff-0.2/tests/test_pytest_arraydiff.py0000644000077000000240000001004013233557651022725 0ustar tomstaff00000000000000import os import sys import subprocess import tempfile import pytest import numpy as np PY2 = sys.version_info[0] == 2 reference_dir = 'baseline' @pytest.mark.array_compare(reference_dir=reference_dir) def test_succeeds_func_default(): return np.arange(3 * 5).reshape((3, 5)) @pytest.mark.array_compare(file_format='text', reference_dir=reference_dir) def test_succeeds_func_text(): return np.arange(3 * 5).reshape((3, 5)) @pytest.mark.array_compare(file_format='fits', reference_dir=reference_dir) def test_succeeds_func_fits(): return np.arange(3 * 5).reshape((3, 5)).astype(np.int64) @pytest.mark.array_compare(file_format='fits', reference_dir=reference_dir) def test_succeeds_func_fits_hdu(): from astropy.io import fits return fits.PrimaryHDU(np.arange(3 * 5).reshape((3, 5)).astype(np.int64)) class TestClass(object): @pytest.mark.array_compare(file_format='fits', reference_dir=reference_dir) def test_succeeds_class(self): return np.arange(2 * 4 * 3).reshape((2, 4, 3)).astype(np.int64) TEST_FAILING = """ import pytest import numpy as np from astropy.io import fits @pytest.mark.array_compare def test_fail(): return np.ones((3, 4)) """ def test_fails(): tmpdir = tempfile.mkdtemp() test_file = os.path.join(tmpdir, 'test.py') with open(test_file, 'w') as f: f.write(TEST_FAILING) # If we use --arraydiff, it should detect that the file is missing code = subprocess.call('py.test --arraydiff {0}'.format(test_file), shell=True) assert code != 0 # If we don't use --arraydiff option, the test should succeed code = subprocess.call('py.test {0}'.format(test_file), shell=True) assert code == 0 TEST_GENERATE = """ import pytest import numpy as np from astropy.io import fits @pytest.mark.array_compare(file_format='{file_format}') def test_gen(): return np.arange(6 * 5).reshape((6, 5)) """ @pytest.mark.parametrize('file_format', ('fits', 'text')) def test_generate(file_format): tmpdir = tempfile.mkdtemp() test_file = os.path.join(tmpdir, 'test.py') with open(test_file, 'w') as f: f.write(TEST_GENERATE.format(file_format=file_format)) gen_dir = os.path.join(tmpdir, 'spam', 'egg') # If we don't generate, the test will fail try: if PY2: subprocess.check_output(['pytest', '--arraydiff', test_file]) else: subprocess.check_output(['pytest', '--arraydiff', test_file], timeout=10) except subprocess.CalledProcessError as grepexc: assert b'File not found for comparison test' in grepexc.output # If we do generate, the test should succeed and a new file will appear if PY2: code = subprocess.call(['pytest', '--arraydiff-generate-path={0}'.format(gen_dir), test_file]) else: code = subprocess.call(['pytest', '--arraydiff-generate-path={0}'.format(gen_dir), test_file], timeout=10) assert code == 0 assert os.path.exists(os.path.join(gen_dir, 'test_gen.' + ('fits' if file_format == 'fits' else 'txt'))) TEST_DEFAULT = """ import pytest import numpy as np from astropy.io import fits @pytest.mark.array_compare def test_default(): return np.arange(6 * 5).reshape((6, 5)) """ @pytest.mark.parametrize('file_format', ('fits', 'text')) def test_default_format(file_format): tmpdir = tempfile.mkdtemp() test_file = os.path.join(tmpdir, 'test.py') with open(test_file, 'w') as f: f.write(TEST_DEFAULT) gen_dir = os.path.join(tmpdir, 'spam', 'egg') # If we do generate, the test should succeed and a new file will appear code = subprocess.call('py.test -s --arraydiff-default-format={0}' ' --arraydiff-generate-path={1} {2}'.format(file_format, gen_dir, test_file), shell=True) assert code == 0 assert os.path.exists(os.path.join(gen_dir, 'test_default.' + ('fits' if file_format == 'fits' else 'txt'))) @pytest.mark.array_compare(reference_dir=reference_dir, rtol=0.5, file_format='fits') def test_tolerance(): return np.ones((3,4)) * 1.6 def test_nofile(): pass pytest-arraydiff-0.2/MANIFEST.in0000644000077000000240000000016113233571576016336 0ustar tomstaff00000000000000include LICENSE include README.rst include CHANGES.md include tox.ini recursive-include tests *.py *.fits *.txt pytest-arraydiff-0.2/pytest_arraydiff.egg-info/0000755000077000000240000000000013233572322021641 5ustar tomstaff00000000000000pytest-arraydiff-0.2/pytest_arraydiff.egg-info/PKG-INFO0000644000077000000240000002253713233572322022747 0ustar tomstaff00000000000000Metadata-Version: 1.1 Name: pytest-arraydiff Version: 0.2 Summary: pytest plugin to help with comparing array output from tests Home-page: https://github.com/astrofrog/pytest-arraydiff Author: Thomas Robitaille Author-email: thomas.robitaille@gmail.com License: BSD Description: |Travis Build Status| |AppVeyor Build status| |Coveralls coverage| About ----- This is a `py.test `__ plugin to facilitate the generation and comparison of data arrays produced during tests (this is a spin-off from `pytest-arraydiff `__). The basic idea is that you can write a test that generates a Numpy array (or other related objects depending on the format). You can then either run the tests in a mode to **generate** reference files from the arrays, or you can run the tests in **comparison** mode, which will compare the results of the tests to the reference ones within some tolerance. At the moment, the supported file formats for the reference files are: - A plain text-based format (baed on Numpy ``loadtxt`` output) - The FITS format (requires `astropy `__). With this format, tests can return either a Numpy array for a FITS HDU object. For more information on how to write tests to do this, see the **Using** section below. Installing ---------- This plugin is compatible with Python 2.7, and 3.5 and later, and requires `pytest `__ and `numpy `__ to be installed. To install, you can do: :: pip install pytest-arraydiff You can check that the plugin is registered with pytest by doing: :: py.test --version which will show a list of plugins: :: This is pytest version 2.7.1, imported from ... setuptools registered plugins: pytest-arraydiff-0.1 at ... Using ----- To use, you simply need to mark the function where you want to compare arrays using ``@pytest.mark.array_compare``, and make sure that the function returns a plain Numpy array: :: python import pytest import numpy as np @pytest.mark.array_compare def test_succeeds(): return np.arange(3 * 5 * 4).reshape((3, 5, 4)) To generate the reference data files, run the tests with the ``--arraydiff-generate-path`` option with the name of the directory where the generated files should be placed: :: py.test --arraydiff-generate-path=reference If the directory does not exist, it will be created. The directory will be interpreted as being relative to where you are running ``py.test``. Make sure you manually check the reference arrays to ensure they are correct. Once you are happy with the generated data files, you should move them to a sub-directory called ``reference`` relative to the test files (this name is configurable, see below). You can also generate the baseline arrays directly in the right directory. You can then run the tests simply with: :: py.test --arraydiff and the tests will pass if the arrays are the same. If you omit the ``--arraydiff`` option, the tests will run but will only check that the code runs without checking the output arrays. Options ------- The ``@pytest.mark.array_compare`` marker take an argument to specify the format to use for the reference files: .. code:: python @pytest.mark.array_compare(file_format='text') def test_array(): ... The default file format can also be specified using the ``--arraydiff-default-format=`` flag when running ``py.test``, and ```` should be either ``fits`` or ``text``. The supported formats at this time are ``text`` and ``fits``, and contributions for other formats are welcome. The default format is ``text``. Another argument is the relative tolerance for floating point values (which defaults to 1e-7): .. code:: python @pytest.mark.array_compare(rtol=20) def test_array(): ... You can also pass keyword arguments to the writers using the ``write_kwargs``. For the ``text`` format, these arguments are passed to ``savetxt`` while for the ``fits`` format they are passed to Astropy's ``fits.writeto`` function. .. code:: python @pytest.mark.array_compare(file_format='fits', write_kwargs={'output_verify': 'silentfix'}) def test_array(): ... Other options include the name of the reference directory (which defaults to ``reference`` ) and the filename for the reference file (which defaults to the name of the test with a format-dependent extension). .. code:: python @pytest.mark.array_compare(reference_dir='baseline_arrays', filename='other_name.fits') def test_array(): ... The reference directory in the decorator above will be interpreted as being relative to the test file. Note that the baseline directory can also be a URL (which should start with ``http://`` or ``https://`` and end in a slash). Finally, you can also set a custom baseline directory globally when running tests by running ``py.test`` with: :: py.test --arraydiff --arraydiff-reference-path=baseline_arrays This directory will be interpreted as being relative to where the tests are run. In addition, if both this option and the ``reference_dir`` option in the ``array_compare`` decorator are used, the one in the decorator takes precedence. Test failure example -------------------- If the arrays produced by the tests are correct, then the test will pass, but if they are not, the test will fail with a message similar to the following: :: E AssertionError: E E a: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/test_to_mask_rect-mode_subpixels-subpixels_18.txt E b: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/reference-test_to_mask_rect-mode_subpixels-subpixels_18.txt E E Not equal to tolerance rtol=1e-07, atol=0 E E (mismatch 47.22222222222222%) E x: array([[ 0. , 0. , 0. , 0. , 0.404012, 0.55 , E 0.023765, 0. , 0. ], E [ 0. , 0. , 0. , 0.112037, 1.028704, 1.1 ,... E y: array([[ 0. , 0. , 0. , 0. , 0.367284, 0.5 , E 0.021605, 0. , 0. ], E [ 0. , 0. , 0. , 0.101852, 0.935185, 1. ,... The file paths included in the exception are then available for inspection. Running the tests for pytest-arraydiff -------------------------------------- If you are contributing some changes and want to run the tests, first install the latest version of the plugin then do: :: cd tests py.test --arraydiff The reason for having to install the plugin first is to ensure that the plugin is correctly loaded as part of the test suite. .. |Travis Build Status| image:: https://travis-ci.org/astrofrog/pytest-arraydiff.svg?branch=master :target: https://travis-ci.org/astrofrog/pytest-arraydiff .. |AppVeyor Build status| image:: https://ci.appveyor.com/api/projects/status/4hfoe954rxhgslsl/branch/master?svg=true :target: https://ci.appveyor.com/project/astrofrog/pytest-arraydiff .. |Coveralls coverage| image:: https://coveralls.io/repos/matplotlib/pytest-arraydiff/badge.svg :target: https://coveralls.io/r/matplotlib/pytest-arraydiff Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Framework :: Pytest Classifier: Intended Audience :: Developers Classifier: Topic :: Software Development :: Testing Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 3 Classifier: Operating System :: OS Independent Classifier: License :: OSI Approved :: BSD License pytest-arraydiff-0.2/pytest_arraydiff.egg-info/SOURCES.txt0000644000077000000240000000117313233572322023527 0ustar tomstaff00000000000000CHANGES.md LICENSE MANIFEST.in README.rst setup.py pytest_arraydiff/__init__.py pytest_arraydiff/plugin.py pytest_arraydiff.egg-info/PKG-INFO pytest_arraydiff.egg-info/SOURCES.txt pytest_arraydiff.egg-info/dependency_links.txt pytest_arraydiff.egg-info/entry_points.txt pytest_arraydiff.egg-info/requires.txt pytest_arraydiff.egg-info/top_level.txt tests/test_pytest_arraydiff.py tests/baseline/test_succeeds_class.fits tests/baseline/test_succeeds_func_default.txt tests/baseline/test_succeeds_func_fits.fits tests/baseline/test_succeeds_func_fits_hdu.fits tests/baseline/test_succeeds_func_text.txt tests/baseline/test_tolerance.fitspytest-arraydiff-0.2/pytest_arraydiff.egg-info/entry_points.txt0000644000077000000240000000006713233572322025142 0ustar tomstaff00000000000000[pytest11] pytest_arraydiff = pytest_arraydiff.plugin pytest-arraydiff-0.2/pytest_arraydiff.egg-info/requires.txt0000644000077000000240000000002113233572322024232 0ustar tomstaff00000000000000numpy six pytest pytest-arraydiff-0.2/pytest_arraydiff.egg-info/top_level.txt0000644000077000000240000000002113233572322024364 0ustar tomstaff00000000000000pytest_arraydiff pytest-arraydiff-0.2/pytest_arraydiff.egg-info/dependency_links.txt0000644000077000000240000000000113233572322025707 0ustar tomstaff00000000000000 pytest-arraydiff-0.2/setup.py0000755000077000000240000000253213233557651016317 0ustar tomstaff00000000000000from setuptools import setup from pytest_arraydiff import __version__ # IMPORTANT: we deliberately use rst here instead of markdown because long_description # needs to be in rst, and requiring pandoc to be installed to convert markdown to rst # on-the-fly is over-complicated and sometimes the generated rst has warnings that # cause PyPI to not display it correctly. with open('README.rst') as infile: long_description = infile.read() setup( version=__version__, url="https://github.com/astrofrog/pytest-arraydiff", name="pytest-arraydiff", description='pytest plugin to help with comparing array output from tests', long_description=long_description, packages=['pytest_arraydiff'], install_requires=['numpy', 'six', 'pytest'], license='BSD', author='Thomas Robitaille', author_email='thomas.robitaille@gmail.com', entry_points={'pytest11': ['pytest_arraydiff = pytest_arraydiff.plugin']}, classifiers=[ 'Development Status :: 4 - Beta', 'Framework :: Pytest', 'Intended Audience :: Developers', 'Topic :: Software Development :: Testing', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 3', 'Operating System :: OS Independent', 'License :: OSI Approved :: BSD License', ], ) pytest-arraydiff-0.2/CHANGES.md0000644000077000000240000000033213233572312016157 0ustar tomstaff000000000000000.2 (2018-01-29) ---------------- - Fix compatibility with recent versions of Astropy and Numpy. [#8, #10] - Add back support for returning HDUs from tests. [#5] 0.1 (2016-11-26) ---------------- - Initial version pytest-arraydiff-0.2/setup.cfg0000644000077000000240000000007313233572322016411 0ustar tomstaff00000000000000[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 pytest-arraydiff-0.2/README.rst0000644000077000000240000001576413233571576016306 0ustar tomstaff00000000000000|Travis Build Status| |AppVeyor Build status| |Coveralls coverage| About ----- This is a `py.test `__ plugin to facilitate the generation and comparison of data arrays produced during tests (this is a spin-off from `pytest-arraydiff `__). The basic idea is that you can write a test that generates a Numpy array (or other related objects depending on the format). You can then either run the tests in a mode to **generate** reference files from the arrays, or you can run the tests in **comparison** mode, which will compare the results of the tests to the reference ones within some tolerance. At the moment, the supported file formats for the reference files are: - A plain text-based format (baed on Numpy ``loadtxt`` output) - The FITS format (requires `astropy `__). With this format, tests can return either a Numpy array for a FITS HDU object. For more information on how to write tests to do this, see the **Using** section below. Installing ---------- This plugin is compatible with Python 2.7, and 3.5 and later, and requires `pytest `__ and `numpy `__ to be installed. To install, you can do: :: pip install pytest-arraydiff You can check that the plugin is registered with pytest by doing: :: py.test --version which will show a list of plugins: :: This is pytest version 2.7.1, imported from ... setuptools registered plugins: pytest-arraydiff-0.1 at ... Using ----- To use, you simply need to mark the function where you want to compare arrays using ``@pytest.mark.array_compare``, and make sure that the function returns a plain Numpy array: :: python import pytest import numpy as np @pytest.mark.array_compare def test_succeeds(): return np.arange(3 * 5 * 4).reshape((3, 5, 4)) To generate the reference data files, run the tests with the ``--arraydiff-generate-path`` option with the name of the directory where the generated files should be placed: :: py.test --arraydiff-generate-path=reference If the directory does not exist, it will be created. The directory will be interpreted as being relative to where you are running ``py.test``. Make sure you manually check the reference arrays to ensure they are correct. Once you are happy with the generated data files, you should move them to a sub-directory called ``reference`` relative to the test files (this name is configurable, see below). You can also generate the baseline arrays directly in the right directory. You can then run the tests simply with: :: py.test --arraydiff and the tests will pass if the arrays are the same. If you omit the ``--arraydiff`` option, the tests will run but will only check that the code runs without checking the output arrays. Options ------- The ``@pytest.mark.array_compare`` marker take an argument to specify the format to use for the reference files: .. code:: python @pytest.mark.array_compare(file_format='text') def test_array(): ... The default file format can also be specified using the ``--arraydiff-default-format=`` flag when running ``py.test``, and ```` should be either ``fits`` or ``text``. The supported formats at this time are ``text`` and ``fits``, and contributions for other formats are welcome. The default format is ``text``. Another argument is the relative tolerance for floating point values (which defaults to 1e-7): .. code:: python @pytest.mark.array_compare(rtol=20) def test_array(): ... You can also pass keyword arguments to the writers using the ``write_kwargs``. For the ``text`` format, these arguments are passed to ``savetxt`` while for the ``fits`` format they are passed to Astropy's ``fits.writeto`` function. .. code:: python @pytest.mark.array_compare(file_format='fits', write_kwargs={'output_verify': 'silentfix'}) def test_array(): ... Other options include the name of the reference directory (which defaults to ``reference`` ) and the filename for the reference file (which defaults to the name of the test with a format-dependent extension). .. code:: python @pytest.mark.array_compare(reference_dir='baseline_arrays', filename='other_name.fits') def test_array(): ... The reference directory in the decorator above will be interpreted as being relative to the test file. Note that the baseline directory can also be a URL (which should start with ``http://`` or ``https://`` and end in a slash). Finally, you can also set a custom baseline directory globally when running tests by running ``py.test`` with: :: py.test --arraydiff --arraydiff-reference-path=baseline_arrays This directory will be interpreted as being relative to where the tests are run. In addition, if both this option and the ``reference_dir`` option in the ``array_compare`` decorator are used, the one in the decorator takes precedence. Test failure example -------------------- If the arrays produced by the tests are correct, then the test will pass, but if they are not, the test will fail with a message similar to the following: :: E AssertionError: E E a: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/test_to_mask_rect-mode_subpixels-subpixels_18.txt E b: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/reference-test_to_mask_rect-mode_subpixels-subpixels_18.txt E E Not equal to tolerance rtol=1e-07, atol=0 E E (mismatch 47.22222222222222%) E x: array([[ 0. , 0. , 0. , 0. , 0.404012, 0.55 , E 0.023765, 0. , 0. ], E [ 0. , 0. , 0. , 0.112037, 1.028704, 1.1 ,... E y: array([[ 0. , 0. , 0. , 0. , 0.367284, 0.5 , E 0.021605, 0. , 0. ], E [ 0. , 0. , 0. , 0.101852, 0.935185, 1. ,... The file paths included in the exception are then available for inspection. Running the tests for pytest-arraydiff -------------------------------------- If you are contributing some changes and want to run the tests, first install the latest version of the plugin then do: :: cd tests py.test --arraydiff The reason for having to install the plugin first is to ensure that the plugin is correctly loaded as part of the test suite. .. |Travis Build Status| image:: https://travis-ci.org/astrofrog/pytest-arraydiff.svg?branch=master :target: https://travis-ci.org/astrofrog/pytest-arraydiff .. |AppVeyor Build status| image:: https://ci.appveyor.com/api/projects/status/4hfoe954rxhgslsl/branch/master?svg=true :target: https://ci.appveyor.com/project/astrofrog/pytest-arraydiff .. |Coveralls coverage| image:: https://coveralls.io/repos/matplotlib/pytest-arraydiff/badge.svg :target: https://coveralls.io/r/matplotlib/pytest-arraydiff