lazyarray-0.1.0/0000755000077300000240000000000011711532254014131 5ustar andrewstaff00000000000000lazyarray-0.1.0/doc/0000755000077300000240000000000011711532254014676 5ustar andrewstaff00000000000000lazyarray-0.1.0/doc/developers.txt0000644000077300000240000000043511710077137017614 0ustar andrewstaff00000000000000================= Developers' guide ================= TO BE COMPLETED Testing ======= In the `test` sub-directory, run:: $ nosetests To see how well the tests cover the code base, run:: $ nosetests --with-coverage --cover-package=lazyarray --cover-erase --cover-html lazyarray-0.1.0/doc/index.txt0000644000077300000240000000262411711466430016554 0ustar andrewstaff00000000000000========= lazyarray ========= lazyarray is a Python package that provides a lazily-evaluated numerical array class, :class:`larray`, based on and compatible with NumPy arrays. Lazy evaluation means that any operations on the array (potentially including array construction) are not performed immediately, but are delayed until evaluation is specifically requested. Evaluation of only parts of the array is also possible. Use of an :class:`larray` can potentially save considerable computation time and memory in cases where: * arrays are used conditionally (i.e. there are cases in which the array is never used); * only parts of an array are used (for example in distributed computation, in which each MPI node operates on a subset of the elements of the array). It appears that much of this functionality may appear in a future version of NumPy (see `this discussion`_ on the numpy-discussion mailing list), but at the time of writing I couldn't find anything equivalent out there. DistNumPy_ might also be an alternative for some of the use cases of lazyarray. Contents ======== .. toctree:: :maxdepth: 2 installation tutorial performance reference developers Licence ======= The code is released under the Modified BSD licence. .. _`this discussion`: http://www.mail-archive.com/numpy-discussion@scipy.org/msg29732.html .. _DistNumPy: http://sites.google.com/site/distnumpy/lazyarray-0.1.0/doc/installation.txt0000644000077300000240000000227211707624432020150 0ustar andrewstaff00000000000000============ Installation ============ Dependencies ============ * Python >= 2.6 * numpy_ >= 1.1 (or >= 1.5 for Python 3) Installing from the Python Package Index ======================================== If you have pip_ installed:: $ pip install lazyarray This will automatically download and install the latest release (you may need to have administrator privileges on the machine you are installing on). To download and install manually, download: http://pypi.python.org/packages/source/L/lazyarray/lazyarray-0.1.0.tar.gz Then:: $ tar xzf lazyarray-0.1.0.tar.gz $ cd lazyarray-0.1.0 $ python setup.py install or:: $ python3 setup.py install depending on which version of Python you are using. Installing from source ====================== To install the latest version of lazyarray from the Mercurial repository:: $ hg clone https://bitbucket.org/apdavison/lazyarray lazyarray_hg $ cd lazyarray_hg $ python setup.py install .. _`numpy`: http://numpy.scipy.org/ .. _`quantities`: http://pypi.python.org/pypi/quantities .. _`pip`: http://pypi.python.org/pypi/pip .. _`setuptools`: http://pypi.python.org/pypi/setuptoolslazyarray-0.1.0/doc/performance.txt0000644000077300000240000000372111711530723017742 0ustar andrewstaff00000000000000=========== Performance =========== The main aim of lazyarray is to improve performance (increased speed and reduced memory use) in two scenarios: * arrays are used conditionally (i.e. there are cases in which the array is never used); * only parts of an array are used (for example in distributed computation, in which each MPI node operates on a subset of the elements of the array). However, at the same time use of :class:`larray` objects should not be too much slower than plain NumPy arrays in normal use. Here we see that using a lazyarray adds minimal overhead compared to using a plain array: :: >>> from timeit import repeat >>> repeat('np.fromfunction(lambda i,j: i*i + 2*i*j + 3, (5000, 5000))', ... setup='import numpy as np', number=1, repeat=5) [1.9397640228271484, 1.92628812789917, 1.8796701431274414, 1.6766629219055176, 1.6844701766967773] >>> repeat('larray(lambda i,j: i*i + 2*i*j + 3, (5000, 5000)).evaluate()', ... setup='from lazyarray import larray', number=1, repeat=5) [1.686661958694458, 1.6836578845977783, 1.6853220462799072, 1.6538069248199463, 1.645576000213623] While if we only need to evaluate part of the array (perhaps because the other parts are being evaluated on other nodes), there is a major gain from using a lazy array. :: >>> repeat('np.fromfunction(lambda i,j: i*i + 2*i*j + 3, (5000, 5000))[:, 0:4999:10]', ... setup='import numpy as np', number=1, repeat=5) [1.691796064376831, 1.668884038925171, 1.647057056427002, 1.6792259216308594, 1.652547836303711] >>> repeat('larray(lambda i,j: i*i + 2*i*j + 3, (5000, 5000))[:, 0:4999:10]', ... setup='from lazyarray import larray', number=1, repeat=5) [0.23157119750976562, 0.16121792793273926, 0.1594078540802002, 0.16096210479736328, 0.16096997261047363] .. note:: These timings were done on a MacBook Pro 2.8 GHz Intel Core 2 Duo with 4 GB RAM, Python 2.7 and NumPy 1.6.1.lazyarray-0.1.0/doc/reference.txt0000644000077300000240000000055711707625441017412 0ustar andrewstaff00000000000000========= Reference ========= .. autoclass:: lazyarray.larray :members: apply, nrows, ncols, is_homogeneous .. method:: evaluate(simplify=False) Return the lazy array as a real NumPy array. If the array is homogeneous and ``simplify`` is ``True``, return a single numerical value. # need to document shape attributelazyarray-0.1.0/doc/tutorial.txt0000644000077300000240000001502711710077444017313 0ustar andrewstaff00000000000000======== Tutorial ======== The :mod:`lazyarray` module contains a single class, :class:`larray`. .. doctest:: >>> from lazyarray import larray Creating a lazy array ===================== Lazy arrays may be created from single numbers, from sequences (lists, NumPy arrays), from iterators, from generators, or from a certain class of functions. Here are some examples: .. doctest:: >>> from_number = larray(20.0) >>> from_list = larray([0, 1, 1, 2, 3, 5, 8]) >>> import numpy as np >>> from_array = larray(np.arange(6).reshape((2, 3))) >>> from_iter = larray(iter(range(8))) >>> from_gen = larray((x**2 + 2*x + 3 for x in range(5))) To create a lazy array from a function or other callable, the function must accept one or more integers as arguments (depending on the dimensionality of the array) and return a single number. .. doctest:: >>> def f(i, j): ... return i*np.sin(np.pi*j/100) >>> from_func = larray(f) Specifying array shape ---------------------- Where the :class:`larray` is created from something that does not already have a known shape (i.e. from something that is not a list or array), it is possible to specify the shape of the array at the time of construction: .. doctest:: >>> from_func2 = larray(lambda i: 2*i, shape=(6,)) >>> print(from_func2.shape) (6,) For sequences, the shape is introspected: .. doctest:: >>> from_list.shape (7,) >>> from_array.shape (2, 3) Otherwise, the :attr:`shape` attribute is set to ``None``, and must be set later before the array can be evaluated. .. doctest:: >>> print(from_number.shape) None >>> print(from_iter.shape) None >>> print(from_gen.shape) None >>> print(from_func.shape) None Evaluating a lazy array ======================= The simplest way to evaluate a lazy array is with the :meth:`evaluate` method, which returns a NumPy array: .. doctest:: >>> from_list.evaluate() array([0, 1, 1, 2, 3, 5, 8]) >>> from_array.evaluate() array([[0, 1, 2], [3, 4, 5]]) >>> from_number.evaluate() Traceback (most recent call last): File "", line 1, in File "/Users/andrew/dev/lazyarray/lazyarray.py", line 35, in wrapped_meth raise ValueError("Shape of larray not specified") ValueError: Shape of larray not specified >>> from_number.shape = (2, 2) >>> from_number.evaluate() array([[ 20., 20.], [ 20., 20.]]) Note that an :class:`larray` can only be evaluated once its shape has been defined. Note also that a lazy array created from a single number evaluates to a homogeneous array containing that number. To obtain just the value, use the ``simplify`` argument: .. doctest:: >>> from_number.evaluate(simplify=True) 20.0 Evaluating a lazy array created from an iterator or generator fills the array in row-first order. The number of values generated by the iterator must fit within the array shape: .. doctest:: >>> from_iter.shape = (2, 4) >>> from_iter.evaluate() array([[ 0., 1., 2., 3.], [ 4., 5., 6., 7.]]) >>> from_gen.shape = (5,) >>> from_gen.evaluate() array([ 3., 6., 11., 18., 27.]) If it doesn't, an Exception is raised: .. doctest:: >>> from_iter.shape = (7,) >>> from_iter.evaluate() Traceback (most recent call last): File "", line 1, in from_iter.evaluate() File "/Users/andrew/dev/lazyarray/lazyarray.py", line 36, in wrapped_meth return meth(self, *args, **kwargs) File "/Users/andrew/dev/lazyarray/lazyarray.py", line 235, in evaluate x = x.reshape(self.shape) ValueError: total size of new array must be unchanged When evaluating a lazy array created from a callable, the function is called with the indices of each element of the array: .. doctest:: >>> from_func.shape = (3, 4) >>> from_func.evaluate() array([[ 0. , 0. , 0. , 0. ], [ 0. , 0.03141076, 0.06279052, 0.09410831], [ 0. , 0.06282152, 0.12558104, 0.18821663]]) It is also possible to evaluate only parts of an array. This is explained below. Performing operations on a lazy array ===================================== Just as with a normal NumPy array, it is possible to perform elementwise arithmetic operations: .. doctest:: >>> a = from_list + 2 >>> b = 2*a >>> print(type(b)) However, these operations are not carried out immediately, rather they are queued up to be carried out later, which can lead to large time and memory savings if the evaluation step turns out later not to be needed, or if only part of the array needs to be evaluated. .. doctest:: >>> b.evaluate() array([ 4, 6, 6, 8, 10, 14, 20]) Some more examples: .. doctest:: >>> a = 1.0/(from_list + 1) >>> a.evaluate() array([ 1. , 0.5 , 0.5 , 0.33333333, 0.25 , 0.16666667, 0.11111111]) >>> (from_list < 2).evaluate() array([ True, True, True, False, False, False, False], dtype=bool) >>> (from_list**2).evaluate() array([ 0, 1, 1, 4, 9, 25, 64]) >>> x = from_list >>> (x**2 - 2*x + 5).evaluate() array([ 5, 4, 4, 5, 8, 20, 53]) Numpy ufuncs cannot be used directly with lazy arrays, as NumPy does not know what to do with :class:`larray` objects. The lazyarray module therefore provides lazy array-compatible versions of a subset of the NumPy ufuncs, e.g.: .. doctest:: >>> from lazyarray import sqrt >>> sqrt(from_list).evaluate() array([ 0. , 1. , 1. , 1.41421356, 1.73205081, 2.23606798, 2.82842712]) For any other function that operates on a NumPy array, it can be applied to a lazy array using the :meth:`apply()` method: .. doctest:: >>> def g(x): ... return x**2 - 2*x + 5 >>> from_list.apply(g) >>> from_list.evaluate() array([ 5, 4, 4, 5, 8, 20, 53]) Partial evaluation ================== When accessing a single element of an array, only that element is evaluated, where possible, not the whole array: .. doctest:: >>> x = larray(lambda i,j: 2*i + 3*j, shape=(4, 5)) >>> x[3, 2] 12 >>> y = larray(lambda i: i*(2-i), shape=(6,)) >>> y[4] -8 The same is true for accessing individual rows or columns: .. doctest:: >>> x[1] array([ 2, 5, 8, 11, 14]) >>> x[:, 4] array([12, 14, 16, 18]) >>> x[:, (0, 4)] array([[ 0, 12], [ 2, 14], [ 4, 16], [ 6, 18]]) lazyarray-0.1.0/lazyarray.py0000644000077300000240000003012311711531563016522 0ustar andrewstaff00000000000000""" lazyarray is a Python package that provides a lazily-evaluated numerical array class, ``larray``, based on and compatible with NumPy arrays. Copyright Andrew P. Davison, 2012 """ from __future__ import division import numpy import operator from copy import deepcopy import collections from functools import wraps __version__ = "0.1.0" # stuff for Python 3 compatibility try: long except NameError: long = int try: reduce except NameError: from functools import reduce def check_shape(meth): """ Decorator for larray magic methods, to ensure that the operand has the same shape as the array. """ @wraps(meth) def wrapped_meth(self, val): if isinstance(val, (larray, numpy.ndarray)): if val.shape != self.shape: raise ValueError("shape mismatch: objects cannot be broadcast to a single shape") return meth(self, val) return wrapped_meth def requires_shape(meth): @wraps(meth) def wrapped_meth(self, *args, **kwargs): if self.shape is None: raise ValueError("Shape of larray not specified") return meth(self, *args, **kwargs) return wrapped_meth def reverse(func): """Given a function f(a, b), returns f(b, a)""" def reversed_func(a, b): return func(b, a) reversed_func.__doc__ = "Reversed argument form of %s" % func.__doc__ return reversed_func def lazy_operation(name, reversed=False): def op(self, val): new_map = deepcopy(self) f = getattr(operator, name) if reversed: f = reverse(f) new_map.operations.append((f, val)) return new_map return check_shape(op) def lazy_inplace_operation(name): def op(self, val): self.operations.append((getattr(operator, name), val)) return self return check_shape(op) def lazy_unary_operation(name): def op(self): new_map = deepcopy(self) new_map.operations.append((getattr(operator, name), None)) return new_map return op class larray(object): """ Optimises storage of and operations on arrays in various ways: - stores only a single value if all the values in the array are the same; - if the array is created from a function `f(i)` or `f(i,j)`, then elements are only evaluated when they are accessed. Any operations performed on the array are also queued up to be executed on access. Two use cases for the latter are: - to save memory for very large arrays by accessing them one row or column at a time: the entire array need never be in memory. - in parallelized code, different rows or columns may be evaluated on different nodes or in different threads. """ def __init__(self, value, shape=None): """ Create a new lazy array. `value` : may be an int, long, float, bool, NumPy array, iterator, generator or a function, `f(i)` or `f(i,j)`, depending on the dimensions of the array. `f(i,j)` should return a single number when `i` and `j` are integers, and a 1D array when either `i` or `j` or both is a NumPy array (in the latter case the two arrays musy have equal lengths). """ if isinstance(value, collections.Sized): # False for numbers, generators, functions, iterators #assert numpy.isreal(value).all() if not isinstance(value, numpy.ndarray): value = numpy.array(value) if shape: assert value.shape == shape, "Array has shape %s, value has shape %s" % (shape, value.shape) self.shape = value.shape else: assert numpy.isreal(value) # also True for callables, generators, iterators self.shape = shape self.base_value = value self.operations = [] def __deepcopy__(self, memo): obj = larray.__new__(larray) try: obj.base_value = deepcopy(self.base_value) except TypeError: # base_value cannot be copied, e.g. is a generator (but see generator_tools from PyPI) obj.base_value = self.base_value # so here we create a reference rather than deepcopying - could cause problems obj.shape = self.shape obj.operations = deepcopy(self.operations) return obj @property @requires_shape def nrows(self): """Size of the first dimension of the array.""" return self.shape[0] @property @requires_shape def ncols(self): """Size of the second dimension (if it exists) of the array.""" if len(self.shape) > 1: return self.shape[1] else: return 1 @property @requires_shape def size(self): return reduce(operator.mul, self.shape) @property def is_homogeneous(self): """True if all the elements of the array are the same.""" hom_base = isinstance(self.base_value, (int, long, float, bool)) hom_ops = all(isinstance(obj.base_value, (int, long, float, bool)) for obj in self.operations if isinstance(obj, larray)) return hom_base and hom_ops def _homogeneous_array(self, addr): self.check_bounds(addr) def size(x, max): if isinstance(x, (int, long)): return 1 elif isinstance(x, slice): return ((x.stop or max) - (x.start or 0)) // (x.step or 1) elif isinstance(x, collections.Sized): return len(x) addr = self._full_address(addr) shape = [size(x, max) for (x, max) in zip(addr, self.shape)] if shape == [1] or shape == [1, 1]: return 1 else: shape = [x for x in shape if x > 1] # remove empty dimensions return numpy.ones(shape, type(self.base_value)) def _full_address(self, addr): if not isinstance(addr, tuple): addr = (addr,) if len(addr) < len(self.shape): full_addr = [slice(None)] * len(self.shape) for i, val in enumerate(addr): full_addr[i] = val addr = full_addr return addr def _array_indices(self, addr): self.check_bounds(addr) def axis_indices(x, max): if isinstance(x, (int, long)): return x elif isinstance(x, slice): # need to handle negative values in slice return numpy.arange((x.start or 0), (x.stop or max), (x.step or 1), dtype=int) elif isinstance(x, collections.Sized): return x addr = self._full_address(addr) indices = [axis_indices(x, max) for (x, max) in zip(addr, self.shape)] if len(indices) == 1: return indices elif len(indices) == 2: if isinstance(indices[0], collections.Sized): if isinstance(indices[1], collections.Sized): mesh_xy = numpy.meshgrid(*indices) return (mesh_xy[0].T, mesh_xy[1].T) # meshgrid works on (x,y), not (i,j) return indices else: raise NotImplementedError("Only 1D and 2D arrays supported") @requires_shape def __getitem__(self, addr): return self._partially_evaluate(addr, simplify=False) def _partially_evaluate(self, addr, simplify=False): """ Return part of the lazy array. """ if self.is_homogeneous: base_val = self._homogeneous_array(addr) * self.base_value elif isinstance(self.base_value, numpy.ndarray): base_val = self.base_value[addr] elif callable(self.base_value): indices = self._array_indices(addr) base_val = self.base_value(*indices) elif isinstance(self.base_value, collections.Iterator): raise NotImplementedError("coming soon...") else: raise ValueError("invalid base value for array") return self._apply_operations(base_val, addr, simplify=simplify) @requires_shape def check_bounds(self, addr): """ Check whether the given address is within the array bounds. """ def check_axis(x, size): if isinstance(x, (int, long)): lower = upper = x elif isinstance(x, slice): lower = x.start or 0 upper = x.stop or size-1 elif isinstance(x, collections.Sized): lower = min(x) upper = max(x) else: raise TypeError("check_bounds() requires a valid array address") if (lower < -size) or (upper >= size): raise IndexError("index out of bounds") addr = self._full_address(addr) for i, size in zip(addr, self.shape): check_axis(i, size) def apply(self, f): """ Add the function `f(x)` to the list of the operations to be performed, where `x` will be a scalar or a numpy array. >>> m = larray(4, shape=(2,2)) >>> m.apply(numpy.sqrt) >>> m.evaluate() array([[ 2., 2.], [ 2., 2.]]) """ self.operations.append((f, None)) def _apply_operations(self, x, addr=None, simplify=False): for f, arg in self.operations: if arg is None: x = f(x) elif isinstance(arg, larray): if addr is None: x = f(x, arg.evaluate(simplify=simplify)) else: x = f(x, arg._partially_evaluate(addr, simplify=simplify)) else: x = f(x, arg) return x @requires_shape def evaluate(self, simplify=False): """ Return the lazy array as a real NumPy array. If the array is homogeneous and ``simplify`` is ``True``, return a single numerical value. """ # need to catch the situation where a generator-based larray is evaluated a second time if self.is_homogeneous: if simplify: x = self.base_value else: x = self.base_value * numpy.ones(self.shape) elif isinstance(self.base_value, numpy.ndarray): x = self.base_value elif callable(self.base_value): x = numpy.fromfunction(self.base_value, shape=self.shape) elif isinstance(self.base_value, collections.Iterator): x = numpy.fromiter(self.base_value, dtype=float, count=self.size) if x.shape != self.shape: x = x.reshape(self.shape) else: raise ValueError("invalid base value for array") return self._apply_operations(x, simplify=simplify) __iadd__ = lazy_inplace_operation('add') __isub__ = lazy_inplace_operation('sub') __imul__ = lazy_inplace_operation('mul') __idiv__ = lazy_inplace_operation('div') __ipow__ = lazy_inplace_operation('pow') __add__ = lazy_operation('add') __radd__ = __add__ __sub__ = lazy_operation('sub') __rsub__ = lazy_operation('sub', reversed=True) __mul__ = lazy_operation('mul') __rmul__ = __mul__ __div__ = lazy_operation('div') __rdiv__ = lazy_operation('div', reversed=True) __truediv__ = lazy_operation('truediv') __truediv__ = lazy_operation('truediv', reversed=True) __pow__ = lazy_operation('pow') __lt__ = lazy_operation('lt') __gt__ = lazy_operation('gt') __le__ = lazy_operation('le') __ge__ = lazy_operation('ge') __neg__ = lazy_unary_operation('neg') __pos__ = lazy_unary_operation('pos') __abs__ = lazy_unary_operation('abs') def _build_ufunc(func): """Return a ufunc that works with lazy arrays""" def larray_compatible_ufunc(x): if isinstance(x, larray): y = deepcopy(x) y.apply(func) return y else: return func(x) return larray_compatible_ufunc # build lazy-array comptible versions of NumPy ufuncs namespace = globals() for name in dir(numpy): obj = getattr(numpy, name) if isinstance(obj, numpy.ufunc): namespace[name] = _build_ufunc(obj) lazyarray-0.1.0/LICENSE0000644000077300000240000000271311704023333015134 0ustar andrewstaff00000000000000Copyright (c) 2012, Andrew P. Davison All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the names of the copyright holders nor the names of the contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. lazyarray-0.1.0/PKG-INFO0000644000077300000240000000320111711532254015222 0ustar andrewstaff00000000000000Metadata-Version: 1.0 Name: lazyarray Version: 0.1.0 Summary: a Python package that provides a lazily-evaluated numerical array class, larray, based on and compatible with NumPy arrays. Home-page: http://bitbucket.org/apdavison/lazyarray/ Author: Andrew P. Davison Author-email: andrew.davison@unic.cnrs-gif.fr License: Modified BSD Description: ========= lazyarray ========= lazyarray is a Python package that provides a lazily-evaluated numerical array class, ``larray``, based on and compatible with NumPy arrays. Lazy evaluation means that any operations on the array (potentially including array construction) are not performed immediately, but are delayed until evaluation is specifically requested. Evaluation of only parts of the array is also possible. Use of an ``larray`` can potentially save considerable computation time and memory in cases where: * arrays are used conditionally (i.e. there are cases in which the array is never used) * only parts of an array are used (for example in distributed computation, in which each MPI node operates on a subset of the elements of the array) Platform: UNKNOWN Classifier: Development Status :: 3 - Alpha Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: BSD License Classifier: Natural Language :: English Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Scientific/Engineering lazyarray-0.1.0/README0000644000077300000240000000141011704031057015002 0ustar andrewstaff00000000000000 ========= lazyarray ========= lazyarray is a Python package that provides a lazily-evaluated numerical array class, ``larray``, based on and compatible with NumPy arrays. Lazy evaluation means that any operations on the array (potentially including array construction) are not performed immediately, but are delayed until evaluation is specifically requested. Evaluation of only parts of the array is also possible. Use of an ``larray`` can potentially save considerable computation time and memory in cases where: * arrays are used conditionally (i.e. there are cases in which the array is never used) * only parts of an array are used (for example in distributed computation, in which each MPI node operates on a subset of the elements of the array)lazyarray-0.1.0/setup.py0000644000077300000240000000165211711531530015643 0ustar andrewstaff00000000000000# -*- coding: utf-8 -*- from distutils.core import setup setup( name='lazyarray', version='0.1.0', py_modules=['lazyarray'], license='Modified BSD', author="Andrew P. Davison", author_email="andrew.davison@unic.cnrs-gif.fr", url="http://bitbucket.org/apdavison/lazyarray/", description="a Python package that provides a lazily-evaluated numerical array class, larray, based on and compatible with NumPy arrays.", long_description=open('README').read(), install_requires=[ "numpy >= 1.5" ], classifiers=[ 'Development Status :: 3 - Alpha', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: BSD License', 'Natural Language :: English', 'Operating System :: OS Independent', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 3', 'Topic :: Scientific/Engineering', ] ) lazyarray-0.1.0/test/0000755000077300000240000000000011711532254015110 5ustar andrewstaff00000000000000lazyarray-0.1.0/test/test_lazyarray.py0000644000077300000240000002750211711524273020547 0ustar andrewstaff00000000000000""" Unit tests for ``larray`` class Copyright Andrew P. Davison, 2012 """ from lazyarray import larray import numpy from nose.tools import assert_raises, assert_equal from numpy.testing import assert_array_equal, assert_array_almost_equal import operator #class MockRNG(random.WrappedRNG): # rng = None # # def __init__(self, parallel_safe): # random.WrappedRNG.__init__(self, parallel_safe=parallel_safe) # self.start = 0.0 # # def _next(self, distribution, n, parameters): # s = self.start # self.start += n*0.1 # return numpy.arange(s, s+n*0.1, 0.1) # test larray def test_create_with_int(): A = larray(3, shape=(5,)) assert A.shape == (5,) assert A.evaluate(simplify=True) == 3 def test_create_with_float(): A = larray(3.0, shape=(5,)) assert A.shape == (5,) assert A.evaluate(simplify=True) == 3.0 def test_create_with_list(): A = larray([1,2,3], shape=(3,)) assert A.shape == (3,) assert_array_equal(A.evaluate(), numpy.array([1,2,3])) def test_create_with_array(): A = larray(numpy.array([1,2,3]), shape=(3,)) assert A.shape == (3,) assert_array_equal(A.evaluate(), numpy.array([1,2,3])) def test_create_with_generator(): def plusone(): i = 0 while True: yield i i += 1 A = larray(plusone(), shape=(5, 11)) assert_array_equal(A.evaluate(), numpy.arange(55).reshape((5, 11))) def test_create_with_function1D(): A = larray(lambda i: 99-i, shape=(3,)) assert_array_equal(A.evaluate(), numpy.array([99, 98, 97])) def test_create_with_function2D(): A = larray(lambda i,j: 3*j-2*i, shape=(2, 3)) assert_array_equal(A.evaluate(), numpy.array([[0, 3, 6], [-2, 1, 4]])) def test_create_inconsistent(): assert_raises(AssertionError, larray, [1,2,3], shape=4) def test_create_with_string(): assert_raises(AssertionError, larray, "123", shape=3) #def test_columnwise_iteration_with_flat_array(): # m = larray(5, shape=(4,3)) # 4 rows, 3 columns # cols = [col for col in m.by_column()] # assert_equal(cols, [5, 5, 5]) # #def test_columnwise_iteration_with_structured_array(): # input = numpy.arange(12).reshape((4,3)) # m = larray(input, shape=(4,3)) # 4 rows, 3 columns # cols = [col for col in m.by_column()] # assert_array_equal(cols[0], input[:,0]) # assert_array_equal(cols[2], input[:,2]) # #def test_columnwise_iteration_with_function(): # input = lambda i,j: 2*i + j # m = larray(input, shape=(4,3)) # cols = [col for col in m.by_column()] # assert_array_equal(cols[0], numpy.array([0, 2, 4, 6])) # assert_array_equal(cols[1], numpy.array([1, 3, 5, 7])) # assert_array_equal(cols[2], numpy.array([2, 4, 6, 8])) # #def test_columnwise_iteration_with_flat_array_and_mask(): # m = larray(5, shape=(4,3)) # 4 rows, 3 columns # mask = numpy.array([True, False, True]) # cols = [col for col in m.by_column(mask=mask)] # assert_equal(cols, [5, 5]) # #def test_columnwise_iteration_with_structured_array_and_mask(): # input = numpy.arange(12).reshape((4,3)) # m = larray(input, shape=(4,3)) # 4 rows, 3 columns # mask = numpy.array([False, True, True]) # cols = [col for col in m.by_column(mask=mask)] # assert_array_equal(cols[0], input[:,1]) # assert_array_equal(cols[1], input[:,2]) def test_size_related_properties(): m1 = larray(1, shape=(9,7)) m2 = larray(1, shape=(13,)) m3 = larray(1) assert_equal(m1.nrows, 9) assert_equal(m1.ncols, 7) assert_equal(m1.size, 63) assert_equal(m2.nrows, 13) assert_equal(m2.ncols, 1) assert_equal(m2.size, 13) assert_raises(ValueError, lambda: m3.nrows) assert_raises(ValueError, lambda: m3.ncols) assert_raises(ValueError, lambda: m3.size) def test_evaluate_with_flat_array(): m = larray(5, shape=(4,3)) assert_array_equal(m.evaluate(), 5*numpy.ones((4,3))) def test_evaluate_with_structured_array(): input = numpy.arange(12).reshape((4,3)) m = larray(input, shape=(4,3)) assert_array_equal(m.evaluate(), input) def test_evaluate_with_functional_array(): input = lambda i,j: 2*i + j m = larray(input, shape=(4,3)) assert_array_equal(m.evaluate(), numpy.array([[0, 1, 2], [2, 3, 4], [4, 5, 6], [6, 7, 8]])) def test_iadd_with_flat_array(): m = larray(5, shape=(4,3)) m += 2 assert_array_equal(m.evaluate(), 7*numpy.ones((4,3))) assert_equal(m.base_value, 5) assert_equal(m.evaluate(simplify=True), 7) def test_add_with_flat_array(): m0 = larray(5, shape=(4,3)) m1 = m0 + 2 assert_equal(m1.evaluate(simplify=True), 7) assert_equal(m0.evaluate(simplify=True), 5) def test_lt_with_flat_array(): m0 = larray(5, shape=(4,3)) m1 = m0 < 10 assert_equal(m1.evaluate(simplify=True), True) assert_equal(m0.evaluate(simplify=True), 5) def test_lt_with_structured_array(): input = numpy.arange(12).reshape((4,3)) m0 = larray(input, shape=(4,3)) m1 = m0 < 5 assert_array_equal(m1.evaluate(simplify=True), input < 5) def test_structured_array_lt_array(): input = numpy.arange(12).reshape((4,3)) m0 = larray(input, shape=(4,3)) comparison = 5*numpy.ones((4,3)) m1 = m0 < comparison assert_array_equal(m1.evaluate(simplify=True), input < comparison) def test_rsub_with_structured_array(): m = larray(numpy.arange(12).reshape((4, 3))) assert_array_equal((11 - m).evaluate(), numpy.arange(11, -1, -1).reshape((4, 3))) def test_inplace_mul_with_structured_array(): m = larray((3*x for x in range(4)), shape=(4,)) m *= 7 assert_array_equal(m.evaluate(), numpy.arange(0, 84, 21)) def test_abs_with_structured_array(): m = larray(lambda i,j: i-j, shape=(3,4)) assert_array_equal(abs(m).evaluate(), numpy.array([[0, 1, 2, 3], [1, 0, 1, 2], [2, 1, 0, 1]])) def test_multiple_operations_with_structured_array(): input = numpy.arange(12).reshape((4,3)) m0 = larray(input, shape=(4,3)) m1 = (m0 + 2) < 5 m2 = (m0 < 5) + 2 assert_array_equal(m1.evaluate(simplify=True), (input+2)<5) assert_array_equal(m2.evaluate(simplify=True), (input<5)+2) assert_array_equal(m0.evaluate(simplify=True), input) def test_apply_function_to_constant_array(): f = lambda m: 2*m + 3 m0 = larray(5, shape=(4,3)) m1 = f(m0) assert isinstance(m1, larray) assert_equal(m1.evaluate(simplify=True), 13) # the following tests the internals, not the behaviour # it is just to check I understand what's going on assert_equal(m1.operations, [(operator.mul, 2), (operator.add, 3)]) def test_apply_function_to_structured_array(): f = lambda m: 2*m + 3 input = numpy.arange(12).reshape((4,3)) m0 = larray(input, shape=(4,3)) m1 = f(m0) assert isinstance(m1, larray) assert_array_equal(m1.evaluate(simplify=True), input*2 + 3) def test_apply_function_to_functional_array(): input = lambda i,j: 2*i + j m0 = larray(input, shape=(4,3)) f = lambda m: 2*m + 3 m1 = f(m0) assert_array_equal(m1.evaluate(), numpy.array([[3, 5, 7], [7, 9, 11], [11, 13, 15], [15, 17, 19]])) def test_add_two_constant_arrays(): m0 = larray(5, shape=(4,3)) m1 = larray(7, shape=(4,3)) m2 = m0 + m1 assert_equal(m2.evaluate(simplify=True), 12) # the following tests the internals, not the behaviour # it is just to check I understand what's going on assert_equal(m2.base_value, m0.base_value) assert_equal(m2.operations, [(operator.add, m1)]) def test_add_incommensurate_arrays(): m0 = larray(5, shape=(4,3)) m1 = larray(7, shape=(5,3)) assert_raises(ValueError, m0.__add__, m1) def test_getitem_from_2D_constant_array(): m = larray(3, shape=(4,3)) assert m[0,0] == m[3,2] == m[-1,2] == m[-4,2] == m[2,-3] == 3 assert_raises(IndexError, m.__getitem__, (4, 0)) assert_raises(IndexError, m.__getitem__, (2, -4)) def test_getitem_from_1D_constant_array(): m = larray(3, shape=(43,)) assert m[0] == m[42] == 3 def test_getitem__with_slice_from_constant_array(): m = larray(3, shape=(4, 3)) assert_array_equal(m[:3, 0], numpy.array([3, 3, 3])) def test_getitem__with_thinslice_from_constant_array(): m = larray(3, shape=(4, 3)) assert_equal(m[2:3, 0:1], 3) def test_getitem__with_mask_from_constant_array(): m = larray(3, shape=(4, 3)) assert_array_equal(m[1, (0, 2)], numpy.array([3, 3])) def test_getslice_from_constant_array(): m = larray(3, shape=(4, 3)) assert_array_equal(m[:2], numpy.array([[3, 3, 3], [3, 3, 3]])) def test_getitem_from_structured_array(): m = larray(3*numpy.ones((4,3)), shape=(4,3)) assert m[0,0] == m[3,2] == m[-1,2] == m[-4,2] == m[2,-3] == 3 assert_raises(IndexError, m.__getitem__, (4,0)) assert_raises(IndexError, m.__getitem__, (2,-4)) def test_getitem_from_2D_functional_array(): m = larray(lambda i,j: 2*i + j, shape=(6,5)) assert_equal(m[5, 4], 14) def test_getitem_from_1D_functional_array(): m = larray(lambda i: i**3, shape=(6,)) assert_equal(m[5], 125) def test_getitem_from_3D_functional_array(): m = larray(lambda i,j,k: i+j+k, shape=(2,3,4)) assert_raises(NotImplementedError, m.__getitem__, (0,1,2)) def test_getitem_with_slice_from_2D_functional_array(): m = larray(lambda i,j: 2*i + j, shape=(6,5)) assert_array_equal(m[2:5, 3:], numpy.array([[7, 8], [9, 10], [11, 12]])) def test_getitem_with_slice_from_2D_functional_array_2(): def test_function(i, j): return i*i + 2*i*j + 3 m = larray(test_function, shape=(3,15)) assert_array_equal(m[:, 3:14:3], numpy.fromfunction(test_function, shape=(3,15))[:, 3:14:3]) def test_getitem_with_mask_from_2D_functional_array(): m = larray(lambda i,j: 2*i + j, shape=(6,5)) assert_array_equal(m[[2, 3, 4], [3, 4]], numpy.array([[7, 8], [9, 10], [11, 12]])) def test_getitem_with_mask_from_1D_functional_array(): m = larray(lambda i: numpy.sqrt(i), shape=(10,)) assert_array_equal(m[[0, 1, 4, 9]], numpy.array([0, 1, 2, 3])) def test_getslice_from_2D_functional_array(): m = larray(lambda i,j: 2*i + j, shape=(6,5)) assert_array_equal(m[1:3], numpy.array([[2, 3, 4, 5, 6], [4, 5, 6, 7, 8]])) def test_getitem_from_iterator_array(): m = larray(iter([1, 2, 3]), shape=(3,)) assert_raises(NotImplementedError, m.__getitem__, 2) def test_getitem_from_array_with_operations(): a1 = numpy.array([[1, 3, 5], [7, 9, 11]]) m1 = larray(a1) f = lambda i,j: numpy.sqrt(i*i + j*j) a2 = numpy.fromfunction(f, shape=(2, 3)) m2 = larray(f, shape=(2, 3)) a3 = 3*a1 + a2 m3 = 3*m1 + m2 assert_array_equal(a3[:,(0,2)], m3[:,(0,2)]) def test_evaluate_with_invalid_base_value(): m = larray(range(5)) m.base_value = "foo" assert_raises(ValueError, m.evaluate) def test_partially_evaluate_with_invalid_base_value(): m = larray(range(5)) m.base_value = "foo" assert_raises(ValueError, m._partially_evaluate, 3) def test_check_bounds_with_invalid_address(): m = larray([[1, 3, 5], [7, 9, 11]]) assert_raises(TypeError, m.check_bounds, (object(), 1))lazyarray-0.1.0/test/test_ufunc.py0000644000077300000240000000235311710013626017640 0ustar andrewstaff00000000000000""" Unit tests for ``larray``-compatible ufuncs Copyright Andrew P. Davison, 2012 """ from lazyarray import larray, sqrt, cos import numpy from numpy.testing import assert_array_equal, assert_array_almost_equal def test_sqrt_from_array(): A = larray(numpy.array([1, 4, 9, 16, 25])) assert_array_equal(sqrt(A).evaluate(), numpy.arange(1, 6)) def test_sqrt_from_iterator(): A = larray(iter([1, 4, 9, 16, 25]), shape=(5,)) assert_array_equal(sqrt(A).evaluate(), numpy.arange(1, 6)) def test_sqrt_from_func(): A = larray(lambda x: (x + 1)**2, shape=(5,)) assert_array_equal(sqrt(A).evaluate(), numpy.arange(1, 6)) def test_sqrt_normal_array(): A = numpy.array([1, 4, 9, 16, 25]) assert_array_equal(sqrt(A), numpy.arange(1, 6)) def test_cos_from_generator(): def clock(): for x in numpy.arange(0, 2*numpy.pi, numpy.pi/2): yield x A = larray(clock(), shape=(2, 2)) assert_array_almost_equal(cos(A).evaluate(), numpy.array([[1.0, 0.0], [-1.0, 0.0]]), decimal=15)