pax_global_header00006660000000000000000000000064135706550200014515gustar00rootroot0000000000000052 comment=c7a3e599190b47db25ab18300266458f92c17dd2 numpysane-0.21/000077500000000000000000000000001357065502000134565ustar00rootroot00000000000000numpysane-0.21/.gitignore000066400000000000000000000001531357065502000154450ustar00rootroot00000000000000*.pyc *~ debian/*.log dist/ MANIFEST .pybuild README build/ *.egg-info *.d *.o *.i *.s *.so */*GENERATED.c numpysane-0.21/Changes000066400000000000000000000005751357065502000147600ustar00rootroot00000000000000numpysane (0.20) * nps.matmult(..., out=out) produces in-place results when one of the arguments is 1D -- Dima Kogan Sat, 30 Nov 2019 18:20:49 -0800 numpysane (0.19) * Added mag() convenience function. mag(x) = sqrt(norm2(x)) * Initial support for C-level broadcasting -- Dima Kogan Thu, 28 Nov 2019 18:50:02 -0800 numpysane-0.21/LICENSE000066400000000000000000000004141357065502000144620ustar00rootroot00000000000000Copyright 2016 Dima Kogan. This program is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License (version 3 or higher) as published by the Free Software Foundation See https://www.gnu.org/licenses/lgpl.html numpysane-0.21/MANIFEST.in000066400000000000000000000000511357065502000152100ustar00rootroot00000000000000include README include test_numpysane.py numpysane-0.21/Makefile000066400000000000000000000012121357065502000151120ustar00rootroot00000000000000all: README README.org # a multiple-target pattern rule means that a single invocation of the command # builds all the targets, which is what I want here %EADME %EADME.org: numpysane.py README.footer.org extract_README.py python3 extract_README.py numpysane test: test2 test3 check: check2 check3 check2: test2 check3: test3 test2 test3: python$(patsubst test%,%,$@) test_numpysane.py .PHONY: check check2 check3 test test2 test3 # make distribution tarball dist: python3 setup.py sdist .PHONY: dist # make and upload the distribution tarball dist_upload: python3 setup.py sdist upload .PHONY: dist_upload EXTRA_CLEAN += README.org README numpysane-0.21/README.footer.org000066400000000000000000000007741357065502000164310ustar00rootroot00000000000000* COMPATIBILITY Python 2 and Python 3 should both be supported. Please report a bug if either one doesn't work. * REPOSITORY https://github.com/dkogan/numpysane * AUTHOR Dima Kogan * LICENSE AND COPYRIGHT Copyright 2016-2017 Dima Kogan. This program is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License (version 3 or higher) as published by the Free Software Foundation See https://www.gnu.org/licenses/lgpl.html numpysane-0.21/README.org000066400000000000000000001331161357065502000151310ustar00rootroot00000000000000* NAME numpysane: more-reasonable core functionality for numpy * SYNOPSIS #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> row = a[0,:] + 1000 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> row array([1000, 1001, 1002]) >>> nps.glue(a,b, axis=-1) array([[ 0, 1, 2, 100, 101, 102], [ 3, 4, 5, 103, 104, 105]]) >>> nps.glue(a,b,row, axis=-2) array([[ 0, 1, 2], [ 3, 4, 5], [ 100, 101, 102], [ 103, 104, 105], [1000, 1001, 1002]]) >>> nps.cat(a,b) array([[[ 0, 1, 2], [ 3, 4, 5]], [[100, 101, 102], [103, 104, 105]]]) >>> @nps.broadcast_define( (('n',), ('n',)) ) ... def inner_product(a, b): ... return a.dot(b) >>> inner_product(a,b) array([ 305, 1250]) #+END_EXAMPLE * DESCRIPTION Numpy is widely used, relatively polished, and has a wide range of libraries available. At the same time, some of its very core functionality is strange, confusing and just plain wrong. This is in contrast with PDL (http://pdl.perl.org), which has a much more reasonable core, but a number of higher-level warts, and a relative dearth of library support. This module intends to improve the developer experience by providing alternate APIs to some core numpy functionality that is much more reasonable, especially for those who have used PDL in the past. Instead of writing a new module (this module), it would be really nice to simply patch numpy to give everybody the more reasonable behavior. I'd be very happy to do that, but the issues lie with some very core functionality, and any changes in behavior would break existing code. Any comments in how to achieve better behaviors in a less forky manner are welcome. Finally, if the existing system DOES make sense in some way that I'm simply not understanding, I'm happy to listen. I have no intention to disparage anyone or anything; I just want a more usable system for numerical computations. The issues addressed by this module fall into two broad categories: 1. Incomplete broadcasting support 2. Strange, special-case-ridden rules for basic array manipulation, especially dealing with dimensionality ** Broadcasting *** Problem Numpy has a limited support for broadcasting (http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html), a generic way to vectorize functions. When making a broadcasted call to a function, you pass in arguments with the inputs to vectorize available in new dimensions, and the broadcasting mechanism automatically calls the function multiple times as needed, and reports the output as an array collecting all the results. A basic example is an inner product: a function that takes in two identically-sized vectors (1-dimensional arrays) and returns a scalar (0-dimensional array). A broadcasted inner product function could take in two arrays of shape (2,3,4), compute the 6 inner products of length-4 each, and report the output in an array of shape (2,3). Numpy puts the most-significant dimension at the end, which is why this isn't 12 inner products of length-2 each. This is an arbitrary design choice, which could have been made differently: PDL puts the most-significant dimension at the front. The user doesn't choose whether to use broadcasting or not: some functions support it, and some do not. In PDL, broadcasting (called "threading" in that system) is a pervasive concept throughout. A PDL user has an expectation that every function can broadcast, and the documentation for every function is very explicit about the dimensionality of the inputs and outputs. Any data above the expected input dimensions is broadcast. By contrast, in numpy very few functions know how to broadcast. On top of that, the documentation is usually silent about the broadcasting status of a function in question. And on top of THAT, broadcasting rules state that an array of dimensions (n,m) is functionally identical to one of dimensions (1,1,1,....1,n,m). Sadly, numpy does not respect its own broadcasting rules, and many functions have special-case logic to create different behaviors for inputs with different numbers of dimensions; and this creates unexpected results. The effect of all this is a messy situation where the user is often not sure of the exact behavior of the functions they're calling, and trial and error is required to make the system do what one wants. *** Solution This module contains functionality to make any arbitrary function broadcastable, in either C or Python. **** Broadcasting rules A detailed description of broadcasting rules is available in the numpy documentation: http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html In short: - The most significant dimension in a numpy array is the LAST one, so the prototype of an input argument must exactly match a given input's trailing shape. So a prototype shape of (a,b,c) accepts an argument shape of (......, a,b,c), with as many or as few leading dimensions as desired. - The extra leading dimensions must be compatible across all the inputs. This means that each leading dimension must either - equal to 1 - be missing (thus assumed to equal 1) - equal to some positive integer >1, consistent across all arguments - The output is collected into an array that's sized as a superset of the above-prototype shape of each argument More involved example: A function with input prototype ( (3,), ('n',3), ('n',), ('m',) ) given inputs of shape #+BEGIN_SRC python (1,5, 3) (2,1, 8,3) ( 8) ( 5, 9) #+END_SRC will return an output array of shape (2,5, ...), where ... is the shape of each output slice. Note again that the prototype dictates the TRAILING shape of the inputs. **** Broadcasting in python This is invoked as a decorator, applied to the arbitrary user function. An example: #+BEGIN_EXAMPLE >>> import numpysane as nps >>> @nps.broadcast_define( (('n',), ('n',)) ) ... def inner_product(a, b): ... return a.dot(b) #+END_EXAMPLE Here we have a simple inner product function to compute ONE inner product. We call 'broadcast_define' to add a broadcasting-aware wrapper that takes two 1D vectors of length 'n' each (same 'n' for the two inputs). This new 'inner_product' function applies broadcasting, as needed: #+BEGIN_EXAMPLE >>> import numpy as np >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> inner_product(a,b) array([ 305, 1250]) #+END_EXAMPLE Another related function in this module broadcast_generate(). It's similar to broadcast_define(), but instead of adding broadcasting-awareness to an existing function, it simply generates tuples from a set of arguments according to a given prototype. Stock numpy has some rudimentary support for all this with its vectorize() function, but it assumes only scalar inputs and outputs, which severaly limits its usefulness. **** Broadcasting in C A C-level flavor of broadcast_define() is available. It wraps C code in C loops. This is an analogue of PDL::PP (http://pdl.perl.org/PDLdocs/PP.html). Here the numpysane_pywrap module is used to produce C code that is compiled and linked into a python extension module. This takes more effort than python-level broadcasting, but the results have much less overhead, and run much faster. Please see the sample (https://github.com/dkogan/numpysane/blob/master/pywrap-sample). This is relatively new, so please let me know if you try it, and stuff does or does not work. *** New planned functionality The C broadcasting is functional, but a few more features are on the roadmap: - It should be possible for some inputs/output to contain different data types - And sometimes one would want to produce more than one output array for each call, possibly with different types - The prototype specification is not flexible enough. Maybe there's some relationship between named dimensions that is known. If so, this should be specify-able - Parallelization for broadcasted slices. Since each broadcasting loop is independent, this is a very natural place to add parallelism. This is fairly simple with OpenMP. ** Strangeness in core routines *** Problem There are some core numpy functions whose behavior is strange, full of special cases and very confusing, at least to me. That makes it difficult to achieve some very basic things. In the following examples, I use a function "arr" that returns a numpy array with given dimensions: #+BEGIN_EXAMPLE >>> def arr(*shape): ... product = reduce( lambda x,y: x*y, shape) ... return np.arange(product).reshape(*shape) >>> arr(1,2,3) array([[[0, 1, 2], [3, 4, 5]]]) >>> arr(1,2,3).shape (1, 2, 3) #+END_EXAMPLE The following sections are an incomplete list of the strange functionality I've encountered. **** Concatenation A prime example of confusing functionality is the array concatenation routines. Numpy has a number of functions to do this, each being strange. ***** hstack() hstack() performs a "horizontal" concatenation. When numpy prints an array, this is the last dimension (remember, the most significant dimensions in numpy are at the end). So one would expect that this function concatenates arrays along this last dimension. In the special case of 1D and 2D arrays, one would be right: #+BEGIN_EXAMPLE >>> np.hstack( (arr(3), arr(3))).shape (6,) >>> np.hstack( (arr(2,3), arr(2,3))).shape (2, 6) #+END_EXAMPLE but in any other case, one would be wrong: #+BEGIN_EXAMPLE >>> np.hstack( (arr(1,2,3), arr(1,2,3))).shape (1, 4, 3) <------ I expect (1, 2, 6) >>> np.hstack( (arr(1,2,3), arr(1,2,4))).shape [exception] <------ I expect (1, 2, 7) >>> np.hstack( (arr(3), arr(1,3))).shape [exception] <------ I expect (1, 6) >>> np.hstack( (arr(1,3), arr(3))).shape [exception] <------ I expect (1, 6) #+END_EXAMPLE I think the above should all succeed, and should produce the shapes as indicated. Cases such as "np.hstack( (arr(3), arr(1,3)))" are maybe up for debate, but broadcasting rules allow adding as many extra length-1 dimensions as we want without changing the meaning of the object, so I claim this should work. Either way, if you print out the operands for any of the above, you too would expect a "horizontal" stack() to work as stated above. It turns out that normally hstack() concatenates along axis=1, unless the first argument only has one dimension, in which case axis=0 is used. This is 100% wrong in a system where the most significant dimension is the last one, unless you assume that everyone has only 2D arrays, where the last dimension and the second dimension are the same. The correct way to do this is to concatenate along axis=-1. It works for n-dimensionsal objects, and doesn't require the special case logic for 1-dimensional objects that hstack() has. ***** vstack() Similarly, vstack() performs a "vertical" concatenation. When numpy prints an array, this is the second-to-last dimension (remember, the most significant dimensions in numpy are at the end). So one would expect that this function concatenates arrays along this second-to-last dimension. In the special case of 1D and 2D arrays, one would be right: #+BEGIN_EXAMPLE >>> np.vstack( (arr(2,3), arr(2,3))).shape (4, 3) >>> np.vstack( (arr(3), arr(3))).shape (2, 3) >>> np.vstack( (arr(1,3), arr(3))).shape (2, 3) >>> np.vstack( (arr(3), arr(1,3))).shape (2, 3) >>> np.vstack( (arr(2,3), arr(3))).shape (3, 3) #+END_EXAMPLE Note that this function appears to tolerate some amount of shape mismatches. It does it in a form one would expect, but given the state of the rest of this system, I found it surprising. For instance "np.hstack( (arr(1,3), arr(3)))" fails, so one would think that "np.vstack( (arr(1,3), arr(3)))" would fail too. And once again, adding more dimensions make it confused, for the same reason: #+BEGIN_EXAMPLE >>> np.vstack( (arr(1,2,3), arr(2,3))).shape [exception] <------ I expect (1, 4, 3) >>> np.vstack( (arr(1,2,3), arr(1,2,3))).shape (2, 2, 3) <------ I expect (1, 4, 3) #+END_EXAMPLE Similarly to hstack(), vstack() concatenates along axis=0, which is "vertical" only for 2D arrays, but not for any others. And similarly to hstack(), the 1D case has special-cased logic to work properly. The correct way to do this is to concatenate along axis=-2. It works for n-dimensionsal objects, and doesn't require the special case for 1-dimensional objects that vstack() has. ***** dstack() I'll skip the detailed description, since this is similar to hstack() and vstack(). The intent was to concatenate across axis=-3, but the implementation takes axis=2 instead. This is wrong, as before. And I find it strange that these 3 functions even exist, since they are all special-cases: the concatenation axis should be an argument, and at most, the edge special case (hstack()) should exist. This brings us to the next function: ***** concatenate() This is a more general function, and unlike hstack(), vstack() and dstack(), it takes as input a list of arrays AND the concatenation dimension. It accepts negative concatenation dimensions to allow us to count from the end, so things should work better. And in many ways that failed previously, they do: #+BEGIN_EXAMPLE >>> np.concatenate( (arr(1,2,3), arr(1,2,3)), axis=-1).shape (1, 2, 6) >>> np.concatenate( (arr(1,2,3), arr(1,2,4)), axis=-1).shape (1, 2, 7) >>> np.concatenate( (arr(1,2,3), arr(1,2,3)), axis=-2).shape (1, 4, 3) #+END_EXAMPLE But many things still don't work as I would expect: #+BEGIN_EXAMPLE >>> np.concatenate( (arr(1,3), arr(3)), axis=-1).shape [exception] <------ I expect (1, 6) >>> np.concatenate( (arr(3), arr(1,3)), axis=-1).shape [exception] <------ I expect (1, 6) >>> np.concatenate( (arr(1,3), arr(3)), axis=-2).shape [exception] <------ I expect (3, 3) >>> np.concatenate( (arr(3), arr(1,3)), axis=-2).shape [exception] <------ I expect (2, 3) >>> np.concatenate( (arr(2,3), arr(2,3)), axis=-3).shape [exception] <------ I expect (2, 2, 3) #+END_EXAMPLE This function works as expected only if - All inputs have the same number of dimensions - All inputs have a matching shape, except for the dimension along which we're concatenating - All inputs HAVE the dimension along which we're concatenating A legitimate use case that violates these conditions: I have an object that contains N 3D vectors, and I want to add another 3D vector to it. This is essentially the first failing example above. ***** stack() The name makes it sound exactly like concatenate(), and it takes the same arguments, but it is very different. stack() requires that all inputs have EXACTLY the same shape. It then concatenates all the inputs along a new dimension, and places that dimension in the location given by the 'axis' input. If this is the exact type of concatenation you want, this function works fine. But it's one of many things a user may want to do. **** inner() and dot() Another arbitrary example of a strange API is np.dot() and np.inner(). In a real-valued n-dimensional Euclidean space, a "dot product" is just another name for an "inner product". Numpy disagrees. It looks like np.dot() is matrix multiplication, with some wonky behaviors when given higher-dimension objects, and with some special-case behaviors for 1-dimensional and 0-dimensional objects: #+BEGIN_EXAMPLE >>> np.dot( arr(4,5,2,3), arr(3,5)).shape (4, 5, 2, 5) <--- expected result for a broadcasted matrix multiplication >>> np.dot( arr(3,5), arr(4,5,2,3)).shape [exception] <--- np.dot() is not commutative. Expected for matrix multiplication, but not for a dot product >>> np.dot( arr(4,5,2,3), arr(1,3,5)).shape (4, 5, 2, 1, 5) <--- don't know where this came from at all >>> np.dot( arr(4,5,2,3), arr(3)).shape (4, 5, 2) <--- 1D special case. This is a dot product. >>> np.dot( arr(4,5,2,3), 3).shape (4, 5, 2, 3) <--- 0D special case. This is a scaling. #+END_EXAMPLE It looks like np.inner() is some sort of quasi-broadcastable inner product, also with some funny dimensioning rules. In many cases it looks like np.dot(a,b) is the same as np.inner(a, transpose(b)) where transpose() swaps the last two dimensions: #+BEGIN_EXAMPLE >>> np.inner( arr(4,5,2,3), arr(5,3)).shape (4, 5, 2, 5) <--- All the length-3 inner products collected into a shape with not-quite-broadcasting rules >>> np.inner( arr(5,3), arr(4,5,2,3)).shape (5, 4, 5, 2) <--- np.inner() is not commutative. Unexpected for an inner product >>> np.inner( arr(4,5,2,3), arr(1,5,3)).shape (4, 5, 2, 1, 5) <--- No idea >>> np.inner( arr(4,5,2,3), arr(3)).shape (4, 5, 2) <--- 1D special case. This is a dot product. >>> np.inner( arr(4,5,2,3), 3).shape (4, 5, 2, 3) <--- 0D special case. This is a scaling. #+END_EXAMPLE **** atleast_xd() Numpy has 3 special-case functions atleast_1d(), atleast_2d() and atleast_3d(). For 4d and higher, you need to do something else. As expected by now, these do surprising things: #+BEGIN_EXAMPLE >>> np.atleast_3d( arr(3)).shape (1, 3, 1) #+END_EXAMPLE I don't know when this is what I would want, so we move on. *** Solution This module introduces new functions that can be used for this core functionality instead of the builtin numpy functions. These new functions work in ways that (I think) are more intuitive and more reasonable. They do not refer to anything being "horizontal" or "vertical", nor do they talk about "rows" or "columns"; these concepts simply don't apply in a generic N-dimensional system. These functions are very explicit about the dimensionality of the inputs/outputs, and fit well into a broadcasting-aware system. Furthermore, the names and semantics of these new functions come directly from PDL, which is more consistent in this area. Since these functions assume that broadcasting is an important concept in the system, the given axis indices should be counted from the most significant dimension: the last dimension in numpy. This means that where an axis index is specified, negative indices are encouraged. glue() forbids axis>=0 outright. Example for further justification: An array containing N 3D vectors would have shape (N,3). Another array containing a single 3D vector would have shape (3). Counting the dimensions from the end, each vector is indexed in dimension -1. However, counting from the front, the vector is indexed in dimension 0 or 1, depending on which of the two arrays we're looking at. If we want to add the single vector to the array containing the N vectors, and we mistakenly try to concatenate along the first dimension, it would fail if N != 3. But if we're unlucky, and N=3, then we'd get a nonsensical output array of shape (3,4). Why would an array of N 3D vectors have shape (N,3) and not (3,N)? Because if we apply python iteration to it, we'd expect to get N iterates of arrays with shape (3,) each, and numpy iterates from the first dimension: #+BEGIN_EXAMPLE >>> a = np.arange(2*3).reshape(2,3) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> [x for x in a] [array([0, 1, 2]), array([3, 4, 5])] #+END_EXAMPLE New functions this module provides (documented fully in the next section): **** glue Concatenates arrays along a given axis ('axis' must be given in a kwarg). Implicit length-1 dimensions are added at the start as needed. Dimensions other than the glueing axis must match exactly. **** cat Concatenate a given list of arrays along a new least-significant (leading) axis. Again, implicit length-1 dimensions are added, and the resulting shapes must match, and no data duplication occurs. **** clump Reshapes the array by grouping together 'n' dimensions, where 'n' is given in a kwarg. If 'n' > 0, then n leading dimensions are clumped; if 'n' < 0, then -n trailing dimensions are clumped **** atleast_dims Adds length-1 dimensions at the front of an array so that all the given dimensions are in-bounds. Given axis<0 can expand the shape; given axis>=0 MUST already be in-bounds. This preserves the alignment of the most-significant axis index. **** mv Moves a dimension from one position to another **** xchg Exchanges the positions of two dimensions **** transpose Reverses the order of the two most significant dimensions in an array. The whole array is seen as being an array of 2D matrices, each matrix living in the 2 most significant dimensions, which implies this definition. **** dummy Adds a single length-1 dimension at the given position **** reorder Completely reorders the dimensions in an array **** dot Broadcast-aware non-conjugating dot product. Identical to inner **** vdot Broadcast-aware conjugating dot product **** inner Broadcast-aware inner product. Identical to dot **** outer Broadcast-aware outer product. **** norm2 Broadcast-aware 2-norm. norm2(x) is identical to inner(x,x) **** mag Broadcast-aware vector magnitude. mag(x) is functionally identical to sqrt(inner(x,x)) **** trace Broadcast-aware trace. **** matmult Broadcast-aware matrix multiplication *** New planned functionality The functions listed above are a start, but more will be added with time. * INTERFACE ** broadcast_define() Vectorizes an arbitrary function, expecting input as in the given prototype. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> @nps.broadcast_define( (('n',), ('n',)) ) ... def inner_product(a, b): ... return a.dot(b) >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> inner_product(a,b) array([ 305, 1250]) #+END_EXAMPLE The prototype defines the dimensionality of the inputs. In the inner product example above, the input is two 1D n-dimensional vectors. In particular, the 'n' is the same for the two inputs. This function is intended to be used as a decorator, applied to a function defining the operation to be vectorized. Each element in the prototype list refers to each input, in order. In turn, each such element is a list that describes the shape of that input. Each of these shape descriptors can be any of - a positive integer, indicating an input dimension of exactly that length - a string, indicating an arbitrary, but internally consistent dimension The normal numpy broadcasting rules (as described elsewhere) apply. In summary: - Dimensions are aligned at the end of the shape list, and must match the prototype - Extra dimensions left over at the front must be consistent for all the input arguments, meaning: - All dimensions !=1 must be identical - Missing dimensions are implicitly set to 1 - The output has a shape where - The trailing dimensions are whatever the function being broadcasted outputs - The leading dimensions come from the extra dimensions in the inputs Scalars are represented as 0-dimensional numpy arrays: arrays with shape (), and these broadcast as one would expect: #+BEGIN_EXAMPLE >>> @nps.broadcast_define( (('n',), ('n',), ())) ... def scaled_inner_product(a, b, scale): ... return a.dot(b)*scale >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> scale = np.array((10,100)) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> scale array([ 10, 100]) >>> scaled_inner_product(a,b,scale) array([[ 3050], [125000]]) #+END_EXAMPLE Let's look at a more involved example. Let's say we have a function that takes a set of points in R^2 and a single center point in R^2, and finds a best-fit least-squares line that passes through the given center point. Let it return a 3D vector containing the slope, y-intercept and the RMS residual of the fit. This broadcasting-enabled function can be defined like this: #+BEGIN_SRC python import numpy as np import numpysane as nps @nps.broadcast_define( (('n',2), (2,)) ) def fit(xy, c): # line-through-origin-model: y = m*x # E = sum( (m*x - y)**2 ) # dE/dm = 2*sum( (m*x-y)*x ) = 0 # ----> m = sum(x*y)/sum(x*x) x,y = (xy - c).transpose() m = np.sum(x*y) / np.sum(x*x) err = m*x - y err **= 2 rms = np.sqrt(err.mean()) # I return m,b because I need to translate the line back b = c[1] - m*c[0] return np.array((m,b,rms)) #+END_SRC And I can use broadcasting to compute a number of these fits at once. Let's say I want to compute 4 different fits of 5 points each. I can do this: #+BEGIN_SRC python n = 5 m = 4 c = np.array((20,300)) xy = np.arange(m*n*2, dtype=np.float64).reshape(m,n,2) + c xy += np.random.rand(*xy.shape)*5 res = fit( xy, c ) mb = res[..., 0:2] rms = res[..., 2] print "RMS residuals: {}".format(rms) #+END_SRC Here I had 4 different sets of points, but a single center point c. If I wanted 4 different center points, I could pass c as an array of shape (4,2). I can use broadcasting to plot all the results (the points and the fitted lines): #+BEGIN_SRC python import gnuplotlib as gp gp.plot( *nps.mv(xy,-1,0), _with='linespoints', equation=['{}*x + {}'.format(mb_single[0], mb_single[1]) for mb_single in mb], unset='grid', square=1) #+END_SRC The examples above all create a separate output array for each broadcasted slice, and copy the contents from each such slice into the large output array that contains all the results. This is inefficient, and it is possible to pre-allocate an array to forgo these extra allocations and copies. There are several settings to control this. If the function being broadcasted can write its output to a given array instead of creating a new one, most of the inefficiency goes away. broadcast_define() supports the case where this function takes this array in a kwarg: the name of this kwarg can be given to broadcast_define() like so: #+BEGIN_SRC python @nps.broadcast_define( ....., out_kwarg = "out" ) def func( ....., out): ..... out[:] = result #+END_SRC In order for broadcast_define() to pass such an output array to the inner function, this output array must be available, which means that it must be given to us somehow, or we must create it. The most efficient way to make a broadcasted call is to create the full output array beforehand, and to pass that to the broadcasted function. In this case, nothing extra will be allocated, and no unnecessary copies will be made. This can be done like this: #+BEGIN_SRC python @nps.broadcast_define( (('n',), ('n',)), ....., out_kwarg = "out" ) def inner_product(a, b, out): ..... out.setfield(a.dot(b), out.dtype) return out out = np.empty((2,4), float) inner_product( np.arange(3), np.arange(2*4*3).reshape(2,4,3), out=out) #+END_SRC In this example, the caller knows that it's calling an inner_product function, and that the shape of each output slice would be (). The caller also knows the input dimensions and that we have an extra broadcasting dimension (2,4), so the output array will have shape (2,4) + () = (2,4). With this knowledge, the caller preallocates the array, and passes it to the broadcasted function call. Furthermore, in this case the inner function will be called with an output array EVERY time, and this is the only mode the inner function needs to support. If the caller doesn't know (or doesn't want to pre-compute) the shape of the output, it can let the broadcasting machinery create this array for them. In order for this to be possible, the shape of the output should be pre-declared, and the dtype of the output should be known: #+BEGIN_SRC python @nps.broadcast_define( (('n',), ('n',)), (), out_kwarg = "out" ) def inner_product(a, b, out): ..... out.setfield(a.dot(b), out.dtype) return out out = inner_product( np.arange(3), np.arange(2*4*3).reshape(2,4,3), dtype=int) #+END_SRC Note that the caller didn't need to specify the prototype of the output or the extra broadcasting dimensions (output prototype is in the broadcast_define() call, but not the inner_product() call). Specifying the dtype here is optional: it defaults to float if omitted. If we want the output array to be pre-allocated, the output prototype (it is () in this example) is required: we must know the shape of the output array in order to create it. Without a declared output prototype, we can still make mostly- efficient calls: the broadcasting mechanism can call the inner function for the first slice as we showed earlier, by creating a new array for the slice. This new array required an extra allocation and copy, but it contains the required shape information. This infomation will be used to allocate the output, and the subsequent calls to the inner function will be efficient: #+BEGIN_SRC python @nps.broadcast_define( (('n',), ('n',)), out_kwarg = "out" ) def inner_product(a, b, out=None): ..... if out is None: return a.dot(b) out.setfield(a.dot(b), out.dtype) return out out = inner_product( np.arange(3), np.arange(2*4*3).reshape(2,4,3)) #+END_SRC Here we were slighly inefficient, but the ONLY required extra specification was out_kwarg: that's mostly all you need. Also it is important to note that in this case the inner function is called both with passing it an output array to fill in, and with asking it to create a new one (by passing out=None to the inner function). This inner function then must support both modes of operation. If the inner function does not support filling in an output array, none of these efficiency improvements are possible. broadcast_define() is analogous to thread_define() in PDL. ** broadcast_generate() A generator that produces broadcasted slices Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> for s in nps.broadcast_generate( (('n',), ('n',)), (a,b)): ... print "slice: {}".format(s) slice: (array([0, 1, 2]), array([100, 101, 102])) slice: (array([3, 4, 5]), array([103, 104, 105])) #+END_EXAMPLE ** glue() Concatenates a given list of arrays along the given 'axis' keyword argument. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> row = a[0,:] + 1000 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> row array([1000, 1001, 1002]) >>> nps.glue(a,b, axis=-1) array([[ 0, 1, 2, 100, 101, 102], [ 3, 4, 5, 103, 104, 105]]) # empty arrays ignored when glueing. Useful for initializing an accumulation >>> nps.glue(a,b, np.array(()), axis=-1) array([[ 0, 1, 2, 100, 101, 102], [ 3, 4, 5, 103, 104, 105]]) >>> nps.glue(a,b,row, axis=-2) array([[ 0, 1, 2], [ 3, 4, 5], [ 100, 101, 102], [ 103, 104, 105], [1000, 1001, 1002]]) >>> nps.glue(a,b, axis=-3) array([[[ 0, 1, 2], [ 3, 4, 5]], [[100, 101, 102], [103, 104, 105]]]) #+END_EXAMPLE The 'axis' must be given in a keyword argument. In order to count dimensions from the inner-most outwards, this function accepts only negative axis arguments. This is because numpy broadcasts from the last dimension, and the last dimension is the inner-most in the (usual) internal storage scheme. Allowing glue() to look at dimensions at the start would allow it to unalign the broadcasting dimensions, which is never what you want. To glue along the last dimension, pass axis=-1; to glue along the second-to-last dimension, pass axis=-2, and so on. Unlike in PDL, this function refuses to create duplicated data to make the shapes fit. In my experience, this isn't what you want, and can create bugs. For instance, PDL does this: #+BEGIN_SRC python pdl> p sequence(3,2) [ [0 1 2] [3 4 5] ] pdl> p sequence(3) [0 1 2] pdl> p PDL::glue( 0, sequence(3,2), sequence(3) ) [ [0 1 2 0 1 2] <--- Note the duplicated "0,1,2" [3 4 5 0 1 2] ] #+END_SRC while numpysane.glue() does this: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a[0:1,:] >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[0, 1, 2]]) >>> nps.glue(a,b,axis=-1) [exception] #+END_EXAMPLE Finally, this function adds as many length-1 dimensions at the front as required. Note that this does not create new data, just new degenerate dimensions. Example: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> res = nps.glue(a,b, axis=-5) >>> res array([[[[[ 0, 1, 2], [ 3, 4, 5]]]], [[[[100, 101, 102], [103, 104, 105]]]]]) >>> res.shape (2, 1, 1, 2, 3) #+END_EXAMPLE In numpysane older than 0.10 the semantics were slightly different: the axis kwarg was optional, and glue(*args) would glue along a new leading dimension, and thus would be equivalent to cat(*args). This resulted in very confusing error messages if the user accidentally omitted the kwarg. To request the legacy behavior, do #+BEGIN_SRC python nps.glue.legacy_version = '0.9' #+END_SRC ** cat() Concatenates a given list of arrays along a new first (outer) dimension. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> c = a - 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> c array([[-100, -99, -98], [ -97, -96, -95]]) >>> res = nps.cat(a,b,c) >>> res array([[[ 0, 1, 2], [ 3, 4, 5]], [[ 100, 101, 102], [ 103, 104, 105]], [[-100, -99, -98], [ -97, -96, -95]]]) >>> res.shape (3, 2, 3) >>> [x for x in res] [array([[0, 1, 2], [3, 4, 5]]), array([[100, 101, 102], [103, 104, 105]]), array([[-100, -99, -98], [ -97, -96, -95]])] #+END_EXAMPLE This function concatenates the input arrays into an array shaped like the highest-dimensioned input, but with a new outer (at the start) dimension. The concatenation axis is this new dimension. As usual, the dimensions are aligned along the last one, so broadcasting will continue to work as expected. Note that this is the opposite operation from iterating a numpy array; see the example above. ** clump() Groups the given n dimensions together. Synopsis: #+BEGIN_EXAMPLE >>> import numpysane as nps >>> nps.clump( arr(2,3,4), n = -2).shape (2, 12) #+END_EXAMPLE Reshapes the array by grouping together 'n' dimensions, where 'n' is given in a kwarg. If 'n' > 0, then n leading dimensions are clumped; if 'n' < 0, then -n trailing dimensions are clumped So for instance, if x.shape is (2,3,4) then nps.clump(x, n = -2).shape is (2,12) and nps.clump(x, n = 2).shape is (6, 4) In numpysane older than 0.10 the semantics were different: n > 0 was required, and we ALWAYS clumped the trailing dimensions. Thus the new clump(-n) is equivalent to the old clump(n). To request the legacy behavior, do #+BEGIN_SRC python nps.clump.legacy_version = '0.9' #+END_SRC ** atleast_dims() Returns an array with extra length-1 dimensions to contain all given axes. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> nps.atleast_dims(a, -1).shape (2, 3) >>> nps.atleast_dims(a, -2).shape (2, 3) >>> nps.atleast_dims(a, -3).shape (1, 2, 3) >>> nps.atleast_dims(a, 0).shape (2, 3) >>> nps.atleast_dims(a, 1).shape (2, 3) >>> nps.atleast_dims(a, 2).shape [exception] >>> l = [-3,-2,-1,0,1] >>> nps.atleast_dims(a, l).shape (1, 2, 3) >>> l [-3, -2, -1, 1, 2] #+END_EXAMPLE If the given axes already exist in the given array, the given array itself is returned. Otherwise length-1 dimensions are added to the front until all the requested dimensions exist. The given axis>=0 dimensions MUST all be in-bounds from the start, otherwise the most-significant axis becomes unaligned; an exception is thrown if this is violated. The given axis<0 dimensions that are out-of-bounds result in new dimensions added at the front. If new dimensions need to be added at the front, then any axis>=0 indices become offset. For instance: #+BEGIN_EXAMPLE >>> x.shape (2, 3, 4) >>> [x.shape[i] for i in (0,-1)] [2, 4] >>> x = nps.atleast_dims(x, 0, -1, -5) >>> x.shape (1, 1, 2, 3, 4) >>> [x.shape[i] for i in (0,-1)] [1, 4] #+END_EXAMPLE Before the call, axis=0 refers to the length-2 dimension and axis=-1 refers to the length=4 dimension. After the call, axis=-1 refers to the same dimension as before, but axis=0 now refers to a new length=1 dimension. If it is desired to compensate for this offset, then instead of passing the axes as separate arguments, pass in a single list of the axes indices. This list will be modified to offset the axis>=0 appropriately. Ideally, you only pass in axes<0, and this does not apply. Doing this in the above example: #+BEGIN_EXAMPLE >>> l [0, -1, -5] >>> x.shape (2, 3, 4) >>> [x.shape[i] for i in (l[0],l[1])] [2, 4] >>> x=nps.atleast_dims(x, l) >>> x.shape (1, 1, 2, 3, 4) >>> l [2, -1, -5] >>> [x.shape[i] for i in (l[0],l[1])] [2, 4] #+END_EXAMPLE We passed the axis indices in a list, and this list was modified to reflect the new indices: The original axis=0 becomes known as axis=2. Again, if you pass in only axis<0, then you don't need to care about this. ** mv() Moves a given axis to a new position. Similar to numpy.moveaxis(). Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.mv( a, -1, 0).shape (4, 2, 3) >>> nps.mv( a, -1, -5).shape (4, 1, 1, 2, 3) >>> nps.mv( a, 0, -5).shape (2, 1, 1, 3, 4) #+END_EXAMPLE New length-1 dimensions are added at the front, as required, and any axis>=0 that are passed in refer to the array BEFORE these new dimensions are added. ** xchg() Exchanges the positions of the two given axes. Similar to numpy.swapaxes() Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.xchg( a, -1, 0).shape (4, 3, 2) >>> nps.xchg( a, -1, -5).shape (4, 1, 2, 3, 1) >>> nps.xchg( a, 0, -5).shape (2, 1, 1, 3, 4) #+END_EXAMPLE New length-1 dimensions are added at the front, as required, and any axis>=0 that are passed in refer to the array BEFORE these new dimensions are added. ** transpose() Reverses the order of the last two dimensions. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.transpose(a).shape (2, 4, 3) >>> nps.transpose( np.arange(3) ).shape (3, 1) #+END_EXAMPLE A "matrix" is generally seen as a 2D array that we can transpose by looking at the 2 dimensions in the opposite order. Here we treat an n-dimensional array as an n-2 dimensional object containing 2D matrices. As usual, the last two dimensions contain the matrix. New length-1 dimensions are added at the front, as required, meaning that 1D input of shape (n,) is interpreted as a 2D input of shape (1,n), and the transpose is 2 of shape (n,1). ** dummy() Adds a single length-1 dimension at the given position. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.dummy(a, 0).shape (1, 2, 3, 4) >>> nps.dummy(a, 1).shape (2, 1, 3, 4) >>> nps.dummy(a, -1).shape (2, 3, 4, 1) >>> nps.dummy(a, -2).shape (2, 3, 1, 4) >>> nps.dummy(a, -5).shape (1, 1, 2, 3, 4) #+END_EXAMPLE This is similar to numpy.expand_dims(), but handles out-of-bounds dimensions better. New length-1 dimensions are added at the front, as required, and any axis>=0 that are passed in refer to the array BEFORE these new dimensions are added. ** reorder() Reorders the dimensions of an array. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.reorder( a, 0, -1, 1 ).shape (2, 4, 3) >>> nps.reorder( a, -2 , -1, 0 ).shape (3, 4, 2) >>> nps.reorder( a, -4 , -2, -5, -1, 0 ).shape (1, 3, 1, 4, 2) #+END_EXAMPLE This is very similar to numpy.transpose(), but handles out-of-bounds dimensions much better. New length-1 dimensions are added at the front, as required, and any axis>=0 that are passed in refer to the array BEFORE these new dimensions are added. ** dot() Non-conjugating dot product of two 1-dimensional n-long vectors. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3) >>> b = a+5 >>> a array([0, 1, 2]) >>> b array([5, 6, 7]) >>> nps.dot(a,b) 20 #+END_EXAMPLE this is identical to numpysane.inner(). for a conjugating version of this function, use nps.vdot(). note that the numpy dot() has some special handling when its dot() is given more than 1-dimensional input. this function has no special handling: normal broadcasting rules are applied. ** vdot() Conjugating dot product of two 1-dimensional n-long vectors. vdot(a,b) is equivalent to dot(np.conj(a), b) Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.array(( 1 + 2j, 3 + 4j, 5 + 6j)) >>> b = a+5 >>> a array([ 1.+2.j, 3.+4.j, 5.+6.j]) >>> b array([ 6.+2.j, 8.+4.j, 10.+6.j]) >>> nps.vdot(a,b) array((136-60j)) >>> nps.dot(a,b) array((24+148j)) #+END_EXAMPLE For a non-conjugating version of this function, use nps.dot(). Note that the numpy vdot() has some special handling when its vdot() is given more than 1-dimensional input. THIS function has no special handling: normal broadcasting rules are applied. ** outer() Outer product of two 1-dimensional n-long vectors. Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3) >>> b = a+5 >>> a array([0, 1, 2]) >>> b array([5, 6, 7]) >>> nps.outer(a,b) array([[ 0, 0, 0], [ 5, 6, 7], [10, 12, 14]]) #+END_EXAMPLE This function is broadcast-aware through numpysane.broadcast_define(). The expected inputs have input prototype: #+BEGIN_SRC python (('n',), ('n',)) #+END_SRC and output prototype #+BEGIN_SRC python ('n', 'n') #+END_SRC The first 2 positional arguments will broadcast. The trailing shape of those arguments must match the input prototype; the leading shape must follow the standard broadcasting rules. Positional arguments past the first 2 and all the keyword arguments are passed through untouched. ** norm2() Broadcast-aware 2-norm. norm2(x) is identical to inner(x,x) Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3) >>> a array([0, 1, 2]) >>> nps.norm2(a) 5 #+END_EXAMPLE This is a convenience function to compute a 2-norm ** mag() Magnitude of a vector. mag(x) is functionally identical to sqrt(inner(x,x)) Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3) >>> a array([0, 1, 2]) >>> nps.mag(a) 2.23606797749979 #+END_EXAMPLE This is a convenience function to compute a magnitude of a vector, with full broadcasting support. If and explicit "out" array isn't given, we produce output of dtype=float. Otherwise "out" retains its dtype ** trace() Broadcast-aware trace Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3*4*4).reshape(3,4,4) >>> a array([[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15]], [[16, 17, 18, 19], [20, 21, 22, 23], [24, 25, 26, 27], [28, 29, 30, 31]], [[32, 33, 34, 35], [36, 37, 38, 39], [40, 41, 42, 43], [44, 45, 46, 47]]]) >>> nps.trace(a) array([ 30, 94, 158]) #+END_EXAMPLE This function is broadcast-aware through numpysane.broadcast_define(). The expected inputs have input prototype: #+BEGIN_SRC python (('n', 'n'),) #+END_SRC and output prototype #+BEGIN_SRC python () #+END_SRC The first 1 positional arguments will broadcast. The trailing shape of those arguments must match the input prototype; the leading shape must follow the standard broadcasting rules. Positional arguments past the first 1 and all the keyword arguments are passed through untouched. ** matmult2() Multiplication of two matrices Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6) .reshape(2,3) >>> b = np.arange(12).reshape(3,4) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> nps.matmult2(a,b) array([[20, 23, 26, 29], [56, 68, 80, 92]]) #+END_EXAMPLE This multiplies exactly 2 matrices, and the output object can be given in the 'out' argument. If the usual case where the you let numpysane create and return the result, you can use numpysane.matmult() instead. An advantage of that function is that it can multiply an arbitrary N matrices together, not just 2. ** matmult() Multiplication of N matrices Synopsis: #+BEGIN_EXAMPLE >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6) .reshape(2,3) >>> b = np.arange(12).reshape(3,4) >>> c = np.arange(4) .reshape(4,1) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> c array([[0], [1], [2], [3]]) >>> nps.matmult(a,b,c) array([[162], [504]]) #+END_EXAMPLE This multiplies N matrices together by repeatedly calling matmult2() for each adjacent pair. Unlike matmult2(), the arguments MUST all be matrices to multiply. The 'out' kwarg for the output is not supported here. This function supports broadcasting fully, in C internally * COMPATIBILITY Python 2 and Python 3 should both be supported. Please report a bug if either one doesn't work. * REPOSITORY https://github.com/dkogan/numpysane * AUTHOR Dima Kogan * LICENSE AND COPYRIGHT Copyright 2016-2017 Dima Kogan. This program is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License (version 3 or higher) as published by the Free Software Foundation See https://www.gnu.org/licenses/lgpl.html numpysane-0.21/extract_README.py000077500000000000000000000112301357065502000165170ustar00rootroot00000000000000#!/usr/bin/python r'''Constructs README, README.org files The README files are generated by this script. They are made from: - The main module docstring, with some org markup applied to the README.org, but not to the README - The docstrings from each API function in the module, with some org markup applied to the README.org, but not to the README - README.footer.org, copied verbatim The main module name must be passed in as the first cmdline argument. ''' import sys try: modname = sys.argv[1] except: raise Exception("Need main module name as the first cmdline arg") exec( 'import {} as mod'.format(modname) ) import inspect import re try: from StringIO import StringIO ## for Python 2 except ImportError: from io import StringIO ## for Python 3 def dirmod(): r'''Same as dir(mod), but returns only functions, in the definition order''' with open('{}.py'.format(modname), 'r') as f: for l in f: m = re.match(r'def +([a-zA-Z0-9_]+)\(', l) if m: yield m.group(1) with open('README.org', 'w') as f_target_org: with open('README', 'w') as f_target: def write(s): f_target. write(s) f_target_org.write(s) def write_orgized(s): r'''Writes the given string, reformatted slightly with org in mind. The only change this function applies, is to look for indented block (signifying examples) and to wrap then in a #+BEGIN_SRC or #+BEGIN_EXAMPLE. ''' # the non-org version is written as is f_target.write(s) # the org version neeeds massaging f = f_target_org in_quote = None # can be None or 'example' or 'src' queued_blanks = 0 indent_size = 4 prev_indented = False sio = StringIO(s) for l in sio: if in_quote is None: if len(l) <= 1: # blank line f.write(l) continue if not re.match(' '*indent_size, l): # don't have full indent. not quote start prev_indented = re.match(' ', l) f.write(l) continue if re.match(' '*indent_size + '-', l): # Start of indented list. not quote start prev_indented = re.match(' ', l) f.write(l) continue if prev_indented: # prev line(s) were indented, so this can't start a quote f.write(l) continue # start of quote. What kind? if re.match(' >>>', l): in_quote = 'example' f.write('#+BEGIN_EXAMPLE\n') else: in_quote = 'src' f.write('#+BEGIN_SRC python\n') f.write(l[indent_size:]) continue # we're in a quote. Skip blank lines for now if len(l) <= 1: queued_blanks = queued_blanks+1 continue if re.match(' '*indent_size, l): # still in quote. Write it out f.write( '\n'*queued_blanks) queued_blanks = 0 f.write(l[indent_size:]) continue # not in quote anymore if in_quote == 'example': f.write('#+END_EXAMPLE\n') else: f.write('#+END_SRC\n') f.write( '\n'*queued_blanks) f.write(l) queued_blanks = 0 in_quote = None prev_indented = False f.write('\n') if in_quote == 'example': f.write('#+END_EXAMPLE\n') elif in_quote == 'src': f.write('#+END_SRC\n') header = '* NAME\n{}: '.format(modname) write( header ) write_orgized(inspect.getdoc(mod)) write( '\n' ) write('* INTERFACE\n') for func in dirmod(): if re.match('_', func): continue if not inspect.isfunction(mod.__dict__[func]): continue doc = inspect.getdoc(mod.__dict__[func]) if doc: write('** {}()\n'.format(func)) write_orgized( doc ) write( '\n' ) with open('README.footer.org', 'r') as f_footer: write( f_footer.read() ) numpysane-0.21/numpysane.py000077500000000000000000002115261357065502000160610ustar00rootroot00000000000000#!/usr/bin/python r'''more-reasonable core functionality for numpy * SYNOPSIS >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> row = a[0,:] + 1000 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> row array([1000, 1001, 1002]) >>> nps.glue(a,b, axis=-1) array([[ 0, 1, 2, 100, 101, 102], [ 3, 4, 5, 103, 104, 105]]) >>> nps.glue(a,b,row, axis=-2) array([[ 0, 1, 2], [ 3, 4, 5], [ 100, 101, 102], [ 103, 104, 105], [1000, 1001, 1002]]) >>> nps.cat(a,b) array([[[ 0, 1, 2], [ 3, 4, 5]], [[100, 101, 102], [103, 104, 105]]]) >>> @nps.broadcast_define( (('n',), ('n',)) ) ... def inner_product(a, b): ... return a.dot(b) >>> inner_product(a,b) array([ 305, 1250]) * DESCRIPTION Numpy is widely used, relatively polished, and has a wide range of libraries available. At the same time, some of its very core functionality is strange, confusing and just plain wrong. This is in contrast with PDL (http://pdl.perl.org), which has a much more reasonable core, but a number of higher-level warts, and a relative dearth of library support. This module intends to improve the developer experience by providing alternate APIs to some core numpy functionality that is much more reasonable, especially for those who have used PDL in the past. Instead of writing a new module (this module), it would be really nice to simply patch numpy to give everybody the more reasonable behavior. I'd be very happy to do that, but the issues lie with some very core functionality, and any changes in behavior would break existing code. Any comments in how to achieve better behaviors in a less forky manner are welcome. Finally, if the existing system DOES make sense in some way that I'm simply not understanding, I'm happy to listen. I have no intention to disparage anyone or anything; I just want a more usable system for numerical computations. The issues addressed by this module fall into two broad categories: 1. Incomplete broadcasting support 2. Strange, special-case-ridden rules for basic array manipulation, especially dealing with dimensionality ** Broadcasting *** Problem Numpy has a limited support for broadcasting (http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html), a generic way to vectorize functions. When making a broadcasted call to a function, you pass in arguments with the inputs to vectorize available in new dimensions, and the broadcasting mechanism automatically calls the function multiple times as needed, and reports the output as an array collecting all the results. A basic example is an inner product: a function that takes in two identically-sized vectors (1-dimensional arrays) and returns a scalar (0-dimensional array). A broadcasted inner product function could take in two arrays of shape (2,3,4), compute the 6 inner products of length-4 each, and report the output in an array of shape (2,3). Numpy puts the most-significant dimension at the end, which is why this isn't 12 inner products of length-2 each. This is an arbitrary design choice, which could have been made differently: PDL puts the most-significant dimension at the front. The user doesn't choose whether to use broadcasting or not: some functions support it, and some do not. In PDL, broadcasting (called "threading" in that system) is a pervasive concept throughout. A PDL user has an expectation that every function can broadcast, and the documentation for every function is very explicit about the dimensionality of the inputs and outputs. Any data above the expected input dimensions is broadcast. By contrast, in numpy very few functions know how to broadcast. On top of that, the documentation is usually silent about the broadcasting status of a function in question. And on top of THAT, broadcasting rules state that an array of dimensions (n,m) is functionally identical to one of dimensions (1,1,1,....1,n,m). Sadly, numpy does not respect its own broadcasting rules, and many functions have special-case logic to create different behaviors for inputs with different numbers of dimensions; and this creates unexpected results. The effect of all this is a messy situation where the user is often not sure of the exact behavior of the functions they're calling, and trial and error is required to make the system do what one wants. *** Solution This module contains functionality to make any arbitrary function broadcastable, in either C or Python. **** Broadcasting rules A detailed description of broadcasting rules is available in the numpy documentation: http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html In short: - The most significant dimension in a numpy array is the LAST one, so the prototype of an input argument must exactly match a given input's trailing shape. So a prototype shape of (a,b,c) accepts an argument shape of (......, a,b,c), with as many or as few leading dimensions as desired. - The extra leading dimensions must be compatible across all the inputs. This means that each leading dimension must either - equal to 1 - be missing (thus assumed to equal 1) - equal to some positive integer >1, consistent across all arguments - The output is collected into an array that's sized as a superset of the above-prototype shape of each argument More involved example: A function with input prototype ( (3,), ('n',3), ('n',), ('m',) ) given inputs of shape (1,5, 3) (2,1, 8,3) ( 8) ( 5, 9) will return an output array of shape (2,5, ...), where ... is the shape of each output slice. Note again that the prototype dictates the TRAILING shape of the inputs. **** Broadcasting in python This is invoked as a decorator, applied to the arbitrary user function. An example: >>> import numpysane as nps >>> @nps.broadcast_define( (('n',), ('n',)) ) ... def inner_product(a, b): ... return a.dot(b) Here we have a simple inner product function to compute ONE inner product. We call 'broadcast_define' to add a broadcasting-aware wrapper that takes two 1D vectors of length 'n' each (same 'n' for the two inputs). This new 'inner_product' function applies broadcasting, as needed: >>> import numpy as np >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> inner_product(a,b) array([ 305, 1250]) Another related function in this module broadcast_generate(). It's similar to broadcast_define(), but instead of adding broadcasting-awareness to an existing function, it simply generates tuples from a set of arguments according to a given prototype. Stock numpy has some rudimentary support for all this with its vectorize() function, but it assumes only scalar inputs and outputs, which severaly limits its usefulness. **** Broadcasting in C A C-level flavor of broadcast_define() is available. It wraps C code in C loops. This is an analogue of PDL::PP (http://pdl.perl.org/PDLdocs/PP.html). Here the numpysane_pywrap module is used to produce C code that is compiled and linked into a python extension module. This takes more effort than python-level broadcasting, but the results have much less overhead, and run much faster. Please see the sample (https://github.com/dkogan/numpysane/blob/master/pywrap-sample). This is relatively new, so please let me know if you try it, and stuff does or does not work. *** New planned functionality The C broadcasting is functional, but a few more features are on the roadmap: - It should be possible for some inputs/output to contain different data types - And sometimes one would want to produce more than one output array for each call, possibly with different types - The prototype specification is not flexible enough. Maybe there's some relationship between named dimensions that is known. If so, this should be specify-able - Parallelization for broadcasted slices. Since each broadcasting loop is independent, this is a very natural place to add parallelism. This is fairly simple with OpenMP. ** Strangeness in core routines *** Problem There are some core numpy functions whose behavior is strange, full of special cases and very confusing, at least to me. That makes it difficult to achieve some very basic things. In the following examples, I use a function "arr" that returns a numpy array with given dimensions: >>> def arr(*shape): ... product = reduce( lambda x,y: x*y, shape) ... return np.arange(product).reshape(*shape) >>> arr(1,2,3) array([[[0, 1, 2], [3, 4, 5]]]) >>> arr(1,2,3).shape (1, 2, 3) The following sections are an incomplete list of the strange functionality I've encountered. **** Concatenation A prime example of confusing functionality is the array concatenation routines. Numpy has a number of functions to do this, each being strange. ***** hstack() hstack() performs a "horizontal" concatenation. When numpy prints an array, this is the last dimension (remember, the most significant dimensions in numpy are at the end). So one would expect that this function concatenates arrays along this last dimension. In the special case of 1D and 2D arrays, one would be right: >>> np.hstack( (arr(3), arr(3))).shape (6,) >>> np.hstack( (arr(2,3), arr(2,3))).shape (2, 6) but in any other case, one would be wrong: >>> np.hstack( (arr(1,2,3), arr(1,2,3))).shape (1, 4, 3) <------ I expect (1, 2, 6) >>> np.hstack( (arr(1,2,3), arr(1,2,4))).shape [exception] <------ I expect (1, 2, 7) >>> np.hstack( (arr(3), arr(1,3))).shape [exception] <------ I expect (1, 6) >>> np.hstack( (arr(1,3), arr(3))).shape [exception] <------ I expect (1, 6) I think the above should all succeed, and should produce the shapes as indicated. Cases such as "np.hstack( (arr(3), arr(1,3)))" are maybe up for debate, but broadcasting rules allow adding as many extra length-1 dimensions as we want without changing the meaning of the object, so I claim this should work. Either way, if you print out the operands for any of the above, you too would expect a "horizontal" stack() to work as stated above. It turns out that normally hstack() concatenates along axis=1, unless the first argument only has one dimension, in which case axis=0 is used. This is 100% wrong in a system where the most significant dimension is the last one, unless you assume that everyone has only 2D arrays, where the last dimension and the second dimension are the same. The correct way to do this is to concatenate along axis=-1. It works for n-dimensionsal objects, and doesn't require the special case logic for 1-dimensional objects that hstack() has. ***** vstack() Similarly, vstack() performs a "vertical" concatenation. When numpy prints an array, this is the second-to-last dimension (remember, the most significant dimensions in numpy are at the end). So one would expect that this function concatenates arrays along this second-to-last dimension. In the special case of 1D and 2D arrays, one would be right: >>> np.vstack( (arr(2,3), arr(2,3))).shape (4, 3) >>> np.vstack( (arr(3), arr(3))).shape (2, 3) >>> np.vstack( (arr(1,3), arr(3))).shape (2, 3) >>> np.vstack( (arr(3), arr(1,3))).shape (2, 3) >>> np.vstack( (arr(2,3), arr(3))).shape (3, 3) Note that this function appears to tolerate some amount of shape mismatches. It does it in a form one would expect, but given the state of the rest of this system, I found it surprising. For instance "np.hstack( (arr(1,3), arr(3)))" fails, so one would think that "np.vstack( (arr(1,3), arr(3)))" would fail too. And once again, adding more dimensions make it confused, for the same reason: >>> np.vstack( (arr(1,2,3), arr(2,3))).shape [exception] <------ I expect (1, 4, 3) >>> np.vstack( (arr(1,2,3), arr(1,2,3))).shape (2, 2, 3) <------ I expect (1, 4, 3) Similarly to hstack(), vstack() concatenates along axis=0, which is "vertical" only for 2D arrays, but not for any others. And similarly to hstack(), the 1D case has special-cased logic to work properly. The correct way to do this is to concatenate along axis=-2. It works for n-dimensionsal objects, and doesn't require the special case for 1-dimensional objects that vstack() has. ***** dstack() I'll skip the detailed description, since this is similar to hstack() and vstack(). The intent was to concatenate across axis=-3, but the implementation takes axis=2 instead. This is wrong, as before. And I find it strange that these 3 functions even exist, since they are all special-cases: the concatenation axis should be an argument, and at most, the edge special case (hstack()) should exist. This brings us to the next function: ***** concatenate() This is a more general function, and unlike hstack(), vstack() and dstack(), it takes as input a list of arrays AND the concatenation dimension. It accepts negative concatenation dimensions to allow us to count from the end, so things should work better. And in many ways that failed previously, they do: >>> np.concatenate( (arr(1,2,3), arr(1,2,3)), axis=-1).shape (1, 2, 6) >>> np.concatenate( (arr(1,2,3), arr(1,2,4)), axis=-1).shape (1, 2, 7) >>> np.concatenate( (arr(1,2,3), arr(1,2,3)), axis=-2).shape (1, 4, 3) But many things still don't work as I would expect: >>> np.concatenate( (arr(1,3), arr(3)), axis=-1).shape [exception] <------ I expect (1, 6) >>> np.concatenate( (arr(3), arr(1,3)), axis=-1).shape [exception] <------ I expect (1, 6) >>> np.concatenate( (arr(1,3), arr(3)), axis=-2).shape [exception] <------ I expect (3, 3) >>> np.concatenate( (arr(3), arr(1,3)), axis=-2).shape [exception] <------ I expect (2, 3) >>> np.concatenate( (arr(2,3), arr(2,3)), axis=-3).shape [exception] <------ I expect (2, 2, 3) This function works as expected only if - All inputs have the same number of dimensions - All inputs have a matching shape, except for the dimension along which we're concatenating - All inputs HAVE the dimension along which we're concatenating A legitimate use case that violates these conditions: I have an object that contains N 3D vectors, and I want to add another 3D vector to it. This is essentially the first failing example above. ***** stack() The name makes it sound exactly like concatenate(), and it takes the same arguments, but it is very different. stack() requires that all inputs have EXACTLY the same shape. It then concatenates all the inputs along a new dimension, and places that dimension in the location given by the 'axis' input. If this is the exact type of concatenation you want, this function works fine. But it's one of many things a user may want to do. **** inner() and dot() Another arbitrary example of a strange API is np.dot() and np.inner(). In a real-valued n-dimensional Euclidean space, a "dot product" is just another name for an "inner product". Numpy disagrees. It looks like np.dot() is matrix multiplication, with some wonky behaviors when given higher-dimension objects, and with some special-case behaviors for 1-dimensional and 0-dimensional objects: >>> np.dot( arr(4,5,2,3), arr(3,5)).shape (4, 5, 2, 5) <--- expected result for a broadcasted matrix multiplication >>> np.dot( arr(3,5), arr(4,5,2,3)).shape [exception] <--- np.dot() is not commutative. Expected for matrix multiplication, but not for a dot product >>> np.dot( arr(4,5,2,3), arr(1,3,5)).shape (4, 5, 2, 1, 5) <--- don't know where this came from at all >>> np.dot( arr(4,5,2,3), arr(3)).shape (4, 5, 2) <--- 1D special case. This is a dot product. >>> np.dot( arr(4,5,2,3), 3).shape (4, 5, 2, 3) <--- 0D special case. This is a scaling. It looks like np.inner() is some sort of quasi-broadcastable inner product, also with some funny dimensioning rules. In many cases it looks like np.dot(a,b) is the same as np.inner(a, transpose(b)) where transpose() swaps the last two dimensions: >>> np.inner( arr(4,5,2,3), arr(5,3)).shape (4, 5, 2, 5) <--- All the length-3 inner products collected into a shape with not-quite-broadcasting rules >>> np.inner( arr(5,3), arr(4,5,2,3)).shape (5, 4, 5, 2) <--- np.inner() is not commutative. Unexpected for an inner product >>> np.inner( arr(4,5,2,3), arr(1,5,3)).shape (4, 5, 2, 1, 5) <--- No idea >>> np.inner( arr(4,5,2,3), arr(3)).shape (4, 5, 2) <--- 1D special case. This is a dot product. >>> np.inner( arr(4,5,2,3), 3).shape (4, 5, 2, 3) <--- 0D special case. This is a scaling. **** atleast_xd() Numpy has 3 special-case functions atleast_1d(), atleast_2d() and atleast_3d(). For 4d and higher, you need to do something else. As expected by now, these do surprising things: >>> np.atleast_3d( arr(3)).shape (1, 3, 1) I don't know when this is what I would want, so we move on. *** Solution This module introduces new functions that can be used for this core functionality instead of the builtin numpy functions. These new functions work in ways that (I think) are more intuitive and more reasonable. They do not refer to anything being "horizontal" or "vertical", nor do they talk about "rows" or "columns"; these concepts simply don't apply in a generic N-dimensional system. These functions are very explicit about the dimensionality of the inputs/outputs, and fit well into a broadcasting-aware system. Furthermore, the names and semantics of these new functions come directly from PDL, which is more consistent in this area. Since these functions assume that broadcasting is an important concept in the system, the given axis indices should be counted from the most significant dimension: the last dimension in numpy. This means that where an axis index is specified, negative indices are encouraged. glue() forbids axis>=0 outright. Example for further justification: An array containing N 3D vectors would have shape (N,3). Another array containing a single 3D vector would have shape (3). Counting the dimensions from the end, each vector is indexed in dimension -1. However, counting from the front, the vector is indexed in dimension 0 or 1, depending on which of the two arrays we're looking at. If we want to add the single vector to the array containing the N vectors, and we mistakenly try to concatenate along the first dimension, it would fail if N != 3. But if we're unlucky, and N=3, then we'd get a nonsensical output array of shape (3,4). Why would an array of N 3D vectors have shape (N,3) and not (3,N)? Because if we apply python iteration to it, we'd expect to get N iterates of arrays with shape (3,) each, and numpy iterates from the first dimension: >>> a = np.arange(2*3).reshape(2,3) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> [x for x in a] [array([0, 1, 2]), array([3, 4, 5])] New functions this module provides (documented fully in the next section): **** glue Concatenates arrays along a given axis ('axis' must be given in a kwarg). Implicit length-1 dimensions are added at the start as needed. Dimensions other than the glueing axis must match exactly. **** cat Concatenate a given list of arrays along a new least-significant (leading) axis. Again, implicit length-1 dimensions are added, and the resulting shapes must match, and no data duplication occurs. **** clump Reshapes the array by grouping together 'n' dimensions, where 'n' is given in a kwarg. If 'n' > 0, then n leading dimensions are clumped; if 'n' < 0, then -n trailing dimensions are clumped **** atleast_dims Adds length-1 dimensions at the front of an array so that all the given dimensions are in-bounds. Given axis<0 can expand the shape; given axis>=0 MUST already be in-bounds. This preserves the alignment of the most-significant axis index. **** mv Moves a dimension from one position to another **** xchg Exchanges the positions of two dimensions **** transpose Reverses the order of the two most significant dimensions in an array. The whole array is seen as being an array of 2D matrices, each matrix living in the 2 most significant dimensions, which implies this definition. **** dummy Adds a single length-1 dimension at the given position **** reorder Completely reorders the dimensions in an array **** dot Broadcast-aware non-conjugating dot product. Identical to inner **** vdot Broadcast-aware conjugating dot product **** inner Broadcast-aware inner product. Identical to dot **** outer Broadcast-aware outer product. **** norm2 Broadcast-aware 2-norm. norm2(x) is identical to inner(x,x) **** mag Broadcast-aware vector magnitude. mag(x) is functionally identical to sqrt(inner(x,x)) **** trace Broadcast-aware trace. **** matmult Broadcast-aware matrix multiplication *** New planned functionality The functions listed above are a start, but more will be added with time. ''' import numpy as np from functools import reduce import itertools import types import inspect from distutils.version import StrictVersion # setup.py assumes the version is a simple string in '' quotes __version__ = '0.21' def _product(l): r'''Returns product of all values in the given list''' return reduce( lambda a,b: a*b, l ) def _clone_function(f, name): r'''Returns a clone of a given function. This is useful to copy a function, updating its metadata, such as the documentation, name, etc. There are also differences here between python 2 and python 3 that this function handles. ''' def get(f, what): what2 = 'func_{}'.format(what) what3 = '__{}__' .format(what) try: return getattr(f, what2) except: try: return getattr(f, what3) except: pass return None return types.FunctionType(get(f, 'code'), get(f, 'globals'), name, get(f, 'defaults'), get(f, 'closure')) class NumpysaneError(Exception): def __init__(self, err): self.err = err def __str__(self): return self.err def _eval_broadcast_dims( args, prototype ): r'''Helper function to evaluate a given list of arguments in respect to a given broadcasting prototype. This function will flag any errors in the dimensionality of the inputs. If no errors are detected, it returns dims_extra,dims_named where dims_extra is the outer shape of the broadcast This is a list: the union of all the leading shapes of all the arguments, after the trailing shapes of the prototype have been stripped dims_named is the sizes of the named dimensions This is a dict mapping dimension names to their sizes ''' # First I initialize dims_extra: the array containing the broadcasted # slices. Each argument calls for some number of extra dimensions, and the # overall array is as large as the biggest one of those Ndims_extra = 0 for i_arg in range(len(args)): Ndims_extra_here = len(args[i_arg].shape) - len(prototype[i_arg]) if Ndims_extra_here > Ndims_extra: Ndims_extra = Ndims_extra_here dims_extra = [1] * Ndims_extra def parse_dim( name_arg, shape_prototype, shape_arg, dims_named ): def range_rev(n): r'''Returns a range from -1 to -n. Useful to index variable-sized lists while aligning their ends.''' return range(-1, -n-1, -1) # first, I make sure the input is at least as dimension-ful as the # prototype. I do this by prepending dummy dimensions of length-1 as # necessary if len(shape_prototype) > len(shape_arg): ndims_missing_here = len(shape_prototype) - len(shape_arg) shape_arg = (1,) * ndims_missing_here + shape_arg # MAKE SURE THE PROTOTYPE DIMENSIONS MATCH (the trailing dimensions) # # Loop through the dimensions. Set the dimensionality of any new named # argument to whatever the current argument has. Any already-known # argument must match for i_dim in range_rev(len(shape_prototype)): dim_prototype = shape_prototype[i_dim] if not isinstance(dim_prototype, int): # This is a named dimension. These can have any value, but ALL # dimensions of the same name must thave the SAME value # EVERYWHERE if dim_prototype not in dims_named: dims_named[dim_prototype] = shape_arg[i_dim] dim_prototype = dims_named[dim_prototype] # The prototype dimension (named or otherwise) now has a numeric # value. Make sure it matches what I have if dim_prototype != shape_arg[i_dim]: raise NumpysaneError("Argument {} dimension '{}': expected {} but got {}". format(name_arg, shape_prototype[i_dim], dim_prototype, shape_arg[i_dim])) # I now know that this argument matches the prototype. I look at the # extra dimensions to broadcast, and make sure they match with the # dimensions I saw previously Ndims_extra_here = len(shape_arg) - len(shape_prototype) # MAKE SURE THE BROADCASTED DIMENSIONS MATCH (the leading dimensions) # # This argument has Ndims_extra_here dimensions to broadcast. The # current shape to broadcast must be at least as large, and must match for i_dim in range_rev(Ndims_extra_here): dim_arg = shape_arg[i_dim - len(shape_prototype)] if dim_arg != 1: if dims_extra[i_dim] == 1: dims_extra[i_dim] = dim_arg elif dims_extra[i_dim] != dim_arg: raise NumpysaneError("Argument {} prototype {} extra broadcast dim {} mismatch: previous arg set this to {}, but this arg wants {}". format(name_arg, shape_prototype, i_dim, dims_extra[i_dim], dim_arg)) dims_named = {} # parse_dim() adds to this for i_arg in range(len(args)): parse_dim( i_arg, prototype[i_arg], args[i_arg].shape, dims_named ) return dims_extra,dims_named def _broadcast_iter_dim( args, prototype, dims_extra ): r'''Generator to iterate through all the broadcasting slices. ''' # pad the dimension of each arg with ones. This lets me use the full # dims_extra index on each argument, without worrying about overflow args = [ atleast_dims(args[i], -(len(prototype[i])+len(dims_extra)) ) for i in range(len(args)) ] # per-arg dims_extra indexing varies: len-1 dimensions always index at 0. I # make a mask that I apply each time idx_slice_mask = np.ones( (len(args), len(dims_extra)), dtype=int) for i in range(len(args)): idx_slice_mask[i, np.array(args[i].shape,dtype=int)[:len(dims_extra)]==1] = 0 for idx_slice in itertools.product( *(range(x) for x in dims_extra) ): # tuple(idx) because of wonky behavior differences: # >>> a # array([[0, 1, 2], # [3, 4, 5]]) # # >>> a[tuple((1,1))] # 4 # # >>> a[list((1,1))] # array([[3, 4, 5], # [3, 4, 5]]) yield tuple( args[i][tuple(idx_slice * idx_slice_mask[i])] for i in range(len(args)) ) def broadcast_define(prototype, prototype_output=None, out_kwarg=None): r'''Vectorizes an arbitrary function, expecting input as in the given prototype. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> @nps.broadcast_define( (('n',), ('n',)) ) ... def inner_product(a, b): ... return a.dot(b) >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> inner_product(a,b) array([ 305, 1250]) The prototype defines the dimensionality of the inputs. In the inner product example above, the input is two 1D n-dimensional vectors. In particular, the 'n' is the same for the two inputs. This function is intended to be used as a decorator, applied to a function defining the operation to be vectorized. Each element in the prototype list refers to each input, in order. In turn, each such element is a list that describes the shape of that input. Each of these shape descriptors can be any of - a positive integer, indicating an input dimension of exactly that length - a string, indicating an arbitrary, but internally consistent dimension The normal numpy broadcasting rules (as described elsewhere) apply. In summary: - Dimensions are aligned at the end of the shape list, and must match the prototype - Extra dimensions left over at the front must be consistent for all the input arguments, meaning: - All dimensions !=1 must be identical - Missing dimensions are implicitly set to 1 - The output has a shape where - The trailing dimensions are whatever the function being broadcasted outputs - The leading dimensions come from the extra dimensions in the inputs Scalars are represented as 0-dimensional numpy arrays: arrays with shape (), and these broadcast as one would expect: >>> @nps.broadcast_define( (('n',), ('n',), ())) ... def scaled_inner_product(a, b, scale): ... return a.dot(b)*scale >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> scale = np.array((10,100)) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> scale array([ 10, 100]) >>> scaled_inner_product(a,b,scale) array([[ 3050], [125000]]) Let's look at a more involved example. Let's say we have a function that takes a set of points in R^2 and a single center point in R^2, and finds a best-fit least-squares line that passes through the given center point. Let it return a 3D vector containing the slope, y-intercept and the RMS residual of the fit. This broadcasting-enabled function can be defined like this: import numpy as np import numpysane as nps @nps.broadcast_define( (('n',2), (2,)) ) def fit(xy, c): # line-through-origin-model: y = m*x # E = sum( (m*x - y)**2 ) # dE/dm = 2*sum( (m*x-y)*x ) = 0 # ----> m = sum(x*y)/sum(x*x) x,y = (xy - c).transpose() m = np.sum(x*y) / np.sum(x*x) err = m*x - y err **= 2 rms = np.sqrt(err.mean()) # I return m,b because I need to translate the line back b = c[1] - m*c[0] return np.array((m,b,rms)) And I can use broadcasting to compute a number of these fits at once. Let's say I want to compute 4 different fits of 5 points each. I can do this: n = 5 m = 4 c = np.array((20,300)) xy = np.arange(m*n*2, dtype=np.float64).reshape(m,n,2) + c xy += np.random.rand(*xy.shape)*5 res = fit( xy, c ) mb = res[..., 0:2] rms = res[..., 2] print "RMS residuals: {}".format(rms) Here I had 4 different sets of points, but a single center point c. If I wanted 4 different center points, I could pass c as an array of shape (4,2). I can use broadcasting to plot all the results (the points and the fitted lines): import gnuplotlib as gp gp.plot( *nps.mv(xy,-1,0), _with='linespoints', equation=['{}*x + {}'.format(mb_single[0], mb_single[1]) for mb_single in mb], unset='grid', square=1) The examples above all create a separate output array for each broadcasted slice, and copy the contents from each such slice into the large output array that contains all the results. This is inefficient, and it is possible to pre-allocate an array to forgo these extra allocations and copies. There are several settings to control this. If the function being broadcasted can write its output to a given array instead of creating a new one, most of the inefficiency goes away. broadcast_define() supports the case where this function takes this array in a kwarg: the name of this kwarg can be given to broadcast_define() like so: @nps.broadcast_define( ....., out_kwarg = "out" ) def func( ....., out): ..... out[:] = result In order for broadcast_define() to pass such an output array to the inner function, this output array must be available, which means that it must be given to us somehow, or we must create it. The most efficient way to make a broadcasted call is to create the full output array beforehand, and to pass that to the broadcasted function. In this case, nothing extra will be allocated, and no unnecessary copies will be made. This can be done like this: @nps.broadcast_define( (('n',), ('n',)), ....., out_kwarg = "out" ) def inner_product(a, b, out): ..... out.setfield(a.dot(b), out.dtype) return out out = np.empty((2,4), float) inner_product( np.arange(3), np.arange(2*4*3).reshape(2,4,3), out=out) In this example, the caller knows that it's calling an inner_product function, and that the shape of each output slice would be (). The caller also knows the input dimensions and that we have an extra broadcasting dimension (2,4), so the output array will have shape (2,4) + () = (2,4). With this knowledge, the caller preallocates the array, and passes it to the broadcasted function call. Furthermore, in this case the inner function will be called with an output array EVERY time, and this is the only mode the inner function needs to support. If the caller doesn't know (or doesn't want to pre-compute) the shape of the output, it can let the broadcasting machinery create this array for them. In order for this to be possible, the shape of the output should be pre-declared, and the dtype of the output should be known: @nps.broadcast_define( (('n',), ('n',)), (), out_kwarg = "out" ) def inner_product(a, b, out): ..... out.setfield(a.dot(b), out.dtype) return out out = inner_product( np.arange(3), np.arange(2*4*3).reshape(2,4,3), dtype=int) Note that the caller didn't need to specify the prototype of the output or the extra broadcasting dimensions (output prototype is in the broadcast_define() call, but not the inner_product() call). Specifying the dtype here is optional: it defaults to float if omitted. If we want the output array to be pre-allocated, the output prototype (it is () in this example) is required: we must know the shape of the output array in order to create it. Without a declared output prototype, we can still make mostly- efficient calls: the broadcasting mechanism can call the inner function for the first slice as we showed earlier, by creating a new array for the slice. This new array required an extra allocation and copy, but it contains the required shape information. This infomation will be used to allocate the output, and the subsequent calls to the inner function will be efficient: @nps.broadcast_define( (('n',), ('n',)), out_kwarg = "out" ) def inner_product(a, b, out=None): ..... if out is None: return a.dot(b) out.setfield(a.dot(b), out.dtype) return out out = inner_product( np.arange(3), np.arange(2*4*3).reshape(2,4,3)) Here we were slighly inefficient, but the ONLY required extra specification was out_kwarg: that's mostly all you need. Also it is important to note that in this case the inner function is called both with passing it an output array to fill in, and with asking it to create a new one (by passing out=None to the inner function). This inner function then must support both modes of operation. If the inner function does not support filling in an output array, none of these efficiency improvements are possible. broadcast_define() is analogous to thread_define() in PDL. ''' def inner_decorator_for_some_reason(func): # args broadcast, kwargs do not. All auxillary data should go into the # kwargs def broadcast_loop(*args, **kwargs): if len(args) < len(prototype): raise NumpysaneError("Mismatched number of input arguments. Wanted at least {} but got {}". \ format(len(prototype), len(args))) args_passthru = args[ len(prototype):] args = args[0:len(prototype) ] # make sure all the arguments are numpy arrays args = tuple(np.asarray(arg) for arg in args) # dims_extra: extra dimensions to broadcast through # dims_named: values of the named dimensions dims_extra,dims_named = \ _eval_broadcast_dims( args, prototype) # if no broadcasting involved, just call the function if not dims_extra: sliced_args = args + args_passthru return func( *sliced_args, **kwargs ) # I checked all the dimensions and aligned everything. I have my # to-broadcast dimension counts. Iterate through all the broadcasting # output, and gather the results output = None # substitute named variable values into the output prototype prototype_output_expanded = None if prototype_output is not None: prototype_output_expanded = [d if type(d) is int else dims_named[d] for d in prototype_output] # if the output was supposed to go to a particular place, set that if out_kwarg and out_kwarg in kwargs: output = kwargs[out_kwarg] if prototype_output_expanded is not None: expected_shape = dims_extra + prototype_output_expanded if output.shape != tuple(expected_shape): raise NumpysaneError("Inconsistent output shape: expected {}, but got {}".format(expected_shape, output.shape)) # if we know enough to allocate the output, do that elif prototype_output_expanded is not None: kwargs_dtype = {} if 'dtype' in kwargs: kwargs_dtype['dtype'] = kwargs['dtype'] output = np.empty(dims_extra + prototype_output_expanded, **kwargs_dtype) # reshaped output. I write to this array if output is not None: output_flattened = clump(output, n=len(dims_extra)) i_slice = 0 for x in _broadcast_iter_dim( args, prototype, dims_extra ): # if the function knows how to write directly to an array, # request that if output is not None and out_kwarg: kwargs[out_kwarg] = output_flattened[i_slice, ...] sliced_args = x + args_passthru result = func( *sliced_args, **kwargs ) if not isinstance(result, np.ndarray): result = np.array(result) if output is None: output = np.empty( dims_extra + list(result.shape), dtype = result.dtype) output_flattened = output.reshape( (_product(dims_extra),) + result.shape) output_flattened[i_slice, ...] = result elif not out_kwarg: output_flattened[i_slice, ...] = result if prototype_output_expanded is None: prototype_output_expanded = result.shape elif result.shape != tuple(prototype_output_expanded): raise NumpysaneError("Inconsistent slice output shape: expected {}, but got {}".format(prototype_output_expanded, result.shape)) i_slice = i_slice+1 return output func_out = _clone_function( broadcast_loop, func.__name__ ) func_out.__doc__ = inspect.getdoc(func) if func_out.__doc__ is None: func_out.__doc__ = '' func_out.__doc__+= \ '''\n\nThis function is broadcast-aware through numpysane.broadcast_define(). The expected inputs have input prototype: {prototype} {output_prototype_text} The first {nargs} positional arguments will broadcast. The trailing shape of those arguments must match the input prototype; the leading shape must follow the standard broadcasting rules. Positional arguments past the first {nargs} and all the keyword arguments are passed through untouched.'''. \ format(prototype = prototype, output_prototype_text = 'No output prototype is defined.' if prototype_output is None else 'and output prototype\n\n {}'.format(prototype_output), nargs = len(prototype)) return func_out return inner_decorator_for_some_reason def broadcast_generate(prototype, args): r'''A generator that produces broadcasted slices Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> for s in nps.broadcast_generate( (('n',), ('n',)), (a,b)): ... print "slice: {}".format(s) slice: (array([0, 1, 2]), array([100, 101, 102])) slice: (array([3, 4, 5]), array([103, 104, 105])) ''' if len(args) != len(prototype): raise NumpysaneError("Mismatched number of input arguments. Wanted {} but got {}". \ format(len(prototype), len(args))) # make sure all the arguments are numpy arrays args = tuple(np.asarray(arg) for arg in args) # dims_extra: extra dimensions to broadcast through # dims_named: values of the named dimensions dims_extra,dims_named = \ _eval_broadcast_dims( args, prototype ) # I checked all the dimensions and aligned everything. I have my # to-broadcast dimension counts. Iterate through all the broadcasting # output, and gather the results for x in _broadcast_iter_dim( args, prototype, dims_extra ): yield x def glue(*args, **kwargs): r'''Concatenates a given list of arrays along the given 'axis' keyword argument. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> row = a[0,:] + 1000 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> row array([1000, 1001, 1002]) >>> nps.glue(a,b, axis=-1) array([[ 0, 1, 2, 100, 101, 102], [ 3, 4, 5, 103, 104, 105]]) # empty arrays ignored when glueing. Useful for initializing an accumulation >>> nps.glue(a,b, np.array(()), axis=-1) array([[ 0, 1, 2, 100, 101, 102], [ 3, 4, 5, 103, 104, 105]]) >>> nps.glue(a,b,row, axis=-2) array([[ 0, 1, 2], [ 3, 4, 5], [ 100, 101, 102], [ 103, 104, 105], [1000, 1001, 1002]]) >>> nps.glue(a,b, axis=-3) array([[[ 0, 1, 2], [ 3, 4, 5]], [[100, 101, 102], [103, 104, 105]]]) The 'axis' must be given in a keyword argument. In order to count dimensions from the inner-most outwards, this function accepts only negative axis arguments. This is because numpy broadcasts from the last dimension, and the last dimension is the inner-most in the (usual) internal storage scheme. Allowing glue() to look at dimensions at the start would allow it to unalign the broadcasting dimensions, which is never what you want. To glue along the last dimension, pass axis=-1; to glue along the second-to-last dimension, pass axis=-2, and so on. Unlike in PDL, this function refuses to create duplicated data to make the shapes fit. In my experience, this isn't what you want, and can create bugs. For instance, PDL does this: pdl> p sequence(3,2) [ [0 1 2] [3 4 5] ] pdl> p sequence(3) [0 1 2] pdl> p PDL::glue( 0, sequence(3,2), sequence(3) ) [ [0 1 2 0 1 2] <--- Note the duplicated "0,1,2" [3 4 5 0 1 2] ] while numpysane.glue() does this: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a[0:1,:] >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[0, 1, 2]]) >>> nps.glue(a,b,axis=-1) [exception] Finally, this function adds as many length-1 dimensions at the front as required. Note that this does not create new data, just new degenerate dimensions. Example: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> res = nps.glue(a,b, axis=-5) >>> res array([[[[[ 0, 1, 2], [ 3, 4, 5]]]], [[[[100, 101, 102], [103, 104, 105]]]]]) >>> res.shape (2, 1, 1, 2, 3) In numpysane older than 0.10 the semantics were slightly different: the axis kwarg was optional, and glue(*args) would glue along a new leading dimension, and thus would be equivalent to cat(*args). This resulted in very confusing error messages if the user accidentally omitted the kwarg. To request the legacy behavior, do nps.glue.legacy_version = '0.9' ''' legacy = \ hasattr(glue, 'legacy_version') and \ StrictVersion(glue.legacy_version) <= StrictVersion('0.9') axis = kwargs.get('axis') if legacy: if axis is not None and axis >= 0: raise NumpysaneError("axis >= 0 can make broadcasting dimensions inconsistent, and is thus not allowed") else: if axis is None: raise NumpysaneError("glue() requires the axis to be given in the 'axis' kwarg") if axis >= 0: raise NumpysaneError("axis >= 0 can make broadcasting dimensions inconsistent, and is thus not allowed") # deal with scalar (non-ndarray) args args = [ np.asarray(x) for x in args ] # ignore empty arrays( shape == (0,) ) but not scalars ( shape == ()) args = [ x for x in args if x.shape != (0,) ] if not args: return np.zeros((0,)) # Legacy behavior: if no axis is given, add a new axis at the front, and # glue along it max_ndim = max( x.ndim for x in args ) if axis is None: axis = -1 - max_ndim # if we're glueing along a dimension beyond what we already have, expand the # target dimension count if max_ndim < -axis: max_ndim = -axis # Now I add dummy dimensions at the front of each array, to bring the source # arrays to the same dimensionality. After this is done, ndims for all the # matrices will be the same, and np.concatenate() should know what to do. args = [ x[(np.newaxis,)*(max_ndim - x.ndim) + (Ellipsis,)] for x in args ] return np.concatenate( args, axis=axis ) def cat(*args): r'''Concatenates a given list of arrays along a new first (outer) dimension. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> b = a + 100 >>> c = a - 100 >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[100, 101, 102], [103, 104, 105]]) >>> c array([[-100, -99, -98], [ -97, -96, -95]]) >>> res = nps.cat(a,b,c) >>> res array([[[ 0, 1, 2], [ 3, 4, 5]], [[ 100, 101, 102], [ 103, 104, 105]], [[-100, -99, -98], [ -97, -96, -95]]]) >>> res.shape (3, 2, 3) >>> [x for x in res] [array([[0, 1, 2], [3, 4, 5]]), array([[100, 101, 102], [103, 104, 105]]), array([[-100, -99, -98], [ -97, -96, -95]])] This function concatenates the input arrays into an array shaped like the highest-dimensioned input, but with a new outer (at the start) dimension. The concatenation axis is this new dimension. As usual, the dimensions are aligned along the last one, so broadcasting will continue to work as expected. Note that this is the opposite operation from iterating a numpy array; see the example above. ''' if len(args) == 0: return np.array(()) max_ndim = max( x.ndim for x in args ) return glue(*args, axis = -1 - max_ndim) def clump(x, **kwargs): r'''Groups the given n dimensions together. Synopsis: >>> import numpysane as nps >>> nps.clump( arr(2,3,4), n = -2).shape (2, 12) Reshapes the array by grouping together 'n' dimensions, where 'n' is given in a kwarg. If 'n' > 0, then n leading dimensions are clumped; if 'n' < 0, then -n trailing dimensions are clumped So for instance, if x.shape is (2,3,4) then nps.clump(x, n = -2).shape is (2,12) and nps.clump(x, n = 2).shape is (6, 4) In numpysane older than 0.10 the semantics were different: n > 0 was required, and we ALWAYS clumped the trailing dimensions. Thus the new clump(-n) is equivalent to the old clump(n). To request the legacy behavior, do nps.clump.legacy_version = '0.9' ''' legacy = \ hasattr(clump, 'legacy_version') and \ StrictVersion(clump.legacy_version) <= StrictVersion('0.9') n = kwargs.get('n') if n is None: raise NumpysaneError("clump() requires a dimension count in the 'n' kwarg") if legacy: # old PDL-like clump(). Takes positive dimension counts, and acts from # the most-significant dimension (from the back) if n < 0: raise NumpysaneError("clump() requires n > 0") if n <= 1: return x if x.ndim < n: n = x.ndim s = list(x.shape[:-n]) + [ _product(x.shape[-n:]) ] return x.reshape(s) if -1 <= n and n <= 1: return x if x.ndim < n: n = x.ndim if -x.ndim > n: n = -x.ndim if n < 0: s = list(x.shape[:n]) + [ _product(x.shape[n:]) ] else: s = [ _product(x.shape[:n]) ] + list(x.shape[n:]) return x.reshape(s) def atleast_dims(x, *dims): r'''Returns an array with extra length-1 dimensions to contain all given axes. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6).reshape(2,3) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> nps.atleast_dims(a, -1).shape (2, 3) >>> nps.atleast_dims(a, -2).shape (2, 3) >>> nps.atleast_dims(a, -3).shape (1, 2, 3) >>> nps.atleast_dims(a, 0).shape (2, 3) >>> nps.atleast_dims(a, 1).shape (2, 3) >>> nps.atleast_dims(a, 2).shape [exception] >>> l = [-3,-2,-1,0,1] >>> nps.atleast_dims(a, l).shape (1, 2, 3) >>> l [-3, -2, -1, 1, 2] If the given axes already exist in the given array, the given array itself is returned. Otherwise length-1 dimensions are added to the front until all the requested dimensions exist. The given axis>=0 dimensions MUST all be in-bounds from the start, otherwise the most-significant axis becomes unaligned; an exception is thrown if this is violated. The given axis<0 dimensions that are out-of-bounds result in new dimensions added at the front. If new dimensions need to be added at the front, then any axis>=0 indices become offset. For instance: >>> x.shape (2, 3, 4) >>> [x.shape[i] for i in (0,-1)] [2, 4] >>> x = nps.atleast_dims(x, 0, -1, -5) >>> x.shape (1, 1, 2, 3, 4) >>> [x.shape[i] for i in (0,-1)] [1, 4] Before the call, axis=0 refers to the length-2 dimension and axis=-1 refers to the length=4 dimension. After the call, axis=-1 refers to the same dimension as before, but axis=0 now refers to a new length=1 dimension. If it is desired to compensate for this offset, then instead of passing the axes as separate arguments, pass in a single list of the axes indices. This list will be modified to offset the axis>=0 appropriately. Ideally, you only pass in axes<0, and this does not apply. Doing this in the above example: >>> l [0, -1, -5] >>> x.shape (2, 3, 4) >>> [x.shape[i] for i in (l[0],l[1])] [2, 4] >>> x=nps.atleast_dims(x, l) >>> x.shape (1, 1, 2, 3, 4) >>> l [2, -1, -5] >>> [x.shape[i] for i in (l[0],l[1])] [2, 4] We passed the axis indices in a list, and this list was modified to reflect the new indices: The original axis=0 becomes known as axis=2. Again, if you pass in only axis<0, then you don't need to care about this. ''' if any( not isinstance(d, int) for d in dims ): if len(dims) == 1 and isinstance(dims[0], list): dims = dims[0] else: raise NumpysaneError("atleast_dims() takes in axes as integers in separate arguments or\n" "as a single modifiable list") if max(dims) >= x.ndim: raise NumpysaneError("Axis {} out of bounds because x.ndim={}.\n" "To keep the last dimension anchored, " "only <0 out-of-bounds axes are allowed".format(max(dims), x.ndim)) need_ndim = -min(d if d<0 else -1 for d in dims) if x.ndim >= need_ndim: return x num_new_axes = need_ndim-x.ndim # apply an offset to any axes that need it if isinstance(dims, list): dims[:] = [ d+num_new_axes if d >= 0 else d for d in dims ] return x[ (np.newaxis,)*(num_new_axes) ] def mv(x, axis_from, axis_to): r'''Moves a given axis to a new position. Similar to numpy.moveaxis(). Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.mv( a, -1, 0).shape (4, 2, 3) >>> nps.mv( a, -1, -5).shape (4, 1, 1, 2, 3) >>> nps.mv( a, 0, -5).shape (2, 1, 1, 3, 4) New length-1 dimensions are added at the front, as required, and any axis>=0 that are passed in refer to the array BEFORE these new dimensions are added. ''' axes = [axis_from, axis_to] x = atleast_dims(x, axes) # The below is equivalent to # return np.moveaxis( x, *axes ) # but some older installs have numpy 1.8, where this isn't available axis_from = axes[0] if axes[0] >= 0 else x.ndim + axes[0] axis_to = axes[1] if axes[1] >= 0 else x.ndim + axes[1] # python3 needs the list() cast order = list(range(0, axis_from)) + list(range((axis_from+1), x.ndim)) order.insert(axis_to, axis_from) return np.transpose(x, order) def xchg(x, axis_a, axis_b): r'''Exchanges the positions of the two given axes. Similar to numpy.swapaxes() Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.xchg( a, -1, 0).shape (4, 3, 2) >>> nps.xchg( a, -1, -5).shape (4, 1, 2, 3, 1) >>> nps.xchg( a, 0, -5).shape (2, 1, 1, 3, 4) New length-1 dimensions are added at the front, as required, and any axis>=0 that are passed in refer to the array BEFORE these new dimensions are added. ''' axes = [axis_a, axis_b] x = atleast_dims(x, axes) return np.swapaxes( x, *axes ) def transpose(x): r'''Reverses the order of the last two dimensions. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.transpose(a).shape (2, 4, 3) >>> nps.transpose( np.arange(3) ).shape (3, 1) A "matrix" is generally seen as a 2D array that we can transpose by looking at the 2 dimensions in the opposite order. Here we treat an n-dimensional array as an n-2 dimensional object containing 2D matrices. As usual, the last two dimensions contain the matrix. New length-1 dimensions are added at the front, as required, meaning that 1D input of shape (n,) is interpreted as a 2D input of shape (1,n), and the transpose is 2 of shape (n,1). ''' return xchg( atleast_dims(x, -2), -1, -2) def dummy(x, axis=None): r'''Adds a single length-1 dimension at the given position. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.dummy(a, 0).shape (1, 2, 3, 4) >>> nps.dummy(a, 1).shape (2, 1, 3, 4) >>> nps.dummy(a, -1).shape (2, 3, 4, 1) >>> nps.dummy(a, -2).shape (2, 3, 1, 4) >>> nps.dummy(a, -5).shape (1, 1, 2, 3, 4) This is similar to numpy.expand_dims(), but handles out-of-bounds dimensions better. New length-1 dimensions are added at the front, as required, and any axis>=0 that are passed in refer to the array BEFORE these new dimensions are added. ''' need_ndim = axis+1 if axis >= 0 else -axis if x.ndim >= need_ndim: # referring to an axis that already exists. expand_dims() thus works return np.expand_dims(x, axis) # referring to a non-existing axis. I simply add sufficient new axes, and # I'm done return atleast_dims(x, axis) def reorder(x, *dims): r'''Reorders the dimensions of an array. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(24).reshape(2,3,4) >>> a.shape (2, 3, 4) >>> nps.reorder( a, 0, -1, 1 ).shape (2, 4, 3) >>> nps.reorder( a, -2 , -1, 0 ).shape (3, 4, 2) >>> nps.reorder( a, -4 , -2, -5, -1, 0 ).shape (1, 3, 1, 4, 2) This is very similar to numpy.transpose(), but handles out-of-bounds dimensions much better. New length-1 dimensions are added at the front, as required, and any axis>=0 that are passed in refer to the array BEFORE these new dimensions are added. ''' dims = list(dims) x = atleast_dims(x, dims) return np.transpose(x, dims) # Note that this explicitly isn't done with @broadcast_define. Instead I # implement the internals with core numpy routines. The advantage is that these # are some of very few numpy functions that support broadcasting, and they do so # in C, so their broadcasting loop is FAST. Much more so than my current # @broadcast_define loop def dot(a, b, out=None, dtype=None): r'''Non-conjugating dot product of two 1-dimensional n-long vectors. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3) >>> b = a+5 >>> a array([0, 1, 2]) >>> b array([5, 6, 7]) >>> nps.dot(a,b) 20 this is identical to numpysane.inner(). for a conjugating version of this function, use nps.vdot(). note that the numpy dot() has some special handling when its dot() is given more than 1-dimensional input. this function has no special handling: normal broadcasting rules are applied. ''' if out is not None and dtype is not None and out.dtype != dtype: raise NumpysaneError("'out' and 'dtype' given explicitly, but the dtypes are mismatched!") v = np.sum(a*b, axis=-1, out=out, dtype=dtype ) if out is None: return v return out # nps.inner and nps.dot are equivalent. Set the functionality and update the # docstring inner = _clone_function( dot, "inner" ) doc = dot.__doc__ doc = doc.replace("vdot", "aaa") doc = doc.replace("dot", "bbb") doc = doc.replace("inner", "ccc") doc = doc.replace("ccc", "dot") doc = doc.replace("bbb", "inner") doc = doc.replace("aaa", "vdot") inner.__doc__ = doc # Note that this explicitly isn't done with @broadcast_define. Instead I # implement the internals with core numpy routines. The advantage is that these # are some of very few numpy functions that support broadcasting, and they do so # on the C level, so their broadcasting loop is FAST. Much more so than my # current @broadcast_define loop def vdot(a, b, out=None, dtype=None): r'''Conjugating dot product of two 1-dimensional n-long vectors. vdot(a,b) is equivalent to dot(np.conj(a), b) Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.array(( 1 + 2j, 3 + 4j, 5 + 6j)) >>> b = a+5 >>> a array([ 1.+2.j, 3.+4.j, 5.+6.j]) >>> b array([ 6.+2.j, 8.+4.j, 10.+6.j]) >>> nps.vdot(a,b) array((136-60j)) >>> nps.dot(a,b) array((24+148j)) For a non-conjugating version of this function, use nps.dot(). Note that the numpy vdot() has some special handling when its vdot() is given more than 1-dimensional input. THIS function has no special handling: normal broadcasting rules are applied. ''' return dot(np.conj(a), b, out=out, dtype=dtype) @broadcast_define( (('n',), ('n',)), prototype_output=('n','n'), out_kwarg='out' ) def outer(a, b, out=None): r'''Outer product of two 1-dimensional n-long vectors. Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3) >>> b = a+5 >>> a array([0, 1, 2]) >>> b array([5, 6, 7]) >>> nps.outer(a,b) array([[ 0, 0, 0], [ 5, 6, 7], [10, 12, 14]]) ''' if out is None: return np.outer(a,b) out.setfield(np.outer(a,b), out.dtype) return out # Note that this explicitly isn't done with @broadcast_define. Instead I # implement the internals with core numpy routines. The advantage is that these # are some of very few numpy functions that support broadcasting, and they do so # in C, so their broadcasting loop is FAST. Much more so than my current # @broadcast_define loop def norm2(a, **kwargs): r'''Broadcast-aware 2-norm. norm2(x) is identical to inner(x,x) Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3) >>> a array([0, 1, 2]) >>> nps.norm2(a) 5 This is a convenience function to compute a 2-norm ''' return inner(a,a, **kwargs) def mag(a, out=None): r'''Magnitude of a vector. mag(x) is functionally identical to sqrt(inner(x,x)) Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3) >>> a array([0, 1, 2]) >>> nps.mag(a) 2.23606797749979 This is a convenience function to compute a magnitude of a vector, with full broadcasting support. If and explicit "out" array isn't given, we produce output of dtype=float. Otherwise "out" retains its dtype ''' if out is None: out = inner(a,a, dtype=float) if not isinstance(out, np.ndarray): # given two vectors, and without and 'out' array, inner() produces a # scalar, not an array. So I can't updated it inplace, and just # return a copy return np.sqrt(out) else: inner(a,a, out=out) # in-place sqrt np.sqrt.at(out,()) return out @broadcast_define( (('n','n',),), prototype_output=() ) def trace(a): r'''Broadcast-aware trace Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(3*4*4).reshape(3,4,4) >>> a array([[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15]], [[16, 17, 18, 19], [20, 21, 22, 23], [24, 25, 26, 27], [28, 29, 30, 31]], [[32, 33, 34, 35], [36, 37, 38, 39], [40, 41, 42, 43], [44, 45, 46, 47]]]) >>> nps.trace(a) array([ 30, 94, 158]) ''' return np.trace(a) # Could be implemented with a simple loop around np.dot(): # # @broadcast_define( (('n', 'm'), ('m', 'l')), prototype_output=('n','l'), out_kwarg='out' ) # def matmult2(a, b, out=None): # return np.dot(a,b) # # but this would produce a python broadcasting loop, which is potentially slow. # Instead I'm using the np.matmul() primitive to get C broadcasting loops. This # function has stupid special-case rules for low-dimensional arrays, so I make # sure to do the normal broadcasting thing in those cases def matmult2(a, b, out=None): r'''Multiplication of two matrices Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6) .reshape(2,3) >>> b = np.arange(12).reshape(3,4) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> nps.matmult2(a,b) array([[20, 23, 26, 29], [56, 68, 80, 92]]) This multiplies exactly 2 matrices, and the output object can be given in the 'out' argument. If the usual case where the you let numpysane create and return the result, you can use numpysane.matmult() instead. An advantage of that function is that it can multiply an arbitrary N matrices together, not just 2. ''' if not isinstance(a, np.ndarray) and not isinstance(b, np.ndarray): # two non-arrays (assuming two scalars) if out is not None: o = a*b out.setfield(o, out.dtype) out.resize([]) return out return a*b if not isinstance(a, np.ndarray) or len(a.shape) == 0: # one non-array (assuming one scalar) if out is not None: out.setfield(a*b, out.dtype) out.resize(b.shape) return out return a*b if not isinstance(b, np.ndarray) or len(b.shape) == 0: # one non-array (assuming one scalar) if out is not None: out.setfield(a*b, out.dtype) out.resize(a.shape) return out return a*b if len(b.shape) == 1: b = b[np.newaxis, :] o = np.matmul(a,b, out) return o def matmult( *args ): r'''Multiplication of N matrices Synopsis: >>> import numpy as np >>> import numpysane as nps >>> a = np.arange(6) .reshape(2,3) >>> b = np.arange(12).reshape(3,4) >>> c = np.arange(4) .reshape(4,1) >>> a array([[0, 1, 2], [3, 4, 5]]) >>> b array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> c array([[0], [1], [2], [3]]) >>> nps.matmult(a,b,c) array([[162], [504]]) This multiplies N matrices together by repeatedly calling matmult2() for each adjacent pair. Unlike matmult2(), the arguments MUST all be matrices to multiply. The 'out' kwarg for the output is not supported here. This function supports broadcasting fully, in C internally ''' return reduce( matmult2, args ) # I use np.matmul at one point. This was added in numpy 1.10.0, but # apparently I want to support even older releases. I thus provide a # compatibility function in that case. This is slower (python loop instead of C # loop), but at least it works if not hasattr(np, 'matmul'): @broadcast_define( (('n','m'),('m','o')), ('n','o')) def matmul(a,b, out=None): return np.dot(a,b,out) np.matmul = matmul numpysane-0.21/numpysane_pywrap.py000066400000000000000000000345121357065502000174560ustar00rootroot00000000000000import sys import time import numpy as np import os # Technically I'm supported to use some "resource extractor" something to # unbreak setuptools. But I'm instead assuming that this was installed via # Debian or by using the eager_resources tag in setup(). This allows files to # remain files, and to appear in a "normal" directory, where this script can # grab them and use them # # Aand I try two different directories, in case I'm running in-place _pywrap_path = ( os.path.dirname( __file__ ) + '/pywrap-templates', sys.prefix + '/share/python-numpysane/pywrap-templates' ) for p in _pywrap_path: _module_header_filename = p + '/pywrap_module_header.c' _module_footer_filename = p + '/pywrap_module_footer_generic.c' _function_filename = p + '/pywrap_function_generic.c' if os.path.exists(_module_header_filename): break else: raise Exception("Couldn't find pywrap templates! Looked in {}".format(_pywrap_path)) def _quote(s, convert_newlines=False): r'''Quote string for inclusion in C code There should be a library for this. Hopefuly this is correct. ''' s = s.replace('\\', '\\\\') # Pass all \ through verbatim if convert_newlines: s = s.replace('\n', '\\n') # All newlines -> \n s = s.replace('"', '\\"') # Quote all " return s def _substitute(s, convert_newlines=False, **kwargs): r'''format() with specific semantics - {xxx} substitutions founc in kwargs are made - {xxx} expressions not found in kwargs are left as is - {{ }} escaping is not respected: any '{xxx}' is replaced - \ and " and \n are handled for C strings - if convert_newlines: newlines are converted to \n (useful for C strings). Otherwise they're left alone (useful for C code) ''' for k in kwargs.keys(): v = kwargs[k] if isinstance(v, str): v = _quote(v, convert_newlines) else: v = str(v) s = s.replace('{' + k + '}', kwargs[k]) return s class module: def __init__(self, MODULE_NAME, MODULE_DOCSTRING, HEADER=''): r'''Initialize the python-wrapper-generator Arguments: - MODULE_NAME The name of the python module we're creating - MODULE_DOCSTRING The docstring for this module - HEADER C code to include verbatim. Any #includes or utility functions can go here ''' with open( _module_header_filename, 'r') as f: self.module_header = f.read() + "\n" + HEADER + "\n" with open( _module_footer_filename, 'r') as f: self.module_footer = _substitute(f.read(), MODULE_NAME = MODULE_NAME, MODULE_DOCSTRING = MODULE_DOCSTRING, convert_newlines = True) self.functions = [] def function(self, FUNCTION_NAME, FUNCTION_DOCSTRING, argnames, prototype_input, prototype_output, FUNCTION__slice_code, VALIDATE_code = ''): r'''Add a function to the python module we're creating SYNOPSIS If we're wrapping a simple inner product you can do this: function( "inner", "Inner-product pywrapped with npsp", prototype_input = (('n',), ('n',)), prototype_output = (), FUNCTION__slice_code = r""" output.data[0] = inner(a.data, b.data, a.strides[0], b.strides[0], a.shape[0]); return true; """, "a", "b'") Here we generate code to wrap a chunk of C code. The main chunk of user-supplied glue code is passed-in with FUNCTION__slice_code. This function is given all the input and output buffers, and it's the job of the glue code to read and write them. ARGUMENTS - FUNCTION_NAME The name of this function - FUNCTION_DOCSTRING The docstring for this function - argnames The names of the arguments. Must have the same number of elements as prototype_input - prototype_input An iterable defining the shapes of the inputs. Each element describes the trailing shape of each argument. Each element of this shape definition is either an integer (if this dimension of the broadcasted slice MUST have this size) or a string (naming this dimension; any size is allowed, but the sizes of all dimensions with this name must match) - prototype_output A single shape definition. Similar to prototype_input, but there's just one. Named dimensions in prototype_input must match the ones here - FUNCTION__slice_code This is a dict from numpy type objects to code snippets that form the body of a slice_function_t function. This is C code that will be included verbatim into the python-wrapping code. For instance, if we're wrapping a function called FUNCTION that works on 64-bit floating-point values, here we specify the way we call this function. typedef struct { void* data; const npy_intp* strides; const npy_intp* shape; } nps_slice_t; bool __FUNCTION__float64_slice( nps_slice_t output, nps_slice_t a, nps_slice_t b ) { // THE PASSED-IN STRING FOR THE 'float' KEY ENDS UP HERE ... ... // This string eventually contains a FUNCTION() call FUNCTION(...); } This function is called for each broadcasting slice. The number of arguments and their names are generated from the "prototype_input" and "argnames" arguments. The strides and shape define the memory layout of the data in memory for this slice. The 'shape' is only knowable at runtime because of named dimensions. The inner slice function returns true on success. Currently any number of data types are supported, but ALL of the inputs AND the output MUST share a single, consistent type. No implicit type conversions are performed, but the system does check for, and report type mismatches - VALIDATE_code C code that will be included verbatim into the python-wrapping code. Any special variable-validation code can be specified here. Dimensionality checks against the prototypes are generated automatically, but things like stride or type checking can be done here. ''' if len(prototype_input) != len(argnames): raise Exception("Input prototype says we have {} arguments, but names for {} were given. These must match". \ format(len(prototype_input), len(argnames))) function_slice_template = r''' static bool {SLICE_FUNCTION_NAME}(nps_slice_t output{SLICE_DEFINITIONS}) { {FUNCTION__slice} } ''' # I enumerate each named dimension, starting from -1, and counting DOWN named_dims = {} for i_arg in range(len(prototype_input)): shape = prototype_input[i_arg] for i_dim in range(len(shape)): dim = shape[i_dim] if isinstance(dim,int): if dim <= 0: raise Exception("Dimension {} in argument '{}' must be a string (named dimension) or an integer>0. Got '{}'". \ format(i_dim, argnames[i_arg], dim)) elif isinstance(dim, str): if dim not in named_dims: named_dims[dim] = -1-len(named_dims) else: raise Exception("Dimension {} in argument '{}' must be a string (named dimension) or an integer>0. Got '{}' (type '{}')". \ format(i_dim, argnames[i_arg], dim, type(dim))) # The output is allowed to have named dimensions, but ONLY those that # appear in the input for i_dim in range(len(prototype_output)): dim = prototype_output[i_dim] if isinstance(dim,int): if dim <= 0: raise Exception("Dimension {} in the output must be a string (named dimension) or an integer>0. Got '{}'". \ format(i_dim, dim)) elif isinstance(dim, str): if dim not in named_dims: raise Exception("Dimension {} in the output is a NEW named dimension: '{}'. Output named dimensions MUST match those already seen in the input". \ format(i_dim, dim)) else: raise Exception("Dimension {} in the ouptut must be a string (named dimension) or an integer>0. Got '{}' (type '{}')". \ format(i_dim, dim, type(dim))) for dim in prototype_output: if not isinstance(dim,int) and \ dim not in named_dims: named_dims[dim] = -1-len(named_dims) def expand_prototype(shape): r'''Produces a shape string for each argument This is the dimensions passed-into this function except, named dimensions are consolidated, and set to -1,-2,..., and the whole thing is stringified and joined ''' shape = [ dim if isinstance(dim,int) else named_dims[dim] for dim in shape ] return ','.join(str(dim) for dim in shape) PROTOTYPE_DIM_DEFS = '' for i_arg_input in range(len(argnames)): PROTOTYPE_DIM_DEFS += " const npy_intp PROTOTYPE_{}[{}] = {{{}}};\n". \ format(argnames[i_arg_input], len(prototype_input[i_arg_input]), expand_prototype(prototype_input[i_arg_input])); PROTOTYPE_DIM_DEFS += " /* Not const. updating the named dimensions in-place */\n" PROTOTYPE_DIM_DEFS += " npy_intp PROTOTYPE__output__[{}] = {{{}}};\n". \ format(len(prototype_output), expand_prototype(prototype_output)); PROTOTYPE_DIM_DEFS += " int Ndims_named = {};\n". \ format(len(named_dims)) known_types = tuple(FUNCTION__slice_code.keys()) slice_functions = [ "__{}__{}__slice".format(FUNCTION_NAME,np.dtype(t).name) for t in known_types] TYPE_DEFS = ' int Nknown_typenums = {};\n'.format(len(known_types)); TYPE_DEFS += \ ' int known_typenums[] = {' + \ ','.join(str(np.dtype(t).num) for t in known_types) + \ '};\n' TYPE_DEFS += \ ' slice_function_t* slice_functions[] = {' + \ ','.join(slice_functions) + \ '};\n' KNOWN_TYPES_LIST_STRING = ','.join(np.dtype(t).name for t in known_types) ARGUMENTS_LIST = ['#define ARGUMENTS(_)'] for i_arg_input in range(len(argnames)): ARGUMENTS_LIST.append( '_({})'.format(argnames[i_arg_input]) ) if not hasattr(self, 'function_template'): with open(_function_filename, 'r') as f: self.function_template = f.read() text = '' for i in range(len(known_types)): text += \ _substitute(function_slice_template, SLICE_FUNCTION_NAME = slice_functions[i], SLICE_DEFINITIONS = ''.join([", nps_slice_t " + n for n in argnames]), FUNCTION__slice = FUNCTION__slice_code[known_types[i]]) text += \ ' \\\n '.join(ARGUMENTS_LIST) + \ '\n\n' + \ _substitute(self.function_template, FUNCTION_NAME = FUNCTION_NAME, PROTOTYPE_DIM_DEFS = PROTOTYPE_DIM_DEFS, KNOWN_TYPES_LIST_STRING = KNOWN_TYPES_LIST_STRING, TYPE_DEFS = TYPE_DEFS, VALIDATE = VALIDATE_code) self.functions.append( (FUNCTION_NAME, _quote(FUNCTION_DOCSTRING, convert_newlines=True), text) ) def write(self, file=sys.stdout): r'''Write out the module definition to stdout Call this after the constructor and all the function() calls ''' # Get shellquote from the right place in python2 and python3 try: import pipes shellquote = pipes.quote except: # python3 puts this into a different module import shlex shellquote = shlex.quote print("// THIS IS A GENERATED FILE. CHANGES WILL BE OVERWRITTEN", file=file) print("// Generated on {} with {}\n\n". \ format(time.strftime("%Y-%m-%d %H:%M:%S"), ' '.join(shellquote(s) for s in sys.argv)), file=file) print('#define FUNCTIONS(_) \\', file=file) print(' \\\n'.join( ' _({}, "{}")'.format(f[0],f[1]) for f in self.functions), file=file) print("\n") print('///////// {{{{{{{{{ ' + _module_header_filename, file=file) print(self.module_header, file=file) print('///////// }}}}}}}}} ' + _module_header_filename, file=file) for f in self.functions: print('///////// {{{{{{{{{ ' + _function_filename, file=file) print('///////// for function ' + f[0], file=file) print(f[2], file=file) print('///////// }}}}}}}}} ' + _function_filename, file=file) print('\n', file=file) print('///////// {{{{{{{{{ ' + _module_footer_filename, file=file) print(self.module_footer, file=file) print('///////// }}}}}}}}} ' + _module_footer_filename, file=file) numpysane-0.21/pywrap-sample/000077500000000000000000000000001357065502000162575ustar00rootroot00000000000000numpysane-0.21/pywrap-sample/Makefile000066400000000000000000000020711357065502000177170ustar00rootroot00000000000000PYTHON_VERSION_FOR_EXTENSIONS := 3 # Minimal part of https://github.com/dkogan/mrbuild to provide python Makefile # rules include Makefile.common.header # I build a python extension module called "testlibmodule" from the C library # (testlib) and from the numpysane_pywrap wrapper. The wrapper is generated with # a demo script genpywrap.py all: testlibmodule$(PY_EXT_SUFFIX) testlibmodule$(PY_EXT_SUFFIX): testlib_pywrap_GENERATED.o testlib.o $(PY_MRBUILD_LINKER) $(PY_MRBUILD_LDFLAGS) $^ -o $@ testlib_pywrap_GENERATED.o: CFLAGS += $(PY_MRBUILD_CFLAGS) CC ?= gcc %.o:%.c $(CC) -Wall -Wextra $(CFLAGS) $(CPPFLAGS) -c -o $@ $< testlib_pywrap_GENERATED.c: genpywrap.py ./$< > $@ # In the python api I have to cast a PyCFunctionWithKeywords to a PyCFunction, # and the compiler complains. But that's how Python does it! So I tell the # compiler to chill testlib_pywrap_GENERATED.o: CFLAGS += -Wno-cast-function-type CFLAGS += -Wno-missing-field-initializers clean: rm -rf *.[do] *.so *.so.* testlib_pywrap_GENERATED.c .PHONY: clean # keep intermediate files .SECONDARY: numpysane-0.21/pywrap-sample/Makefile.common.header000066400000000000000000000132321357065502000224360ustar00rootroot00000000000000# -*- Makefile -*- # This is a part of the mrbuild project: https://github.com/dkogan/mrbuild # # Released under an MIT-style license. Modify and distribute as you like: # # Copyright 2016-2019 California Institute of Technology # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in all # copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE. # This stuff defines variables (PY_EXT_SUFFIX) that could be used by the user # Makefile at parsing time. So this must be included BEFORE the rest of the user # Makefile PYTHON_VERSION_FOR_EXTENSIONS ?= 3 # 2 or 3 # Flags for python extension modules. See # http://notes.secretsauce.net/notes/2017/11/14_python-extension-modules-without-setuptools-or-distutils.html # # I build the python extension module without any setuptools or anything. # Instead I ask python about the build flags it likes, and build the DSO # normally using those flags. # # There's some sillyness in Make I need to work around. First, I produce a # python script to query the various build flags, but replacing all whitespace # with __whitespace__. The string I get when running this script will then have # a number of whitespace-separated tokens, each setting ONE variable # # I set up a number of variables: # # These come from Python queries. I ask Python about XXX and store the result # into PY_XXX # # PY_CC # PY_CFLAGS # PY_CCSHARED # PY_INCLUDEPY # PY_BLDSHARED # PY_LDFLAGS # PY_EXT_SUFFIX # PY_MULTIARCH # # These process the above into a single set of CFLAGS: # # PY_MRBUILD_CFLAGS # # These process the above into a single set of LDFLAGS: # # PY_MRBUILD_LDFLAGS # # These process the above into a DSO-building linker command # # PY_MRBUILD_LINKER # # When the user Makefile evaluates ANY of these variables I query python, and # memoize the results. So the python is invoked at MOST one time. Any Makefiles # that don't touch the PY_... variables will not end up invoking the python # thing at all # # Variables to ask Python about _PYVARS_LIST := CC CFLAGS CCSHARED INCLUDEPY BLDSHARED BLDLIBRARY LDFLAGS EXT_SUFFIX MULTIARCH # Python script to query those variables define _PYVARS_SCRIPT from __future__ import print_function import sysconfig import re conf = sysconfig.get_config_vars() for v in ($(foreach v,$(_PYVARS_LIST),"$v",)): if v in conf: print(re.sub("[\t ]+", "__whitespace__", "_PY_{}:={}".format(v, conf[v]))) endef # I eval this to actually invoke the Python and to ingest its results. I only # eval this ONLY when necessary. define query_python_extension_building_flags _PYVARS := $$(shell python$(PYTHON_VERSION_FOR_EXTENSIONS) -c '$$(_PYVARS_SCRIPT)') # I then $(eval) these tokens one at a time, restoring the whitespace $$(foreach setvarcmd,$$(_PYVARS),$$(eval $$(subst __whitespace__, ,$$(setvarcmd)))) # pull out flags from CC, throw out the compiler itself, since I know better _FLAGS_FROM_PYCC := $$(wordlist 2,$$(words $$(_PY_CC)),$$(_PY_CC)) _PY_MRBUILD_CFLAGS := $$(filter-out -O%,$$(_FLAGS_FROM_PYCC) $$(_PY_CFLAGS) $$(_PY_CCSHARED) -I$$(_PY_INCLUDEPY)) # I add an RPATH to the python extension DSO so that it runs in-tree. Will pull # it out at install time _PY_MRBUILD_LDFLAGS := $$(_PY_LDFLAGS) -L$$(abspath .) -Wl,-rpath=$$(abspath .) _PY_MRBUILD_LINKER := $$(_PY_BLDSHARED) $$(_PY_BLDLIBRARY) endef # List of variables a user Makefile could touch _PYVARS_API := $(foreach v,$(_PYVARS_LIST),PY_$v) PY_MRBUILD_CFLAGS PY_MRBUILD_LDFLAGS PY_MRBUILD_LINKER # The first time the user touches these variables, ask Python. Each subsequent # time, use the previously-returned value. So we query Python at most once. If a # project isn't using the Python extension modules, we will not query Python at # all # # I handle all the Python API variables identically, except for PY_EXT_SUFFIX. # If Python gives me a suffix, Iuse it (this is available in python3; it has # ABI, architecture details). Otherwise, I try the multiarch suffix, or if even # THAT isn't available, just do .so. I need to handle it specially to make the # self-referential logic work with the memoization logic define _PY_DEFINE_API_VAR $1 = $$(or $$(_$1),$$(eval $$(query_python_extension_building_flags))$$(_$1)) endef define _PY_DEFINE_API_VAR_EXTSUFFIX $1 = $$(or $$(_$1),$$(eval $$(query_python_extension_building_flags))$$(or $$(_$1),$$(if $$(PY_MULTIARCH),.$$(PY_MULTIARCH)).so)) endef $(foreach v,$(filter-out PY_EXT_SUFFIX,$(_PYVARS_API)),$(eval $(call _PY_DEFINE_API_VAR,$v))) $(eval $(call _PY_DEFINE_API_VAR_EXTSUFFIX,PY_EXT_SUFFIX)) # Useful to pull in a local build of some library. Sets the compiler and linker # (runtime and build-time) flags. Invoke like this: # $(eval $(call add_local_library_path,/home/user/library)) define add_local_library_path CFLAGS += -I$1 CXXFLAGS += -I$1 LDFLAGS += -L$1 -Wl,-rpath=$1 endef numpysane-0.21/pywrap-sample/README000066400000000000000000000013371357065502000171430ustar00rootroot00000000000000This is a small project to demonstrate broadcast-aware python wrapping functionality provided by numpysane_pywrap. The wrapper can be produced and built by running "make". Then you can use the wrapper (to check that the output is correct) by running tst.py We produce a broadcast-aware python wrapper to a tiny C library called "testlib". This library is in testlib.[ch], and can compute inner and outer products. A demo script to generate broadcast-aware python wrapping for testlib is in genpywrap.py. The Makefile is set up to run this script, and to compile the results into a python extension module. This extension module can then be used by calling testlibmodule.inner() and testlibmodule.outer(). Samples appear in tst.py numpysane-0.21/pywrap-sample/genpywrap.py000077500000000000000000000067321357065502000206600ustar00rootroot00000000000000#!/usr/bin/python3 r'''A demo script to generate broadcast-aware python wrapping to testlib testlib is a tiny demo library that can compute inner and outer products. Here we wrap each available function. For each one we provide a code snipped that takes raw data arrays for each slice, and invokes the testlib library for each one ''' import sys import os dir_path = os.path.dirname(os.path.realpath(__file__)) sys.path[:0] = dir_path + '/..', import numpy as np import numpysane as nps import numpysane_pywrap as npsp m = npsp.module( MODULE_NAME = "testlibmodule", MODULE_DOCSTRING = "Test module", HEADER = '#include "testlib.h"') m.function( "inner", "Inner-product pywrapped with npsp", argnames = ("a", "b"), prototype_input = (('n',), ('n',)), prototype_output = (), FUNCTION__slice_code = \ {float: r''' ((double*)output.data)[0] = inner_double((double*)a.data, (double*)b.data, a.strides[0], b.strides[0], a.dims[0]); return true; ''', np.int64: r''' ((int64_t*)output.data)[0] = inner_int64_t((int64_t*)a.data, (int64_t*)b.data, a.strides[0], b.strides[0], a.dims[0]); return true; ''', np.int32: r''' ((int32_t*)output.data)[0] = inner_int32_t((int32_t*)a.data, (int32_t*)b.data, a.strides[0], b.strides[0], a.dims[0]); return true; '''}) m.function( "outer", "Outer-product pywrapped with npsp", argnames = ("a", "b"), prototype_input = (('n',), ('n',)), prototype_output = ('n', 'n'), FUNCTION__slice_code = \ {float: r''' outer((double*)output.data, (double*)a.data, (double*)b.data, a.strides[0], b.strides[0], a.dims[0]); return true; '''}) # Tests. Try to wrap functions using illegal output prototypes. The wrapper code # should barf try: m.function( "outer2", "Outer-product pywrapped with npsp", argnames = ("a", "b"), prototype_input = (('n',), ('n',)), prototype_output = ('n', 'fn'), FUNCTION__slice_code = '') except: pass # known error else: raise Exception("Expected error didn't happen") try: m.function( "outer3", "Outer-product pywrapped with npsp", argnames = ("a", "b"), prototype_input = (('n',), ('n',)), prototype_output = ('n', -1), FUNCTION__slice_code = '') except: pass # known error else: raise Exception("Expected error didn't happen") try: m.function( "outer4", "Outer-product pywrapped with npsp", argnames = ("a", "b"), prototype_input = (('n',), (-1,)), prototype_output = ('n', 'n'), FUNCTION__slice_code = '') except: pass # known error else: raise Exception("Expected error didn't happen") m.write() numpysane-0.21/pywrap-sample/testlib.c000066400000000000000000000030511357065502000200700ustar00rootroot00000000000000#include #include // Header for a demo C library being wrapped by numpysane_pywrap. This library // can compute inner and outer products with arbitrary strides. The inner // product is implemented with 32-bit integers, 64-bit integers and 64-bit // floats #define DEFINE_INNER_T(T) \ T inner_ ## T(const T* a, \ const T* b, \ int stride_a, \ int stride_b, \ int n) \ { \ T s = 0.0; \ for(int i=0; i #define DECLARE_INNER_T(T) \ T inner_ ## T(const T* a, \ const T* b, \ int stride_a, \ int stride_b, \ int n); DECLARE_INNER_T(int32_t) DECLARE_INNER_T(int64_t) DECLARE_INNER_T(double) #define inner inner_double void outer(// out assumed contiguous double* out, const double* a, const double* b, int stride_a, int stride_b, int n); numpysane-0.21/pywrap-sample/tst.py000066400000000000000000000121761357065502000174520ustar00rootroot00000000000000#!/usr/bin/python3 r'''A demo script to test the python wrapping produced with genpywrap.py testlib is a tiny demo library that can compute inner and outer products. We wrapped each available function by running genpywrap.py, and we ran "make" to build the C wrapper code into a python extension module. This script runs the wrapped-in-C inner and outer product, and compares the results against existing implementations in numpysane. ''' import numpy as np import numpysane as nps import sys # The extension module we're testing import testlibmodule def check(matching_functions, A, B): r'''Compare results of pairs of matching functions matching_functions is a list of pairs of functions that are supposed to produce identical results (testlibmodule and numpysane implementations of inner and outer products). A and B are lists of arguments that we try out. These support broadcasting, so either one is allowed to be a single array, which is then used for all the checks. I check both dynamically-created and inlined "out" arrays ''' N = 1 if type(A) is tuple and len(A) > N: N = len(A) if type(B) is tuple and len(B) > N: N = len(B) if type(A) is not tuple: A = (A,) * N if type(B) is not tuple: B = (B,) * N for f0,f1 in matching_functions: for i in range(N): out0 = f0(A[i], B[i]) out1 = f1(A[i], B[i]) if np.linalg.norm(out0 - out1) < 1e-10: print("Test passed") else: print("Dynamically-allocated test failed! {}({}, {}) should equal {}({}, {}), but the second one is {}". \ format(f0, A[i], B[i], f1, A[i], B[i], out1)) outshape = out0.shape out0 = np.zeros(outshape, dtype=np.array(A[i]).dtype) out1 = np.ones (outshape, dtype=np.array(A[i]).dtype) f0(A[i], B[i], out=out0) f1(A[i], B[i], out=out1) if np.linalg.norm(out0 - out1) < 1e-10: print("Test passed") else: print("Inlined 'out' test failed! {}({}, {}) should equal {}({}, {}), but the second one is {}". \ format(f0, A[i], B[i], f1, A[i], B[i], out1)) # pairs of functions that should produce identical results matching_functions = ( (nps.inner, testlibmodule.inner), (nps.outer, testlibmodule.outer) ) # Basic 1D arrays a0 = np.arange(5, dtype=float) b = a0+3 # a needs to broadcast; contiguous and strided a1 = np.arange(10, dtype=float).reshape(2,5) a2 = nps.transpose(np.arange(10, dtype=float).reshape(5,2)) # Try it! check(matching_functions, (a0,a1,a2), b) # Try it again, but use the floating-point version check( ((nps.inner, testlibmodule.inner),), tuple([a.astype(int) for a in (a0,a1,a2)]), b.astype(int)) try: check( ((nps.inner, testlibmodule.inner),), (a0,a1,a2), b.astype(int)) except: pass # expected barf. Types don't match else: print("should have barfed but didn't!") # Too few input dimensions (passing a scalar where a vector is expected). This # should be ok. It can be viewed as a length-1 vector check( ((nps.inner, testlibmodule.inner),), 6., (5., np.array(5, dtype=float), np.array((5,), dtype=float), ),) # Too few output dimensions. Again, this should be ok out = np.zeros((), dtype=float) testlibmodule.inner( nps.atleast_dims(np.array(6.,dtype=float), -5), nps.atleast_dims(np.array(5.,dtype=float), -2), out=out) if np.linalg.norm(out - 6*5) < 1e-10: print("Test passed") else: print("Inlined 'out' test failed! inner(6,5)=30, but I got {}".format(out)) # Broadcasting. Should be ok. No barf. testlibmodule.inner(np.arange(10, dtype=float).reshape( 2,5), np.arange(15, dtype=float).reshape(3,1,5)) try: testlibmodule.inner(np.arange(10, dtype=float).reshape(2,5), np.arange(15, dtype=float).reshape(3,5)) except: pass # expected barf else: print("should have barfed but didn't!") try: testlibmodule.inner(np.arange(5), np.arange(6)) except: pass # expected barf else: print("should have barfed but didn't!") try: testlibmodule.outer_only3(np.arange(5), np.arange(5)) except: pass # expected barf else: print("should have barfed but didn't!") testlibmodule.outer(a0,b, out=np.zeros((5,5), dtype=float)) # wrong dimensions on out. These all should barf try: testlibmodule.outer(a0,b, out=np.zeros((3,3), dtype=float)) except: pass # expected barf else: print("should have barfed but didn't!") try: testlibmodule.outer(a0,b, out=np.zeros((4,5), dtype=float)) except: pass # expected barf else: print("should have barfed but didn't!") try: testlibmodule.outer(a0,b, out=np.zeros((5,), dtype=float)) except: pass # expected barf else: print("should have barfed but didn't!") try: testlibmodule.outer(a0,b, out=np.zeros((), dtype=float)) except: pass # expected barf else: print("should have barfed but didn't!") try: testlibmodule.outer(a0,b, out=np.zeros((5,5,5), dtype=float)) except: pass # expected barf else: print("should have barfed but didn't!") numpysane-0.21/pywrap-templates/000077500000000000000000000000001357065502000167745ustar00rootroot00000000000000numpysane-0.21/pywrap-templates/pywrap_function_generic.c000066400000000000000000000374701357065502000240760ustar00rootroot00000000000000#undef ARG_DEFINE #undef NAMELIST #undef PARSECODE #undef PARSEARG #undef DECLARE_DIM_VARS #undef PARSE_DIMS #undef DEFINE_SLICE #undef ADVANCE_SLICE #undef ARGLIST_SLICE #undef FREE_PYARRAY static PyObject* __pywrap__{FUNCTION_NAME}(PyObject* NPY_UNUSED(self), PyObject* args, PyObject* kwargs) { PyObject* __py__result__ = NULL; PyArrayObject* __py__output__ = NULL; #define ARG_DEFINE(name) PyArrayObject* __py__ ## name = NULL; ARGUMENTS(ARG_DEFINE); SET_SIGINT(); #define NAMELIST(name) #name , char* keywords[] = { ARGUMENTS(NAMELIST) "out", NULL }; #define PARSECODE(name) "O&" #define PARSEARG(name) PyArray_Converter_leaveNone, &__py__ ## name, if(!PyArg_ParseTupleAndKeywords( args, kwargs, ARGUMENTS(PARSECODE) "|O&", keywords, ARGUMENTS(PARSEARG) PyArray_Converter_leaveNone, &__py__output__, NULL)) goto done; // Helper function to evaluate a given list of arguments in respect to a // given broadcasting prototype. This function will flag any errors in the // dimensionality of the inputs. If no errors are detected, it returns // dims_extra,dims_named // where // dims_extra is the outer dimensions of the broadcast // dims_named is the values of the named dimensions // First I initialize dims_extra: the array containing the broadcasted // slices. Each argument calls for some number of extra dimensions, and the // overall array is as large as the biggest one of those {PROTOTYPE_DIM_DEFS}; {TYPE_DEFS}; const int PROTOTYPE_LEN__output__ = (int)sizeof(PROTOTYPE__output__)/sizeof(PROTOTYPE__output__[0]); // the maximum of Ndims_extra_this for all the arguments int Ndims_extra = 0; { // It's possible for my arguments (and the output) to have fewer dimensions // than required by the prototype, and still pass all the dimensionality // checks, assuming implied leading dimensions of length 1. For instance I // could receive a scalar where a ('n',) dimension is expected, or a ('n',) // vector where an ('m','n') array is expected. I thus make a local (and // padded) copy of the strides and dims arrays, and use those where needed. // Most of the time these will just be copies of the input. The dimension // counts and argument counts will be relatively small, so this is only a // tiny bit wasteful #define DECLARE_DIM_VARS(name) \ const int PROTOTYPE_LEN_ ## name = (int)sizeof(PROTOTYPE_ ## name)/sizeof(PROTOTYPE_ ## name[0]); \ int __ndim__ ## name = PyArray_NDIM(__py__ ## name); \ if( __ndim__ ## name < PROTOTYPE_LEN_ ## name ) \ /* Too few input dimensions. Add dummy dimension of length-1 */ \ __ndim__ ## name = PROTOTYPE_LEN_ ## name; \ npy_intp __dims__ ## name[__ndim__ ## name]; \ npy_intp __strides__## name[__ndim__ ## name]; \ { \ const npy_intp* dims_orig = PyArray_DIMS (__py__ ## name); \ const npy_intp* strides_orig = PyArray_STRIDES(__py__ ## name); \ npy_intp ndim_orig = PyArray_NDIM (__py__ ## name); \ int i_dim = -1; \ for(; i_dim >= -ndim_orig; i_dim--) \ { \ __dims__ ## name[i_dim + __ndim__ ## name] = dims_orig [i_dim + ndim_orig]; \ __strides__## name[i_dim + __ndim__ ## name] = strides_orig[i_dim + ndim_orig]; \ } \ for(; i_dim >= -__ndim__ ## name; i_dim--) \ { \ __dims__ ## name[i_dim + __ndim__ ## name] = 1; \ __strides__ ## name[i_dim + __ndim__ ## name] = 0; \ } \ } \ void* __data__ ## name = PyArray_DATA (__py__ ## name); \ \ \ /* guaranteed >= 0 because of the padding */ \ int Ndims_extra_ ## name = __ndim__ ## name - PROTOTYPE_LEN_ ## name; \ if(Ndims_extra < Ndims_extra_ ## name) Ndims_extra = Ndims_extra_ ## name; ARGUMENTS(DECLARE_DIM_VARS); npy_intp dims_extra[Ndims_extra]; for(int i=0; i= 0) PROTOTYPE__output__[i] = dims_named[-PROTOTYPE__output__[i]-1]; else { PyErr_Format(PyExc_RuntimeError, "Output prototype has some named dimension not encountered in the input. The pywrap generator shouldn't have let this happen"); goto done; } } int selected_typenum = NPY_NOTYPE; slice_function_t* slice_function; for( int i=0; itype_num if( (PyObject*)__py__output__ != Py_None && __py__output__ != NULL && known_typenums[i] != PyArray_DESCR(__py__output__)->type_num ) // output type doesn't match continue; if(true ARGUMENTS(TYPE_MATCHES)) { // all arguments match this type! selected_typenum = known_typenums[i]; slice_function = slice_functions[i]; break; } } if(selected_typenum == NPY_NOTYPE) { #define INPUT_PERCENT_S(name) "%S," #define INPUT_TYPEOBJ(name) PyArray_DESCR(__py__ ## name)->typeobj, PyErr_Format(PyExc_RuntimeError, "ALL inputs and outputs must have consistent type: one of ({KNOWN_TYPES_LIST_STRING}), instead I got (inputs,output) of type (" ARGUMENTS(INPUT_PERCENT_S) "%S", ARGUMENTS(INPUT_TYPEOBJ) ((PyObject*)__py__output__ != Py_None && __py__output__ != NULL) ? (PyObject*)PyArray_DESCR(__py__output__)->typeobj : (PyObject*)Py_None); goto done; } {VALIDATE}; // The dimensions of the output must be (dims_extra + PROTOTYPE__output__) int Ndims_output = Ndims_extra + PROTOTYPE_LEN__output__; npy_intp dims_output_want[Ndims_output]; for(int i=0; i= -Ndims_to_compare; i_dim--) { int i_dim_var = i_dim + PyArray_NDIM(__py__output__); int i_dim_output = i_dim + Ndims_output; int dim_var = i_dim_var >= 0 ? PyArray_DIMS(__py__output__)[i_dim_var ] : 1; int dim_output_want = i_dim_output >= 0 ? dims_output_want [i_dim_output] : 1; if(dim_var != dim_output_want) { PyErr_Format(PyExc_RuntimeError, "Given output array dimension %d mismatch. Expected %d but got %d", i_dim, dim_output_want, dim_var); goto done; } } } else { // No output array available. Make one __py__output__ = (PyArrayObject*)PyArray_SimpleNew(Ndims_output, dims_output_want, selected_typenum); if(__py__output__ == NULL) { // Error already set. I simply exit goto done; } } // similarly to how I treated the inputs above, I handle the // dimensionality of the output. I make sure that output arrays with too // few dimensions (but enough elements) work properly. This effectively // does nothing useful if we created a new output array (we used exactly // the "right" shape), but it's required for passed-in output arrays // with funny dimensions int __ndim__output = PyArray_NDIM(__py__output__); if( __ndim__output < PROTOTYPE_LEN__output__ ) /* Too few output dimensions. Add dummy dimension of length-1 */ __ndim__output = PROTOTYPE_LEN__output__; npy_intp __dims__output[__ndim__output]; npy_intp __strides__output[__ndim__output]; { const npy_intp* dims_orig = PyArray_DIMS (__py__output__); const npy_intp* strides_orig = PyArray_STRIDES(__py__output__); npy_intp ndim_orig = PyArray_NDIM (__py__output__); int i_dim = -1; for(; i_dim >= -ndim_orig; i_dim--) { __dims__output[i_dim + __ndim__output] = dims_orig [i_dim + ndim_orig]; __strides__output[i_dim + __ndim__output] = strides_orig[i_dim + ndim_orig]; } for(; i_dim >= -__ndim__output; i_dim--) { __dims__output[i_dim + __ndim__output] = 1; __strides__output[i_dim + __ndim__output] = 0; } } // if no broadcasting involved, just call the function if(Ndims_extra == 0) { #define DEFINE_SLICE(name) char* slice_ ## name = __data__ ## name; ARGUMENTS(DEFINE_SLICE); char* slice_output = PyArray_DATA(__py__output__); #define ARGLIST_SLICE(name) \ , \ (nps_slice_t){ .data = (void*)slice_ ## name, \ .strides = &__strides__ ## name[ Ndims_extra_ ## name ], \ .dims = &__dims__ ## name[ Ndims_extra_ ## name ] } if( ! slice_function ( (nps_slice_t){ .data = (void*)slice_output, .strides = __strides__output, .dims = __dims__output } ARGUMENTS(ARGLIST_SLICE) ) ) { PyErr_Format(PyExc_RuntimeError, "__{FUNCTION_NAME}__slice failed!"); } else __py__result__ = (PyObject*)__py__output__; goto done; } #if 0 // How many elements (not bytes) to advance for each broadcasted dimension. // Takes into account the length-1 slieces (implicit and explicit) int stride_extra_elements_a[Ndims_extra]; int stride_extra_elements_b[Ndims_extra]; for(int idim_extra=0; idim_extra=0 && __dims__a[idim] != 1) stride_extra_elements_a[idim_extra] = __strides__a[idim] / sizeof(double); else stride_extra_elements_a[idim_extra] = 0; idim = idim_extra + Ndims_extra_b - Ndims_extra; if(idim>=0 && __dims__b[idim] != 1) stride_extra_elements_b[idim_extra] = __strides__b[idim] / sizeof(double); else stride_extra_elements_b[idim_extra] = 0; } #endif // I checked all the dimensions and aligned everything. I have my // to-broadcast dimension counts. // Iterate through all the broadcasting output, and gather the results int idims_extra[Ndims_extra]; for(int i=0; i=0; i--) { if(++idims[i] < Ndims[i]) return true; idims[i] = 0; } return false; } do { ARGUMENTS(DEFINE_SLICE); char* slice_output = PyArray_DATA(__py__output__); for( int i_dim=-1; i_dim >= -Ndims_extra; i_dim--) { #define ADVANCE_SLICE(name) \ if(i_dim + Ndims_extra_ ## name >= 0 && \ __dims__ ## name[i_dim + Ndims_extra_ ## name] != 1) \ slice_ ## name += idims_extra[i_dim + Ndims_extra]*__strides__ ## name[i_dim + Ndims_extra_ ## name]; ARGUMENTS(ADVANCE_SLICE); if(i_dim + Ndims_extra >= 0 && __dims__output[i_dim + Ndims_extra] != 1) slice_output += idims_extra[i_dim + Ndims_extra]*__strides__output[i_dim + Ndims_extra]; } if( ! slice_function ( (nps_slice_t){ .data = (void*)slice_output, .strides = __strides__output, .dims = __dims__output } ARGUMENTS(ARGLIST_SLICE) ) ) { PyErr_Format(PyExc_RuntimeError, "__{FUNCTION_NAME}__slice failed!"); goto done; } } while(next(idims_extra, dims_extra, Ndims_extra)); __py__result__ = (PyObject*)__py__output__; } done: #define FREE_PYARRAY(name) Py_XDECREF(__py__ ## name); ARGUMENTS(FREE_PYARRAY); if(__py__result__ == NULL) // An error occurred. If an output array was passed-in or created, I // release it Py_XDECREF(__py__output__); RESET_SIGINT(); return __py__result__; } #undef ARGUMENTS numpysane-0.21/pywrap-templates/pywrap_module_footer_generic.c000066400000000000000000000014261357065502000251040ustar00rootroot00000000000000#define PYMETHODDEF_ENTRY(name,docstring) \ { #name, \ (PyCFunction)__pywrap__ ## name, \ METH_VARARGS | METH_KEYWORDS, \ docstring }, static PyMethodDef methods[] = { FUNCTIONS(PYMETHODDEF_ENTRY) {} }; #if PY_MAJOR_VERSION == 2 PyMODINIT_FUNC init{MODULE_NAME}(void) { PyObject* module = Py_InitModule3("{MODULE_NAME}", methods, "{MODULE_DOCSTRING}"); import_array(); } #else static struct PyModuleDef module_def = { PyModuleDef_HEAD_INIT, "{MODULE_NAME}", "{MODULE_DOCSTRING}", -1, methods }; PyMODINIT_FUNC PyInit_{MODULE_NAME}(void) { PyObject* module = PyModule_Create(&module_def); import_array(); return module; } #endif numpysane-0.21/pywrap-templates/pywrap_module_header.c000066400000000000000000000131521357065502000233410ustar00rootroot00000000000000#define NPY_NO_DEPRECATED_API NPY_API_VERSION #include #include #include #include #include // Python is silly. There's some nuance about signal handling where it sets a // SIGINT (ctrl-c) handler to just set a flag, and the python layer then reads // this flag and does the thing. Here I'm running C code, so SIGINT would set a // flag, but not quit, so I can't interrupt the solver. Thus I reset the SIGINT // handler to the default, and put it back to the python-specific version when // I'm done #define SET_SIGINT() struct sigaction sigaction_old; \ do { \ if( 0 != sigaction(SIGINT, \ &(struct sigaction){ .sa_handler = SIG_DFL }, \ &sigaction_old) ) \ { \ PyErr_SetString(PyExc_RuntimeError, "sigaction() failed"); \ goto done; \ } \ } while(0) #define RESET_SIGINT() do { \ if( 0 != sigaction(SIGINT, \ &sigaction_old, NULL )) \ PyErr_SetString(PyExc_RuntimeError, "sigaction-restore failed"); \ } while(0) // just like PyArray_Converter(), but leave None as None static int PyArray_Converter_leaveNone(PyObject* obj, PyObject** address) { if(obj == Py_None) { *address = Py_None; Py_INCREF(Py_None); return 1; } return PyArray_Converter(obj,address); } typedef struct { void* data; const npy_intp* strides; const npy_intp* dims; } nps_slice_t; typedef bool (slice_function_t)(nps_slice_t output, nps_slice_t a, nps_slice_t b); static bool parse_dim(// input and output npy_intp* dims_named, npy_intp* dims_extra, // input int Ndims_extra, const char* arg_name, int Ndims_extra_var, const npy_intp* dims_want, int Ndims_want, const npy_intp* dims_var, int Ndims_var ) { // MAKE SURE THE PROTOTYPE DIMENSIONS MATCH (the trailing dimensions) // // Loop through the dimensions. Set the dimensionality of any new named // argument to whatever the current argument has. Any already-known // argument must match for( int i_dim=-1; i_dim >= -Ndims_want; i_dim--) { int i_dim_want = i_dim + Ndims_want; int dim_want = dims_want[i_dim_want]; int i_dim_var = i_dim + Ndims_var; // I assume i_dim_var>=0 because the caller prepended dimensions of // length-1 as needed int dim_var = dims_var[i_dim_var]; if(dim_want < 0) { // This is a named dimension. These can have any value, but // ALL dimensions of the same name must thave the SAME value // EVERYWHERE if(dims_named[-dim_want-1] < 0) dims_named[-dim_want-1] = dim_var; dim_want = dims_named[-dim_want-1]; } // The prototype dimension (named or otherwise) now has a numeric // value. Make sure it matches what I have if(dim_want != dim_var) { if(dims_want[i_dim_want] < 0) PyErr_Format(PyExc_RuntimeError, "Argument '%s': prototype says dimension %d (named dimension %d) has length %d, but got %d", arg_name, i_dim, dims_want[i_dim_want], dim_want, dim_var); else PyErr_Format(PyExc_RuntimeError, "Argument '%s': prototype says dimension %d has length %d, but got %d", arg_name, i_dim, dim_want, dim_var); return false; } } // I now know that this argument matches the prototype. I look at the // extra dimensions to broadcast, and make sure they match with the // dimensions I saw previously // MAKE SURE THE BROADCASTED DIMENSIONS MATCH (the leading dimensions) // // This argument has Ndims_extra_var dimensions to broadcast. The // current dimensions to broadcast must be at least as large, and must // match for( int i_dim=-1; i_dim >= -Ndims_extra_var; i_dim--) { int i_dim_var = i_dim - Ndims_want + Ndims_var; // I assume i_dim_var>=0 because the caller prepended dimensions of // length-1 as needed int dim_var = dims_var[i_dim_var]; int i_dim_extra = i_dim + Ndims_extra; if (dim_var != 1) { if( dims_extra[i_dim_extra] == 1) dims_extra[i_dim_extra] = dim_var; else if(dims_extra[i_dim_extra] != dim_var) { PyErr_Format(PyExc_RuntimeError, "Argument '%s' dimension %d (broadcasted dimension %d) mismatch. Previously saw length %d, but here have length %d", arg_name, i_dim-Ndims_want, i_dim, dims_extra[i_dim_extra], dim_var); return false; } } } return true; } numpysane-0.21/setup.py000077500000000000000000000025311357065502000151740ustar00rootroot00000000000000#!/usr/bin/python # using distutils not setuptools because setuptools puts dist_files in the root # of the host prefix, not the target prefix. Life is too short to fight this # nonsense from distutils.core import setup import re import glob version = None with open("numpysane.py", "r") as f: for l in f: m = re.match("__version__ *= *'(.*?)' *$", l) if m: version = m.group(1) break if version is None: raise Exception("Couldn't find version in 'numpysane.py'") pywrap_templates = glob.glob('pywrap-templates/*.c') setup(name = 'numpysane', version = version, author = 'Dima Kogan', author_email = 'dima@secretsauce.net', url = 'http://github.com/dkogan/numpysane', description = 'more-reasonable core functionality for numpy', long_description = """numpysane is a collection of core routines to provide basic numpy functionality in a more reasonable way""", license = 'LGPL-3+', py_modules = ['numpysane', 'numpysane_pywrap'], data_files = [ ('share/python-numpysane/pywrap-templates', pywrap_templates)], # Tell setuputils to not deal with the egg nonsense. numpysane_pywrap.py # assumes this eager_resources = pywrap_templates, test_suite = 'test_numpysane', install_requires = 'numpy') numpysane-0.21/test_numpysane.py000077500000000000000000001166641357065502000171270ustar00rootroot00000000000000#!/usr/bin/python2 import numpy as np import numpysane as nps from functools import reduce # Local test harness. The python standard ones all suck from testutils import * def arr(*shape, **kwargs): dtype = kwargs.get('dtype',float) r'''Return an arange() array of the given shape.''' if len(shape) == 0: return np.array(3, dtype=dtype) product = reduce( lambda x,y: x*y, shape) return np.arange(product, dtype=dtype).reshape(*shape) def test_broadcasting(): r'''Checking broadcasting rules.''' @nps.broadcast_define( (('n',), ('n',)) ) def f1(a, b): r'''Basic inner product.''' return a.dot(b) assertValueShape( np.array(5), (), f1, arr(3), arr(3)) assertValueShape( np.array((5,14)), (2,), f1, arr(2,3), arr(3)) assertValueShape( np.array((5,14)), (2,), f1, arr(3), arr(2,3)) assertValueShape( np.array(((5,14),)), (1,2,), f1, arr(1,2,3), arr(3)) assertValueShape( np.array(((5,),(14,))), (2,1,), f1, arr(2,1,3), arr(3)) assertValueShape( np.array((5,14)), (2,), f1, arr(2,3), arr(1,3)) assertValueShape( np.array((5,14)), (2,), f1, arr(1,3), arr(2,3)) assertValueShape( np.array(((5,14),)), (1,2,), f1, arr(1,2,3), arr(1,3)) assertValueShape( np.array(((5,),(14,))), (2,1,), f1, arr(2,1,3), arr(1,3)) assertValueShape( np.array(((5,14),(14,50))), (2,2,), f1, arr(2,1,3), arr(2,3)) assertValueShape( np.array(((5,14),(14,50))), (2,2,), f1, arr(2,1,3), arr(1,2,3)) confirm_raises( lambda: f1(arr(3)), msg='right number of args' ) confirm_raises( lambda: f1(arr(3),arr(5)), msg='matching args') confirm_raises( lambda: f1(arr(2,3),arr(4,3)), msg='matching args') confirm_raises( lambda: f1(arr(3,3,3),arr(2,1,3)), msg='matching args') confirm_raises( lambda: f1(arr(1,2,4),arr(2,1,3)), msg='matching args') # fancier function, has some preset dimensions @nps.broadcast_define( ((3,), ('n',3), ('n',), ('m',)) ) def f2(a,b,c,d): return d n=4 m=6 d = np.arange(m) assertValueShape( d, (m,), f2, arr( 3), arr(n,3), arr( n), arr( m)) assertValueShape( np.array((d,)), (1,m), f2, arr(1, 3), arr(1, n,3), arr( n), arr(1, m)) assertValueShape( np.array((d,)), (1,m,), f2, arr(1, 3), arr(1, n,3), arr( n), arr( m)) assertValueShape( np.array((d,d+m,d+2*m,d+3*m,d+4*m)), (5,m), f2, arr(5, 3), arr(5, n,3), arr( n), arr(5, m)) assertValueShape( np.array(((d,d+m,d+2*m,d+3*m,d+4*m),)), (1,5,m), f2, arr(1,5, 3), arr( 5, n,3), arr( n), arr( 5, m)) assertValueShape( np.array(((d,d+m,d+2*m,d+3*m,d+4*m), (d,d+m,d+2*m,d+3*m,d+4*m))), (2,5,m), f2, arr(1,5, 3), arr(2,5, n,3), arr( n), arr( 5, m)) assertValueShape( np.array(((d,d+m,d+2*m,d+3*m,d+4*m), (d,d+m,d+2*m,d+3*m,d+4*m))), (2,5,m), f2, arr(1,5, 3), arr(2,1, n,3), arr( n), arr( 5, m)) assertValueShape( np.array((((d,d,d,d,d), (d,d,d,d,d)),)), (1,2,5,m), f2, arr(1,1,5, 3), arr(1,2,1, n,3), arr(1, n), arr(1, 1, m)) confirm_raises( lambda: f2( arr(5, 3), arr(5, n,3), arr( m), arr(5, m)), msg='matching args') confirm_raises( lambda: f2( arr(5, 2), arr(5, n,3), arr( n), arr(5, m)), msg='matching args') confirm_raises( lambda: f2( arr(5, 2), arr(5, n,2), arr( n), arr(5, m)), msg='matching args') confirm_raises( lambda: f2( arr(1, 3), arr(1, n,3), arr( 5*n), arr(1, m)), msg='matching args') # Make sure extra args and the kwargs are passed through @nps.broadcast_define( ((3,), ('n',3), ('n',), ('m',)) ) def f3(a,b,c,d, e,f, *args, **kwargs): def val_or_0(x): return x if x else 0 return np.array( (a[0], val_or_0(e), val_or_0(f), val_or_0(args[0]), val_or_0( kwargs.get('xxx'))) ) assertValueShape( np.array( ((0, 1, 2, 3, 6), (3, 1, 2, 3, 6)) ), (2,5), f3, arr(2, 3), arr(1, n,3), arr( n), arr( m), 1, 2, 3, 4., dummy=5, xxx=6) # Make sure scalars (0-dimensional array) can broadcast @nps.broadcast_define( (('n',), ('n','m'), (2,), ()) ) def f4(a,b,c,d): return d @nps.broadcast_define( (('n',), ('n','m'), (2,), ()) ) def f5(a,b,c,d): return nps.glue( c, d, axis=-1 ) assertValueShape( np.array((5,5)), (2,), f4, arr( 3), arr(1, 3,4), arr(2, 2), np.array(5)) assertValueShape( np.array((5,5)), (2,), f4, arr( 3), arr(1, 3,4), arr(2, 2), 5) assertValueShape( np.array(((0,1,5),(2,3,5))), (2,3), f5, arr( 3), arr(1, 3,4), arr(2, 2), np.array(5)) assertValueShape( np.array(((0,1,5),(2,3,5))), (2,3), f5, arr( 3), arr(1, 3,4), arr(2, 2), 5) confirm_raises( lambda: f5( arr( 3), arr(1, 3,4), arr(2, 2), arr(5)) ) # Test the generator i=0 for s in nps.broadcast_generate( (('n',), ('n','m'), (2,), ()), (arr( 3), arr(1, 3,4), arr(2, 2), np.array(5)) ): confirm_equal( arr(3), s[0] ) confirm_equal( arr(3,4), s[1] ) confirm_equal( arr(2) + 2*i, s[2] ) confirm_equal( np.array(5), s[3] ) confirm_equal ( s[3].shape, ()) i = i+1 i=0 for s in nps.broadcast_generate( (('n',), ('n','m'), (2,), ()), (arr( 3), arr(1, 3,4), arr(2, 2), 5) ): confirm_equal( arr(3), s[0] ) confirm_equal( arr(3,4), s[1] ) confirm_equal( arr(2) + 2*i, s[2] ) confirm_equal( np.array(5), s[3] ) confirm_equal( s[3].shape, ()) i = i+1 i=0 for s in nps.broadcast_generate( (('n',), ('n','m'), (2,), ()), (arr( 3), arr(1, 3,4), arr(2, 2), arr(2)) ): confirm_equal( arr(3), s[0] ) confirm_equal( arr(3,4), s[1] ) confirm_equal( arr(2) + 2*i, s[2] ) confirm_equal( np.array(i), s[3] ) confirm_equal( s[3].shape, ()) i = i+1 # Make sure we add dummy length-1 dimensions assertValueShape( None, (3,), nps.matmult, arr(4), arr(4,3) ) assertValueShape( None, (3,), nps.matmult2, arr(4), arr(4,3) ) assertValueShape( None, (1,3,), nps.matmult, arr(1,4), arr(4,3) ) assertValueShape( None, (1,3,), nps.matmult2, arr(1,4), arr(4,3) ) assertValueShape( None, (10,3,), nps.matmult, arr(4), arr(10,4,3) ) assertValueShape( None, (10,3,), nps.matmult2, arr(4), arr(10,4,3) ) assertValueShape( None, (10,1,3,), nps.matmult, arr(1,4), arr(10,4,3) ) assertValueShape( None, (10,1,3,), nps.matmult2, arr(1,4), arr(10,4,3) ) # scalar output shouldn't barf @nps.broadcast_define( ((),), ) def f6(x): return 6 @nps.broadcast_define( ((),), ()) def f7(x): return 7 assertValueShape( 6, (), f6, 5) assertValueShape( 6*np.ones((5,)), (5,), f6, np.arange(5)) assertValueShape( 7, (), f7, 5) assertValueShape( 7*np.ones((5,)), (5,), f7, np.arange(5)) def test_broadcasting_into_output(): r'''Checking broadcasting with the output array defined.''' # I think about all 2^4 = 16 combinations: # # broadcast_define(): yes/no prototype_output, out_kwarg # broadcasted call: yes/no dtype, output prototype = (('n',), ('n',)) in1, in2 = arr(3), arr(2,4,3) out_ref = np.array([[ 5, 14, 23, 32], [41, 50, 59, 68]]) outshape_ref = (2,4) def f(a, b, out=None, dtype=None): r'''Basic inner product.''' if out is None: if dtype is None: return a.dot(b) else: return a.dot(b).astype(dtype) if f.do_dtype_check: if dtype is not None: confirm_equal( out.dtype, dtype ) if f.do_base_check: if f.base is not None: confirm_is(out.base, f.base) f.base_check_count = f.base_check_count+1 else: f.base_check_count = 0 f.base = out.base if f.do_dim_check: if out.shape != (): raise nps.NumpysaneError("mismatched lists") out.setfield(a.dot(b), out.dtype) return out # First we look at the case where broadcast_define() has no out_kwarg. # Then the output cannot be specified at all. If prototype_output # exists, then it is either used to create the output array, or to # validate the dimensions of output slices obtained from elsewhere. The # dtype is simply passed through to the inner function, is free to use # it, to not use it, or to crash in response (the f() function above # will take it; created arrays will be of that type; passed-in arrays # will create an error for a wrong type) f1 = nps.broadcast_define(prototype) (f) f2 = nps.broadcast_define(prototype, prototype_output=() )(f) f3 = nps.broadcast_define(prototype, prototype_output=(1,) )(f) f.do_base_check = False f.do_dtype_check = False f.do_dim_check = True assertValueShape( out_ref, outshape_ref, f1, in1, in2) assertValueShape( out_ref, outshape_ref, f1, in1, in2, dtype=float) assertValueShape( out_ref, outshape_ref, f1, in1, in2, dtype=int) assertValueShape( out_ref, outshape_ref, f2, in1, in2) confirm_raises ( lambda: f3(in1, in2) ) # OK then. Let's now pass in an out_kwarg. Here we do not yet # pre-allocate an output. Thus if we don't pass in a prototype_output # either, the first slice will dictate the output shape, and we'll have # 7 inner calls into an output array (6 base comparisons). If we DO pass # in a prototype_output, then we will allocate immediately, and we'll # see 8 inner calls into an output array (7 base comparisons) f1 = nps.broadcast_define(prototype, out_kwarg="out") (f) f2 = nps.broadcast_define(prototype, out_kwarg="out", prototype_output=() )(f) f3 = nps.broadcast_define(prototype, out_kwarg="out", prototype_output=(1,) )(f) f.do_base_check = True f.do_dtype_check = True f.do_dim_check = True f.base = None assertValueShape( out_ref, outshape_ref, f1, in1, in2) confirm_equal( 6, f.base_check_count ) f.base = None assertValueShape( out_ref, outshape_ref, f1, in1, in2, dtype=float) confirm_equal( 6, f.base_check_count ) f.base = None assertValueShape( out_ref, outshape_ref, f1, in1, in2, dtype=int) confirm_equal( 6, f.base_check_count ) f.base = None assertValueShape( out_ref, outshape_ref, f2, in1, in2) confirm_equal( 7, f.base_check_count ) f.base = None assertValueShape( out_ref, outshape_ref, f2, in1, in2, dtype=float) confirm_equal( 7, f.base_check_count ) f.base = None assertValueShape( out_ref, outshape_ref, f2, in1, in2, dtype=int) confirm_equal( 7, f.base_check_count ) # Here the inner function will get an improperly-sized array to fill in. # broadcast_define() itself won't see any issues with this, but the # inner function is free to detect the error f.do_dim_check = False f.base = None assertValueShape( None, None, f3, in1, in2) f.do_dim_check = True f.base = None confirm_raises( lambda: f3(in1, in2) ) # Now pre-allocate the full output array ourselves. Any prototype_output # we pass in is used for validation. Any dtype passed in does nothing, # but assertValueShape() will flag discrepancies. We use the same # f1,f2,f3 as above f.do_base_check = True f.do_dtype_check = False f.do_dim_check = True # correct shape, varying dtypes out0 = np.empty( outshape_ref, dtype=float ) out1 = np.empty( outshape_ref, dtype=int ) # shape has too many dimensions out2 = np.empty( outshape_ref + (1,), dtype=int ) out3 = np.empty( outshape_ref + (2,), dtype=int ) out4 = np.empty( (1,) + outshape_ref, dtype=int ) out5 = np.empty( (2,) + outshape_ref, dtype=int ) # shape has the correct number of dimensions, but they aren't right out6 = np.empty( (1,) + outshape_ref[1:], dtype=int ) out7 = np.empty( outshape_ref[:1] + (1,), dtype=int ) # f1 and f2 should work exactly the same, since prototype_output is just # a validating parameter for f12 in f1,f2: f.base = None assertValueShape( out_ref, outshape_ref, f12, in1, in2, out=out0) confirm_equal( 7, f.base_check_count ) f.base = None assertValueShape( out_ref, outshape_ref, f12, in1, in2, out=out0, dtype=float) confirm_equal( 7, f.base_check_count ) f.base = None assertValueShape( out_ref, outshape_ref, f12, in1, in2, out=out1) confirm_equal( 7, f.base_check_count ) f.base = None assertValueShape( out_ref, outshape_ref, f12, in1, in2, out=out1, dtype=int) confirm_equal( 7, f.base_check_count ) # any improperly-sized output matrices WILL be flagged if # prototype_output is given, and will likely be flagged if it isn't # also, although there are cases where this wouldn't happen. I simply # expect all of these to fail for out_misshaped in out2,out3,out4,out5,out6,out7: f.do_dim_check = False f.base = None confirm_raises( lambda: f2(in1, in2, out=out_misshaped) ) f.do_dim_check = True f.base = None confirm_raises( lambda: f1(in1, in2, out=out_misshaped) ) def test_concatenation(): r'''Checking the various concatenation functions.''' confirm_raises( lambda: nps.glue( arr(2,3), arr(2,3), axis=0), msg='axes are negative' ) confirm_raises( lambda: nps.glue( arr(2,3), arr(2,3), axis=1), msg='axes are negative' ) # basic glueing assertValueShape( None, (2,6), nps.glue, arr(2,3), arr(2,3), axis=-1 ) assertValueShape( None, (4,3), nps.glue, arr(2,3), arr(2,3), axis=-2 ) assertValueShape( None, (2,2,3), nps.glue, arr(2,3), arr(2,3), axis=-3 ) assertValueShape( None, (2,1,2,3), nps.glue, arr(2,3), arr(2,3), axis=-4 ) confirm_raises ( lambda: nps.glue( arr(2,3), arr(2,3)) ) assertValueShape( None, (2,2,3), nps.cat, arr(2,3), arr(2,3) ) # extra length-1 dims added as needed, data not duplicated as needed confirm_raises( lambda: nps.glue( arr(3), arr(2,3), axis=-1) ) assertValueShape( None, (3,3), nps.glue, arr(3), arr(2,3), axis=-2 ) confirm_raises( lambda: nps.glue( arr(3), arr(2,3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(3), arr(2,3), axis=-4) ) confirm_raises( lambda: nps.glue( arr(3), arr(2,3)) ) confirm_raises( lambda: nps.cat( arr(3), arr(2,3)) ) confirm_raises( lambda: nps.glue( arr(2,3), arr(3), axis=-1) ) assertValueShape( None, (3,3), nps.glue, arr(2,3), arr(3), axis=-2 ) confirm_raises( lambda: nps.glue( arr(2,3), arr(3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(2,3), arr(3), axis=-4) ) confirm_raises( lambda: nps.cat( arr(2,3), arr(3)) ) confirm_raises( lambda: nps.glue( arr(1,3), arr(2,3), axis=-1) ) assertValueShape( None, (3,3), nps.glue, arr(1,3), arr(2,3), axis=-2 ) confirm_raises( lambda: nps.glue( arr(1,3), arr(2,3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(1,3), arr(2,3), axis=-4) ) confirm_raises( lambda: nps.cat( arr(1,3), arr(2,3)) ) confirm_raises( lambda: nps.glue( arr(2,3), arr(1,3), axis=-1) ) assertValueShape( None, (3,3), nps.glue, arr(2,3), arr(1,3), axis=-2 ) confirm_raises( lambda: nps.glue( arr(2,3), arr(1,3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(2,3), arr(1,3), axis=-4) ) confirm_raises( lambda: nps.cat( arr(2,3), arr(1,3)) ) confirm_raises( lambda: nps.glue( arr(1,3), arr(2,3), axis=-1) ) assertValueShape( None, (3,3), nps.glue, arr(1,3), arr(2,3), axis=-2 ) confirm_raises( lambda: nps.glue( arr(1,3), arr(2,3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(1,3), arr(2,3), axis=-4) ) confirm_raises( lambda: nps.cat( arr(1,3), arr(2,3)) ) # empty arrays are accepted and ignored assertValueShape( None, (2,3), nps.glue, np.array(()), arr(2,3), axis=-2 ) assertValueShape( None, (1,2,3), nps.glue, np.array(()), arr(2,3), axis=-3 ) assertValueShape( None, (3,), nps.glue, arr(3), np.array(()), axis=-1 ) assertValueShape( None, (1,3), nps.glue, arr(3), np.array(()), axis=-2 ) assertValueShape( None, (3,3), nps.glue, arr(3), arr(2,3), np.array(()), axis=-2 ) # zero-length arrays do the right thing confirm_raises( lambda: nps.glue( arr(0,3), arr(2,3), axis=-1) ) assertValueShape( None, (2,3), nps.glue, arr(0,3), arr(2,3), axis=-2 ) confirm_raises( lambda: nps.glue( arr(0,3), arr(2,3), axis=-3) ) assertValueShape( None, (2,3), nps.glue, arr(2,0), arr(2,3), axis=-1 ) confirm_raises( lambda: nps.glue( arr(2,0), arr(2,3), axis=-2) ) confirm_raises( lambda: nps.glue( arr(2,0), arr(2,3), axis=-3) ) assertValueShape( None, (2,3), nps.glue, arr(2,0), arr(2,3), axis=-1 ) confirm_raises( lambda: nps.glue( arr(2,0), arr(2,3), axis=-2) ) confirm_raises( lambda: nps.glue( arr(2,0), arr(2,3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(2,3), arr(0,3), axis=-1) ) assertValueShape( None, (2,3), nps.glue, arr(2,3), arr(0,3), axis=-2 ) confirm_raises( lambda: nps.glue( arr(2,3), arr(0,3), axis=-3) ) assertValueShape( None, (0,5), nps.glue, arr(0,2), arr(0,3), axis=-1 ) confirm_raises( lambda: nps.glue( arr(0,2), arr(0,3), axis=-2) ) confirm_raises( lambda: nps.glue( arr(0,2), arr(0,3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(2,0), arr(0,3), axis=-1) ) confirm_raises( lambda: nps.glue( arr(2,0), arr(0,3), axis=-2) ) confirm_raises( lambda: nps.glue( arr(2,0), arr(0,3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(0,2), arr(3,0), axis=-1) ) confirm_raises( lambda: nps.glue( arr(0,2), arr(3,0), axis=-2) ) confirm_raises( lambda: nps.glue( arr(0,2), arr(3,0), axis=-3) ) confirm_raises( lambda: nps.glue( arr(2,0), arr(3,0), axis=-1) ) assertValueShape( None, (5,0), nps.glue, arr(2,0), arr(3,0), axis=-2 ) confirm_raises( lambda: nps.glue( arr(2,0), arr(3,0), axis=-3) ) assertValueShape( None, (0,), nps.glue, arr(0,), arr(0,), axis=-1 ) assertValueShape( None, (2,), nps.glue, arr(2,), arr(0,), axis=-1 ) assertValueShape( None, (2,), nps.glue, arr(0,), arr(2,), axis=-1 ) assertValueShape( None, (1,2,), nps.glue, arr(2,), arr(0,), axis=-2 ) assertValueShape( None, (1,2,), nps.glue, arr(0,), arr(2,), axis=-2 ) # same as before, but np.array(()) instead of np.arange(0) assertValueShape( None, (0,), nps.glue, np.array(()), np.array(()), axis=-1 ) assertValueShape( None, (2,), nps.glue, arr(2,), np.array(()), axis=-1 ) assertValueShape( None, (2,), nps.glue, np.array(()),arr(2,), axis=-1 ) assertValueShape( None, (1,2,), nps.glue, arr(2,), np.array(()), axis=-2 ) assertValueShape( None, (1,2,), nps.glue, np.array(()), arr(2,), axis=-2 ) assertValueShape( None, (0,6), nps.glue, arr(0,3), arr(0,3), axis=-1 ) assertValueShape( None, (0,3), nps.glue, arr(0,3), arr(0,3), axis=-2 ) assertValueShape( None, (2,0,3), nps.glue, arr(0,3), arr(0,3), axis=-3 ) confirm_raises( lambda: nps.glue( arr(3,0), arr(0,3), axis=-1) ) confirm_raises( lambda: nps.glue( arr(3,0), arr(0,3), axis=-2) ) confirm_raises( lambda: nps.glue( arr(3,0), arr(0,3), axis=-3) ) confirm_raises( lambda: nps.glue( arr(0,3), arr(3,0), axis=-1) ) confirm_raises( lambda: nps.glue( arr(0,3), arr(3,0), axis=-2) ) confirm_raises( lambda: nps.glue( arr(0,3), arr(3,0), axis=-3) ) assertValueShape( None, (3,0), nps.glue, arr(3,0), arr(3,0), axis=-1 ) assertValueShape( None, (6,0), nps.glue, arr(3,0), arr(3,0), axis=-2 ) assertValueShape( None, (2,3,0), nps.glue, arr(3,0), arr(3,0), axis=-3 ) # legacy behavior allows one to omit the 'axis' kwarg nps.glue.legacy_version = '0.9' assertValueShape( None, (2,2,3), nps.glue, arr(2,3), arr(2,3) ) delattr(nps.glue, 'legacy_version') def test_dimension_manipulation(): r'''Checking the various functions that manipulate dimensions.''' assertValueShape( None, (24,), nps.clump, arr(2,3,4), n=5 ) assertValueShape( None, (24,), nps.clump, arr(2,3,4), n=4 ) assertValueShape( None, (24,), nps.clump, arr(2,3,4), n=3 ) assertValueShape( None, (6,4), nps.clump, arr(2,3,4), n=2 ) assertValueShape( None, (2,3,4), nps.clump, arr(2,3,4), n=1 ) assertValueShape( None, (2,3,4), nps.clump, arr(2,3,4), n=0 ) assertValueShape( None, (2,3,4), nps.clump, arr(2,3,4), n=1 ) assertValueShape( None, (2,12), nps.clump, arr(2,3,4), n=-2 ) assertValueShape( None, (24,), nps.clump, arr(2,3,4), n=-3 ) assertValueShape( None, (24,), nps.clump, arr(2,3,4), n=-4 ) assertValueShape( None, (24,), nps.clump, arr(2,3,4), n=-5 ) # legacy behavior: n>0 required, and always clumps the trailing dimensions nps.clump.legacy_version = '0.9' confirm_raises ( lambda: nps.clump( arr(2,3,4), n=-1) ) assertValueShape( None, (2,3,4), nps.clump, arr(2,3,4), n=0 ) assertValueShape( None, (2,3,4), nps.clump, arr(2,3,4), n=1 ) assertValueShape( None, (2,12), nps.clump, arr(2,3,4), n=2 ) assertValueShape( None, (24,), nps.clump, arr(2,3,4), n=3 ) assertValueShape( None, (24,), nps.clump, arr(2,3,4), n=4 ) delattr(nps.clump, 'legacy_version') assertValueShape( None, (2,3,4), nps.atleast_dims, arr(2,3,4), -1, 1 ) assertValueShape( None, (2,3,4), nps.atleast_dims, arr(2,3,4), -2, 1 ) assertValueShape( None, (2,3,4), nps.atleast_dims, arr(2,3,4), -3, 1 ) assertValueShape( None, (1,2,3,4), nps.atleast_dims, arr(2,3,4), -4, 1 ) assertValueShape( None, (2,3,4), nps.atleast_dims, arr(2,3,4), -2, 0 ) assertValueShape( None, (2,3,4), nps.atleast_dims, arr(2,3,4), -2, 1 ) assertValueShape( None, (2,3,4), nps.atleast_dims, arr(2,3,4), -2, 2 ) confirm_raises ( lambda: nps.atleast_dims( arr(2,3,4), -2, 3) ) assertValueShape( None, (3,), nps.atleast_dims, arr(3), 0 ) confirm_raises ( lambda: nps.atleast_dims( arr(3), 1) ) assertValueShape( None, (3,), nps.atleast_dims, arr(3), -1 ) assertValueShape( None, (1,3,), nps.atleast_dims, arr(3), -2 ) confirm_raises ( lambda: nps.atleast_dims( arr(), 0) ) confirm_raises ( lambda: nps.atleast_dims( arr(), 1) ) assertValueShape( None, (1,), nps.atleast_dims, arr(), -1 ) assertValueShape( None, (1,1), nps.atleast_dims, arr(), -2 ) l = (-4,1) confirm_raises ( lambda: nps.atleast_dims( arr(2,3,4), l) ) l = [-4,1] confirm_raises ( lambda: nps.atleast_dims( arr(2,3,4), l, -1) ) assertValueShape( None, (1,2,3,4), nps.atleast_dims, arr(2,3,4), l ) confirm_equal ( l, [-4, 2]) assertValueShape( None, (3,4,2), nps.mv, arr(2,3,4), -3, -1 ) assertValueShape( None, (3,2,4), nps.mv, arr(2,3,4), -3, 1 ) assertValueShape( None, (2,1,1,3,4), nps.mv, arr(2,3,4), -3, -5 ) assertValueShape( None, (2,1,1,3,4), nps.mv, arr(2,3,4), 0, -5 ) assertValueShape( None, (4,3,2), nps.xchg, arr(2,3,4), -3, -1 ) assertValueShape( None, (3,2,4), nps.xchg, arr(2,3,4), -3, 1 ) assertValueShape( None, (2,1,1,3,4), nps.xchg, arr(2,3,4), -3, -5 ) assertValueShape( None, (2,1,1,3,4), nps.xchg, arr(2,3,4), 0, -5 ) assertValueShape( None, (2,4,3), nps.transpose, arr(2,3,4) ) assertValueShape( None, (4,3), nps.transpose, arr(3,4) ) assertValueShape( None, (4,1), nps.transpose, arr(4) ) assertValueShape( None, (1,2,3,4), nps.dummy, arr(2,3,4), 0 ) assertValueShape( None, (2,1,3,4), nps.dummy, arr(2,3,4), 1 ) assertValueShape( None, (2,3,4,1), nps.dummy, arr(2,3,4), -1 ) assertValueShape( None, (2,3,1,4), nps.dummy, arr(2,3,4), -2 ) assertValueShape( None, (2,1,3,4), nps.dummy, arr(2,3,4), -3 ) assertValueShape( None, (1,2,3,4), nps.dummy, arr(2,3,4), -4 ) assertValueShape( None, (1,1,2,3,4), nps.dummy, arr(2,3,4), -5 ) assertValueShape( None, (2,3,1,4), nps.dummy, arr(2,3,4), 2 ) confirm_raises ( lambda: nps.dummy( arr(2,3,4), 3) ) assertValueShape( None, (2,4,3), nps.reorder, arr(2,3,4), 0, -1, 1 ) assertValueShape( None, (3,4,2), nps.reorder, arr(2,3,4), -2, -1, 0 ) assertValueShape( None, (1,3,1,4,2), nps.reorder, arr(2,3,4), -4, -2, -5, -1, 0 ) confirm_raises ( lambda: nps.reorder( arr(2,3,4), -4, -2, -5, -1, 0, 5), msg='reorder barfs on out-of-bounds dimensions' ) def test_inner(): r'''Testing the broadcasted inner product''' assertResult_inoutplace( np.array([[[ 30, 255, 730], [ 180, 780, 1630]], [[ 180, 780, 1630], [1455, 2430, 3655]], [[ 330, 1305, 2530], [2730, 4080, 5680]], [[ 480, 1830, 3430], [4005, 5730, 7705.0]]]), nps.inner, arr(2,3,5), arr(4,1,3,5), out_inplace_dtype=float ) assertResult_inoutplace( np.array([[[ 30, 255, 730], [ 180, 780, 1630]], [[ 180, 780, 1630], [1455, 2430, 3655]], [[ 330, 1305, 2530], [2730, 4080, 5680]], [[ 480, 1830, 3430], [4005, 5730, 7705.0]]]), nps.inner, arr(2,3,5), arr(4,1,3,5), dtype=float, out_inplace_dtype=float ) output = np.empty((4,2,3), dtype=float) confirm_raises( lambda: nps.inner( arr(2,3,5), arr(4,1,3,5), dtype=int, out=output ), "inner(out=out, dtype=dtype) have out=dtype==dtype" ) assertResult_inoutplace( np.array((24+148j)), nps.dot, np.array(( 1 + 2j, 3 + 4j, 5 + 6j)), np.array(( 1 + 2j, 3 + 4j, 5 + 6j)) + 5, out_inplace_dtype=np.complex) assertResult_inoutplace( np.array((136-60j)), nps.vdot, np.array(( 1 + 2j, 3 + 4j, 5 + 6j)), np.array(( 1 + 2j, 3 + 4j, 5 + 6j)) + 5, out_inplace_dtype=np.complex) # complex values AND non-trivial dimensions a = arr( 2,3,5).astype(np.complex) b = arr(4,1,3,5).astype(np.complex) a += a*a * 1j b -= b * 1j dot_ref = np.array([[[ 130.0 +70.0j, 2180.0 +1670.0j, 9730.0 +8270.0j], [ 3430.0 +3070.0j, 18230.0 +16670.0j, 46030.0 +42770.0j]], [[ 730.0 +370.0j, 6530.0 +4970.0j, 21580.0 +18320.0j], [ 26530.0 +23620.0j, 56330.0 +51470.0j, 102880.0 +95570.0j]], [[ 1330.0 +670.0j, 10880.0 +8270.0j, 33430.0 +28370.0j], [ 49630.0 +44170.0j, 94430.0 +86270.0j, 159730.0 +148370.0j]], [[ 1930.0 +970.0j, 15230.0 +11570.0j, 45280.0 +38420.0j], [ 72730.0 +64720.0j, 132530.0 +121070.0j, 216580.0 +201170.0j]]]) vdot_ref = np.array([[[ -70.0 -130.0j, -1670.0 -2180.0j, -8270.0 -9730.0j], [ -3070.0 -3430.0j, -16670.0 -18230.0j, -42770.0 -46030.0j]], [[ -370.0 -730.0j, -4970.0 -6530.0j, -18320.0 -21580.0j], [ -23620.0 -26530.0j, -51470.0 -56330.0j, -95570.0 -102880.0j]], [[ -670.0 -1330.0j, -8270.0 -10880.0j, -28370.0 -33430.0j], [ -44170.0 -49630.0j, -86270.0 -94430.0j, -148370.0 -159730.0j]], [[ -970.0 -1930.0j, -11570.0 -15230.0j, -38420.0 -45280.0j], [ -64720.0 -72730.0j, -121070.0 -132530.0j, -201170.0 -216580.0j]]]) assertResult_inoutplace( dot_ref, nps.dot, a, b, out_inplace_dtype=np.complex) assertResult_inoutplace( vdot_ref, nps.vdot, a, b, out_inplace_dtype=np.complex) def test_mag(): r'''Testing the broadcasted magnitude product''' # input is a 1D array of integers, no output dtype specified. Output # should be a floating-point scalar assertResult_inoutplace( np.sqrt(nps.norm2(np.arange(5))), nps.mag, arr(5, dtype=int) ) # input is a 1D array of floats, no output dtype specified. Output # should be a floating-point scalar assertResult_inoutplace( np.sqrt(nps.norm2(np.arange(5))), nps.mag, arr(5, dtype=float) ) # input is a 1D array of integers, output dtype=int. Output should be an # integer scalar output = np.empty((), dtype=int) nps.mag( np.arange(5, dtype=int), out = output ) confirm_equal(int(np.sqrt(nps.norm2(np.arange(5)))), output) # input is a 1D array of integers, output dtype=float. Output should be an # float scalar output = np.empty((), dtype=float) nps.mag( np.arange(5, dtype=int), out = output ) confirm_equal(np.sqrt(nps.norm2(np.arange(5))), output) # input is a 2D array of integers, no output dtype specified. Output # should be a floating-point 1D vector assertResult_inoutplace( np.sqrt(np.array(( nps.norm2(np.arange(5)), nps.norm2(np.arange(5,10))))), nps.mag, arr(2,5, dtype=int) ) # input is a 2D array of floats, no output dtype specified. Output # should be a floating-point 1D vector assertResult_inoutplace( np.sqrt(np.array(( nps.norm2(np.arange(5)), nps.norm2(np.arange(5,10))))), nps.mag, arr(2,5, dtype=float) ) # input is a 2D array of integers, output dtype=int. Output should be an # array of integers output = np.empty((2,), dtype=int) nps.mag( arr(2,5, dtype=int), out = output ) confirm_equal(np.sqrt(np.array(( nps.norm2(np.arange(5)), nps.norm2(np.arange(5,10))))).astype(int), output) # input is a 2D array of integers, output dtype=float. Output should be an # array of floats output = np.empty((2,), dtype=float) nps.mag( arr(2,5, dtype=int), out = output ) confirm_equal(np.sqrt(np.array(( nps.norm2(np.arange(5)), nps.norm2(np.arange(5,10))))), output) def test_outer(): r'''Testing the broadcasted outer product''' # comes from PDL. numpy has a reversed axis ordering convention from # PDL, so I transpose the array before comparing ref = nps.transpose( np.array([[[[[0,0,0,0,0],[0,1,2,3,4],[0,2,4,6,8],[0,3,6,9,12],[0,4,8,12,16]], [[25,30,35,40,45],[30,36,42,48,54],[35,42,49,56,63],[40,48,56,64,72],[45,54,63,72,81]], [[100,110,120,130,140],[110,121,132,143,154],[120,132,144,156,168],[130,143,156,169,182],[140,154,168,182,196]]], [[[0,0,0,0,0],[15,16,17,18,19],[30,32,34,36,38],[45,48,51,54,57],[60,64,68,72,76]], [[100,105,110,115,120],[120,126,132,138,144],[140,147,154,161,168],[160,168,176,184,192],[180,189,198,207,216]], [[250,260,270,280,290],[275,286,297,308,319],[300,312,324,336,348],[325,338,351,364,377],[350,364,378,392,406]]]], [[[[0,15,30,45,60],[0,16,32,48,64],[0,17,34,51,68],[0,18,36,54,72],[0,19,38,57,76]], [[100,120,140,160,180],[105,126,147,168,189],[110,132,154,176,198],[115,138,161,184,207],[120,144,168,192,216]], [[250,275,300,325,350],[260,286,312,338,364],[270,297,324,351,378],[280,308,336,364,392],[290,319,348,377,406]]], [[[225,240,255,270,285],[240,256,272,288,304],[255,272,289,306,323],[270,288,306,324,342],[285,304,323,342,361]], [[400,420,440,460,480],[420,441,462,483,504],[440,462,484,506,528],[460,483,506,529,552],[480,504,528,552,576]], [[625,650,675,700,725],[650,676,702,728,754],[675,702,729,756,783],[700,728,756,784,812],[725,754,783,812,841]]]], [[[[0,30,60,90,120],[0,31,62,93,124],[0,32,64,96,128],[0,33,66,99,132],[0,34,68,102,136]], [[175,210,245,280,315],[180,216,252,288,324],[185,222,259,296,333],[190,228,266,304,342],[195,234,273,312,351]], [[400,440,480,520,560],[410,451,492,533,574],[420,462,504,546,588],[430,473,516,559,602],[440,484,528,572,616]]], [[[450,480,510,540,570],[465,496,527,558,589],[480,512,544,576,608],[495,528,561,594,627],[510,544,578,612,646]], [[700,735,770,805,840],[720,756,792,828,864],[740,777,814,851,888],[760,798,836,874,912],[780,819,858,897,936]], [[1000,1040,1080,1120,1160],[1025,1066,1107,1148,1189],[1050,1092,1134,1176,1218],[1075,1118,1161,1204,1247],[1100,1144,1188,1232,1276]]]], [[[[0,45,90,135,180],[0,46,92,138,184],[0,47,94,141,188],[0,48,96,144,192],[0,49,98,147,196]], [[250,300,350,400,450],[255,306,357,408,459],[260,312,364,416,468],[265,318,371,424,477],[270,324,378,432,486]], [[550,605,660,715,770],[560,616,672,728,784],[570,627,684,741,798],[580,638,696,754,812],[590,649,708,767,826]]], [[[675,720,765,810,855],[690,736,782,828,874],[705,752,799,846,893],[720,768,816,864,912],[735,784,833,882,931]], [[1000,1050,1100,1150,1200],[1020,1071,1122,1173,1224],[1040,1092,1144,1196,1248],[1060,1113,1166,1219,1272],[1080,1134,1188,1242,1296]], [[1375,1430,1485,1540,1595],[1400,1456,1512,1568,1624],[1425,1482,1539,1596,1653],[1450,1508,1566,1624,1682],[1475,1534,1593,1652,1711]]]]])) assertResult_inoutplace( ref, nps.outer, arr(2,3,5), arr(4,1,3,5), out_inplace_dtype=float ) def test_matmult(): r'''Testing the broadcasted matrix multiplication''' assertValueShape( None, (4,2,3,5), nps.matmult, arr(2,3,7), arr(4,1,7,5) ) ref = np.array([[[[ 42, 48, 54], [ 114, 136, 158]], [[ 114, 120, 126], [ 378, 400, 422]]], [[[ 186, 224, 262], [ 258, 312, 366]], [[ 642, 680, 718], [ 906, 960, 1014]]]]) assertResult_inoutplace( ref, nps.matmult2, arr(2,1,2,4), arr(2,4,3), out_inplace_dtype=float ) ref2 = np.array([[[[ 156.], [ 452.]], [[ 372.], [ 1244.]]], [[[ 748.], [ 1044.]], [[ 2116.], [ 2988.]]]]) assertResult_inoutplace(ref2, nps.matmult2, arr(2,1,2,4), nps.matmult2(arr(2,4,3), arr(3,1))) # not doing assertResult_inoutplace() because matmult() doesn't take an # 'out' kwarg confirm_equal(ref2, nps.matmult(arr(2,1,2,4), arr(2,4,3), arr(3,1))) # checking the null-dimensionality logic A = arr(2,3) assertResult_inoutplace( nps.inner(nps.transpose(A), np.arange(2)), nps.matmult2, np.arange(2), A ) A = arr(3) assertResult_inoutplace( A*2, nps.matmult2, np.array([2]), A ) A = arr(3) assertResult_inoutplace( A*2, nps.matmult2, np.array(2), A ) test_broadcasting() test_broadcasting_into_output() test_concatenation() test_dimension_manipulation() test_inner() test_mag() test_outer() test_matmult() finish() numpysane-0.21/testutils.py000066400000000000000000000163401357065502000160740ustar00rootroot00000000000000r'''A simple test harness These should be trivial, but all the standard ones in python suck. This one sucks far less. ''' import sys import numpy as np import os import re from inspect import currentframe Nchecks = 0 NchecksFailed = 0 # no line breaks. Useful for test reporting. Yes, this sets global state, but # we're running a test harness. This is fine np.set_printoptions(linewidth=1e10, suppress=True) def test_location(): r'''Reports string describing current location in the test Skips over the backtrace entries that are in the test harness itself ''' filename_this = os.path.split( __file__ )[1] if filename_this.endswith(".pyc"): filename_this = filename_this[:-1] frame = currentframe().f_back.f_back while frame: if frame.f_back is None or \ not frame.f_code.co_filename.endswith(filename_this): break frame = frame.f_back testfile = os.path.split(frame.f_code.co_filename)[1] try: return "{}:{} {}()".format(testfile, frame.f_lineno, frame.f_code.co_name) except: return '' def print_red(x): """print the message in red""" sys.stdout.write("\x1b[31m" + test_location() + ": " + x + "\x1b[0m\n") def print_green(x): """Print the message in green""" sys.stdout.write("\x1b[32m" + test_location() + ": " + x + "\x1b[0m\n") def confirm_equal(x, xref, msg='', eps=1e-6): r'''If x is equal to xref, report test success. msg identifies this check. eps sets the RMS equality tolerance. The x,xref arguments can be given as many different types. This function tries to do the right thing. ''' global Nchecks global NchecksFailed Nchecks = Nchecks + 1 # strip all trailing whitespace in each line, in case these are strings if isinstance(x, str): x = re.sub('[ \t]+(\n|$)', '\\1', x) if isinstance(xref, str): xref = re.sub('[ \t]+(\n|$)', '\\1', xref) # convert data to numpy if possible try: xref = np.array(xref) except: pass try: x = np.array(x) except: pass try: # flatten array if possible x = x.ravel() xref = xref.ravel() except: pass try: N = x.shape[0] except: N = 1 try: Nref = xref.shape[0] except: Nref = 1 if N != Nref: print_red(("FAILED{}: mismatched array sizes: N = {} but Nref = {}. Arrays: \n" + "x = {}\n" + "xref = {}"). format((': ' + msg) if msg else '', N, Nref, x, xref)) NchecksFailed = NchecksFailed + 1 return False if N != 0: try: # I I can subtract, get the error that way diff = x - xref def norm2sq(x): """Return 2 norm""" return np.inner(x, x) rms = np.sqrt(norm2sq(diff) / N) if not np.all(np.isfinite(rms)): print_red("FAILED{}: Some comparison results are NaN or Inf. " "rms error = {}. x = {}, xref = {}".format( (': ' + msg) if msg else '', rms, x, xref)) NchecksFailed = NchecksFailed + 1 return False if rms > eps: print_red("FAILED{}: rms error = {}.\nx,xref,err =\n{}".format( (': ' + msg) if msg else '', rms, np.vstack((x, xref, diff)).transpose())) NchecksFailed = NchecksFailed + 1 return False except: # Can't subtract. Do == instead if not np.array_equal(x, xref): print_red("FAILED{}: x =\n'{}', xref =\n'{}'".format( (': ' + msg) if msg else '', x, xref)) NchecksFailed = NchecksFailed + 1 return False print_green("OK{}".format((': ' + msg) if msg else '')) return True def confirm(x, msg=''): r'''If x is true, report test success. msg identifies this check''' global Nchecks global NchecksFailed Nchecks = Nchecks + 1 if not x: print_red("FAILED{}".format((': ' + msg) if msg else '')) NchecksFailed = NchecksFailed + 1 return False print_green("OK{}".format((': ' + msg) if msg else '')) return True def confirm_is(x, xref, msg=''): r'''If x is xref, report test success. msg identifies this check ''' global Nchecks global NchecksFailed Nchecks = Nchecks + 1 if x is xref: print_green("OK{}".format((': ' + msg) if msg else '')) return True print_red("FAILED{}".format((': ' + msg) if msg else '')) NchecksFailed = NchecksFailed + 1 return False def confirm_raises(f, msg=''): r'''If f() raises an exception, report test success. msg identifies this check''' global Nchecks global NchecksFailed Nchecks = Nchecks + 1 try: f() print_red("FAILED{}".format((': ' + msg) if msg else '')) NchecksFailed = NchecksFailed + 1 return False except: print_green("OK{}".format((': ' + msg) if msg else '')) return True def finish(): r'''Finalize the executed tests. Prints the test summary. Exits successfully iff all the tests passed. ''' if not Nchecks and not NchecksFailed: print_red("No tests defined") sys.exit(0) if NchecksFailed: print_red("Some tests failed: {} out of {}".format(NchecksFailed, Nchecks)) sys.exit(1) print_green("All tests passed: {} total".format(Nchecks)) sys.exit(0) # numpysane-specific tests. Keep these in this file to make sure test-harness # line numbers are not reported def assertValueShape(value_ref, s, f, *args, **kwargs): r'''Makes sure a given call produces a given value and shape. It is redundant to specify both, but it makes it clear I'm asking for what I think I'm asking. The value check can be skipped by passing None. ''' try: res = f(*args, **kwargs) except Exception as e: print_red("FAILED: Exception \"{}\" calling \"{}\"".format(e,f)) return if 'out' in kwargs: confirm(res is kwargs['out'], msg='returning same matrix as the given "out"') if s is not None: try: shape = res.shape except: shape = () confirm_equal(shape, s, msg='shape matches') if value_ref is not None: confirm_equal(value_ref, res, msg='value matches') if 'dtype' in kwargs: confirm_equal(res.dtype, kwargs['dtype'], msg='matching dtype') def assertResult_inoutplace( ref, func, *args, **kwargs ): r'''makes sure func(a,b) == ref. Tests both a pre-allocated array and a slice-at-a-time allocate/copy mode Only one test-specific kwarg is known: 'out_inplace_dtype'. The rest are passed down to the test function ''' out_inplace_dtype = None if 'out_inplace_dtype' in kwargs: out_inplace_dtype = kwargs['out_inplace_dtype'] del kwargs['out_inplace_dtype'] assertValueShape( ref, ref.shape, func, *args, **kwargs ) output = np.empty(ref.shape, dtype=out_inplace_dtype) assertValueShape( ref, ref.shape, func, *args, out=output, **kwargs) confirm_equal(ref, output)