dap-2.2.6.7/0000755000175000017500000000000011123574304012344 5ustar robertorobertodap-2.2.6.7/dap.egg-info/0000755000175000017500000000000011123574304014602 5ustar robertorobertodap-2.2.6.7/dap.egg-info/requires.txt0000644000175000017500000000012211123574301017172 0ustar robertorobertohttplib2 [proxy] Paste WSGIFilter [server] Paste PasteScript PasteDeploy Cheetahdap-2.2.6.7/dap.egg-info/dependency_links.txt0000644000175000017500000000000111123574301020645 0ustar robertoroberto dap-2.2.6.7/dap.egg-info/not-zip-safe0000644000175000017500000000000111006072350017022 0ustar robertoroberto dap-2.2.6.7/dap.egg-info/SOURCES.txt0000644000175000017500000000274211123574301016470 0ustar robertorobertoLICENSE MANIFEST.in README TODO ez_setup.py setup.py dap/__init__.py dap/client.py dap/dtypes.py dap/exceptions.py dap/helper.py dap/lib.py dap/proxy.py dap/server.py dap/xdr.py dap.egg-info/PKG-INFO dap.egg-info/SOURCES.txt dap.egg-info/dependency_links.txt dap.egg-info/entry_points.txt dap.egg-info/namespace_packages.txt dap.egg-info/not-zip-safe dap.egg-info/requires.txt dap.egg-info/top_level.txt dap/parsers/__init__.py dap/parsers/das.py dap/parsers/dds.py dap/plugins/__init__.py dap/plugins/csvfiles.py dap/plugins/lib.py dap/plugins/templates.py dap/plugins/paster_templates/setup.cfg dap/plugins/paster_templates/setup.py_tmpl dap/plugins/paster_templates/dap/__init__.py dap/plugins/paster_templates/dap/plugins/__init__.py dap/plugins/paster_templates/dap/plugins/+package+/__init__.py dap/responses/__init__.py dap/responses/ascii.py dap/responses/das.py dap/responses/dds.py dap/responses/dods.py dap/responses/help.py dap/responses/version.py dap/util/__init__.py dap/util/filter.py dap/util/http.py dap/util/ordereddict.py dap/util/safeeval.py dap/util/socks.py dap/util/wsgi_intercept.py dap/wsgi/__init__.py dap/wsgi/application.py dap/wsgi/proxy.py dap/wsgi/templates.py dap/wsgi/paster_templates/server.ini_tmpl dap/wsgi/paster_templates/+package+.egg-info/not-zip-safe dap/wsgi/paster_templates/data/sample.csv dap/wsgi/paster_templates/template/index.tmpl docs/Changelog docs/bugs docs/copying docs/history tests/test0.py tests/test1.py tests/test2.py tests/test3.py tests/test4.pydap-2.2.6.7/dap.egg-info/PKG-INFO0000644000175000017500000000455411123574301015704 0ustar robertorobertoMetadata-Version: 1.0 Name: dap Version: 2.2.6.7 Summary: DAP (Data Access Protocol) client and server for Python. Home-page: http://pydap.org/ Author: Roberto De Almeida Author-email: rob@pydap.org License: MIT Description: Implementation of the `Data Access Protocol `_. This is a Python implementation of the Data Access Protocol, a scientific protocol for data access developed by the OPeNDAP team (http://opendap.org). This implementation is developed from scratch, following the latest specification of the protocol (DAP 2.0 Draft Community Standard 2005/04/27) and based on my experience with OPeNDAP servers on the wild. Using this module one can access hundreds of scientific datasets from Python programs, accessing data in an efficient, transparent and pythonic way. Arrays are manipulated like normal multi-dimensional arrays (like numpy.array, e.g.), with the fundamental difference that data is downloaded on-the-fly when a variable is sliced. Sequential data can be filtered on the server side before being downloaded, saving bandwith and time. The module also implements a DAP server, allowing datasets from a multitude of formats (netCDF, Matlab, CSV, GrADS/GRIB files, SQL RDBMS) to be served on the internet. The server specifies a plugin API for supporting new data formats in an easy way. The DAP server is implemented as a WSGI application (see PEP 333), running on a variery of servers, and can be combined with WSGI middleware to support authentication, gzip compression and much more. The latest version is available in a `Subversion repository `_. Keywords: dap opendap dods data science climate meteorology oceanography Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: MIT License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Topic :: Internet Classifier: Topic :: Scientific/Engineering Classifier: Topic :: Software Development :: Libraries :: Python Modules dap-2.2.6.7/dap.egg-info/namespace_packages.txt0000644000175000017500000000003211123574301021125 0ustar robertorobertodap.plugins dap.responses dap-2.2.6.7/dap.egg-info/top_level.txt0000644000175000017500000000000411123574301017323 0ustar robertorobertodap dap-2.2.6.7/dap.egg-info/entry_points.txt0000644000175000017500000000117411123574301020100 0ustar robertoroberto # -*- Entry points: -*- [dap.response] dds = dap.responses.dds das = dap.responses.das dods = dap.responses.dods asc = dap.responses.ascii ascii = dap.responses.ascii ver = dap.responses.version version = dap.responses.version help = dap.responses.help [dap.plugin] csv = dap.plugins.csvfiles [paste.app_factory] main = dap.wsgi.application:make_app proxy = dap.wsgi.proxy:make_proxy [paste.paster_create_template] dap_server = dap.wsgi.templates:DapServerTemplate dap_plugin = dap.plugins.templates:DapPluginTemplate dap-2.2.6.7/LICENSE0000644000175000017500000000207310774147132013361 0ustar robertorobertoCopyright (c) 2003-2006 Roberto De Almeida Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. dap-2.2.6.7/ez_setup.py0000644000175000017500000001777510774147132014603 0ustar robertoroberto#!python """Bootstrap setuptools installation If you want to use setuptools in your package's setup.py, just include this file in the same directory with it, and add this to the top of your setup.py:: from ez_setup import use_setuptools use_setuptools() If you want to require a specific version of setuptools, set a download mirror, or use an alternate download directory, you can do so by supplying the appropriate options to ``use_setuptools()``. This file can also be run as a script to install or upgrade setuptools. """ import sys DEFAULT_VERSION = "0.6a8" DEFAULT_URL = "http://cheeseshop.python.org/packages/%s/s/setuptools/" % sys.version[:3] md5_data = { 'setuptools-0.5a13-py2.3.egg': '85edcf0ef39bab66e130d3f38f578c86', 'setuptools-0.5a13-py2.4.egg': 'ede4be600e3890e06d4ee5e0148e092a', 'setuptools-0.6a1-py2.3.egg': 'ee819a13b924d9696b0d6ca6d1c5833d', 'setuptools-0.6a1-py2.4.egg': '8256b5f1cd9e348ea6877b5ddd56257d', 'setuptools-0.6a2-py2.3.egg': 'b98da449da411267c37a738f0ab625ba', 'setuptools-0.6a2-py2.4.egg': 'be5b88bc30aed63fdefd2683be135c3b', 'setuptools-0.6a3-py2.3.egg': 'ee0e325de78f23aab79d33106dc2a8c8', 'setuptools-0.6a3-py2.4.egg': 'd95453d525a456d6c23e7a5eea89a063', 'setuptools-0.6a4-py2.3.egg': 'e958cbed4623bbf47dd1f268b99d7784', 'setuptools-0.6a4-py2.4.egg': '7f33c3ac2ef1296f0ab4fac1de4767d8', 'setuptools-0.6a5-py2.3.egg': '748408389c49bcd2d84f6ae0b01695b1', 'setuptools-0.6a5-py2.4.egg': '999bacde623f4284bfb3ea77941d2627', 'setuptools-0.6a6-py2.3.egg': '7858139f06ed0600b0d9383f36aca24c', 'setuptools-0.6a6-py2.4.egg': 'c10d20d29acebce0dc76219dc578d058', 'setuptools-0.6a7-py2.3.egg': 'cfc4125ddb95c07f9500adc5d6abef6f', 'setuptools-0.6a7-py2.4.egg': 'c6d62dab4461f71aed943caea89e6f20', 'setuptools-0.6a8-py2.3.egg': '2f18eaaa3f544f5543ead4a68f3b2e1a', 'setuptools-0.6a8-py2.4.egg': '799018f2894f14c9f8bcb2b34e69b391', } import sys, os def _validate_md5(egg_name, data): if egg_name in md5_data: from md5 import md5 digest = md5(data).hexdigest() if digest != md5_data[egg_name]: print >>sys.stderr, ( "md5 validation of %s failed! (Possible download problem?)" % egg_name ) sys.exit(2) return data def use_setuptools( version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, download_delay=15 ): """Automatically find/download setuptools and make it available on sys.path `version` should be a valid setuptools version number that is available as an egg for download under the `download_base` URL (which should end with a '/'). `to_dir` is the directory where setuptools will be downloaded, if it is not already available. If `download_delay` is specified, it should be the number of seconds that will be paused before initiating a download, should one be required. If an older version of setuptools is installed, this routine will print a message to ``sys.stderr`` and raise SystemExit in an attempt to abort the calling script. """ try: import setuptools if setuptools.__version__ == '0.0.1': print >>sys.stderr, ( "You have an obsolete version of setuptools installed. Please\n" "remove it from your system entirely before rerunning this script." ) sys.exit(2) except ImportError: egg = download_setuptools(version, download_base, to_dir, download_delay) sys.path.insert(0, egg) import setuptools; setuptools.bootstrap_install_from = egg import pkg_resources try: pkg_resources.require("setuptools>="+version) except pkg_resources.VersionConflict: # XXX could we install in a subprocess here? print >>sys.stderr, ( "The required version of setuptools (>=%s) is not available, and\n" "can't be installed while this script is running. Please install\n" " a more recent version first." ) % version sys.exit(2) def download_setuptools( version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, delay = 15 ): """Download setuptools from a specified location and return its filename `version` should be a valid setuptools version number that is available as an egg for download under the `download_base` URL (which should end with a '/'). `to_dir` is the directory where the egg will be downloaded. `delay` is the number of seconds to pause before an actual download attempt. """ import urllib2, shutil egg_name = "setuptools-%s-py%s.egg" % (version,sys.version[:3]) url = download_base + egg_name saveto = os.path.join(to_dir, egg_name) src = dst = None if not os.path.exists(saveto): # Avoid repeated downloads try: from distutils import log if delay: log.warn(""" --------------------------------------------------------------------------- This script requires setuptools version %s to run (even to display help). I will attempt to download it for you (from %s), but you may need to enable firewall access for this script first. I will start the download in %d seconds. ---------------------------------------------------------------------------""", version, download_base, delay ); from time import sleep; sleep(delay) log.warn("Downloading %s", url) src = urllib2.urlopen(url) # Read/write all in one block, so we don't create a corrupt file # if the download is interrupted. data = _validate_md5(egg_name, src.read()) dst = open(saveto,"wb"); dst.write(data) finally: if src: src.close() if dst: dst.close() return os.path.realpath(saveto) def main(argv, version=DEFAULT_VERSION): """Install or upgrade setuptools and EasyInstall""" try: import setuptools except ImportError: import tempfile, shutil tmpdir = tempfile.mkdtemp(prefix="easy_install-") try: egg = download_setuptools(version, to_dir=tmpdir, delay=0) sys.path.insert(0,egg) from setuptools.command.easy_install import main main(list(argv)+[egg]) finally: shutil.rmtree(tmpdir) else: if setuptools.__version__ == '0.0.1': # tell the user to uninstall obsolete version use_setuptools(version) req = "setuptools>="+version import pkg_resources try: pkg_resources.require(req) except pkg_resources.VersionConflict: try: from setuptools.command.easy_install import main except ImportError: from easy_install import main main(list(argv)+[download_setuptools(delay=0)]) sys.exit(0) # try to force an exit else: if argv: from setuptools.command.easy_install import main main(argv) else: print "Setuptools version",version,"or greater has been installed." print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)' def update_md5(filenames): """Update our built-in md5 registry""" import re from md5 import md5 for name in filenames: base = os.path.basename(name) f = open(name,'rb') md5_data[base] = md5(f.read()).hexdigest() f.close() data = [" %r: %r,\n" % it for it in md5_data.items()] data.sort() repl = "".join(data) import inspect srcfile = inspect.getsourcefile(sys.modules[__name__]) f = open(srcfile, 'rb'); src = f.read(); f.close() match = re.search("\nmd5_data = {\n([^}]+)}", src) if not match: print >>sys.stderr, "Internal error!" sys.exit(2) src = src[:match.start(1)] + repl + src[match.end(1):] f = open(srcfile,'w') f.write(src) f.close() if __name__=='__main__': if len(sys.argv)>2 and sys.argv[1]=='--md5update': update_md5(sys.argv[2:]) else: main(sys.argv[1:]) dap-2.2.6.7/dap/0000755000175000017500000000000011123574304013110 5ustar robertorobertodap-2.2.6.7/dap/responses/0000755000175000017500000000000011123574304015131 5ustar robertorobertodap-2.2.6.7/dap/responses/dods.py0000644000175000017500000000407710774147126016455 0ustar robertoroberto"""DODS DAP response. This module implements the DODS DAP response, building it dynamically from datasets objects. """ __author__ = "Roberto De Almeida " import itertools from dap import dtypes from dap.lib import __dap__, isiterable from dap.xdr import DapPacker def build(self, constraints=None): dataset = self._parseconstraints(constraints) headers = [('Content-description', 'dods_data'), ('XDODS-Server', 'dods/%s' % '.'.join([str(i) for i in __dap__])), ('Content-type', 'application/octet-stream'), ] def output(dataset): foo, dds = self.dds(constraints) for line in dds: yield line yield 'Data:\n' for line in _dispatch(dataset): yield line return headers, output(dataset) def _dispatch(dapvar, data=None): r"""Build a DODS from a DAP object. >>> dataset = dtypes.DatasetType(name='temp.dat') >>> dataset['Tmp'] = dtypes.ArrayType(name='Tmp', shape=[5], type='Int32', data=range(5)) >>> for line in _dispatch(dataset): ... print repr(line) '\x00\x00\x00\x05\x00\x00\x00\x05' '\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04' """ func = {dtypes.DatasetType : _dataset, dtypes.StructureType: _structure, dtypes.SequenceType : _sequence, dtypes.BaseType : _base, dtypes.ArrayType : _array, dtypes.GridType : _grid, }[type(dapvar)] return func(dapvar, data) def _dataset(dapvar, data=None): for var in dapvar.walk(): for line in _dispatch(var): yield line _structure = _dataset _grid = _dataset def _sequence(dapvar, data=None): for struct_ in dapvar: yield '\x5a\x00\x00\x00' for line in _dispatch(struct_): yield line yield '\xa5\x00\x00\x00' def _base(dapvar, data=None): return DapPacker(dapvar) _array = _base def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/responses/version.py0000644000175000017500000000067210774147126017206 0ustar robertoroberto"""Version response.""" from dap.lib import __dap__, __version__ def build(self, constraints=None): headers = [('XDODS-Server', 'dods/%s' % '.'.join([str(i) for i in __dap__])), ('Content-type', 'text/plain'), ] output = ['Core version: dods/%s\n' % '.'.join([str(i) for i in __dap__]), 'Server version: pydap/%s\n' % '.'.join([str(i) for i in __version__])] return headers, output dap-2.2.6.7/dap/responses/das.py0000644000175000017500000000733710774147126016275 0ustar robertoroberto"""DAS DAP response. This module implements the DAS DAP response, building it dynamically from datasets objects. """ __author__ = "Roberto De Almeida " from dap.lib import INDENT, __dap__, encode_atom, to_list from dap.dtypes import * from dap.dtypes import _basetypes def build(self, constraints=None): dataset = self._parseconstraints(None) headers = [('Content-description', 'dods_das'), ('XDODS-Server', 'dods/%s' % '.'.join([str(i) for i in __dap__])), ('Content-type', 'text/plain'), ] output = _dispatch(dataset) return headers, output # Type conversion between Python and DODS, for the DAS. typeconvert = {basestring: 'String', unicode : 'String', str : 'String', float : 'Float64', long : 'Int32', int : 'Int32', } def _recursive_build(attr, values, level=0): """ Recursive function to build the DAS. This function checks for attribute nodes that do not belong to any variable, and append them as metadata. """ # Check for metadata. if isinstance(values, dict): yield '%s%s {\n' % ((level+1) * INDENT, attr) for k,v in values.items(): for line in _recursive_build(k, v, level+1): yield line yield '%s}\n' % ((level+1) * INDENT) else: # Convert values to list. values = to_list(values) # this takes care of numpy arrays if not isinstance(values, list): values = [values] # Get value type and encode properly. attrtype = typeconvert[type(values[0])] values = [encode_atom(v) for v in values] values = ', '.join(values) yield '%s%s %s %s;\n' % ((level+1) * INDENT, attrtype, attr.replace(' ', '_'), values) def _dispatch(dapvar, level=0): func = {DatasetType : _dataset, StructureType: _structure, SequenceType : _sequence, GridType : _grid, ArrayType : _array, BaseType : _base, }[type(dapvar)] return func(dapvar, level) def _dataset(dapvar, level=0): yield '%sAttributes {\n' % (level * INDENT) # Global attributes and metadata. for attr,values in dapvar.attributes.items(): for line in _recursive_build(attr, values, level): yield line # Get the DAS from stored variables. for var in dapvar.walk(): for line in _dispatch(var, level=level+1): yield line yield '%s}\n' % (level * INDENT) def _structure(dapvar, level=0): yield '%s%s {\n' % (level * INDENT, dapvar.name) # Global attributes and metadata. for attr,values in dapvar.attributes.items(): for line in _recursive_build(attr, values, level): yield line # Get the DAS from stored variables. for var in dapvar.walk(): for line in _dispatch(var, level=level+1): yield line yield '%s}\n' % (level * INDENT) _sequence = _structure #_grid = _structure def _grid(dapvar, level=0): yield '%s%s {\n' % (level * INDENT, dapvar.name) # Global attributes and metadata. for attr,values in dapvar.attributes.items(): for line in _recursive_build(attr, values, level): yield line yield '%s}\n' % (level * INDENT) def _array(dapvar, level=0): yield '%s%s {\n' % (level * INDENT, dapvar.name) # Global attributes and metadata. for attr,values in dapvar.attributes.items(): for line in _recursive_build(attr, values, level): yield line yield '%s}\n' % (level * INDENT) _base = _array def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/responses/__init__.py0000644000175000017500000000007010774147126017250 0ustar robertoroberto__import__('pkg_resources').declare_namespace(__name__) dap-2.2.6.7/dap/responses/ascii.py0000644000175000017500000000626410774147126016614 0ustar robertoroberto"""ASCII DAP response. This module implements the ASCII DAP response, building it dynamically from datasets objects. """ __author__ = "Roberto De Almeida " import itertools from dap.lib import INDENT, __dap__, encode_atom, isiterable from dap.dtypes import * def n_iterate(t): """Iterator over n-tuple. This function is used when representing data in the ASCII response. It iterates over a n-dimensional tuple/list, yielding all indexes in order. >>> for l in n_iterate([1,2,3]): ... print l [0, 0, 0] [0, 0, 1] [0, 0, 2] [0, 1, 0] [0, 1, 1] [0, 1, 2] """ if not t: raise StopIteration output = [0] * len(t) # Check if any length is zero. if 0 in t: raise StopIteration while 1: # Check digits from right to left. for i in range(len(t)-1,0,-1): if output[i] >= t[i]: # Carry numbers to the left. output[i] = 0 output[i-1] += 1 else: break if output[0] >= t[0]: raise StopIteration yield output # Add 1. output[-1] += 1 def build(self, constraints=None): dataset = self._parseconstraints('') dataset = self._parseconstraints(constraints) headers = [('Content-description', 'dods_ascii'), ('XDODS-Server', 'dods/%s' % '.'.join([str(i) for i in __dap__])), ('Content-type', 'text/plain'), ] def output(dataset): foo, dds = self.dds(constraints) for line in dds: yield line yield 45 * '-' yield '\n' for line in _dispatch(dataset): yield line return headers, output(dataset) def _dispatch(dapvar, printname=True): func = {DatasetType : _dataset, StructureType: _structure, SequenceType : _sequence, GridType : _grid, ArrayType : _array, BaseType : _base, }[type(dapvar)] return func(dapvar, printname) def _dataset(dapvar, printname): for var in dapvar: for line in _dispatch(var, printname): yield line yield '\n\n' _structure = _dataset _grid = _dataset def _sequence(dapvar, printname): yield ', '.join([var.id for var in dapvar.values()]) yield '\n' for struct_ in dapvar: out = [] for var in struct_: for line in _dispatch(var, printname=False): out.append(line) yield ', '.join(out) yield '\n' def _array(dapvar, printname): if printname: yield dapvar.id yield '\n' first = True data = getattr(dapvar.data, 'flat', dapvar.data) for indexes, value in itertools.izip(n_iterate(dapvar.shape), data): if first: first = False else: yield '\n' index = ']['.join([str(idx) for idx in indexes]) index = '[%s]' % index yield '%s %s' % (index, encode_atom(value)) def _base(dapvar, printname): if printname: yield dapvar.id yield '\n' yield encode_atom(dapvar.data) def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/responses/help.py0000644000175000017500000000122410774147126016443 0ustar robertorobertofrom paste.request import construct_url from dap.lib import __dap__ def build(self, constraints=None, message=''): """Help response.""" headers = [('XDODS-Server', 'dods/%s' % '.'.join([str(i) for i in __dap__])), ('Content-type', 'text/html'), ] if message: output = [message] else: location = construct_url(self.environ, with_query_string=False)[:-len('.help')] output = ["

To access this file, use the URL %s.

" % location] return headers, output dap-2.2.6.7/dap/responses/dds.py0000644000175000017500000000560010774147126016267 0ustar robertoroberto""" DDS DAP response. This module implements the DDS DAP response, building it dynamically from datasets objects. """ __author__ = "Roberto De Almeida " from dap.lib import INDENT, __dap__ from dap.dtypes import * from dap.dtypes import _basetypes def build(self, constraints=None): dataset = self._parseconstraints(constraints) headers = [('Content-description', 'dods_dds'), ('XDODS-Server', 'dods/%s' % '.'.join([str(i) for i in __dap__])), ('Content-type', 'text/plain'), ] output = _dispatch(dataset) return headers, output def _dispatch(dapvar, level=0): func = {DatasetType : _dataset, StructureType: _structure, SequenceType : _sequence, GridType : _grid, ArrayType : _array, BaseType : _base, }[type(dapvar)] return func(dapvar, level) def _dataset(dapvar, level=0): yield '%sDataset {\n' % (level * INDENT) # Get the DDS from stored variables. for var in dapvar.walk(): for line in _dispatch(var, level=level+1): yield line yield '%s} %s;\n' % (level * INDENT, dapvar.name) def _structure(dapvar, level=0): yield '%sStructure {\n' % (level * INDENT) # Get the DDS from stored variables. for var in dapvar.walk(): for line in _dispatch(var, level=level+1): yield line yield '%s} %s;\n' % (level * INDENT, dapvar.name) def _sequence(dapvar, level=0): yield '%sSequence {\n' % (level * INDENT) # Get the DDS from stored variables. for var in dapvar.walk(): for line in _dispatch(var, level=level+1): yield line yield '%s} %s;\n' % (level * INDENT, dapvar.name) def _grid(dapvar, level=0): yield '%sGrid {\n' % (level * INDENT) # Get the DDS from the array... yield '%sArray:\n' % ((level+1) * INDENT) for line in _dispatch(dapvar.array, level=level+2): yield line # ...and also from the maps. yield '%sMaps:\n' % ((level+1) * INDENT) for map_ in dapvar.maps.values(): for line in _dispatch(map_, level=level+2): yield line yield '%s} %s;\n' % (level * INDENT, dapvar.name) def _array(dapvar, level=0): # Get the var shape and dimensions, if any. if dapvar.dimensions: dims = ['%s = %d' % dim for dim in zip(dapvar.dimensions, dapvar.shape)] else: if len(dapvar.shape) == 1: dims = ['%s = %d' % (dapvar.name, dapvar.shape[0])] else: dims = ['%d' % i for i in dapvar.shape] shape = ']['.join(dims) shape = '[%s]' % shape yield '%s%s %s%s;\n' % (level * INDENT, dapvar.type, dapvar.name, shape) def _base(dapvar, level=0): yield '%s%s %s;\n' % (level * INDENT, dapvar.type, dapvar.name) def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/client.py0000644000175000017500000000407210774147131014750 0ustar robertoroberto__author__ = "Roberto De Almeida " import dap.lib from dap.util.http import openurl from dap.exceptions import ClientError def open(url, cache=None, username=None, password=None, verbose=False): """Connect to a remote dataset. This function opens a dataset stored in a DAP server: >>> dataset = open(url, cache=None, username=None, password=None, verbose=False): You can specify a cache location (a directory), so that repeated accesses to the same URL avoid the network. The username and password may be necessary if the DAP server requires authentication. The 'verbose' option will make pydap print all the URLs that are acessed. """ # Set variables on module namespace. dap.lib.VERBOSE = verbose if url.startswith('http'): for response in [_ddx, _ddsdas]: dataset = response(url, cache, username, password) if dataset: return dataset else: raise ClientError("Unable to open dataset.") else: from dap.plugins.lib import loadhandler from dap.helper import walk # Open a local file. This is a clever hack. :) handler = loadhandler(url) dataset = handler._parseconstraints() # Unwrap any arrayterators in the dataset. for var in walk(dataset): try: var.data = var.data._var except: pass return dataset def _ddsdas(baseurl, cache, username, password): ddsurl, dasurl = '%s.dds' % baseurl, '%s.das' % baseurl # Get metadata. respdds, dds = openurl(ddsurl, cache, username, password) respdas, das = openurl(dasurl, cache, username, password) if respdds['status'] == '200' and respdas['status'] == '200': from dap.parsers.dds import DDSParser from dap.parsers.das import DASParser # Build dataset. dataset = DDSParser(dds, ddsurl, cache, username, password).parse() # Add attributes. dataset = DASParser(das, dasurl, dataset).parse() return dataset def _ddx(baseurl, cache, username, password): pass dap-2.2.6.7/dap/lib.py0000644000175000017500000000574611123542065014243 0ustar robertorobertofrom __future__ import division """Basic functions concerning the DAP. These functions are mostly related to encoding data according to the DAP. """ from urllib import quote as _quote __author__ = 'Roberto De Almeida ' __version__ = (2,2,6,7) # module version __dap__ = (2,0) # protocol version # Constants that used to live in __init__.py but had to be moved # because we share the namespace with plugins and responses. USER_AGENT = 'pydap/%s' % '.'.join([str(_) for _ in __version__]) INDENT = ' ' * 4 VERBOSE = False CACHE = None TIMEOUT = None PROXY = None def isiterable(o): """Tests if an object is iterable. >>> print isiterable(range(10)) True >>> print isiterable({}) True >>> def a(): ... for i in range(10): yield i >>> print isiterable(a()) True >>> print isiterable('string') False >>> print isiterable(1) False """ # We DON'T want to iterate over strings. if isinstance(o, basestring): return False try: iter(o) return True except TypeError: return False def to_list(L): if hasattr(L, 'tolist'): return L.tolist() # shortcut for numpy arrays elif isiterable(L): return [to_list(item) for item in L] else: return L def quote(name): """Extended quote for the DAP spec. The period MUST be escaped in names (DAP spec, item 5.1): >>> quote("White space") 'White%20space' >>> _quote("Period.") 'Period.' >>> quote("Period.") 'Period%2E' """ return _quote(name).replace('.', '%2E') def encode_atom(atom): r"""Atomic types encoding. Encoding atomic types for the DAS. Integers should be printed using the base 10 ASCII representation of its value: >>> encode_atom(42) '42' Floating point values are printed using the base 10 ASCII representation of its value, conforming to ANSI C's description of printf using the %g format and precision 6. >>> encode_atom(1/3) '0.333333' String and URLs are printed in ASCII, escaped according to escape(). >>> encode_atom('This is a "string".') '"This is a \\"string\\"."' """ return {basestring: lambda s: escape(s), unicode : lambda s: escape(s), str : lambda s: escape(s), float : lambda f: '%.6g' % f, long : lambda f: '%.6g' % f, int : lambda i: repr(i), }.get(type(atom), lambda obj: '%s' % obj)(atom) def escape(s): r"""Escape a string. Enclose strings with double quotes, escape double quotes and backslashes: >>> escape('String') '"String"' >>> escape('This is a "test".') '"This is a \\"test\\"."' """ s = s.replace(r'\\', r'\\\\') s = s.replace(r'"', r'\"') s = '"%s"' % s return s def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/plugins/0000755000175000017500000000000011123574304014571 5ustar robertorobertodap-2.2.6.7/dap/plugins/lib.py0000644000175000017500000000307710774147124015727 0ustar robertoroberto"""Plugins for accessing data formats. This module contains plugins for accessing different data formats, together with functions to locate and load the proper handlers. """ import os import re from pkg_resources import iter_entry_points from dap.exceptions import ExtensionNotSupportedError def loadplugins(throw_errors=False): """Load all available plugins. This function returns a list of all available plugins as modules. """ plugins = [] for entrypoint in iter_entry_points("dap.plugin"): try: plugins.append(entrypoint.load()) except ImportError: if throw_errors: raise return plugins def loadhandler(file_, environ=None, plugins=None): """Load a handler for a given file. This function returns a Handler object able to process a given data file. Eg: >>> H = loadhandler('file.nc') # load a netCDF file >>> dataset = H._parsecontraints() This will load the full (unconstrained) dataset to ``dataset``. """ if environ is None: environ = os.environ.copy() if plugins is None: plugins = loadplugins(throw_errors=environ.get('x-wsgiorg.throw_errors', False)) # Check each plugin to see which one handles this file. for plugin in plugins: p = re.compile(plugin.extensions) m = p.match(file_) if m: try: return plugin.Handler(file_, environ) except: if environ.get('x-wsgiorg.throw_errors'): raise raise ExtensionNotSupportedError('No handler available for file %s.' % file_) dap-2.2.6.7/dap/plugins/csvfiles.py0000644000175000017500000001237110774147124016774 0ustar robertoroberto"""Plugin for CSV (comma separated values) files. This plugin serves sequential data from a CSV file. It's a bit hackish and abuses ``lambda`` and ``itertools``, but it works *very* nice. The plugin uses the ``buildfilter()`` function to create a filter from the constraint expression, and applies it on-the-fly on the data as it is being read. """ __author__ = "Roberto De Almeida " import sys import os.path import re import csv import itertools import urllib from dap import dtypes from dap.responses.das import typeconvert from dap.server import BaseHandler from dap.exceptions import OpenFileError from dap.helper import buildfilter, parse_querystring from dap.util.safeeval import expr_eval extensions = r"""^.*\.(csv|CSV)$""" def lazy_eval(s): """Try to evalute expression or fallback to string. >>> lazy_eval("1") 1 >>> lazy_eval("None") 'None' """ try: s = expr_eval(s) except: pass return s class Handler(BaseHandler): def __init__(self, filepath, environ): """Handler constructor. """ self.filepath = filepath self.environ = environ dir, self.filename = os.path.split(filepath) # Add dummy description. self.description = "Comma Separated Values from file %s." % self.filename def _parseconstraints(self, constraints=None): """Dataset builder. This method opens a CSV reader, extracts the variable names from the first line and returns an iterator to the data. Constraint expressions or handled by the ``get_filter()`` function and a filter to return only data from the columns corresponding to the requested variables. """ try: self._file = open(self.filepath) reader = csv.reader(self._file) except: message = 'Unable to open file %s.' % self.filepath raise OpenFileError(message) # Parse constraints. fields, queries = parse_querystring(constraints) # Build the dataset. dataset = dtypes.DatasetType(name=self.filename) dataset.attributes['filename'] = self.filename # Create sequence. name = self.filename[:-4].split('_', 1)[0] seq = dataset[name] = dtypes.SequenceType(name=name) # Read variables names. fieldnames = reader.next() ids = ['%s.%s' % (seq.name, n) for n in fieldnames] # We need to read the first line to grab the fields names and peek types. line = reader.next() types_ = [lazy_eval(i) for i in line] types_ = [typeconvert[type(i)] for i in types_] # Get list of requested variables. if seq.id in fields.keys(): req_ids = [] # put everything else: # Check for shorthand notation. Ugly, ugly hack. If the requested # var is not in the list of ids we append the sequence id to it, # assuming that is was requested using the shorthand notation syntax. req_ids = [['%s.%s' % (seq.id, var), var][var in ids] for var in fields.keys()] # Add requested variables. if req_ids: indexes = [] for id_ in req_ids: if id_ in ids: i = ids.index(id_) indexes.append(i) name = fieldnames[i] type_ = types_[i] seq[name] = dtypes.BaseType(name=name, type=type_) else: for name, type_ in zip(fieldnames, types_): seq[name] = dtypes.BaseType(name=name, type=type_) # Reinsert first data line. data = itertools.chain([line], reader) data = itertools.imap(lambda l: map(lazy_eval, l), data) # Filter results. if queries: # Get filter. filter1 = buildfilter(queries, ids) data = itertools.ifilter(filter1, data) # Select only requested variables. if req_ids: filter2 = lambda x: [x[i] for i in indexes] data = itertools.imap(filter2, data) # Apply stride to sequence? slice_ = fields.get(seq.id) if slice_: slice_ = slice_[0] data = itertools.islice(data, slice_.start or 0, slice_.stop or sys.maxint, slice_.step or 1) else: # Check stored variables. If more than one variable is selected, # and they have different slices, use the most restritive start, # step and stop. # # Behaviour rev-eng'ed from http://test.opendap.org/dap/data/ff/1998-6-avhrr.dat slices = [] for var in seq.walk(): slice_ = fields.get(var.id) if slice_: slices.append(slice_[0]) if slices: start, step, stop = zip(*[(s.start or 0, s.step or 1, s.stop or sys.maxint) for s in slices]) data = itertools.islice(data, max(start), min(stop), max(step)) # Insert data directly into sequence. seq.data = data return dataset def close(self): """Close the CSV file.""" if hasattr(self, '_file'): self._file.close() def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/plugins/__init__.py0000644000175000017500000000007010774147124016706 0ustar robertoroberto__import__('pkg_resources').declare_namespace(__name__) dap-2.2.6.7/dap/plugins/paster_templates/0000755000175000017500000000000011123574304020145 5ustar robertorobertodap-2.2.6.7/dap/plugins/paster_templates/setup.py_tmpl0000644000175000017500000000216210774147124022723 0ustar robertoroberto# If true, then the svn revision won't be used to calculate the # revision (set to True for real releases) RELEASE = False from setuptools import setup, find_packages import sys, os classifiers = """\ Environment :: Console Intended Audience :: Developers Intended Audience :: Science/Research Operating System :: OS Independent Programming Language :: Python Topic :: Internet Topic :: Scientific/Engineering Topic :: Software Development :: Libraries :: Python Modules """ version = '0.0' setup(name=${repr("dap.plugins.%s" % $package)}, version=version, description="A plugin for pydap server", long_description="""\ """, classifiers=filter(None, classifiers.split("\n")), keywords='dap opendap dods data', author='', author_email='', url='', license='', packages=find_packages(exclude=['ez_setup', 'examples', 'tests']), namespace_packages=['dap.plugins'], include_package_data=True, zip_safe=True, install_requires=[ # -*- Extra requirements: -*- ], entry_points=""" # -*- Entry points: -*- """, ) dap-2.2.6.7/dap/plugins/paster_templates/dap/0000755000175000017500000000000011123574304020711 5ustar robertorobertodap-2.2.6.7/dap/plugins/paster_templates/dap/plugins/0000755000175000017500000000000011123574304022372 5ustar robertorobertodap-2.2.6.7/dap/plugins/paster_templates/dap/plugins/__init__.py0000644000175000017500000000007010774147124024507 0ustar robertoroberto__import__('pkg_resources').declare_namespace(__name__) dap-2.2.6.7/dap/plugins/paster_templates/dap/plugins/+package+/0000755000175000017500000000000011123574304024113 5ustar robertorobertodap-2.2.6.7/dap/plugins/paster_templates/dap/plugins/+package+/__init__.py0000644000175000017500000000213010774147124026227 0ustar robertorobertoimport os.path from dap import dtypes from dap.server import BaseHandler from dap.helper import parse_querystring # This is a regular expression that should match the # files supported by your plugin. extensions = r"""^.*\.(ext|EXT)$""" class Handler(BaseHandler): def __init__(self, filepath, environ): """ This method receives the full path to the file and the WSGI environment (you probably won't need this. """ dir, self.filename = os.path.split(filepath) def _parseconstraints(self, constraints=None): """ This method should build the dataset according to the constraint expression. """ # Build dataset. dataset = dtypes.DatasetType(name=self.filename) # Grab requested variables. fields, queries = parse_querystring(constraints) # Add variables to dataset here depending on # ``fields`` and ``queries``. # ... return dataset def close(self): """ Close files, connections, etc. """ pass dap-2.2.6.7/dap/plugins/paster_templates/dap/__init__.py0000644000175000017500000000007010774147124023026 0ustar robertoroberto__import__('pkg_resources').declare_namespace(__name__) dap-2.2.6.7/dap/plugins/paster_templates/setup.cfg0000644000175000017500000000006310774147124021774 0ustar robertoroberto[egg_info] tag_build = dev tag_svn_revision = true dap-2.2.6.7/dap/plugins/templates.py0000644000175000017500000000167410774147124017160 0ustar robertorobertoimport os from paste.script import templates, pluginlib # Monkeypatch pluginlib def egg_info_dir(base_dir, dist_name): return os.path.join(base_dir, 'dap.plugins.%s.egg-info' % pluginlib.egg_name(dist_name)) pluginlib.egg_info_dir = egg_info_dir class DapPluginTemplate(templates.Template): summary = "A DAP plugin" egg_plugins = ['dap[server]'] _template_dir = 'paster_templates' use_cheetah = True def post(self, command, output_dir, vars): for prereq in ['dap[server]>=2.2.4']: command.insert_into_file( os.path.join(output_dir, 'setup.py'), 'Extra requirements', '%r,\n' % prereq, indent=True) command.insert_into_file( os.path.join(output_dir, 'setup.py'), 'Entry points', (' [dap.plugin]\n' ' main = dap.plugins.%(package)s\n') % vars, indent=False) dap-2.2.6.7/dap/dtypes.py0000644000175000017500000003717310774147131015012 0ustar robertoroberto"""DAP variables. This module is a Python implementation of the DAP data model. """ __author__ = "Roberto De Almeida " import copy import itertools from dap.lib import quote, to_list, _quote from dap.util.ordereddict import odict from dap.util.filter import get_filters __all__ = ['StructureType', 'SequenceType', 'DatasetType', 'GridType', 'ArrayType', 'BaseType', 'Float', 'Float0', 'Float8', 'Float16', 'Float32', 'Float64', 'Int', 'Int0', 'Int8', 'Int16', 'Int32', 'Int64', 'UInt16', 'UInt32', 'UInt64', 'Byte', 'String', 'Url'] _basetypes = ['Float32', 'Float64', 'Int16', 'Int32', 'UInt16', 'UInt32', 'Byte', 'String', 'Url'] _constructors = ['StructureType', 'SequenceType', 'DatasetType', 'GridType', 'ArrayType'] # Constants. Float = 'Float64' Float0 = 'Float64' Float8 = 'Float32' Float16 = 'Float32' Float32 = 'Float32' Float64 = 'Float64' Int = 'Int32' Int0 = 'Int32' Int8 = 'Byte' Int16 = 'Int16' Int32 = 'Int32' Int64 = 'Int32' UInt16 = 'UInt16' UInt32 = 'UInt32' UInt64 = 'UInt32' UInt8 = 'Byte' Byte = 'Byte' String = 'String' Url = 'Url' typemap = { # numpy 'd': Float64, 'f': Float32, 'l': Int32, 'b': Byte, 'h': Int16, 'q': Int32, 'H': UInt16, 'L': UInt32, 'Q': UInt32, 'B': Byte, 'S': String, } class StructureType(odict): """Structure contructor. A structure is a dict-like object, which can hold other DAP variables. Structures have a 'data' attribute that combines the data from the stored variables when read, and propagates the data to the variables when set. This behaviour can be bypassed by setting the '_data' attribute; in this case, no data is propagated, and further reads do not combine the data from the stored variables. """ def __init__(self, name='', attributes=None): odict.__init__(self) self.name = quote(name) self.attributes = attributes or {} self._id = name self._filters = [] self._data = None def __iter__(self): # Iterate over the variables contained in the structure. return self.itervalues() walk = __iter__ def __getattr__(self, attr): # Try to return stored variable. try: return self[attr] except KeyError: # Try to return attribute from self.attributes. try: return self.attributes[attr] except KeyError: raise AttributeError def __setitem__(self, key, item): # Assign a new variable and apply the proper id. self._dict.__setitem__(key, item) if key not in self._keys: self._keys.append(key) # Ensure that stored objects have the proper id. item._set_id(self._id) def _get_data(self): if self._data is not None: return self._data else: return [var.data for var in self.values()] def _set_data(self, data): # Propagate the data to the stored variables. for data_, var in itertools.izip(data, self.values()): var.data = data_ data = property(_get_data, _set_data) def _get_id(self): return self._id def _set_id(self, parent=None): if parent: self._id = '%s.%s' % (parent, self.name) else: self._id = self.name # Propagate id to stored variables. for var in self.values(): var._set_id(self._id) id = property(_get_id) # Read-only. def _get_filters(self): return self._filters def _set_filters(self, f): self._filters.append(f) # Propagate filter to stored variables. for var in self.values(): var._set_filters(f) filters = property(_get_filters, _set_filters) def __copy__(self): out = self.__class__(name=self.name, attributes=self.attributes.copy()) out._id = self._id out._filters = self._filters[:] out._data = self._data # Stored variables *are not* copied. for k, v in self.items(): out[k] = v return out def __deepcopy__(self, memo=None, _nil=[]): out = self.__class__(name=self.name, attributes=self.attributes.copy()) out._id = self._id out._filters = self._filters[:] out._data = self._data # Stored variables *are* (deep) copied. for k, v in self.items(): out[k] = copy.deepcopy(v) return out class DatasetType(StructureType): """Dataset constructor. A dataset is very similar to a structure -- the main difference is that its name is not used when composing the fully qualified name of stored variables. """ def __setitem__(self, key, item): self._dict.__setitem__(key, item) if key not in self._keys: self._keys.append(key) # Set the id. Here the parent should be None, since the dataset # id is not part of the fully qualified name. item._set_id(None) def _set_id(self, parent=None): self._id = self.name # Propagate id. for var in self.values(): var._set_id(None) class SequenceType(StructureType): """Sequence constructor. A sequence contains ordered data, corresponding to the records in a sequence of structures with the same stored variables. """ # Nesting level. Sequences inside sequences have a level 2, and so on. level = 1 def __setitem__(self, key, item): # Assign a new variable and apply the proper id. self._dict.__setitem__(key, item) if key not in self._keys: self._keys.append(key) # Ensure that stored objects have the proper id. item._set_id(self._id) # If the variable is a sequence, set the nesting level. def set_level(seq, level): if isinstance(seq, SequenceType): seq.level = level for child in seq.walk(): set_level(child, level+1) set_level(item, self.level+1) def walk(self): # Walk over the variables contained in the structure. return self.itervalues() def _get_data(self): # This is similar to the structure _get_data method, except that data # is combined from stored variables using zip(), i.e., grouped values # from each variable. if self._data is not None: return self._data else: return _build_data(self.level, *[var.data for var in self.values()]) def _set_data(self, data): for data_, var in itertools.izip(_propagate_data(self.level, data), self.values()): var.data = data_ data = property(_get_data, _set_data) def __iter__(self): """ When iterating over a sequence, we yield structures containing the corresponding data (first record, second, etc.). """ out = self.__deepcopy__() # Set server-side filters. When the sequence is iterated in a # listcomp/genexp, this function inspects the stack and tries to # build a server-side filter from the client-side filter. This # is voodoo black magic, take care. filters = get_filters(out) for filter_ in filters: out._set_filters(filter_) for values in out.data: # Yield a nice structure. struct_ = StructureType(name=out.name, attributes=out.attributes) for data, name in zip(values, out.keys()): var = struct_[name] = out[name].__deepcopy__() var.data = data # Set the id. This is necessary since the new structure is not # contained inside a dataset. parent = out._id[:-len(out.name)-1] struct_._set_id(parent) yield struct_ def filter(self, *filters): # Make a copy of the sequence. out = self.__deepcopy__() # And filter it according to the selection expression. for filter_ in filters: out._set_filters(_quote(filter_)) return out class GridType(object): """Grid constructor. A grid is a constructor holding an 'array' variable. The array has its dimensions mapped to 'maps' stored in the grid (lat, lon, time, etc.). Most of the requests are simply passed onto the stored array. """ def __init__(self, name='', array=None, maps=None, attributes=None): self.name = quote(name) self.array = array self.maps = maps or odict() self.attributes = attributes or {} self._id = name self._filters = [] def __len__(self): return self.array.shape[0] def __iter__(self): # Iterate over the grid. Yield the array and then the maps. yield self.array for map_ in self.maps.values(): yield map_ walk = __iter__ def __getattr__(self, attr): # Try to return attribute from self.attributes. try: return self.attributes[attr] except KeyError: raise AttributeError def __getitem__(self, index): # Return data from the array. return self.array[index] def _get_data(self): return self.array.data def _set_data(self, data): self.array.data = data data = property(_get_data, _set_data) def _get_id(self): return self._id def _set_id(self, parent=None): if parent: self._id = '%s.%s' % (parent, self.name) else: self._id = self.name # Propagate id to array and maps. if self.array: self.array._set_id(self._id) for map_ in self.maps.values(): map_._set_id(self._id) id = property(_get_id) def _get_filters(self): return self._filters def _set_filters(self, f): self.filters.append(f) # Propagate filter. self.array._set_filters(f) for map_ in self.maps.values(): map_._set_filters(f) filters = property(_get_filters, _set_filters) def _get_dimensions(self): # Return dimensions from stored maps. return tuple(self.maps.keys()) dimensions = property(_get_dimensions) def _get_shape(self): return self.array.shape def _set_shape(self, shape): self.array.shape = shape shape = property(_get_shape, _set_shape) def _get_type(self): return self.array.type def _set_type(self, type): self.array.type = type type = property(_get_type, _set_type) def __copy__(self): out = self.__class__(name=self.name, array=self.array, maps=self.maps, attributes=self.attributes.copy()) out._id = self._id out._filters = self._filters[:] return out def __deepcopy__(self, memo=None, _nil=[]): out = self.__class__(name=self.name, attributes=self.attributes.copy()) out.array = copy.deepcopy(self.array) out.maps = copy.deepcopy(self.maps) out._id = self._id out._filters = self._filters[:] return out class BaseType(object): """DAP Base type. Variable holding a single value, or an iterable if it's stored inside a sequence. It's the fundamental DAP variable, which actually holds data (together with arrays). """ def __init__(self, name='', data=None, type=None, attributes=None): self.name = quote(name) self.data = data self.attributes = attributes or {} if type in _basetypes: self.type = type else: self.type = typemap.get(type, Int32) self._id = name self._filters = [] def __iter__(self): # Yield the stored value. # Perhaps we should raise StopIteration? yield self.data def __getattr__(self, attr): # Try to return attribute from self.attributes. try: return self.attributes[attr] except KeyError: raise AttributeError def __getitem__(self, key): # Return data from the array. return self.data[key] def __setitem__(self, key, item): # Assign a new variable and apply the proper id. self.data.__setitem__(key, item) def _get_id(self): return self._id def _set_id(self, parent=None): if parent: self._id = '%s.%s' % (parent, self.name) else: self._id = self.name id = property(_get_id) # Read-only. def _get_filters(self): return self._filters def _set_filters(self, f): self._filters.append(f) # Propagate to data, if it's a Proxy object. if hasattr(self.data, 'filters'): self.data.filters = self._filters filters = property(_get_filters, _set_filters) def __copy__(self): out = self.__class__(name=self.name, data=self.data, type=self.type, attributes=self.attributes.copy()) out._id = self._id out._filters = self._filters[:] return out def __deepcopy__(self, memo=None, _nil=[]): out = self.__class__(name=self.name, type=self.type, attributes=self.attributes.copy()) try: out.data = copy.copy(self.data) except TypeError: self.data = to_list(self.data) out.data = copy.copy(self.data) out._id = self._id out._filters = self._filters[:] return out # This allows the variable to be compared to numbers. def __ge__(self, other): return self.data >= other def __gt__(self, other): return self.data > other def __le__(self, other): return self.data <= other def __lt__(self, other): return self.data < other def __eq__(self, other): return self.data == other class ArrayType(BaseType): """An array of BaseType variables. Although the DAP supports arrays of any DAP variables, pydap can only handle arrays of base types. This makes the ArrayType class very similar to a BaseType, with the difference that it'll hold an array of data in its 'data' attribute. Array of constructors will not be supported until Python has a native multi-dimensional array type. """ def __init__(self, name='', data=None, shape=None, dimensions=None, type=None, attributes=None): self.name = quote(name) self.data = data self.shape = shape or () self.dimensions = dimensions or () self.attributes = attributes or {} if type in _basetypes: self.type = type else: self.type = typemap.get(type, Int32) self._id = name self._filters = [] def __len__(self): return self.shape[0] def __copy__(self): out = self.__class__(name=self.name, data=self.data, shape=self.shape, dimensions=self.dimensions, type=self.type, attributes=self.attributes.copy()) out._id = self._id out._filters = self._filters[:] return out def __deepcopy__(self, memo=None, _nil=[]): out = self.__class__(name=self.name, shape=self.shape, dimensions=self.dimensions, type=self.type, attributes=self.attributes.copy()) try: out.data = copy.copy(self.data) except TypeError: self.data = to_list(self.data) out.data = copy.copy(self.data) out._id = self._id out._filters = self._filters[:] return out # Functions for propagating data up and down in sequences. # I'm not 100% sure how this works. def _build_data(level, *vars_): if level > 0: out = [_build_data(level-1, *els) for els in itertools.izip(*vars_)] else: out = vars_ return out def _propagate_data(level, vars_): if level > 0: out = zip(*[_propagate_data(level-1, els) for els in vars_]) else: out = vars_ return out dap-2.2.6.7/dap/xdr.py0000644000175000017500000002644010774147131014272 0ustar robertoroberto# This Python file uses the following encoding: utf-8 """Fast(er) implementation of xdrlib for the DAP. This module reimplements Python's xdrlib module for the DAP. It uses the "array" module for speed, and encodes bytes according to the DAP spec. """ __author__ = "Roberto De Almeida " import array import struct import operator try: from numpy import array as array_ except ImportError: array_ = None from dap.lib import isiterable _big_endian = struct.pack('i',1)[0] != '\x01' # Convert from DAP types to array types. typeconvert = {'float64': ('d', 8), 'float32': ('f', 4), 'float' : ('f', 4), 'uint' : ('I', 4), 'uint16' : ('I', 4), 'uint32' : ('I', 4), 'int' : ('i', 4), 'int16' : ('i', 4), 'int32' : ('i', 4), 'byte' : ('b', 1), } class DapPacker(object): r"""Pack variable data into XDR. This is a faster reimplementation of xdrlib using the native module "array". >>> from dap.dtypes import * >>> dapvar = BaseType(data=1, type='Int32') >>> xdrdata = DapPacker(dapvar) >>> for i in xdrdata: ... print repr(i) '\x00\x00\x00\x01' >>> dapvar = ArrayType(data=['one', 'two'], shape=[2], type='String') >>> xdrdata = DapPacker(dapvar) >>> for i in xdrdata: ... print repr(i) '\x00\x00\x00\x02' '\x00\x00\x00\x03one\x00' '\x00\x00\x00\x03two\x00' >>> dapvar = ArrayType(data=range(2), shape=[2], type='Int32') >>> xdrdata = DapPacker(dapvar) >>> for i in xdrdata: ... print repr(i) '\x00\x00\x00\x02\x00\x00\x00\x02' '\x00\x00\x00\x00\x00\x00\x00\x01' >>> dapvar = ArrayType(data=range(2), shape=[2], type='Float64') >>> xdrdata = DapPacker(dapvar) >>> for i in xdrdata: ... print repr(i) '\x00\x00\x00\x02\x00\x00\x00\x02' '\x00\x00\x00\x00\x00\x00\x00\x00?\xf0\x00\x00\x00\x00\x00\x00' """ def __init__(self, dapvar, data=None): self.var = dapvar if data is None: data = dapvar.data # Make data iterable. if not isiterable(data): data = [data] # Put data in blocks. if not hasattr(data, 'shape') or len(data.shape) <= 1: data = [data] self.data = data def __iter__(self): """Iterate over the XDR encoded data.""" # Yield length (twice) if array. if hasattr(self.var, 'shape'): if self.var.type.lower() in ['url', 'string']: yield self._pack_length() else: yield self._pack_length() * 2 # Bytes are sent differently. if self.var.type.lower() == 'byte': for b in self._yield_bytes(): yield b # String are zero padded to 4n. elif self.var.type.lower() in ['url', 'string']: for block in self.data: if hasattr(block, 'flat'): block = block.flat # for numpy str arrays for word in block: yield self._pack_string(word) else: type_, size = typeconvert[self.var.type.lower()] for block in self.data: # Flatten block. if hasattr(block, 'flat'): block = block.flat # Encode block in XDR (big-endian). First we # try to encode using numpy's ``.astype``, # otherwise be use ``array.array`` and ensure # data is in big endian format. try: dtype = ">%s%s" % (type_, size) data = block.base.astype(dtype).data data = str(data) except: data = array.array(type_, list(block)) if not _big_endian: data.byteswap() data = data.tostring() yield data def _pack_length(self): """Yield array length.""" shape = getattr(self.var, 'shape', [1]) length = reduce(operator.mul, shape) return struct.pack('>L', length) def _yield_bytes(self): r"""Yield bytes. Bytes are encoded as is, padded to a four-byte boundary. An array of five bytes, eg, is encoded as eight bytes: >>> from dap.dtypes import * >>> dapvar = ArrayType(data=range(5), shape=[5], type='Byte') >>> xdrdata = DapPacker(dapvar) >>> for i in xdrdata: ... print repr(i) '\x00\x00\x00\x05\x00\x00\x00\x05' '\x00' '\x01' '\x02' '\x03' '\x04' '\x00\x00\x00' Again, the first line correponds to the array size packed twice, followed by the bytes and the padding. """ count = 0 for block in self.data: data = array.array('B', block).tostring() for d in data: yield d count += 1 padding = (4 - (count % 4)) % 4 yield padding * '\0' def _pack_string(self, s): """Pack a string. We first pack the string length, followed by the string padded to size 4n. """ # Pack length first. n = len(s) length = struct.pack('>L', n) n = ((n+3)/4)*4 data = length + s + (n - len(s)) * '\0' return data class DapUnpacker(object): r"""A XDR data unpacker. Unpacking data from a base type: >>> from dap.dtypes import * >>> dapvar = BaseType(data=1, name='dapvar', type='Byte') >>> print dapvar.data 1 >>> from dap.server import SimpleHandler >>> from dap.helper import escape_dods >>> dataset = DatasetType(name='test') >>> dataset['dapvar'] = dapvar >>> headers, output = SimpleHandler(dataset).dods() >>> print escape_dods(''.join(output), pad='') Dataset { Byte dapvar; } test; Data: \x01\x00\x00\x00 >>> headers, output = SimpleHandler(dataset).dods() >>> xdrdata = ''.join(output) >>> start = xdrdata.index('Data:\n') + len('Data:\n') >>> xdrdata = xdrdata[start:] >>> data = DapUnpacker(xdrdata, (), dapvar.type) >>> print data.getvalue() 1 An array of bytes: >>> dapvar = ArrayType(name='dapvar', data=range(5), shape=[5], type='Byte') >>> dataset['dapvar'] = dapvar >>> headers, output = SimpleHandler(dataset).dods() >>> xdrdata = ''.join(output) >>> start = xdrdata.index('Data:\n') + len('Data:\n') >>> xdrdata = xdrdata[start:] >>> data = DapUnpacker(xdrdata, dapvar.shape, dapvar.type, outshape=[5]) >>> print data.getvalue() [0 1 2 3 4] Another array: >>> dapvar = ArrayType(name='dapvar', data=range(25), shape=[5,5], type='Float32') >>> dataset['dapvar'] = dapvar >>> headers, output = SimpleHandler(dataset).dods() >>> xdrdata = ''.join(output) >>> start = xdrdata.index('Data:\n') + len('Data:\n') >>> xdrdata = xdrdata[start:] >>> data = DapUnpacker(xdrdata, dapvar.shape, dapvar.type, outshape=[5,5]) >>> print data.getvalue() [[ 0. 1. 2. 3. 4.] [ 5. 6. 7. 8. 9.] [ 10. 11. 12. 13. 14.] [ 15. 16. 17. 18. 19.] [ 20. 21. 22. 23. 24.]] One more: >>> dapvar = ArrayType(name='dapvar', data=['um', 'dois', 'três'], shape=[3], type='String') >>> dataset['dapvar'] = dapvar >>> headers, output = SimpleHandler(dataset).dods() >>> for line in output: ... print repr(line) 'Dataset {\n' ' String dapvar[dapvar = 3];\n' '} test;\n' 'Data:\n' '\x00\x00\x00\x03' '\x00\x00\x00\x02um\x00\x00' '\x00\x00\x00\x04dois' '\x00\x00\x00\x05tr\xc3\xaas\x00\x00\x00' >>> headers, output = SimpleHandler(dataset).dods() >>> xdrdata = ''.join(output) >>> start = xdrdata.index('Data:\n') + len('Data:\n') >>> xdrdata = xdrdata[start:] >>> data = DapUnpacker(xdrdata, dapvar.shape, dapvar.type, outshape=[3]) >>> print data.getvalue() [um dois três] """ def __init__(self, data, shape, type, outshape=None): self.__buf = data self.shape = shape self.type = type self.outshape = outshape # Buffer position. self.__pos = 0 def getvalue(self): """Unpack the XDR-encoded data.""" # Get current position. i = self.__pos # Check for empty sequence. if self.__buf[i:i+4] == '\xa5\x00\x00\x00': return [] # Check for sequence with data. elif self.__buf[i:i+4] == '\x5a\x00\x00\x00': # Unpack sequence start marker (uint 1509949440). mark = self._unpack_uint() out = [] while mark != 2768240640L: tmp = self.getvalue() out.append(tmp) # Unpack marker. mark = self._unpack_uint() return out # Get data length. n = 1 if self.shape: n = self._unpack_uint() # Strings pack the size only once? if self.type.lower() not in ['url', 'string']: self._unpack_uint() # Bytes are treated differently. if self.type.lower() == 'byte': out = self._unpack_bytes(n) type_, size = typeconvert[self.type.lower()] # As are strings... elif self.type.lower() in ['url', 'string']: out = self._unpack_string(n) type_ = 'S' else: i = self.__pos type_, size = typeconvert[self.type.lower()] out = array.array(type_, self.__buf[i:i+n*size]) self.__pos = i+n*size # Ensure big-endianess. if not _big_endian: out.byteswap() if not self.shape: out = out[0] # BaseType. elif array_: # Convert to array and reshape. out = array_(out, type_) if self.outshape: out.shape = tuple(self.outshape) return out def _unpack_uint(self): i = self.__pos self.__pos = j = i+4 data = self.__buf[i:j] if len(data) < 4: raise EOFError x = struct.unpack('>L', data)[0] try: return int(x) except OverflowError: return x def _unpack_bytes(self, count): i = self.__pos out = array.array('b', self.__buf[i:i+count]) padding = (4 - (count % 4)) % 4 self.__pos = i + count + padding return out def _unpack_string(self, count): out = [] for s in range(count): # Unpack string length. n = self._unpack_uint() i = self.__pos j = i+n data = self.__buf[i:j] # Fix cursor position. padding = (4 - (n % 4)) % 4 self.__pos = j + padding out.append(data) return out def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/proxy.py0000644000175000017500000001044010774147131014647 0ustar robertorobertofrom __future__ import division """Proxy class to DAP data. This module implements a proxy object that behaves like an array and downloads data transparently from a DAP server when sliced. It is used when building a representation of the dataset, and should be not directly used. """ __author__ = "Roberto De Almeida " import sys from dap.xdr import DapUnpacker from dap.helper import fix_slice from dap.util.http import openurl try: from numpy import ndarray as ndarray_ except ImportError: ndarray_ = None class Proxy(object): """ A proxy to data stored in a DODS server. """ def __init__(self, url, id, shape, type, filters=None, cache=None, username=None, password=None): self.url = url self.id = id self.shape = shape self.type = type self.filters = filters or [] self.cache = cache self.username = username self.password = password def __iter__(self): return iter(self[:]) def __getitem__(self, index): """ Download data from DAP server. When the proxy object is sliced, it build an URL from the slice and retrieves the data from the DAP server. """ # Build the base URL. url = '%s.dods?%s' % (self.url, self.id) if self.shape: # Fix the index for incomplete slices or ellipsis. index = fix_slice(len(self.shape), index) # Force index to tuple, to iterate over the slices. if not isinstance(index, tuple): index = index, # Iterate over the sliced dimensions. i = 0 outshape = [] for dimension in index: # If dimension is a slice, get start, step and stop. if isinstance(dimension, slice): start = dimension.start or 0 step = dimension.step or 1 if dimension.stop: stop = dimension.stop-1 else: stop = self.shape[i]-1 # Otherwise, retrieve a single value. else: start = dimension stop = dimension step = 1 # When stop is not specified, use the shape. if stop == sys.maxint or stop > self.shape[i]-1: stop = self.shape[i]-1 # Negative slices. elif stop < 0: stop = self.shape[i]+stop # Negative starting slices. if start < 0: start = self.shape[i]+start # Build the URL used to retrieve the data. url = '%s[%s:%s:%s]' % (url, str(start), str(step), str(stop)) # outshape is a list of the slice dimensions. outshape.append(1+(stop-start)//step) # Update to next dimension. i += 1 else: # No need to resize the data. outshape = None # Make the outshape consistent with the numpy and pytables conventions. if outshape is not None: outshape = self._reduce_outshape(outshape) # Check for filters. if self.filters: ce = '&'.join(self.filters) url = '%s&%s' % (url, ce) # Fetch data. resp, data = openurl(url, self.cache, self.username, self.password) # First lines are ASCII information that end with 'Data:\n'. start = data.index('Data:\n') + len('Data:\n') xdrdata = data[start:] # Unpack data. output = DapUnpacker(xdrdata, self.shape, self.type, outshape).getvalue() # Convert length 1 arrays to scalars if ndarray_: if outshape == () and isinstance(output, ndarray_): output = output[0] return output def _reduce_outshape(self, outshape): """Make the outshape consistent with the numpy and pytables conventions. (1, N) -> (N,) (N, 1) -> (N,) (N, M, 1) -> (N, M) Basically, all ones are removed from the shape. """ return tuple([index for index in outshape if index != 1]) def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/exceptions.py0000644000175000017500000000226510774147131015655 0ustar robertoroberto"""DAP exceptions. These exceptions are mostly used by the server. When an exception is captured, a proper error message is displayed (according to the DAP 2.0 spec), with information about the exception and the error code associated with it. The error codes are attributed using the "first come, first serve" algorithm. """ __author__ = "Roberto De Almeida " class DapError(Exception): """Base DAP exception.""" def __init__(self, value): self.value = value def __str__(self): return repr(self.value) class ClientError(DapError): """Generic error with the client.""" code = 100 class ServerError(DapError): """Generic error with the server.""" code = 200 class ConstraintExpressionError(ServerError): """Exception raised when an invalid constraint expression is given.""" code = 201 class PluginError(DapError): """Generic error with a plugin.""" code = 300 class ExtensionNotSupportedError(PluginError): """Exception raised when trying to open a file not supported by any plugins.""" code = 301 class OpenFileError(PluginError): """Exception raised when unable to open a file.""" code = 302 dap-2.2.6.7/dap/__init__.py0000644000175000017500000000074010774147131015227 0ustar robertoroberto"""A Python implementation of the Data Access Protocol (DAP). Pydap is a Python module implementing the Data Access Protocol (DAP) written from scratch. The module implements a DAP client, allowing transparent and efficient access to dataset stored in DAP server, and also implements a DAP server able to serve data from a variety of formats. For more information about the protocol, please check http://opendap.org. """ __import__('pkg_resources').declare_namespace(__name__) dap-2.2.6.7/dap/util/0000755000175000017500000000000011123574304014065 5ustar robertorobertodap-2.2.6.7/dap/util/safeeval.py0000644000175000017500000000552510774147131016241 0ustar robertoroberto# From: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/286134 import dis _const_codes = map(dis.opmap.__getitem__, [ 'POP_TOP','ROT_TWO','ROT_THREE','ROT_FOUR','DUP_TOP', 'BUILD_LIST','BUILD_MAP','BUILD_TUPLE', 'LOAD_CONST','RETURN_VALUE','STORE_SUBSCR' ]) _expr_codes = _const_codes + map(dis.opmap.__getitem__, [ 'UNARY_POSITIVE','UNARY_NEGATIVE','UNARY_NOT', 'UNARY_INVERT','BINARY_POWER','BINARY_MULTIPLY', 'BINARY_DIVIDE','BINARY_FLOOR_DIVIDE','BINARY_TRUE_DIVIDE', 'BINARY_MODULO','BINARY_ADD','BINARY_SUBTRACT', 'BINARY_LSHIFT','BINARY_RSHIFT','BINARY_AND','BINARY_XOR', 'BINARY_OR', ]) def _get_opcodes(codeobj): """_get_opcodes(codeobj) -> [opcodes] Extract the actual opcodes as a list from a code object >>> c = compile("[1 + 2, (1,2)]", "", "eval") >>> _get_opcodes(c) [100, 100, 23, 100, 100, 102, 103, 83] """ i = 0 opcodes = [] s = codeobj.co_code while i < len(s): code = ord(s[i]) opcodes.append(code) if code >= dis.HAVE_ARGUMENT: i += 3 else: i += 1 return opcodes def test_expr(expr, allowed_codes): """test_expr(expr) -> codeobj Test that the expression contains only the listed opcodes. If the expression is valid and contains only allowed codes, return the compiled code object. Otherwise raise a ValueError """ try: c = compile(expr, "", "eval") except: raise ValueError("%s is not a valid expression", expr) codes = _get_opcodes(c) for code in codes: if code not in allowed_codes: raise ValueError("opcode %s not allowed (%s)" % (dis.opname[code], repr(expr))) return c def const_eval(expr): """const_eval(expression) -> value Safe Python constant evaluation Evaluates a string that contains an expression describing a Python constant. Strings that are not valid Python expressions or that contain other code besides the constant raise ValueError. >>> const_eval("10") 10 >>> const_eval("[1,2, (3,4), {'foo':'bar'}]") [1, 2, (3, 4), {'foo': 'bar'}] >>> const_eval("1+2") Traceback (most recent call last): ... ValueError: opcode BINARY_ADD not allowed """ c = test_expr(expr, _const_codes) return eval(c) def expr_eval(expr): """expr_eval(expression) -> value Safe Python expression evaluation Evaluates a string that contains an expression that only uses Python constants. This can be used to e.g. evaluate a numerical expression from an untrusted source. >>> expr_eval("1+2") 3 >>> expr_eval("[1,2]*2") [1, 2, 1, 2] >>> expr_eval("__import__('sys').modules") Traceback (most recent call last): ... ValueError: opcode LOAD_NAME not allowed """ c = test_expr(expr, _expr_codes) return eval(c) dap-2.2.6.7/dap/util/filter.py0000644000175000017500000000641710774147131015741 0ustar robertorobertoimport sys import inspect import compiler from urllib import quote from dap.lib import encode_atom class ASTVisitor(compiler.visitor.ASTVisitor): def __init__(self, scope, seq): compiler.visitor.ASTVisitor.__init__(self) self.scope = scope self.seq = seq self.filters = [] def visitGenExprFor(self, node): # Check if we're iterating over the sequence. if self.eval(node.iter) == self.seq: # Add sequence with assigned name to the scope. self.scope[node.assign.name] = self.seq # Carry on. self.visit(node.assign) self.visit(node.iter) for if_ in node.ifs: self.visit(if_) def visitListCompFor(self, node): # Check if we're iterating over the sequence. if self.eval(node.list) == self.seq: # Add sequence with assigned name to the scope. self.scope[node.assign.name] = self.seq # Carry on. self.visit(node.assign) self.visit(node.list) for if_ in node.ifs: self.visit(if_) def visitCompare(self, node): # Convert multiple comparisons like 1 < a < 2 to # (1 < a) and (a < 2). if len(node.ops) > 1: left = node.expr out = [] for op in node.ops: out.append(compiler.ast.Compare(left, [op])) left = op[1] new_node = compiler.ast.And(out) self.visit(new_node) else: # Simple comparison. a, op, b = node.getChildren() ops = ['<', '>', '==', '!=', '<=', '>='] if op in ops: a = self.eval(a) b = self.eval(b) # If the objects have an 'id' attribute we use it # instead. if hasattr(a, 'id'): a = a.id else: a = quote(encode_atom(a)) if hasattr(b, 'id'): b = b.id else: b = quote(encode_atom(b)) # Build filter. if op == '==': op = '=' filter_ = '%s%s%s' % (a, op, b) self.filters.append(filter_) def visitOr(self, node): raise Exception('OR not supported by the DAP spec!') def eval(self, node): """ Eval node. This is done by converting the node to bytecode and eval()ing the bytecode in the instance scope. """ ast = compiler.ast.Expression(node) ast.filename = 'dummy' c = compiler.pycodegen.ExpressionCodeGenerator(ast) obj = eval(c.getCode(), self.scope) return obj def get_filters(seq): # We need to get the stack where the sequence is being iterated. # This is called from genexp --> __iter__ --> get_filters, so # we go up 2 frames in the stack. frame = sys._getframe(2) # Inspect frame. fname, lineno, func, src, index = inspect.getframeinfo(frame) scope = frame.f_globals # Fix source. if src: src = src[0].strip() if src.endswith(':'): src = '%s pass' % src # Build and walk AST to parse filters. visitor = ASTVisitor(scope, seq) try: ast = compiler.parse(src) compiler.walk(ast, visitor) except: pass return visitor.filters dap-2.2.6.7/dap/util/__init__.py0000644000175000017500000000000210774147131016173 0ustar robertoroberto# dap-2.2.6.7/dap/util/ordereddict.py0000644000175000017500000000603710774147131016742 0ustar robertoroberto"""Ordered dictionary. This is a dictionary class that preserves the order in which the keys are stored. This is necessary to build Structures and Sequences that follow the requested variable order. """ __author__ = "Roberto De Almeida " import copy class odict(dict): """Ordered dictionary.""" def __init__(self, odict=None): self._keys = [] self._dict = {} if odict is not None: self.update(odict) def __iter__(self): return iter(self._keys[:]) def __setitem__(self, key, item): self._dict.__setitem__(key, item) if key not in self._keys: self._keys.append(key) def __getitem__(self, key): return self._dict.__getitem__(key) def __delitem__(self, key): self._dict.__delitem__(key) self._keys.remove(key) def keys(self): return self._keys[:] def items(self): return [(key, self._dict.__getitem__(key)) for key in self._keys] def values(self): return [self._dict.__getitem__(key) for key in self._keys] def iterkeys(self): for key in self._keys: yield key def iteritems(self): for key in self._keys: yield (key, self._dict.__getitem__(key)) def itervalues(self): for key in self._keys: yield self._dict.__getitem__(key) def clear(self): self._dict.clear() self._keys = [] def copy(self): new = odict(self) return new def update(self, odict): for k, v in odict.items(): self.__setitem__(k, v) def setdefault(self, key, d=None): if key not in self._keys: self._keys.append(key) return self._dict.setdefault(key, d) def get(self, key, d=None): if key in self._keys: return self._dict.__getitem__(key) else: return d def has_key(self, key): return self._dict.has_key(key) def popitem(self): try: key = self._keys[-1] except IndexError: raise KeyError('dictionary is empty') self._keys.remove(key) return self._dict.pop(key) def pop(self, key, d=None): value = self._dict.pop(key, d) try: self._keys.remove(key) except ValueError: pass return value def fromkeys(keys, d=None): new = odict() for key in keys: new.__setitem__(key, d) return new def __contains__(self, key): return self._dict.has_key(key) def __len__(self): return self._dict.__len__() def __repr__(self): return '{%s}' % ', '.join(['%s: %s' % (k.__repr__(), v.__repr__()) for (k, v) in self.items()]) def __str__(self): return '{%s}' % ', '.join(['%s: %s' % (k.__repr__(), v.__repr__()) for (k, v) in self.items()]) def __copy__(self): return self.copy() def __deepcopy__(self, memo=None, _nil=[]): new = odict() for k, v in self.items(): new.__setitem__(k, copy.deepcopy(v)) return new def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/util/http.py0000644000175000017500000000146511123573663015433 0ustar robertorobertoimport re import httplib2 from dap.util import socks httplib2.socks = socks import dap.lib from dap.exceptions import ClientError def openurl(url, cache=None, username=None, password=None): h = httplib2.Http(cache=cache, timeout=dap.lib.TIMEOUT, proxy_info=dap.lib.PROXY) if username and password: h.add_credentials(username, password) if dap.lib.VERBOSE: print url resp, data = h.request(url, "GET", headers={'user-agent': dap.lib.USER_AGENT}) # Check for errors if resp.get("content-description") == "dods_error": # Parse response. m = re.search('code = (?P\d+);\s*message = "(?P.*)"', data, re.DOTALL | re.MULTILINE) msg = 'Server error %(code)s: "%(msg)s"' % m.groupdict() raise ClientError(msg) return resp, data dap-2.2.6.7/dap/util/socks.py0000644000175000017500000003273111123541666015574 0ustar robertoroberto"""SocksiPy - Python SOCKS module. Version 1.00 Copyright 2006 Dan-Haim. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of Dan Haim nor the names of his contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY DAN HAIM "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DAN HAIM OR HIS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMANGE. This module provides a standard socket-like interface for Python for tunneling connections through SOCKS proxies. """ import socket import struct PROXY_TYPE_SOCKS4 = 1 PROXY_TYPE_SOCKS5 = 2 PROXY_TYPE_HTTP = 3 _defaultproxy = None _orgsocket = socket.socket class ProxyError(Exception): def __init__(self, value): self.value = value def __str__(self): return repr(self.value) class GeneralProxyError(ProxyError): def __init__(self, value): self.value = value def __str__(self): return repr(self.value) class Socks5AuthError(ProxyError): def __init__(self, value): self.value = value def __str__(self): return repr(self.value) class Socks5Error(ProxyError): def __init__(self, value): self.value = value def __str__(self): return repr(self.value) class Socks4Error(ProxyError): def __init__(self, value): self.value = value def __str__(self): return repr(self.value) class HTTPError(ProxyError): def __init__(self, value): self.value = value def __str__(self): return repr(self.value) _generalerrors = ("success", "invalid data", "not connected", "not available", "bad proxy type", "bad input") _socks5errors = ("succeeded", "general SOCKS server failure", "connection not allowed by ruleset", "Network unreachable", "Host unreachable", "Connection refused", "TTL expired", "Command not supported", "Address type not supported", "Unknown error") _socks5autherrors = ("succeeded", "authentication is required", "all offered authentication methods were rejected", "unknown username or invalid password", "unknown error") _socks4errors = ("request granted", "request rejected or failed", "request rejected because SOCKS server cannot connect to identd on the client", "request rejected because the client program and identd report different user-ids", "unknown error") def setdefaultproxy(proxytype=None,addr=None,port=None,rdns=True,username=None,password=None): """setdefaultproxy(proxytype, addr[, port[, rdns[, username[, password]]]]) Sets a default proxy which all further socksocket objects will use, unless explicitly changed. """ global _defaultproxy _defaultproxy = (proxytype,addr,port,rdns,username,password) class socksocket(socket.socket): """socksocket([family[, type[, proto]]]) -> socket object Open a SOCKS enabled socket. The parameters are the same as those of the standard socket init. In order for SOCKS to work, you must specify family=AF_INET, type=SOCK_STREAM and proto=0. """ def __init__(self, family=socket.AF_INET, type=socket.SOCK_STREAM, proto=0, _sock=None): _orgsocket.__init__(self,family,type,proto,_sock) if _defaultproxy != None: self.__proxy = _defaultproxy else: self.__proxy = (None, None, None, None, None, None) self.__proxysockname = None self.__proxypeername = None def __recvall(self, bytes): """__recvall(bytes) -> data Receive EXACTLY the number of bytes requested from the socket. Blocks until the required number of bytes have been received. """ data = "" while len(data) < bytes: data = data + self.recv(bytes-len(data)) return data def setproxy(self,proxytype=None,addr=None,port=None,rdns=True,username=None,password=None): """setproxy(proxytype, addr[, port[, rdns[, username[, password]]]]) Sets the proxy to be used. proxytype - The type of the proxy to be used. Three types are supported: PROXY_TYPE_SOCKS4 (including socks4a), PROXY_TYPE_SOCKS5 and PROXY_TYPE_HTTP addr - The address of the server (IP or DNS). port - The port of the server. Defaults to 1080 for SOCKS servers and 8080 for HTTP proxy servers. rdns - Should DNS queries be preformed on the remote side (rather than the local side). The default is True. Note: This has no effect with SOCKS4 servers. username - Username to authenticate with to the server. The default is no authentication. password - Password to authenticate with to the server. Only relevant when username is also provided. """ self.__proxy = (proxytype,addr,port,rdns,username,password) def __negotiatesocks5(self,destaddr,destport): """__negotiatesocks5(self,destaddr,destport) Negotiates a connection through a SOCKS5 server. """ # First we'll send the authentication packages we support. if (self.__proxy[4]!=None) and (self.__proxy[5]!=None): # The username/password details were supplied to the # setproxy method so we support the USERNAME/PASSWORD # authentication (in addition to the standard none). self.sendall("\x05\x02\x00\x02") else: # No username/password were entered, therefore we # only support connections with no authentication. self.sendall("\x05\x01\x00") # We'll receive the server's response to determine which # method was selected chosenauth = self.__recvall(2) if chosenauth[0] != "\x05": self.close() raise GeneralProxyError((1,_generalerrors[1])) # Check the chosen authentication method if chosenauth[1] == "\x00": # No authentication is required pass elif chosenauth[1] == "\x02": # Okay, we need to perform a basic username/password # authentication. self.sendall("\x01" + chr(len(self.__proxy[4])) + self.__proxy[4] + chr(len(self.proxy[5])) + self.__proxy[5]) authstat = self.__recvall(2) if authstat[0] != "\x01": # Bad response self.close() raise GeneralProxyError((1,_generalerrors[1])) if authstat[1] != "\x00": # Authentication failed self.close() raise Socks5AuthError,((3,_socks5autherrors[3])) # Authentication succeeded else: # Reaching here is always bad self.close() if chosenauth[1] == "\xFF": raise Socks5AuthError((2,_socks5autherrors[2])) else: raise GeneralProxyError((1,_generalerrors[1])) # Now we can request the actual connection req = "\x05\x01\x00" # If the given destination address is an IP address, we'll # use the IPv4 address request even if remote resolving was specified. try: ipaddr = socket.inet_aton(destaddr) req = req + "\x01" + ipaddr except socket.error: # Well it's not an IP number, so it's probably a DNS name. if self.__proxy[3]==True: # Resolve remotely ipaddr = None req = req + "\x03" + chr(len(destaddr)) + destaddr else: # Resolve locally ipaddr = socket.inet_aton(socket.gethostbyname(destaddr)) req = req + "\x01" + ipaddr req = req + struct.pack(">H",destport) self.sendall(req) # Get the response resp = self.__recvall(4) if resp[0] != "\x05": self.close() raise GeneralProxyError((1,_generalerrors[1])) elif resp[1] != "\x00": # Connection failed self.close() if ord(resp[1])<=8: raise Socks5Error(ord(resp[1]),_generalerrors[ord(resp[1])]) else: raise Socks5Error(9,_generalerrors[9]) # Get the bound address/port elif resp[3] == "\x01": boundaddr = self.__recvall(4) elif resp[3] == "\x03": resp = resp + self.recv(1) boundaddr = self.__recvall(resp[4]) else: self.close() raise GeneralProxyError((1,_generalerrors[1])) boundport = struct.unpack(">H",self.__recvall(2))[0] self.__proxysockname = (boundaddr,boundport) if ipaddr != None: self.__proxypeername = (socket.inet_ntoa(ipaddr),destport) else: self.__proxypeername = (destaddr,destport) def getproxysockname(self): """getsockname() -> address info Returns the bound IP address and port number at the proxy. """ return self.__proxysockname def getproxypeername(self): """getproxypeername() -> address info Returns the IP and port number of the proxy. """ return _orgsocket.getpeername(self) def getpeername(self): """getpeername() -> address info Returns the IP address and port number of the destination machine (note: getproxypeername returns the proxy) """ return self.__proxypeername def __negotiatesocks4(self,destaddr,destport): """__negotiatesocks4(self,destaddr,destport) Negotiates a connection through a SOCKS4 server. """ # Check if the destination address provided is an IP address rmtrslv = False try: ipaddr = socket.inet_aton(destaddr) except socket.error: # It's a DNS name. Check where it should be resolved. if self.__proxy[3]==True: ipaddr = "\x00\x00\x00\x01" rmtrslv = True else: ipaddr = socket.inet_aton(socket.gethostbyname(destaddr)) # Construct the request packet req = "\x04\x01" + struct.pack(">H",destport) + ipaddr # The username parameter is considered userid for SOCKS4 if self.__proxy[4] != None: req = req + self.__proxy[4] req = req + "\x00" # DNS name if remote resolving is required # NOTE: This is actually an extension to the SOCKS4 protocol # called SOCKS4A and may not be supported in all cases. if rmtrslv==True: req = req + destaddr + "\x00" self.sendall(req) # Get the response from the server resp = self.__recvall(8) if resp[0] != "\x00": # Bad data self.close() raise GeneralProxyError((1,_generalerrors[1])) if resp[1] != "\x5A": # Server returned an error self.close() if ord(resp[1]) in (91,92,93): self.close() raise Socks4Error((ord(resp[1]),_socks4errors[ord(resp[1])-90])) else: raise Socks4Error((94,_socks4errors[4])) # Get the bound address/port self.__proxysockname = (socket.inet_ntoa(resp[4:]),struct.unpack(">H",resp[2:4])[0]) if rmtrslv != None: self.__proxypeername = (socket.inet_ntoa(ipaddr),destport) else: self.__proxypeername = (destaddr,destport) def __negotiatehttp(self,destaddr,destport): """__negotiatehttp(self,destaddr,destport) Negotiates a connection through an HTTP server. """ # If we need to resolve locally, we do this now if self.__proxy[3] == False: addr = socket.gethostbyname(destaddr) else: addr = destaddr self.sendall("CONNECT " + addr + ":" + str(destport) + " HTTP/1.1\r\n" + "Host: " + destaddr + "\r\n\r\n") # We read the response until we get the string "\r\n\r\n" resp = self.recv(1) while resp.find("\r\n\r\n")==-1: resp = resp + self.recv(1) # We just need the first line to check if the connection # was successful statusline = resp.splitlines()[0].split(" ",2) if statusline[0] not in ("HTTP/1.0","HTTP/1.1"): self.close() raise GeneralProxyError((1,_generalerrors[1])) try: statuscode = int(statusline[1]) except ValueError: self.close() raise GeneralProxyError((1,_generalerrors[1])) if statuscode != 200: self.close() raise HTTPError((statuscode,statusline[2])) self.__proxysockname = ("0.0.0.0",0) self.__proxypeername = (addr,destport) def connect(self,destpair): """connect(self,despair) Connects to the specified destination through a proxy. destpar - A tuple of the IP/DNS address and the port number. (identical to socket's connect). To select the proxy server use setproxy(). """ # Do a minimal input check first if (type(destpair) in (list,tuple)==False) or (len(destpair)<2) or (type(destpair[0])!=str) or (type(destpair[1])!=int): raise GeneralProxyError((5,_generalerrors[5])) if self.__proxy[0] == PROXY_TYPE_SOCKS5: if self.__proxy[2] != None: portnum = self.__proxy[2] else: portnum = 1080 _orgsocket.connect(self,(self.__proxy[1],portnum)) self.__negotiatesocks5(destpair[0],destpair[1]) elif self.__proxy[0] == PROXY_TYPE_SOCKS4: if self.__proxy[2] != None: portnum = self.__proxy[2] else: portnum = 1080 _orgsocket.connect(self,(self.__proxy[1],portnum)) self.__negotiatesocks4(destpair[0],destpair[1]) elif self.__proxy[0] == PROXY_TYPE_HTTP: if self.__proxy[2] != None: portnum = self.__proxy[2] else: portnum = 8080 _orgsocket.connect(self,(self.__proxy[1],portnum)) self.__negotiatehttp(destpair[0],destpair[1]) elif self.__proxy[0] == None: _orgsocket.connect(self,(destpair[0],destpair[1])) else: raise GeneralProxyError((4,_generalerrors[4])) dap-2.2.6.7/dap/util/wsgi_intercept.py0000644000175000017500000002473510774147131017505 0ustar robertoroberto""" wsgi_intercept.WSGI_HTTPConnection is a replacement for httplib.HTTPConnection that intercepts certain HTTP connections into a WSGI application. Use 'add_wsgi_intercept' and 'remove_wsgi_intercept' to control this behavior. """ import sys from httplib import HTTPConnection from cStringIO import StringIO import traceback debuglevel = 0 # 1 basic # 2 verbose #### # # Specify which hosts/ports to target for interception to a given WSGI app. # # For simplicity's sake, intercept ENTIRE host/port combinations; # intercepting only specific URL subtrees gets complicated, because we don't # have that information in the HTTPConnection.connect() function that does the # redirection. # # format: key=(host, port), value=(create_app, top_url) # # (top_url becomes the SCRIPT_NAME) _wsgi_intercept = {} def add_wsgi_intercept(host, port, app_create_fn, script_name=''): """ Add a WSGI intercept call for host:port, using the app returned by app_create_fn with a SCRIPT_NAME of 'script_name' (default ''). """ _wsgi_intercept[(host, port)] = (app_create_fn, script_name) def remove_wsgi_intercept(host, port): """ Remove the WSGI intercept call for (host, port). """ key = (host, port) if _wsgi_intercept.has_key(key): del _wsgi_intercept[key] # # make_environ: behave like a Web server. Take in 'input', and behave # as if you're bound to 'host' and 'port'; build an environment dict # for the WSGI app. # # This is where the magic happens, folks. # def make_environ(inp, host, port, script_name): """ Take 'inp' as if it were HTTP-speak being received on host:port, and parse it into a WSGI-ok environment dictionary. Return the dictionary. Set 'SCRIPT_NAME' from the 'script_name' input, and, if present, remove it from the beginning of the PATH_INFO variable. """ # # parse the input up to the first blank line (or its end). # environ = {} method_line = inp.readline() content_type = None content_length = None cookies = [] for line in inp: if not line.strip(): break k, v = line.strip().split(':', 1) v = v.lstrip() # # take care of special headers, and for the rest, put them # into the environ with HTTP_ in front. # if k.lower() == 'content-type': content_type = v elif k.lower() == 'content-length': content_length = v elif k.lower() == 'cookie' or k.lower() == 'cookie2': cookies.append(v) else: h = k.upper() h = h.replace('-', '_') environ['HTTP_' + h] = v if debuglevel >= 2: print 'HEADER:', k, v # # decode the method line # if debuglevel >= 2: print 'METHOD LINE:', method_line method, url, protocol = method_line.split(' ') # clean the script_name off of the url, if it's there. if not url.startswith(script_name): script_name = '' # @CTB what to do -- bad URL. scrap? else: url = url[len(script_name):] url = url.split('?', 1) path_info = url[0] query_string = "" if len(url) == 2: query_string = url[1] if debuglevel: print "method: %s; script_name: %s; path_info: %s; query_string: %s" % (method, script_name, path_info, query_string) r = inp.read() inp = StringIO(r) # # fill out our dictionary. # environ.update({ "wsgi.version" : (1,0), "wsgi.url_scheme": "http", "wsgi.input" : inp, # to read for POSTs "wsgi.errors" : StringIO(), "wsgi.multithread" : 0, "wsgi.multiprocess" : 0, "wsgi.run_once" : 0, "REQUEST_METHOD" : method, "SCRIPT_NAME" : script_name, "PATH_INFO" : path_info, "SERVER_NAME" : host, "SERVER_PORT" : str(port), "SERVER_PROTOCOL" : protocol, "REMOTE_ADDRESS" : '127.0.0.1', }) # # query_string, content_type & length are optional. # if query_string: environ['QUERY_STRING'] = query_string if content_type: environ['CONTENT_TYPE'] = content_type if debuglevel >= 2: print 'CONTENT-TYPE:', content_type if content_length: environ['CONTENT_LENGTH'] = content_length if debuglevel >= 2: print 'CONTENT-LENGTH:', content_length # # handle cookies. # if cookies: environ['HTTP_COOKIE'] = "; ".join(cookies) if debuglevel: print 'WSGI environ dictionary:', environ return environ # # fake socket for WSGI intercept stuff. # class wsgi_fake_socket: """ Handle HTTP traffic and stuff into a WSGI application object instead. Note that this class assumes: 1. 'makefile' is called (by the response class) only after all of the data has been sent to the socket by the request class; 2. non-persistent (i.e. non-HTTP/1.1) connections. """ def __init__(self, app, host, port, script_name): self.app = app # WSGI app object self.host = host self.port = port self.script_name = script_name # SCRIPT_NAME (app mount point) self.inp = StringIO() # stuff written into this "socket" self.results = None # results from running the app self.output = StringIO() # all output from the app, incl headers def makefile(self, *args, **kwargs): """ 'makefile' is called by the HTTPResponse class once all of the data has been written. So, in this interceptor class, we need to: 1. build a start_response function that grabs all the headers returned by the WSGI app; 2. create a wsgi.input file object 'inp', containing all of the traffic; 3. build an environment dict out of the traffic in inp; 4. run the WSGI app & grab the result object; 5. concatenate & return the result(s) read from the result object. @CTB: 'start_response' should return a function that writes directly to self.result, too. """ # dynamically construct the start_response function for no good reason. def start_response(status, headers, exc_info=None): # construct the HTTP request. self.output.write("HTTP/1.0 " + status + "\n") for k, v in headers: self.output.write('%s: %s\n' % (k, v,)) self.output.write('\n') # construct the wsgi.input file from everything that's been # written to this "socket". inp = StringIO(self.inp.getvalue()) # build the environ dictionary. environ = make_environ(inp, self.host, self.port, self.script_name) # run the application. self.result = self.app(environ, start_response) ### # read all of the results. for data in self.result: self.output.write(data) if debuglevel >= 2: print "***", self.output.getvalue(), "***" # return the concatenated results. return StringIO(self.output.getvalue()) def sendall(self, str): """ Save all the traffic to self.inp. """ if debuglevel >= 2: print ">>>", str, ">>>" self.inp.write(str) def close(self): "Do nothing, for now." pass # # WSGI_HTTPConnection # class WSGI_HTTPConnection(HTTPConnection): """ Intercept all traffic to certain hosts & redirect into a WSGI application object. """ def get_app(self, host, port): """ Return the app object for the given (host, port). """ key = (host, int(port)) app, script_name = None, None if _wsgi_intercept.has_key(key): (app_fn, script_name) = _wsgi_intercept[key] app = app_fn() return app, script_name def connect(self): """ Override the connect() function to intercept calls to certain host/ports. """ if debuglevel: sys.stderr.write('connect: %s, %s\n' % (self.host, self.port,)) try: (app, script_name) = self.get_app(self.host, self.port) if app: if debuglevel: sys.stderr.write('INTERCEPTING call to %s:%s\n' % \ (self.host, self.port,)) self.sock = wsgi_fake_socket(app, self.host, self.port, script_name) else: HTTPConnection.connect(self) except Exception, e: if debuglevel: # intercept & print out tracebacks traceback.print_exc() raise ### DEBUGGING CODE -- to help me figure out communications stuff. ### # (ignore me, please) ''' import socket class file_wrapper: def __init__(self, fp): self.fp = fp def readline(self): d = self.fp.readline() if debuglevel: print 'file_wrapper readline:', d return d def __iter__(self): return self def next(self): d = self.fp.next() if debuglevel: print 'file_wrapper next:', d return d def read(self, *args): d = self.fp.read(*args) if debuglevel: print 'file_wrapper read:', d return d def close(self): if debuglevel: print 'file_wrapper close' self.fp.close() class intercept_socket: """ A socket that intercepts everything written to it & read from it. """ def __init__(self): for res in socket.getaddrinfo("floating.caltech.edu", 80, 0, socket.SOCK_STREAM): af, socktype, proto, canonname, sa = res self.sock = socket.socket(af, socktype, proto) self._open = True self.sock.connect(sa) break def makefile(self, *args, **kwargs): fp = self.sock.makefile('rb', 0) return file_wrapper(fp) def sendall(self, str): if not self._open: raise Exception return self.sock.sendall(str) def close(self): self._open = False ''' dap-2.2.6.7/dap/helper.py0000644000175000017500000006576510774147131014771 0ustar robertoroberto"""Helper functions. These are generic functions used mostly for writing plugins. """ __author__ = "Roberto De Almeida " import sys import re import operator import itertools import copy from urllib import quote, unquote from dap.dtypes import * from dap.dtypes import _basetypes from dap.exceptions import ConstraintExpressionError from dap.lib import isiterable from dap.util.safeeval import expr_eval from dap.util.ordereddict import odict def constrain(dataset, constraints): """A simple example. We create a dataset holding three variables: >>> dataset = DatasetType(name='foo') >>> dataset['a'] = BaseType(name='a', type='Byte') >>> dataset['b'] = BaseType(name='b', type='Byte') >>> dataset['c'] = BaseType(name='c', type='Byte') Now we give it a CE requesting only the variables ``a`` and ``b``: >>> dataset2 = constrain(dataset, 'a,b') >>> print dataset2 #doctest: +ELLIPSIS {'a': , 'b': } We can also request the variables in a different order: >>> dataset2 = constrain(dataset, 'b,a') >>> print dataset2 #doctest: +ELLIPSIS {'b': , 'a': } Another example. A dataset with two structures ``a`` and ``b``: >>> dataset = DatasetType(name='foo') >>> dataset['a'] = StructureType(name='a') >>> dataset['a']['a1'] = BaseType(name='a1', type='Byte') >>> dataset['b'] = StructureType(name='b') >>> dataset['b']['b1'] = BaseType(name='b1', type='Byte') >>> dataset['b']['b2'] = BaseType(name='b2', type='Byte') If we request the structure ``b`` we should get it complete: >>> dataset2 = constrain(dataset, 'a.a1,b') >>> print dataset2 #doctest: +ELLIPSIS {'a': {'a1': }, 'b': {'b1': , 'b2': }} >>> dataset2 = constrain(dataset, 'b.b1') >>> print dataset2 #doctest: +ELLIPSIS {'b': {'b1': }} Arrays can be sliced. Here we have a ``(2,3)`` array: >>> dataset = DatasetType(name='foo') >>> from numpy import array >>> data = array([1,2,3,4,5,6]) >>> data.shape = (2,3) >>> dataset['array'] = ArrayType(data=data, name='array', shape=(2,3), type='Int32') >>> dataset2 = constrain(dataset, 'array') >>> from dap.server import SimpleHandler >>> headers, output = SimpleHandler(dataset).dds() >>> print ''.join(output) Dataset { Int32 array[2][3]; } foo; >>> print dataset2['array'].data [[1 2 3] [4 5 6]] But we request only part of it: >>> dataset2 = constrain(dataset, 'array[0:1:1][0:1:1]') >>> headers, output = SimpleHandler(dataset2).dds() >>> print ''.join(output) Dataset { Int32 array[2][2]; } foo; >>> print dataset2['array'].data [[1 2] [4 5]] The same is valid for grids: >>> dataset['grid'] = GridType(name='grid') >>> data = array([1,2,3,4,5,6]) >>> data.shape = (2,3) >>> dataset['grid'].array = ArrayType(name='grid', data=data, shape=(2,3), dimensions=('x', 'y')) >>> dataset['grid'].maps['x'] = ArrayType(name='x', data=array([1,2]), shape=(2,)) >>> dataset['grid'].maps['y'] = ArrayType(name='y', data=array([1,2,3]), shape=(3,)) >>> dataset._set_id() >>> headers, output = SimpleHandler(dataset).dds() >>> print ''.join(output) Dataset { Int32 array[2][3]; Grid { Array: Int32 grid[x = 2][y = 3]; Maps: Int32 x[x = 2]; Int32 y[y = 3]; } grid; } foo; >>> dataset2 = constrain(dataset, 'grid[0:1:0][0:1:0]') >>> headers, output = SimpleHandler(dataset2).dds() >>> print ''.join(output) Dataset { Grid { Array: Int32 grid[x = 1][y = 1]; Maps: Int32 x[x = 1]; Int32 y[y = 1]; } grid; } foo; >>> headers, output = SimpleHandler(dataset2).ascii() >>> print ''.join(output) Dataset { Grid { Array: Int32 grid[x = 1][y = 1]; Maps: Int32 x[x = 1]; Int32 y[y = 1]; } grid; } foo; --------------------------------------------- grid.grid [0][0] 1 grid.x [0] 1 grid.y [0] 1 Selecting a map from a Grid should return a structure: >>> dataset3 = constrain(dataset, 'grid.x') >>> headers, output = SimpleHandler(dataset3).dds() >>> print ''.join(output) Dataset { Structure { Int32 x[x = 2]; } grid; } foo; Short notation also works: >>> dataset3 = constrain(dataset, 'x') >>> headers, output = SimpleHandler(dataset3).dds() >>> print ''.join(output) Dataset { Structure { Int32 x[x = 2]; } grid; } foo; It also works with Sequences: >>> dataset = DatasetType(name='foo') >>> dataset['seq'] = SequenceType(name='seq') >>> dataset['seq']['a'] = BaseType(name='a') >>> dataset['seq']['b'] = BaseType(name='b') >>> dataset['seq']['a'].data = range(5) >>> dataset['seq']['b'].data = range(5,10) >>> for i in dataset['seq'].data: ... print i (0, 5) (1, 6) (2, 7) (3, 8) (4, 9) >>> dataset2 = constrain(dataset, 'seq.a') >>> for i in dataset2['seq'].data: ... print i (0,) (1,) (2,) (3,) (4,) >>> dataset2 = constrain(dataset, 'seq.b') >>> for i in dataset2['seq'].data: ... print i (5,) (6,) (7,) (8,) (9,) >>> dataset2 = constrain(dataset, 'seq.b,seq.a') >>> for i in dataset2['seq'].data: ... print i (5, 0) (6, 1) (7, 2) (8, 3) (9, 4) The function also parses selection expressions. Let's create a dataset with sequential data: >>> dataset = DatasetType(name='foo') >>> dataset['seq'] = SequenceType(name='seq') >>> dataset['seq']['index'] = BaseType(name='index', type='Int32') >>> dataset['seq']['index'].data = [10, 11, 12, 13] >>> dataset['seq']['temperature'] = BaseType(name='temperature', type='Float32') >>> dataset['seq']['temperature'].data = [17.2, 15.1, 15.3, 15.1] >>> dataset['seq']['site'] = BaseType(name='site', type='String') >>> dataset['seq']['site'].data = ['Diamond_St', 'Blacktail_Loop', 'Platinum_St', 'Kodiak_Trail'] Here's the data: >>> for i in dataset['seq'].data: ... print i (10, 17.199999999999999, 'Diamond_St') (11, 15.1, 'Blacktail_Loop') (12, 15.300000000000001, 'Platinum_St') (13, 15.1, 'Kodiak_Trail') Now suppose we only want data where ``index`` is greater than 11: >>> dataset2 = constrain(dataset, 'seq&seq.index>11') >>> for i in dataset2['seq'].data: ... print i (12, 15.300000000000001, 'Platinum_St') (13, 15.1, 'Kodiak_Trail') We can request only a few variables: >>> dataset2 = constrain(dataset, 'seq.site&seq.index>11') >>> for i in dataset2['seq'].data: ... print i ('Platinum_St',) ('Kodiak_Trail',) A few more tests: >>> dataset = DatasetType(name='foo') >>> dataset['a'] = StructureType(name='a') >>> dataset['a']['shn'] = BaseType(name='shn') >>> dataset['b'] = StructureType(name='b') >>> dataset['b']['shn'] = BaseType(name='shn') >>> dataset2 = constrain(dataset, 'a.shn') >>> print dataset2 #doctest: +ELLIPSIS {'a': {'shn': }} >>> dataset3 = constrain(dataset, 'shn') Traceback (most recent call last): ... ConstraintExpressionError: 'Ambiguous shorthand notation request: shn' >>> dataset['shn'] = BaseType(name='shn') >>> dataset3 = constrain(dataset, 'shn') >>> print dataset3 #doctest: +ELLIPSIS {'shn': } """ # Parse constraints. fields, queries = parse_querystring(constraints) # Ids and names are used to check that requests made using the # shorthand notation are not ambiguous. Used names are stored to # make sure that at most only a single variables is returned from # a given name. ids = [var.id for var in walk(dataset)] names = [] new = DatasetType(name=dataset.name, attributes=dataset.attributes.copy()) new = build(dataset, new, fields, queries, ids, names) return new def build(dapvar, new, fields, queries, ids, names): vars_ = fields.keys() order = [] for var in dapvar.walk(): # Make a copy of the variable, so that later we can possibly add it # to the dataset we're building (that's why it's a candidate). candidate = copy.deepcopy(var) # We first filter the data in sequences. This has to be done # before variables are removed, since we can select values based # on conditions on *other* variables. Eg: seq.a where seq.b > 1 if queries and isinstance(candidate, SequenceType): # Filter candidate on the server-side, since the data may be # proxied using ``dap.proxy.Proxy``. candidate = candidate.filter(*queries) # And then filter on the client side. candidate = filter_(candidate, queries) # If the variable was requested, either by id or name, or if no # variables were requested, we simply add this candidate to the # dataset we're building. if not vars_ or candidate.id in vars_ or (candidate.name in vars_ and candidate.name not in ids): new[candidate.name] = candidate # Check if requests done using shn are not ambiguous. if vars_ and candidate.id not in vars_: # request by shn if candidate.name in names: raise ConstraintExpressionError("Ambiguous shorthand notation request: %s" % candidate.name) names.append(candidate.name) # We also need to store the order in which the variables were # requested. Later, we'll rearrange the variables in our built # dataset in the correct order. if vars_: if candidate.id in vars_: index = vars_.index(candidate.id) else: index = vars_.index(candidate.name) order.append((index, candidate.name)) # If the variable was not requested, but it's a constructor, it's # possible that one of its children has been requested. We apply # the algorithm recursively on the variable. elif not isinstance(var, BaseType): # We clear the candidate after storing a copy with the filtered # data and children. We will then append the requested children # to the cleared candidate. ccopy = copy.deepcopy(candidate) if isinstance(candidate, StructureType): candidate.clear() else: # If the variable is a grid we should return it as a # structure with the requested fields. parent = candidate._id[:-len(candidate.name)-1] candidate = StructureType(name=candidate.name, attributes=candidate.attributes.copy()) candidate._set_id(parent) # Check for requested children. candidate = build(ccopy, candidate, fields, queries, ids, names) # If the candidate has any keys, ie, stored variables, we add # it to the dataset we are building. if candidate.keys(): new[candidate.name] = candidate # Check if we need to apply a slice in the variable. slice_ = fields.get(candidate.id) or fields.get(candidate.name) if slice_: candidate = slicevar(candidate, slice_) # Sort variables according to order of requested variables. if len(order) > 1: order.sort() new._keys = [item[1] for item in order] return new def filter_(dapvar, queries): # Get only the queries related to this variable. queries_ = [q for q in queries if q.startswith(dapvar.id)] if queries_: # Build the filter and apply it to the data. ids = [var.id for var in dapvar.values()] f = buildfilter(queries_, ids) data = itertools.ifilter(f, dapvar.data) # Set the data in the stored variables. data = list(data) dapvar.data = data return dapvar def slicevar(dapvar, slice_): if slice_ != (slice(None),): dapvar.data = dapvar.data[slice_] try: dapvar.shape = getattr(dapvar.data, 'shape', (len(dapvar.data),)) except TypeError: pass if isinstance(dapvar, GridType): if not isiterable(slice_): slice_ = (slice_,) # Slice the maps. for map_,mapslice in zip(dapvar.maps.values(), slice_): map_.data = map_.data[mapslice] map_.shape = map_.data.shape return dapvar def order(dataset, fields): """ Order a given dataset according to the requested order. >>> d = DatasetType(name='d') >>> d['a'] = BaseType(name='a') >>> d['b'] = BaseType(name='b') >>> d['c'] = SequenceType(name='c') >>> d['c']['d'] = BaseType(name='d') >>> d['c']['e'] = BaseType(name='e') >>> print order(d, 'b,c.e,c.d,a'.split(',')) #doctest: +ELLIPSIS {'b': , 'c': {'e': , 'd': }, 'a': } >>> print order(d, 'c.e,c.d,a'.split(',')) #doctest: +ELLIPSIS {'c': {'e': , 'd': }, 'a': , 'b': } >>> print order(d, 'b,c,a'.split(',')) #doctest: +ELLIPSIS {'b': , 'c': {'d': , 'e': }, 'a': } """ # Order the dataset. dataset = copy.copy(dataset) orders = [] n = len(dataset._keys) for var in dataset.walk(): # Search for id first. fields_ = [field[:len(var.id)] for field in fields] if var.id in fields_: index = fields_.index(var.id) # Else search by name. elif var.name in fields: index = fields.index(var.name) # Else preserve original order. else: index = n + dataset._keys.index(var.name) orders.append((index, var.name)) # Sort children. if isinstance(var, StructureType): dataset[var.name] = order(var, fields) # Sort dataset. if len(orders) > 1: orders.sort() dataset._keys = [item[1] for item in orders] return dataset def walk(dapvar): """ Iterate over all variables, including dapvar. """ yield dapvar try: for child in dapvar.walk(): for var in walk(child): yield var except: pass def getslice(hyperslab, shape=None): """Parse a hyperslab. Parse a hyperslab to a slice according to variable shape. The hyperslab follows the DAP specification, and ommited dimensions are returned in their entirety. >>> getslice('[0:1:2][0:1:2]') (slice(0, 3, 1), slice(0, 3, 1)) >>> getslice('[0:2][0:2]') (slice(0, 3, 1), slice(0, 3, 1)) >>> getslice('[0][2]') (slice(0, 1, 1), slice(2, 3, 1)) >>> getslice('[0:1:1]') (slice(0, 2, 1),) >>> getslice('[0:2:1]') (slice(0, 2, 2),) >>> getslice('') (slice(None, None, None),) """ # Backwards compatibility. In pydap <= 2.2.3 the ``fields`` dict from # helper.parse_querystring returned the slices as strings (instead of # Python slices). These strings had to be passed to getslice to get a # Python slice. Old plugins still do this, but with pydap >= 2.2.4 # they are already passing the slices, so we simply return them. if not isinstance(hyperslab, basestring): return hyperslab or slice(None) if hyperslab: output = [] dimslices = hyperslab[1:-1].split('][') for dimslice in dimslices: start, size, step = _getsize(dimslice) output.append(slice(start, start+size, step)) output = tuple(output) else: output = (slice(None),) return output def _getsize(dimslice): """Parse a dimension from a hyperslab. Calculates the start, size and step from a DAP formatted hyperslab. >>> _getsize('0:1:9') (0, 10, 1) >>> _getsize('0:2:9') (0, 10, 2) >>> _getsize('0') (0, 1, 1) >>> _getsize('0:9') (0, 10, 1) """ size = dimslice.split(':') start = int(size[0]) if len(size) == 1: stop = start step = 1 elif len(size) == 2: stop = int(size[1]) step = 1 elif len(size) == 3: step = int(size[1]) stop = int(size[2]) else: raise ConstraintExpressionError('Invalid hyperslab: %s.' % dimslice) size = (stop-start) + 1 return start, size, step def buildfilter(queries, vars_): """This function is a filter builder. Given a list of DAP formatted queries and a list of variable names, this function returns a dynamic filter function to filter rows. From the example in the DAP specification: >>> vars_ = ['index', 'temperature', 'site'] >>> data = [] >>> data.append([10, 17.2, 'Diamond_St']) >>> data.append([11, 15.1, 'Blacktail_Loop']) >>> data.append([12, 15.3, 'Platinum_St']) >>> data.append([13, 15.1, 'Kodiak_Trail']) Rows where index is greater-than-or-equal 11: >>> f = buildfilter(['index>=11'], vars_) >>> for line in itertools.ifilter(f, data): ... print line [11, 15.1, 'Blacktail_Loop'] [12, 15.300000000000001, 'Platinum_St'] [13, 15.1, 'Kodiak_Trail'] Rows where site ends with '_St': >>> f = buildfilter(['site=~".*_St"'], vars_) >>> for line in itertools.ifilter(f, data): ... print line [10, 17.199999999999999, 'Diamond_St'] [12, 15.300000000000001, 'Platinum_St'] Index greater-or-equal-than 11 AND site ends with '_St': >>> f = buildfilter(['site=~".*_St"', 'index>=11'], vars_) >>> for line in itertools.ifilter(f, data): ... print line [12, 15.300000000000001, 'Platinum_St'] Site is either 'Diamond_St' OR 'Blacktail_Loop': >>> f = buildfilter(['site={"Diamond_St", "Blacktail_Loop"}'], vars_) >>> for line in itertools.ifilter(f, data): ... print line [10, 17.199999999999999, 'Diamond_St'] [11, 15.1, 'Blacktail_Loop'] Index is either 10 OR 12: >>> f = buildfilter(['index={10, 12}'], vars_) >>> for line in itertools.ifilter(f, data): ... print line [10, 17.199999999999999, 'Diamond_St'] [12, 15.300000000000001, 'Platinum_St'] Python is great, isn't it? :) """ filters = [] p = re.compile(r'''^ # Start of selection {? # Optional { for multi-valued constants (?P.*?) # Anything }? # Closing } (?P<=|>=|!=|=~|>|<|=) # Operators {? # { (?P.*?) # Anything }? # } $ # EOL ''', re.VERBOSE) for query in queries: m = p.match(query) if not m: raise ConstraintExpressionError('Invalid constraint expression: %s.' % query) # Functions associated with each operator. op = {'<' : operator.lt, '>' : operator.gt, '!=': operator.ne, '=' : operator.eq, '>=': operator.ge, '<=': operator.le, '=~': lambda a,b: re.match(b,a), }[m.group('op')] # Allow multiple comparisons in one line. Python rulez! op = multicomp(op) # Build the filter for the first variable. if m.group('var1') in vars_: i = vars_.index(m.group('var1')) var1 = lambda L, i=i: operator.getitem(L, i) # Build the filter for the second variable. It could be either # a name or a constant. if m.group('var2') in vars_: i = vars_.index(m.group('var2')) var2 = lambda L, i=i: operator.getitem(L, i) else: var2 = lambda x, m=m: expr_eval(m.group('var2')) # This is the filter. We apply the function (op) to the variable # filters (var1 and var2). filter0 = lambda x, op=op, var1=var1, var2=var2: op(var1(x), var2(x)) filters.append(filter0) if filters: # We have to join all the filters that were built, using the AND # operator. Believe me, this line does exactly that. # # You are not expected to understand this. filter0 = lambda i: reduce(lambda x,y: x and y, [f(i) for f in filters]) else: filter0 = bool return filter0 def multicomp(function): """Multiple OR comparisons. Given f(a,b), this function returns a new function g(a,b) which performs multiple OR comparisons if b is a tuple. >>> a = 1 >>> b = (0, 1, 2) >>> operator.lt = multicomp(operator.lt) >>> operator.lt(a, b) True """ def f(a, b): if isinstance(b, tuple): for i in b: # Return True if any comparison is True. if function(a, i): return True return False else: return function(a, b) return f def fix_slice(dims, index): """Fix incomplete slices or slices with ellipsis. The behaviour of this function was reversed-engineered from numpy. >>> fix_slice(3, (0, Ellipsis, 0)) (0, slice(None, None, None), 0) >>> fix_slice(4, (0, Ellipsis, 0)) (0, slice(None, None, None), slice(None, None, None), 0) >>> fix_slice(4, (0, 0, Ellipsis, 0)) (0, 0, slice(None, None, None), 0) >>> fix_slice(5, (0, Ellipsis, 0)) (0, slice(None, None, None), slice(None, None, None), slice(None, None, None), 0) >>> fix_slice(5, (0, 0, Ellipsis, 0)) (0, 0, slice(None, None, None), slice(None, None, None), 0) >>> fix_slice(5, (0, Ellipsis, 0, Ellipsis)) (0, slice(None, None, None), slice(None, None, None), 0, slice(None, None, None)) >>> fix_slice(4, slice(None, None, None)) (slice(None, None, None), slice(None, None, None), slice(None, None, None), slice(None, None, None)) >>> fix_slice(4, (slice(None, None, None), 0)) (slice(None, None, None), 0, slice(None, None, None), slice(None, None, None)) """ if not isinstance(index, tuple): index = (index,) out = [] length = len(index) for slice_ in index: if slice_ is Ellipsis: out.extend([slice(None)] * (dims - length + 1)) length += (dims - length) else: out.append(slice_) index = tuple(out) if len(index) < dims: index += (slice(None),) * (dims - len(index)) return index def lenslice(slice_): """ Return the number of values associated with a slice. By Bob Drach. """ step = slice_.step if step is None: step = 1 if step > 0: start = slice_.start stop = slice_.stop else: start = slice_.stop stop = slice_.start step = -step return ((stop-start-1)/step + 1) def parse_querystring(query): """ Parse a query_string returning the requested variables, dimensions, and CEs. >>> parse_querystring('a,b') ({'a': (slice(None, None, None),), 'b': (slice(None, None, None),)}, []) >>> parse_querystring('a[0],b[1]') ({'a': (slice(0, 1, 1),), 'b': (slice(1, 2, 1),)}, []) >>> parse_querystring('a[0],b[1]&foo.bar>1') ({'a': (slice(0, 1, 1),), 'b': (slice(1, 2, 1),)}, ['foo.bar>1']) >>> parse_querystring('a[0],b[1]&foo.bar>1&LAYERS=SST') ({'a': (slice(0, 1, 1),), 'b': (slice(1, 2, 1),)}, ['foo.bar>1', 'LAYERS=SST']) >>> parse_querystring('foo.bar>1&LAYERS=SST') ({}, ['foo.bar>1', 'LAYERS=SST']) """ if query is None: return {}, [] query = unquote(query) constraints = query.split('&') # Check if the first item is either a list of variables (projection) # or a selection. relops = ['<', '<=', '>', '>=', '=', '!=',' =~'] for relop in relops: if relop in constraints[0]: vars_ = [] queries = constraints[:] break else: vars_ = constraints[0].split(',') queries = constraints[1:] fields = odict() p = re.compile(r'(?P[^[]+)(?P(\[[^\]]+\])*)') for var in vars_: if var: # Check if the var has a slice. c = p.match(var).groupdict() id_ = quote(c['name']) fields[id_] = getslice(c['shape']) return fields, queries def escape_dods(dods, pad=''): """ Escape a DODS response. This is useful for debugging. You're probably spending too much time with pydap if you need to use this. """ if 'Data:\n' in dods: index = dods.index('Data:\n') + len('Data:\n') else: index = 0 dds = dods[:index] dods = dods[index:] out = [] for i, char in enumerate(dods): char = hex(ord(char)) char = char.replace('0x', '\\x') if len(char) < 4: char = char.replace('\\x', '\\x0') out.append(char) if pad and (i%4 == 3): out.append(pad) out = ''.join(out) out = out.replace(r'\x5a\x00\x00\x00', '') out = out.replace(r'\xa5\x00\x00\x00', '\n') return dds + out def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/server.py0000644000175000017500000000305311062605662014775 0ustar robertorobertoimport traceback from StringIO import StringIO from pkg_resources import iter_entry_points from dap.helper import constrain from dap.lib import escape, INDENT, __dap__ from dap.exceptions import DapError class BaseHandler(object): description = 'No description available' def __init__(self, filepath=None, environ=None): self.filepath = filepath self.environ = environ or {} def _parseconstraints(self, constraints=None): raise NotImplementedError("Subclasses must implement _parseconstraints") def error(self, info): """Error response.""" headers = [('Content-description', 'dods_error'), ('XDODS-Server', 'dods/%s' % '.'.join([str(i) for i in __dap__])), ('Content-type', 'text/plain'), ] type_, value, traceback_ = info f = StringIO() traceback.print_exception(type_, value, traceback_, file=f) msg = f.getvalue() output = ['Error {\n', INDENT, 'code = %s;\n' % getattr(type_, 'code', -1), INDENT, 'message = %s;\n' % escape(msg), '}'] return headers, output # Load responses from entry points. for ep in iter_entry_points("dap.response"): setattr(BaseHandler, ep.name, ep.load().build) class SimpleHandler(BaseHandler): def __init__(self, dataset, environ=None): self.dataset = dataset self.environ = environ or {} def _parseconstraints(self, constraints=None): return constrain(self.dataset, constraints) dap-2.2.6.7/dap/parsers/0000755000175000017500000000000011123574304014567 5ustar robertorobertodap-2.2.6.7/dap/parsers/das.py0000644000175000017500000002165510774147123015727 0ustar robertoroberto"""DAS parser. This module implements a DAS parser based on ``shlex.shlex``. """ __author__ = "Roberto De Almeida " import array import operator from shlex import shlex from dap import dtypes from dap.parsers import BaseParser from dap.util.safeeval import expr_eval # string.digits + string.ascii_letters + string.punctuation - {};[]:=" #WORDCHARS = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!#$%&\'()*+,-./<>?@\\^_`|~' WORDCHARS = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!#$%&\'()*+-./<>?@\\^_`|~' atomic_types = ('byte', 'int', 'uint', 'int16', 'uint16', 'int32', 'uint32', 'float32', 'float64', 'string', 'url', 'alias') class DASParser(BaseParser): r"""A parser for Dataset Attribute Structure. First we create a dataset and get its DAS: >>> from dap import dtypes >>> dataset = dtypes.DatasetType(name='test', attributes={'GLOBAL': {'name': 'teste'}}) >>> dataset['a'] = dtypes.BaseType(name='a', data=1, type='Int32', attributes={'pi': 3.1415}) >>> from dap.server import SimpleHandler >>> headers, output = SimpleHandler(dataset).das() >>> das_ = ''.join(output) >>> print das_ Attributes { GLOBAL { String name "teste"; } a { Float32 pi 3.1415; } } Now we build a new dataset from the DDS: >>> headers, output = SimpleHandler(dataset).dds() >>> dds_ = ''.join(output) >>> from dap.parsers.dds import DDSParser >>> dataset2 = DDSParser(dds_, '').parse() >>> headers, output = SimpleHandler(dataset2).dds() >>> print ''.join(output) Dataset { Int32 a; } test; The new dataset has no attributes, since it was built only from the DDS. We pass it to the DAS parser, together with the DAS: >>> dataset2 = DASParser(das_, '', dataset2).parse() >>> headers, output = SimpleHandler(dataset2).das() >>> print ''.join(output) Attributes { GLOBAL { String name "teste"; } a { Float32 pi 3.1415; } } The parser should accept additional metadata: >>> das_ = 'Attributes {\nGLOBAL {\nString name "teste";\n}\nMETADATA {\nMETAMETADATA {\nString foo "bar";\n}\n}\na {\nFloat32 pi 3.1415;\n}\n}' >>> dataset2 = DASParser(das_, '', dataset2).parse() >>> print dataset2.attributes {'GLOBAL': {'name': ['teste']}, 'METADATA': {'METAMETADATA': {'foo': ['bar']}}} >>> headers, output = SimpleHandler(dataset2).das() >>> print ''.join(output) Attributes { GLOBAL { String name "teste"; } METADATA { METAMETADATA { String foo "bar"; } } a { Float32 pi 3.1415; } } Checking a bug Rob Cermak found: >>> dataset3 = dtypes.DatasetType(name='dataset') >>> latLonCoordSys = dataset3['latLonCoordSys'] = dtypes.BaseType(name='latLonCoordSys', type='String', attributes={'_CoordinateAxes': "time lat long", 'DODS': {'strlen': 0}}) >>> headers, output = SimpleHandler(dataset3).das() >>> das_ = ''.join(output) >>> print das_ Attributes { latLonCoordSys { DODS { Int32 strlen 0; } String _CoordinateAxes "time lat long"; } } >>> headers, output = SimpleHandler(dataset3).dds() >>> dds_ = ''.join(output) >>> dataset4 = DDSParser(dds_, '').parse() >>> dataset4 = DASParser(das_, '', dataset4).parse() >>> headers, output = SimpleHandler(dataset4).das() >>> print ''.join(output) Attributes { latLonCoordSys { DODS { Int32 strlen 0; } String _CoordinateAxes "time lat long"; } } """ def __init__(self, das, url, dataset): shlex.__init__(self, das) self.url = url self._dataset = dataset # Attribute target when parsing. self._target = self._dataset # Add punctuation to words. #self.wordchars += '/_%.-+' self.wordchars = WORDCHARS def parse(self): """Parse the DAS and return a ``DatasetType`` object with the attributes populated.""" self._consume('attributes') self._consume('{') self._attrconts() self._consume('}') return self._dataset def _attrconts(self): while not self._check('}'): self._attrcont() def _attrcont(self): # Check for attributes or containers. if self._check(*atomic_types): name, values = self._attribute() self._target.attributes[name] = values # Fix attributes for grids. if isinstance(self._target, dtypes.GridType): for map_ in self._target.maps.values(): if not map_.attributes and map_.name in self._dataset: map_.attributes = self._dataset[map_.name].attributes.copy() else: self._container() def _container(self): name = self.get_token() self._consume('{') # Check for flat attributes, since it's relatively common. if '.' in name: names = name.split('.') old_target = self._target # Get the appropriate target: foo.bar should point to dataset['foo']['bar'] d = [old_target] d.extend(names) try: self._target = reduce(operator.getitem, d) except KeyError: # This happens in this dataset, because it's wrong. We just ignore the attributes. # http://motherlode.ucar.edu/cgi-bin/dods/DODS-3.2.1/nph-dods/dods/amsua15_2002.279_22852_0156_0342_GC.eos.das pass self._attrconts() self._target = old_target elif isinstance(self._target, dtypes.StructureType) and name in self._target.keys(): old_target = self._target self._target = old_target[name] self._attrconts() self._target = old_target else: self._target.attributes[name] = self._metadata() self._consume('}') def _metadata(self): output = {} while not self._check('}'): if self._check(*atomic_types): name, values = self._attribute() output[name] = values else: name = self.get_token() self._consume('{') output[name] = self._metadata() self._consume('}') return output def _attribute(self): # Get type and name. type_ = self.get_token() name = self.get_token() # Get values: value_1 (, value_n)* values = [] while not self._check(';'): value = self.get_token() if type_.lower() in ['string', 'url']: value = expr_eval(repr(value)) value = value.strip('"') elif type_.lower() == 'alias': # Support for Alias is not documented in the DAP spec. I based # this on the Java documentation from the OPeNDAP website at: # http://www.opendap.org/api/javaDocs/dods/dap/Alias.html if value.startswith('.'): tokens = value[1:].split('.') value = self._dataset else: tokens = value.split('.') value = self._target for token in tokens: if token in value: value = value[token] elif token: value = value.attributes.get(token, "Alias pointing to non-existing attribute.") break else: value = value.attributes elif type_.lower() == 'float32': # Convert to right precision; otherwise floats # are converted to Float64 automatically by # Python. value = array.array('f', [float(value)])[0] elif type_.lower() == 'float64': value = array.array('d', [float(value)])[0] else: value = int(value) values.append(value) if self._check(','): self._consume(',') self._consume(';') # Return single attributes as values, not list. if len(values) == 1: values = values[0] return name, values def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/parsers/__init__.py0000644000175000017500000000201710774147123016706 0ustar robertoroberto"""Parsing routines. This module implements parsers for the dataset descriptors: DAS/DDS, DDX, JSON, etc. """ __author__ = "Roberto De Almeida " import shlex class BaseParser(shlex.shlex): """A base parser with support functions. This module is a simple parser based on ``shlex.shlex``. It has support function for peeking and consuming tokens. """ def _consume(self, token): """Consume the specified token or raise exception.""" _token = self.get_token() if token.lower() == _token.lower(): return _token e = "%s Found '%s' (expected '%s')" % (self.error_leader(self.url), _token, token) raise Exception(e) def _peek(self): """Inspect the next token.""" _token = self.get_token() self.push_token(_token) return _token def _check(self, *tokens): """Check for token(s).""" _token = self._peek() if _token.lower() in [t.lower() for t in tokens]: return _token return None dap-2.2.6.7/dap/parsers/dds.py0000644000175000017500000001312410774147123015722 0ustar robertoroberto"""DDS parser. This module implements a DDS parser based on ``shlex.shlex``. """ __author__ = "Roberto De Almeida " from shlex import shlex from dap.dtypes import * from dap import proxy from dap.parsers import BaseParser # string.digits + string.ascii_letters + string.punctuation - {};[]:=" WORDCHARS = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!#$%&\'()*+-./<>?@\\^_`|~' atomic_types = ('byte', 'int', 'uint', 'int16', 'uint16', 'int32', 'uint32', 'float32', 'float64', 'string', 'url') constructors = ('grid', 'sequence', 'structure') class DDSParser(BaseParser): """A parser for Dataset Descriptor Structure. First we create a dataset and get its DDS: >>> dataset = DatasetType(name='test') >>> dataset['a'] = BaseType(name='a', data=1) >>> from dap.server import SimpleHandler >>> headers, output = SimpleHandler(dataset).dds() >>> dds = ''.join(output) >>> print dds Dataset { Int32 a; } test; Now we try to parse it. We'll get a second dataset that should have the same DDS: >>> dataset2 = DDSParser(dds, '').parse() >>> headers, output = SimpleHandler(dataset2).dds() >>> print ''.join(output) Dataset { Int32 a; } test; QED. """ def __init__(self, dds, url, cache=None, username=None, password=None): shlex.__init__(self, dds) self.url = url[:-4] # strip .dds from url # Info for the proxy. self.cache = cache self.username = username self.password = password # Add punctuation to words. self.wordchars = WORDCHARS def parse(self): """Parse the DDS and return a ``DatasetType`` object.""" return self._dataset() def _dataset(self): self._consume('Dataset') self._consume('{') dataset = DatasetType() # Scan type declarations. while self._check(*(atomic_types + constructors)): var = self._declaration() dataset[var.name] = var self._consume('}') # Scan the filename. dataset.name = self.get_token() self._consume(';') # Fix all ids. dataset._set_id() # Add Proxy to all BaseType instances. def walk(dapvar): for var in dapvar.walk(): if isinstance(var, BaseType): # Set the data. var.data = proxy.Proxy(self.url, var.id, getattr(var, 'shape', ()), var.type, cache=self.cache, username=self.username, password=self.password) else: walk(var) walk(dataset) return dataset def _declaration(self): if self._check('grid'): return self._grid() elif self._check('sequence'): return self._sequence() elif self._check('structure'): return self._structure() else: return self._base_declaration() def _base_declaration(self): type_ = self.get_token() name = self.get_token() # Get the dimensions, if any. shape, dimensions = self._dimensions() self._consume(';') if shape: var = ArrayType(name=name, shape=shape, dimensions=dimensions, type=type_) else: var = BaseType(name=name, type=type_) return var def _dimensions(self): shape = [] names = [] while not self._check(';'): self._consume('[') _token = self.get_token() if self._check('='): names.append(_token) self._consume('=') shape.append(int(self.get_token())) else: shape.append(int(_token)) self._consume(']') return tuple(shape), tuple(names) def _sequence(self): sequence = SequenceType() self._consume('sequence') self._consume('{') while 1: var = self._declaration() sequence[var.name] = var if self._check('}'): break self._consume('}') # Get the sequence name. sequence.name = self.get_token() self._consume(';') return sequence def _structure(self): structure = StructureType() self._consume('structure') self._consume('{') while 1: var = self._declaration() structure[var.name] = var if self._check('}'): break self._consume('}') # Get structure name. structure.name = self.get_token() self._consume(';') return structure def _grid(self): grid = GridType() self._consume('grid') self._consume('{') # Scan the array. self._consume('array') self._consume(':') var = self._base_declaration() grid.array = var # Scan the maps. self._consume('maps') self._consume(':') while 1: var = self._base_declaration() grid.maps[var.name] = var if self._check('}'): break self._consume('}') grid.name = self.get_token() self._consume(';') return grid def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() dap-2.2.6.7/dap/wsgi/0000755000175000017500000000000011123574304014061 5ustar robertorobertodap-2.2.6.7/dap/wsgi/application.py0000644000175000017500000001703211062605607016744 0ustar robertoroberto"""DAP server WSGI application. This module implements a DAP server over the WSGI specification. For more information, read http://www.python.org/peps/pep-0333.html. """ __author__ = "Roberto De Almeida " import sys import os import re import urllib from urlparse import urljoin from paste.httpexceptions import HTTPException, HTTPNotFound from paste.request import construct_url from paste.deploy.converters import asbool from paste.wsgilib import intercept_output from pkg_resources import iter_entry_points from Cheetah.Template import Template import dap.lib from dap.plugins.lib import loadplugins, loadhandler from dap.server import BaseHandler, SimpleHandler index_tmpl = """ $title

DODS directory for $location

#if $parent #end if #for $dir in $dirs: #set $dirname = '%s/' % $dir.split('/')[-1] #end for #for $file in $files: #set $filename = $file.split('/')[-1] #end for
Parent directory
$dirname
$filename[DDS][DAS]

pydap/$version © Roberto De Almeida

""" def make_app(global_conf, root, **kwargs): return DapServerApplication(root, **kwargs) class SimpleApplication(object): def __init__(self, dataset): self.dataset = dataset def __call__(self, environ, start_response): status = '200 OK' try: headers, output = self.handlerequest(environ) except HTTPException, exc: # Raise these exceptions so they get # captured by the middleware. status, headers, output = intercept_output(environ, exc) except: status = '500 Internal Error' self.handler = BaseHandler(environ=environ) headers, output = self.handler.error(sys.exc_info()) if environ.get('x-wsgiorg.throw_errors'): raise start_response(status, headers) for line in output: yield line try: self.handler.close() except: pass def handlerequest(self, environ): # Remove the leading '/'. request = environ.get('PATH_INFO', '').lstrip('/') request = urllib.unquote(request) if not request: # Show list of responses. message = ['%s' % (ep.name, ep.name) for ep in iter_entry_points("dap.response")] message = ' | '.join(message) self.handler = BaseHandler(environ=environ) response = self.handler.help(message=message) else: entity = request.split('.')[-1] self.handler = SimpleHandler(self.dataset, environ=environ) call = getattr(self.handler, entity) response = call(environ.get('QUERY_STRING', '')) return response class DapServerApplication(SimpleApplication): def __init__(self, root, name="pyDAP server", template=None, verbose=False, **kwargs): # Ensure that root ends with /. if not root.endswith('/'): root = '%s/' % root self.root = root self.name = name self.template = template # Set verbose option. dap.lib.VERBOSE = asbool(verbose) # Set additional keywords and ``throw_errors`` key. # http://wsgi.org/wsgi/Specifications/throw_errors if kwargs.get('x-wsgiorg.throw_errors'): kwargs['x-wsgiorg.throw_errors'] = asbool(kwargs['x-wsgiorg.throw_errors']) self.to_environ = kwargs # And load plugins for data handling. self.plugins = loadplugins(throw_errors=kwargs.get('x-wsgiorg.throw_errors', False)) def handlerequest(self, environ): # Update environ with user-defined keys. environ.update(self.to_environ) # Remove the leading '/'. request = environ.get('PATH_INFO', '').lstrip('/') request = urllib.unquote(request) # Special requests. if request in ['version', 'help']: self.handler = BaseHandler(environ=environ) call = getattr(self.handler, request) return call() request = os.path.join(self.root, request) # Directory listing. if os.path.exists(request) and os.path.isdir(request): return self.listfiles(request, environ) # Strip file suffix. entity = request.split('.')[-1] file_ = request[:-len(entity)-1] if not os.path.exists(file_): raise HTTPNotFound # 404 # Call appropriate plugin. self.handler = loadhandler(file_, environ, self.plugins) call = getattr(self.handler, entity) response = call(environ.get('QUERY_STRING')) return response def listfiles(self, dir, environ): headers = [('XDODS-Server', 'dods/%s' % '.'.join([str(i) for i in dap.lib.__dap__])), ('Content-Type', 'text/html'), ('Content-Encoding', 'utf-8'), ] location = construct_url(environ, with_query_string=False) title = 'DODS directory for %s' % location.rstrip('/') # Check which files are supported by plugins. res = [plugin.extensions for plugin in self.plugins] dirs, files = [], [] # Sort files. file_listing = os.listdir(dir) file_listing.sort() for item in file_listing: path = os.path.join(dir, item) url = urljoin(location, item) if os.path.isdir(path): dirs.append(url) else: # Add only supported files. for regexp in res: if re.match(regexp, url): files.append(url) break # Add parent dir. if dir != self.root: parent = '..' else: parent = None namespace = {'title': title, 'location': location, 'parent': parent, 'dirs': dirs, 'files': files, 'version': '.'.join([str(_) for _ in dap.lib.__version__]) } if self.template: index = os.path.join(self.template, 'index.tmpl') t = Template(file=index, searchList=[namespace], filter='EncodeUnicode') else: t = Template(index_tmpl, searchList=[namespace], filter='EncodeUnicode') output = [unicode(t).encode('utf-8')] return headers, output if __name__ == "__main__": import os import optparse from paste.httpserver import serve parser = optparse.OptionParser() parser.add_option('-p', '--port', dest='port', default=8080, type='int', help="port to serve on (default 8080)") options, args = parser.parse_args() # Get the app root. Defaults to current directory. pwd = os.environ['PWD'] if not len(args) == 1: root = pwd else: root = os.path.join(pwd, args[0]) app = DapServerApplication(root) serve(app, port=options.port, host='127.0.0.1') dap-2.2.6.7/dap/wsgi/proxy.py0000644000175000017500000000572310774147131015630 0ustar robertoroberto""" Pydap proxy. Works as a "frontend" to another DAP server. Normal responses are simply proxied, but additional ones (like JSON, KML, etc.) are generated by connecting to the server as a DAP client. Sample usage. Create a file called ``server.ini``:: [server:main] use = egg:Paste#http # Change to 0.0.0.0 to make public host = 127.0.0.1 port = 8080 [app:main] use = egg:dap#proxy url = http://nomad2.ncep.noaa.gov:9090/dods/sst responses = json kml wms verbose = 1 debug = 1 cache = .pydap-cache And run ``paster serve server.ini``. This will forward a request originally to:: http://localhost:8080/oiv2.dds To the URL:: http://nomad2.ncep.noaa.gov:9090/dods/sst/oiv2.dds But a request to:: http://localhost:8080/oiv2.json Will be processed by pydap by connecting to the remote dataset as a client, and generating the JSON response on the fly. """ import urlparse from paste.proxy import TransparentProxy from paste.request import construct_url from wsgifilter import Filter from wsgigilter.fixuplinks import fixup_text_links from dap.client import open from dap.wsgi.application import SimpleApplication def make_proxy(global_conf, url, responses, verbose=False, debug=False, cache=None, **kwargs): from paste.deploy.converters import aslist, asbool responses = aslist(responses) verbose = asbool(verbose) debug = asbool(debug) return DapDispatcher(url, responses, verbose, debug, cache, **kwargs) class URLFilter(Filter): def filter(self, environ, headers, data): repl = environ.get('repl') # TODO: check relocatereponse etc. func = lambda s: s.replace(*repl) return fixup_text_links(data, func) class DapDispatcher(object): def __init__(self, url, responses, verbose=False, debug=False, cache=None): self.url = url self.responses = responses self.verbose = verbose self.debug = debug self.cache = cache def __call__(self, environ, start_response): environ['x-wsgiorg.throw_errors'] = self.debug base = construct_url(environ, with_query_string=False, with_path_info=False) # Extract response. request = environ.get('PATH_INFO', '').lstrip('/') response = request.split('.')[-1] if response not in self.responses: proxy = TransparentProxy() scheme, netloc, path, queries, fragment = urlparse.urlsplit(self.url) environ['wsgi.url_scheme'] = scheme environ['HTTP_HOST'] = netloc environ['SCRIPT_NAME'] = path.rstrip('/') environ['repl'] = (self.url, base) app = URLFilter(proxy) else: # Connect to server. request = request[:-len(response)-1] url = urlparse.urljoin(self.url + '/', request) dataset = open(url, verbose=self.verbose, cache=self.cache) app = SimpleApplication(dataset) return app(environ, start_response) dap-2.2.6.7/dap/wsgi/__init__.py0000644000175000017500000000041010774147131016172 0ustar robertoroberto"""Python Web Server Gateway Interface (WSGI) application and middleware. This is an implementation of the DAP server as a WSGI application. This allows the DAP server to be run on a variety of environments and combined with third-party middleware (filters). """ dap-2.2.6.7/dap/wsgi/paster_templates/0000755000175000017500000000000011123574304017435 5ustar robertorobertodap-2.2.6.7/dap/wsgi/paster_templates/server.ini_tmpl0000644000175000017500000000053711006072314022477 0ustar robertoroberto[server:main] use = egg:Paste#http # Change to 0.0.0.0 to make public host = 127.0.0.1 port = 8080 [composit:main] use = egg:Paste#cascade app1 = static app2 = pydap catch = 404 [app:static] use = egg:Paste#static document_root = %(here)s/data [app:pydap] use = egg:dap name = $project root = %(here)s/data verbose = 0 template = %(here)s/template dap-2.2.6.7/dap/wsgi/paster_templates/template/0000755000175000017500000000000011123574304021250 5ustar robertorobertodap-2.2.6.7/dap/wsgi/paster_templates/template/index.tmpl0000644000175000017500000000210610774147131023261 0ustar robertoroberto $title

DODS directory for $location

#if $parent #end if #for $dir in $dirs: #set $dirname = '%s/' % $dir.split('/')[-1] #end for #for $file in $files: #set $filename = $file.split('/')[-1] #end for
Parent directory
$dirname
$filename [DDS] [DAS]

pydap/$version © Roberto De Almeida

dap-2.2.6.7/dap/wsgi/paster_templates/+package+.egg-info/0000755000175000017500000000000011123574304022650 5ustar robertorobertodap-2.2.6.7/dap/wsgi/paster_templates/+package+.egg-info/not-zip-safe0000644000175000017500000000000110774147131025103 0ustar robertoroberto dap-2.2.6.7/dap/wsgi/paster_templates/data/0000755000175000017500000000000011123574304020346 5ustar robertorobertodap-2.2.6.7/dap/wsgi/paster_templates/data/sample.csv0000644000175000017500000000003510774147131022347 0ustar robertorobertoa,b,c row1,1,1.0 row2,2,10.0 dap-2.2.6.7/dap/wsgi/templates.py0000644000175000017500000000100410774147131016431 0ustar robertorobertoimport os from paste.script import templates class DapServerTemplate(templates.Template): summary = "A DAP server deployed through paste.deploy" egg_plugins = ['dap[server]'] _template_dir = 'paster_templates' use_cheetah = True def post(self, command, output_dir, vars): if command.verbose: print '*'*72 print '* Run "paster serve %s/server.ini" to run' % output_dir print '* the DAP server on http://localhost:8080' print '*'*72 dap-2.2.6.7/PKG-INFO0000644000175000017500000000455411123574304013451 0ustar robertorobertoMetadata-Version: 1.0 Name: dap Version: 2.2.6.7 Summary: DAP (Data Access Protocol) client and server for Python. Home-page: http://pydap.org/ Author: Roberto De Almeida Author-email: rob@pydap.org License: MIT Description: Implementation of the `Data Access Protocol `_. This is a Python implementation of the Data Access Protocol, a scientific protocol for data access developed by the OPeNDAP team (http://opendap.org). This implementation is developed from scratch, following the latest specification of the protocol (DAP 2.0 Draft Community Standard 2005/04/27) and based on my experience with OPeNDAP servers on the wild. Using this module one can access hundreds of scientific datasets from Python programs, accessing data in an efficient, transparent and pythonic way. Arrays are manipulated like normal multi-dimensional arrays (like numpy.array, e.g.), with the fundamental difference that data is downloaded on-the-fly when a variable is sliced. Sequential data can be filtered on the server side before being downloaded, saving bandwith and time. The module also implements a DAP server, allowing datasets from a multitude of formats (netCDF, Matlab, CSV, GrADS/GRIB files, SQL RDBMS) to be served on the internet. The server specifies a plugin API for supporting new data formats in an easy way. The DAP server is implemented as a WSGI application (see PEP 333), running on a variery of servers, and can be combined with WSGI middleware to support authentication, gzip compression and much more. The latest version is available in a `Subversion repository `_. Keywords: dap opendap dods data science climate meteorology oceanography Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: MIT License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Topic :: Internet Classifier: Topic :: Scientific/Engineering Classifier: Topic :: Software Development :: Libraries :: Python Modules dap-2.2.6.7/setup.cfg0000644000175000017500000000007311123574304014165 0ustar robertoroberto[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 dap-2.2.6.7/tests/0000755000175000017500000000000011123574304013506 5ustar robertorobertodap-2.2.6.7/tests/test2.py0000644000175000017500000000353610774147122015135 0ustar robertorobertofrom dap.wsgi.application import SimpleApplication from dap.util import wsgi_intercept from dap.dtypes import * from dap.client import open from dap.server import SimpleHandler from dap.helper import escape_dods from paste.lint import middleware import httplib httplib.HTTPConnection = wsgi_intercept.WSGI_HTTPConnection dataset = DatasetType(name='dataset') dataset['seq'] = seq = SequenceType(name='seq') seq['lat'] = BaseType(name='lat', type='Int32') seq['lon'] = BaseType(name='lon', type='Int32') seq['profile'] = SequenceType(name='profile') seq['profile']['depth'] = BaseType(name='depth', type='Int32') seq['profile']['temp'] = BaseType(name='temp', type='Int32') seq['profile']['saln'] = BaseType(name='saln', type='Int32') data = [] data.append([-10, 70, [(100, 24, 35), (200, 23, 36)]]) data.append([-11, 71, [(101, 23, 34), (201, 21, 35)]]) seq.data = data print list(seq.data) app = SimpleApplication(dataset) #app = middleware(app) wsgi_intercept.add_wsgi_intercept('localhost', 8080, lambda: app, script_name='') dataset = open('http://localhost:8080/', verbose=1) seq = dataset['seq'] print list(seq.data) for position in seq: print position.lat.data, position.lon.data for cast in position.profile: print cast.depth.data, cast.temp.data, cast.saln.data print dataset = open('http://localhost:8080/', verbose=1) seq = dataset['seq'] seq2 = seq.filter('seq.lat<-10') for position in seq2: print position.lat.data, position.lon.data for cast in position.profile: print cast.depth.data, cast.temp.data, cast.saln.data print dataset = open('http://localhost:8080/', verbose=1) seq = dataset['seq'] seq2 = (struct_ for struct_ in seq if struct_['lat'] < -10) for position in seq2: print position.lat.data, position.lon.data for cast in position.profile: print cast.depth.data, cast.temp.data, cast.saln.data print dap-2.2.6.7/tests/test0.py0000644000175000017500000000137210774147122015127 0ustar robertorobertofrom dap.wsgi.application import SimpleApplication from dap.util import wsgi_intercept from dap.dtypes import * from dap.client import open from paste.lint import middleware from numpy import * import httplib httplib.HTTPConnection = wsgi_intercept.WSGI_HTTPConnection dataset = DatasetType(name='dataset') data = arange(16) data.shape = (4, 4) a = dataset['a'] = ArrayType(name='a', data=data, shape=data.shape, type=data.dtype.char) app = SimpleApplication(dataset) app = middleware(app) wsgi_intercept.add_wsgi_intercept('localhost', 8080, lambda: app, script_name='') dataset = open('http://localhost:8080/', verbose=1) print dataset print dataset.a.shape, dataset.a.type print dataset.a[:] print dataset.a[0] print dataset.a[:,0] print dataset.a[::2] dap-2.2.6.7/tests/test1.py0000644000175000017500000000271310774147122015130 0ustar robertorobertofrom dap.wsgi.application import SimpleApplication from dap.util import wsgi_intercept from dap.dtypes import * from dap.client import open from paste.lint import middleware import httplib httplib.HTTPConnection = wsgi_intercept.WSGI_HTTPConnection dataset = DatasetType(name='dataset') dataset['seq'] = seq = SequenceType(name='seq') seq['a'] = BaseType(name='a', type='Int32') seq['b'] = BaseType(name='b', type='Int32') seq['a'].data = range(5) seq['b'].data = range(5, 10) print list(seq.data) app = SimpleApplication(dataset) #app = middleware(app) wsgi_intercept.add_wsgi_intercept('localhost', 8080, lambda: app, script_name='') dataset = open('http://localhost:8080/', verbose=1) seq = dataset['seq'] print list(seq.data) for struct_ in seq: print struct_.data seq2 = (struct_ for struct_ in seq if struct_['a'] > 1 and struct_['b'] < 9) for struct_ in seq2: print struct_.data seq2 = [struct_ for struct_ in seq if struct_.a > 1 and struct_.b < 9] for struct_ in seq2: print struct_.data dataset = open('http://localhost:8080/', verbose=1) seq = dataset.seq amin = 1 bmax = 9 seq2 = (struct_ for struct_ in seq if struct_.a > amin and struct_.b < bmax) for struct_ in seq2: print struct_.data dataset = open('http://localhost:8080/', verbose=1) for struct_ in (s for s in dataset.seq if s.a > 2): print struct_ dataset = open('http://localhost:8080/', verbose=1) for struct_ in dataset.seq: if struct_.a > 2: print struct_.data dap-2.2.6.7/tests/test4.py0000644000175000017500000000357410774147122015141 0ustar robertorobertofrom dap.wsgi.application import SimpleApplication from dap.util import wsgi_intercept from dap.dtypes import * from dap.client import open from dap.server import SimpleHandler from dap.helper import escape_dods from paste.lint import middleware import httplib httplib.HTTPConnection = wsgi_intercept.WSGI_HTTPConnection dataset = DatasetType(name='dataset') dataset['seq'] = seq = SequenceType(name='seq') seq['lat'] = BaseType(name='lat', type='Int32') seq['lon'] = BaseType(name='lon', type='Int32') seq['profile'] = SequenceType(name='profile') seq['profile']['depth'] = BaseType(name='depth', type='Int32') seq['profile']['temp'] = BaseType(name='temp', type='Int32') seq['profile']['saln'] = BaseType(name='saln', type='Int32') data = [] data.append([-10, 70, [(100, 24, 35), (200, 23, 36)]]) data.append([-11, 71, [(101, 23, 34), (201, 21, 35)]]) seq.data = data print list(seq.data) app = SimpleApplication(dataset) #app = middleware(app) wsgi_intercept.add_wsgi_intercept('localhost', 8080, lambda: app, script_name='') dataset = open('http://localhost:8080/', verbose=1) seq = dataset['seq'] print list(seq.data) for position in seq: print position.lat.data, position.lon.data for cast in position.profile: print cast.depth.data, cast.temp.data, cast.saln.data print dataset = open('http://localhost:8080/', verbose=1) seq = dataset['seq'] seq2 = seq.filter('seq.lat<-10', 'seq.profile.depth>150') for position in seq2: print position.lat.data, position.lon.data for cast in position.profile: print cast.depth.data, cast.temp.data, cast.saln.data print dataset = open('http://localhost:8080/', verbose=1) seq = dataset['seq'] seq2 = (s for s in seq if s.lat < -10 and s.profile.depth > 150) for position in seq2: print position.lat.data, position.lon.data for cast in position.profile: print cast.depth.data, cast.temp.data, cast.saln.data print dap-2.2.6.7/tests/test3.py0000644000175000017500000000276610774147122015142 0ustar robertorobertoimport re from dap.wsgi.application import SimpleApplication from dap.util import wsgi_intercept from dap.dtypes import * from dap.client import open from paste.lint import middleware import httplib httplib.HTTPConnection = wsgi_intercept.WSGI_HTTPConnection dataset = DatasetType(name='dataset') dataset['seq'] = seq = SequenceType(name='seq') dataset['seq']['index'] = BaseType(name='index', type='Int32') dataset['seq']['index'].data = (i for i in [10, 11, 12, 13]) dataset['seq']['temperature'] = BaseType(name='temperature', type='Float64') dataset['seq']['temperature'].data = [17.2, 15.1, 15.3, 15.1] dataset['seq']['site'] = BaseType(name='site', type='String') dataset['seq']['site'].data = ['Diamond_St', 'Blacktail_Loop', 'Platinum_St', 'Kodiak_Trail'] app = SimpleApplication(dataset) #app = middleware(app) wsgi_intercept.add_wsgi_intercept('localhost', 8080, lambda: app, script_name='') dataset = open('http://localhost:8080/', verbose=1) seq = dataset['seq'] print list(seq.data) seq2 = (struct_ for struct_ in seq if struct_.index >= 12) for index, temperature, site in seq2: print index.data, temperature.data, site.data seq2 = (struct_ for struct_ in seq if struct_.index >= 11 and struct_.temperature < 15.2) for index, temperature, site in seq2: print index.data, temperature.data, site.data # Currently not working fine. seq2 = (struct_ for struct_ in seq if re.match(".*_St", struct_.site.data)) for index, temperature, site in seq2: print index.data, temperature.data, site.data dap-2.2.6.7/MANIFEST.in0000644000175000017500000000032410774147132014107 0ustar robertoroberto#recursive-include examples * include LICENSE include README include docs/* include ez_setup.py include dap/wsgi/paster_templates/+package+.egg-info/not-zip-safe include dap/wsgi/paster_templates/data/sample.csv dap-2.2.6.7/TODO0000644000175000017500000000135010774147132013041 0ustar robertorobertoThings to consider: - BaseType[:] raises exception, we need BaseType.data - Check GRADS plugin Responses: - netCDF - Excel - google spreadsheet api WMS response: - transparent gifs - generate plots for inner variables of sequences, using color for value - customize plots: line vs. points, animated vs. averages 2.3: - simplify dap.dtypes and add client functionality (filters, proxies) to dap.proxy.BaseType, dap.plugins.sql.SequenceType, etc. - rewrite parse_querystring and constrain (from dap.helper) - rewrite all responses to dispatch based on ``isinstance`` - allow attributes to have a specific type (Float32, eg), when the plugin passes a ``numpy.array`` or ``array.array`` instead of a list. - matlab plugin: use scipy.io.loadmat dap-2.2.6.7/setup.py0000644000175000017500000000676310774147132014100 0ustar robertoroberto# If true, then the svn revision won't be used to calculate the # revision (set to True for real releases) RELEASE = True from setuptools import setup, find_packages import sys, os classifiers = """\ Development Status :: 4 - Beta Environment :: Console Intended Audience :: Developers Intended Audience :: Science/Research License :: OSI Approved :: MIT License Operating System :: OS Independent Programming Language :: Python Topic :: Internet Topic :: Scientific/Engineering Topic :: Software Development :: Libraries :: Python Modules """ import dap.lib version = '.'.join([str(_) for _ in dap.lib.__version__]) setup(name='dap', version=version, description="DAP (Data Access Protocol) client and server for Python.", long_description="""\ Implementation of the `Data Access Protocol `_. This is a Python implementation of the Data Access Protocol, a scientific protocol for data access developed by the OPeNDAP team (http://opendap.org). This implementation is developed from scratch, following the latest specification of the protocol (DAP 2.0 Draft Community Standard 2005/04/27) and based on my experience with OPeNDAP servers on the wild. Using this module one can access hundreds of scientific datasets from Python programs, accessing data in an efficient, transparent and pythonic way. Arrays are manipulated like normal multi-dimensional arrays (like numpy.array, e.g.), with the fundamental difference that data is downloaded on-the-fly when a variable is sliced. Sequential data can be filtered on the server side before being downloaded, saving bandwith and time. The module also implements a DAP server, allowing datasets from a multitude of formats (netCDF, Matlab, CSV, GrADS/GRIB files, SQL RDBMS) to be served on the internet. The server specifies a plugin API for supporting new data formats in an easy way. The DAP server is implemented as a WSGI application (see PEP 333), running on a variery of servers, and can be combined with WSGI middleware to support authentication, gzip compression and much more. The latest version is available in a `Subversion repository `_.""", classifiers=filter(None, classifiers.split("\n")), keywords='dap opendap dods data science climate meteorology oceanography', author='Roberto De Almeida', author_email='rob@pydap.org', url='http://pydap.org/', license='MIT', packages=find_packages(exclude=['ez_setup', 'examples', 'tests']), namespace_packages=['dap.plugins', 'dap.responses'], include_package_data=True, zip_safe=False, install_requires=[ # -*- Extra requirements: -*- 'httplib2', ], extras_require={ 'server': ['Paste', 'PasteScript', 'PasteDeploy', 'Cheetah'], 'proxy': ['Paste', 'WSGIFilter'], }, entry_points=""" # -*- Entry points: -*- [dap.response] dds = dap.responses.dds das = dap.responses.das dods = dap.responses.dods asc = dap.responses.ascii ascii = dap.responses.ascii ver = dap.responses.version version = dap.responses.version help = dap.responses.help [dap.plugin] csv = dap.plugins.csvfiles [paste.app_factory] main = dap.wsgi.application:make_app proxy = dap.wsgi.proxy:make_proxy [paste.paster_create_template] dap_server = dap.wsgi.templates:DapServerTemplate dap_plugin = dap.plugins.templates:DapPluginTemplate """, ) dap-2.2.6.7/README0000644000175000017500000000341510774147132013235 0ustar robertorobertoThis is a Python implementation of the Data Access Protocol, a scientific protocol for data access developed by the OPeNDAP team (http://opendap.org). This implementation is developed from scratch, following the latest specification of the protocol (DAP 2.0 Draft Community Standard 2005/04/27) and based on my experience with OPeNDAP servers on the wild. Using this module one can access hundreds of scientific datasets from Python programs, accessing data in an efficient, transparent and pythonic way. Arrays are manipulated like normal multi-dimensional arrays (like numpy.array, e.g.), with the fundamental difference that data is downloaded on-the-fly when a variable is sliced. Sequential data can be filtered on the server side before being downloaded, saving bandwith and time. The module also implements a DAP server, allowing datasets from a multitude of formats (netCDF, Matlab, CSV files, SQL RDBMS) to be served on the internet. The server specifies a plugin API for supporting new data formats in an easy way. The DAP server is implemented as a WSGI application (see PEP 333), running on a variery of servers, and can be combined with WSGI middleware to support authentication, gzip compression and much more. For more information, please check the official website (http://pydap.org) or the included documentation. This module was supported in 2005 by the Google Summer of Code, mentored by Paul Dubois of Numeric Python fame, and a few donations through PayPal. I also received a grant from CNPq to implement the server at CPTEC/INPE, giving me some generous time to work on the module while being paid at the same time (to think that it all started from fun!). Thanks for everyone that helped me with this project. (c) 2003-2007 Roberto De Almeida http://pydap.org/ dap-2.2.6.7/docs/0000755000175000017500000000000011123574304013274 5ustar robertorobertodap-2.2.6.7/docs/Changelog0000644000175000017500000001456011123542134015110 0ustar robertoroberto2.2.2.7 (Sun Dec 21 22:33:42 GMT 2008) Added proxy support to the client. 2.2.6.6 (Fri Sep 13 00:39:40 GMT 2008) Error response now is always verbose, showing the full traceback, even for know exceptions. 2.2.6.5 (Fri Sep 5 18:52:00 GMT 2008) Added a better error handler, with more information. Fixed problem when building URLs in Windows, was using \ instead of /. 2.2.6.4 (Wed Apr 30 13:30:55 GMT 2008) Fixed bug in that caused the HTML response to fail with recent versions of Paste. The ``httpexceptions`` WSGI middleware is no longer required. 2.2.6.3 (Fri Jan 4 21:12:56 GMT 2008) Fixed bug in WSGI application, closing the handler properly. 2.2.6.2 (Mon Dec 10 19:30:11 GMT 2007) Fixed bug in the DAS parser, found by Charles Carleton, with an overflow on Int32s bigger than Python's integers. The cast to long inside the ``array.array`` object would cause the overflow. 2.2.6.1 (Mon Nov 5 15:30:47 GMT 2007) Added timeout option to the client. Removed implicit genexp in client to make code compatible with Python 2.3. 2.2.6 (Thu Sep 13 13:10:30 GMT 2007) Fixed concurrency issues on the server. 2.2.5.10 (Thu Jul 19 11:57:55 GMT 2007) Fixed a small bug in the client where the cache information was not being used to retrieve the DAS and the DDS (only the DODS response) -- thanks to Darren Hardy . Brian Granger made the ArrayType object more compatible with numpy arrays. 2.2.5.9 (Tue Apr 10 20:05:09 GMT 2007) Added proper support in the client for the "Alias" attribute type. 2.2.5.8 Fixed bug in the evaluation of values from DAS. Bytes are decoded assuming unsigned values (0:255) instead of signed (-128:127). 2.2.5.7 Values from the DAS are now evaluated using the right precision. This means missing_values defined as Float32, eg, will have the same representation as the data. 2.2.5.6 Moved DAP proxy to the main tree (``dap.wsgi.proxy``). When opening local files, client tries to unwrap arrayterators. 2.2.5.5 Fixed bug in client when requesting data with step > 1. DAS parser now returns single attributes as simple values, instead of a list of only one element. DAS parser add attributes to maps from a variable of the same name from the root dataset. Eg, a grid with a map ``latitude`` will have its attributes copied from a variable ``latitude`` in the main dataset, if it exists and it the map has no attributes. 2.2.5.4 Fixed bug when generating the DAS response from attributes contained in 0D numpy arrays. Additional parameters in the Paste Deploy configuration file are now being passed to the ``environ`` dict. WSGI app now honors ``x-wsgiorg.throw_errors`` specification from http://wsgi.org/wsgi/Specifications/throw_errors. 2.2.5.3 Fixed bug where the requested variables were being passed unquoted in the response from ``dap.helper.parse_querystring``. 2.2.5.2 Fixed another small bug in xdr.py when encoding numpy arrays of strings. 2.2.5.1 Fixed a small bug in the XDR encoding that only appeared in the SCGI server, thanks to Dallas Masters. 2.2.5 Fixed directory listing in ``DapServerApplication``: output was being returned as a string instead of a list, making the WSGI server iterate over each character. Changed server (``dap.wsgi.application.SimpleApplication``) to ignore exceptions derived from ``paste.httpexceptions.HTTPException``. This allows the HTML response to raise a ``HTTPSeeOther`` exception, redirecting the user to the corresponding ASCII response after a POST. Made server **much** faster by using numpy's ``array.astype`` instead of Python's native ``array.array``. This makes it at least 10x faster for big datasets. 2.2.4 Changes mostly related to plugin development: added a namespace for the packages; made ``parse_querystring()`` return Python slices instead of strings. Changed the DAS response to work with GrADS by ommiting attributes from array and maps in grids. Fixed USER_AGENT that was not being passed to httplib2. 2.2.3 Moved server template to ``dap/wsgi/``. Created template for writing new plugins in ``dap/plugins/``. 2.2.2 Empty directories from paster template where not being included. Added stub files in MANIFEST.in to include them. 2.2.1 Fixed small bug when deepcopying dtypes (data attribute was not begin copied). Added a generic error catcher when the server fails, returning a DAP-formatted error message. 2.2.0 Moved client to use httplib2, giving us cache and authentication for free. Sequences are now iterable (this should've always been the case). Sequence filtering is now much more pythonic: sequences can be filtered using generator expressions or list comprehensions. Removed old logger because it was too slow. Rewrote dtypes.py for consistency. Moved ``trim()`` function to a ``constrain()`` function that builds the dataset instead of trimming it down. Module now uses Numpy exclusively. Moved plugins (netCDF, Matlab, SQL, compress) out of module. Responses (DDS, DAS, DODS, ASCII, HTML, JSON, etc.) are now pluggable, like plugins. Cleaned up the netCDF plugin *a lot*. Major improvements to the SQL plugin. Server should use environ['wsgi.errors'] for logging. Server uses Cheetah template for directory listing; templates can be overwritten to customize the server. Created a server template based on Paste Script. 2.1.6 Removed fpconst dependency. 2.1.5 Added patch from Rob Cermak to support Alias tag. Added patch from Peter Desnoyers making server compatible with ncks. Added patch from Peter Desnoyers to run dap-server.py in foreground. Added a "host" option to dap-server.py. Fixed ordering of Grid maps in the DDS to match the dimensions. This is necessary for compatibility with Ferret. Fixed bug when unpacking arrays of bytes (thanks to David Poulter). Fixed bug with Grids inside constructors (thanks to Nelson Castillo). 2.1.4 Fixed bug in xdr.py (thanks to Gerald Manipon). 2.1.3 Allow multiple filters when filtering sequences. 2.1.2 Fixed bug when PATH_INFO is empty. 2.1.1 Fixed netCDF shape (thanks to Bob Drach). Fixed bug in THREDDS catalog generator. 2.1.0 New release. dap-2.2.6.7/docs/bugs0000644000175000017500000000034610774147132014170 0ustar robertorobertopydap is unable to handle datasets containing Arrays of other constructor variables, eg, Arrays of Sequences. Although datasets like these are theoretically possible, they are not commonly used, so this is probably not a problem. dap-2.2.6.7/docs/copying0000644000175000017500000000013010774147132014667 0ustar robertorobertoThis module is distributed according to the MIT license, described in the LICENSE file. dap-2.2.6.7/docs/history0000644000175000017500000000362510774147132014734 0ustar robertorobertoAn informal history of the module: Version 1.0 of the module implemented only a DAP client. The client downloaded data in ASCII format, and worked only with Arrays and Grids. It was developed by reverse-engineering the responses from the official DAP server. It really should have been released as 0.0.1, but I'm good at marketing. Further 1.x versions brought binary data transfer using Python's xdrlib module, a DDS/DAS parser that wasn't based on regular expressions, a simple server architecture based on plugins and a common core that could run as CGI, with BaseHTTPServer or Twisted. Version 2.0 was a rewrite from scratch, after extensively reading the DAP 2.0 specification draft. The DDS/DAS parser was written by hand, a faster xdrlib module was designed, and the server was built based on the WSGI specification. The client and the server were able to handle almost *any* dataset structure, metadata in the DAS, and much more. This version was supported by Google during the 2005 Summer of Code, and supervised by Paul Dubois. Version 2.1 implemented a fully buffered server, from the plugin layer to the HTTP mechanism, allowing datasets of infinite size to be served. The module moved from distutils to setuptools, plugins got "smarter" and a THREDDS catalog WSGI middleware generator was built. The server was also Paste-deployified. In version 2.2 a lot of code was rewriten. The code was cleaned up and made more modular. The client got a few improvements, with support for authentication, cache, and a more pythonic handling of sequences. The server was streamlined and became much more fast. Version 2.3 brought indexable sequences (both on client and server side), and a handy rewrite of some legacy code from 2.0 (xdr.py and proxy.py specially). This release was motivated by the Dapper specification for in-situ data. A lot of parallel work on pupynere, arrayterator, a thredds WSGI app and plugins (HDF5, GrADS).