debian/0000755000000000000000000000000012146131776007176 5ustar debian/control0000644000000000000000000000307712146125743010605 0ustar Source: drslib Section: utils Priority: optional Maintainer: Alastair McKinstry Build-Depends: debhelper (>= 9), python-all, python-setuptools,python3-all, python3-setuptools Standards-Version: 3.9.4 X-Python-Version: >= 2.6 X-Python3-Version: >= 3.2 Homepage: http://esgf.org/esgf-drslib-site/ Package: drslib Section: science Architecture: all Conflicts: python-drslib (<= 0.3.0a3-2) Depends: python-drslib, python-pkg-resources, ${misc:Depends}, python Description: Command-line tools for the Data Reference Syntax library This is a set of command-line tools for using the DRSLIB python API. Package: python-drslib Section: python Architecture: all Depends: ${python:Depends},${misc:Depends}, python-lxml, python-xlrd Recommends: cmip5-cmor-tables Description: Library for processing the CMIP5 Data Reference Syntax CMIP5 is the Climate Model Intercomparison Project5. This library supports the generation of paths and filenames corresponding to version 0.28 of thei CMIP Data Reference Syntax. It includes tools for converting from previous CMIP3 forms. Package: python3-drslib Section: python Architecture: all Depends: ${misc:Depends}, python3-lxml, python3 (>= 3.2.3-3~), python3-metaconfig Recommends: cmip5-cmor-tables, python3-xlrd Description: Library for processing the CMIP5 Data Reference Syntax (Python3) CMIP5 is the Climate Model Intercomparison Project5. This library supports the generation of paths and filenames corresponding to version 0.28 of thei CMIP Data Reference Syntax. It includes tools for converting from previous CMIP3 forms. debian/drslibCompletion.bash0000644000000000000000000000070711755217627013357 0ustar complete -W " list \ todo \ upgrade \ mapfile \ init \ history \ diff \ --help \ --version \ --root \ --incoming \ --activity \ --product \ --institute \ --model \ --experiment \ --frequency \ --realm \ --profile \ --detect-product \ --shelve-dir \ --p-cmip5-config \ --move-cmd \ " -f drs_tool complete -W " --help \ --incude \ --exclude \ --copy \ --dry-run \ --loglevel \ " -f translate_cmip3 complete -W " --help \ --catalog \ " -f drs_checkthredds debian/source/0000755000000000000000000000000011755217627010503 5ustar debian/source/format0000644000000000000000000000001411755217627011711 0ustar 3.0 (quilt) debian/patches/0000755000000000000000000000000012146131373010616 5ustar debian/patches/deb-config.patch0000644000000000000000000000167411755217627013660 0ustar Description: Set default MIP_TABLE_PATH on debian. Author: Alastair McKinstry Last-Updated: 2011-11-23 Forwarded: no Index: drslib-0.3.0a3/drslib/config.py =================================================================== --- drslib-0.3.0a3.orig/drslib/config.py 2011-11-22 22:44:26.000000000 +0000 +++ drslib-0.3.0a3/drslib/config.py 2011-11-23 11:54:37.000000000 +0000 @@ -36,7 +36,11 @@ try: table_path = config.get('tables', 'path') except: - raise Exception("Please configure your MIP table path using MIP_TABLE_PATH or a config file") + # On Debian, the user may have cmip5-cmor-tables installed. Fall back to these. + if os.path.exists('/usr/share/cmor'): + table_path ='/usr/share/cmor' + else: + raise Exception("Please configure your MIP table path using MIP_TABLE_PATH or a config file") try: table_path_csv = config.get('tables', 'path_csv') debian/patches/series0000644000000000000000000000003712142712113012024 0ustar deb-config.patch python3.patch debian/patches/python3.patch0000644000000000000000000007276512146131373013264 0ustar Author: Alastair McKinstry Description: Python3 porting patches Last-Updated: 2013-05-09 Forwarded: no Index: drslib-0.3.0a3/drslib/cmip5.py =================================================================== --- drslib-0.3.0a3.orig/drslib/cmip5.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/cmip5.py 2013-05-19 12:02:14.000000000 +0100 @@ -77,7 +77,7 @@ path_i = T.CMIP5_DRS.PATH_INSTITUTE file_i = None component = 'institute' - vocab = model_institute_map.values() + vocab = list(model_institute_map.values()) def filename_to_drs(self, context): context.drs.institute = self._deduce_institute(context) @@ -111,13 +111,13 @@ path_i = T.CMIP5_DRS.PATH_MODEL file_i = T.CMIP5_DRS.FILE_MODEL component = 'model' - vocab = model_institute_map.keys() + vocab = list(model_institute_map.keys()) def _validate(self, s): # Demote validation errors to a warning. try: return super(ModelTranslator, self)._validate(s) - except T.TranslationError, e: + except T.TranslationError as e: log.warning('Model validation error: %s', e) return s @@ -134,7 +134,7 @@ super(ExperimentTranslator, self).__init__(table_store) # Get valid experiment ids from MIP tables - for table in self.table_store.tables.values(): + for table in list(self.table_store.tables.values()): self.vocab.update(table.experiments) # Get valid experiment ids from metaconfig @@ -149,7 +149,7 @@ super(FrequencyTranslator, self).__init__(table_store) self.vocab = set() - for table in self.table_store.tables.values(): + for table in list(self.table_store.tables.values()): try: self.vocab.add(table.frequency) except AttributeError: @@ -188,7 +188,7 @@ # Extract valid realms from the MIP tables self.vocab = set() - for table in table_store.tables.values(): + for table in list(table_store.tables.values()): for var in table.variables: try: realms = table.get_variable_attr(var, 'modeling_realm')[0] Index: drslib-0.3.0a3/drslib/drs.py =================================================================== --- drslib-0.3.0a3.orig/drslib/drs.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/drs.py 2013-05-19 12:02:14.000000000 +0100 @@ -198,7 +198,7 @@ """ parts = dataset_id.split('.') - for attr, val in itertools.izip(DRS_ATTRS, parts): + for attr, val in zip(DRS_ATTRS, parts): if val is '%': continue if attr is 'ensemble': @@ -245,7 +245,7 @@ attrs = ['product', 'institute', 'model', 'experiment', 'frequency', 'realm', 'table', 'ensemble'] drs = DRS(activity=activity) - for val, attr in itertools.izip(p, attrs): + for val, attr in zip(p, attrs): if attr == 'ensemble': mo = re.match(r'r(\d+)i(\d+)p(\d+)', val) drs[attr] = tuple(int(x) for x in mo.groups()) Index: drslib-0.3.0a3/drslib/drs_command.py =================================================================== --- drslib-0.3.0a3.orig/drslib/drs_command.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/drs_command.py 2013-05-19 12:02:14.000000000 +0100 @@ -10,10 +10,13 @@ """ + +from __future__ import print_function + import sys, os from optparse import OptionParser -from ConfigParser import NoSectionError, NoOptionError +from configparser import NoSectionError, NoOptionError from drslib.drs_tree import DRSTree from drslib import config @@ -180,21 +183,21 @@ def print_header(self): - print """\ + print("""\ ============================================================================== DRS Tree at %s ------------------------------------------------------------------------------\ -""" % self.drs_root +""" % self.drs_root) def print_sep(self): - print """\ + print("""\ ------------------------------------------------------------------------------\ -""" +""") def print_footer(self): - print """\ + print("""\ ==============================================================================\ -""" +""") @@ -213,18 +216,18 @@ if pt.state != pt.STATE_VERSIONED: to_upgrade += 1 #!TODO: print update summary - print '%-70s %s' % (pt.version_drs().to_dataset_id(with_version=True), state_msg) + print('%-70s %s' % (pt.version_drs().to_dataset_id(with_version=True), state_msg)) if self.drs_tree.incomplete: self.print_sep() - print 'Incompletely specified incoming datasets' + print('Incompletely specified incoming datasets') self.print_sep() for dataset_id in sorted(self.drs_tree.incomplete_dataset_ids()): - print '%-70s' % dataset_id + print('%-70s' % dataset_id) if to_upgrade: self.print_sep() - print '%d datasets awaiting upgrade' % to_upgrade + print('%d datasets awaiting upgrade' % to_upgrade) self.print_footer() class TodoCommand(Command): @@ -237,7 +240,7 @@ if pt.count_todo() == 0: if not first: self.print_sep() - print 'Nothing todo for %s' % pt.drs.to_dataset_id() + print('Nothing todo for %s' % pt.drs.to_dataset_id()) first = False continue @@ -249,11 +252,11 @@ todos = pt.list_todo(next_version) if not first: self.print_sep() - print "Publisher Tree %s todo for version %d" % (pt.drs.to_dataset_id(), - next_version) + print("Publisher Tree %s todo for version %d" % (pt.drs.to_dataset_id(), + next_version)) first = False - print - print '\n'.join(todos) + print() + print('\n'.join(todos)) self.print_footer() class UpgradeCommand(Command): @@ -268,12 +271,12 @@ next_version = pt._next_version() if pt.state == pt.STATE_VERSIONED: - print 'Publisher Tree %s has no pending upgrades' % pt.drs.to_dataset_id() + print('Publisher Tree %s has no pending upgrades' % pt.drs.to_dataset_id()) else: - print ('Upgrading %s to version %d ...' % (pt.drs.to_dataset_id(), next_version)), + print(('Upgrading %s to version %d ...' % (pt.drs.to_dataset_id(), next_version)), end=' ') to_process = pt.count_todo() pt.do_version(next_version) - print 'done %d' % to_process + print('done %d' % to_process) self.print_footer() @@ -292,7 +295,7 @@ if len(self.drs_tree.pub_trees) == 0: raise Exception("No datasets selected") - pt = self.drs_tree.pub_trees.values()[0] + pt = list(self.drs_tree.pub_trees.values())[0] #!TODO: better argument handling if self.args: @@ -314,14 +317,14 @@ """ if len(self.drs_tree.pub_trees) != 1: raise Exception("You must select 1 dataset to list history. %d selected" % len(self.drs_tree.pub_trees)) - pt = self.drs_tree.pub_trees.values()[0] + pt = list(self.drs_tree.pub_trees.values())[0] self.print_header() - print "History of %s" % pt.drs.to_dataset_id() + print("History of %s" % pt.drs.to_dataset_id()) self.print_sep() for version in sorted(pt.versions, reverse=True): vdrs = DRS(pt.drs, version=version) - print vdrs.to_dataset_id(with_version=True) + print(vdrs.to_dataset_id(with_version=True)) self.print_footer() @@ -337,7 +340,7 @@ from drslib.p_cmip5.init import init init(self.shelve_dir, config.table_path) - print "CMIP5 configuration data written to %s" % repr(self.shelve_dir) + print("CMIP5 configuration data written to %s" % repr(self.shelve_dir)) @@ -357,7 +360,7 @@ if len(self.drs_tree.pub_trees) == 0: raise Exception("No datasets selected") - pt =self.drs_tree.pub_trees.values()[0] + pt =list(self.drs_tree.pub_trees.values())[0] #!TODO: better argument handling args = self.args[:] @@ -378,7 +381,7 @@ v2_msg = v2 else: v2_msg = 'todo list' - print 'Diff between %s and %s' % (v1, v2_msg) + print('Diff between %s and %s' % (v1, v2_msg)) self.print_sep() #!FIXME: Just compare file sizes at the moment! @@ -388,15 +391,15 @@ continue # Don't compare by path if using the todo list elif (v2 is not None) and (diff_type & pt.DIFF_PATH == pt.DIFF_PATH): - print 'PATH\t\t%s' % filename + print('PATH\t\t%s' % filename) elif diff_type & pt.DIFF_SIZE == pt.DIFF_SIZE: - print 'SIZE\t\t%s' % filename + print('SIZE\t\t%s' % filename) elif diff_type & pt.DIFF_TRACKING_ID == pt.DIFF_TRACKING_ID: - print 'TRACKING_ID\t%s' % filename + print('TRACKING_ID\t%s' % filename) elif diff_type & pt.DIFF_V1_ONLY == pt.DIFF_V1_ONLY: - print '%s\t\t%s' % (v1, filename) + print('%s\t\t%s' % (v1, filename)) elif diff_type & pt.DIFF_V2_ONLY == pt.DIFF_V2_ONLY: - print '%s\t\t%s' % (v2_msg, filename) + print('%s\t\t%s' % (v2_msg, filename)) else: assert False Index: drslib-0.3.0a3/drslib/drs_tree.py =================================================================== --- drslib-0.3.0a3.orig/drslib/drs_tree.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/drs_tree.py 2013-05-19 12:02:14.000000000 +0100 @@ -164,7 +164,7 @@ continue log.debug('File %s => %s' % (repr(filename), drs)) - for k, v in components.items(): + for k, v in list(components.items()): if v is None: continue # If component is present in drs act as a filter @@ -279,7 +279,7 @@ """ items = self - for k, v in kw.items(): + for k, v in list(kw.items()): if type(v) == list: items = [x for x in items if x[1].get(k, None) in v] else: Index: drslib-0.3.0a3/drslib/mapfile.py =================================================================== --- drslib-0.3.0a3.orig/drslib/mapfile.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/mapfile.py 2013-05-19 12:02:14.000000000 +0100 @@ -10,6 +10,8 @@ """ +from __future__ import print_function + #!TODO: check againsts similar code in datanode_admin and merge import stat, os @@ -45,5 +47,5 @@ size = file_stat[stat.ST_SIZE] mtime = file_stat[stat.ST_MTIME] - print >>fh, ' | '.join([drs_to_id(drs), path, str(size), "mod_time=%f"%float(mtime)]) + print(' | '.join([drs_to_id(drs), path, str(size), "mod_time=%f"%float(mtime)]), file=fh) Index: drslib-0.3.0a3/drslib/mip_table.py =================================================================== --- drslib-0.3.0a3.orig/drslib/mip_table.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/mip_table.py 2013-05-19 12:02:14.000000000 +0100 @@ -143,11 +143,11 @@ @property def variables(self): - return self._vardict.keys() + return list(self._vardict.keys()) @property def experiments(self): - return self._exptdict.keys() + return list(self._exptdict.keys()) @property def frequency(self): @@ -282,8 +282,8 @@ table_reader = csv.reader(fh) # Check first 2 lines look like the right file - header1 = table_reader.next() - header2 = table_reader.next() + header1 = next(table_reader) + header2 = next(table_reader) assert "CMIP5 Modeling Groups" in header1[0] assert 'Abbreviated name of center or group' in header2[1] Index: drslib-0.3.0a3/drslib/publisher_tree.py =================================================================== --- drslib-0.3.0a3.orig/drslib/publisher_tree.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/publisher_tree.py 2013-05-19 12:02:14.000000000 +0100 @@ -205,7 +205,7 @@ for filepath, drs in fl: files2[os.path.basename(filepath)] = filepath - for file in set(files1.keys() + files2.keys()): + for file in set(list(files1.keys()) + list(files2.keys())): if file in files1 and file in files2: yield (self._diff_file(files1[file], files2[file], by_tracking_id), Index: drslib-0.3.0a3/drslib/thredds.py =================================================================== --- drslib-0.3.0a3.orig/drslib/thredds.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/thredds.py 2013-05-19 12:07:04.000000000 +0100 @@ -22,8 +22,12 @@ from lxml import etree as ET from drslib.drs import DRS from drslib.cmip5 import make_translator -import urlparse from optparse import OptionParser +try: + from urllib.parse import urlparse +except ImportError: + # Python 2 + from urlparse import urlparse import logging log = logging.getLogger(__name__) @@ -102,7 +106,7 @@ check = CheckClass(environ) try: check.check(etree) - except InvalidThreddsException, e: + except InvalidThreddsException as e: log.error(e) except CheckNotPossible: log.warn('Check %s aborted' % CheckClass.__name__) Index: drslib-0.3.0a3/drslib/translate_cmip3.py =================================================================== --- drslib-0.3.0a3.orig/drslib/translate_cmip3.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/translate_cmip3.py 2013-05-19 12:02:14.000000000 +0100 @@ -81,7 +81,7 @@ log.debug('dirnames remaining %s' % dirnames) -def _mkdirs(name, mode=0777): +def _mkdirs(name, mode=0o777): log.info('mkdir -p %s' % name) if not dry_run: os.makedirs(name, mode) @@ -113,7 +113,7 @@ try: drs = cmip3_t.path_to_drs(dirpath) path = cmip5_t.drs_to_path(drs) - except TranslationError, e: + except TranslationError as e: log.error('Failed to translate path %s: %s' % (dirpath, e)) continue except: @@ -130,7 +130,7 @@ try: drs2 = cmip3_t.filepath_to_drs(os.path.join(dirpath, filename)) filename2 = cmip5_t.drs_to_file(drs2) - except TranslationError, e: + except TranslationError as e: log.error('Failed to translate filename %s: %s' % (filename, e)) continue Index: drslib-0.3.0a3/drslib/p_cmip5/__init__.py =================================================================== --- drslib-0.3.0a3.orig/drslib/p_cmip5/__init__.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/p_cmip5/__init__.py 2013-05-19 12:02:14.000000000 +0100 @@ -28,8 +28,8 @@ """ -import product -import init +from . import product +from . import init # Exception raised by product detection failures -from product import ProductScope, cmip5_product, version, version_date +from .product import ProductScope, cmip5_product, version, version_date Index: drslib-0.3.0a3/drslib/p_cmip5/init.py =================================================================== --- drslib-0.3.0a3.orig/drslib/p_cmip5/init.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/p_cmip5/init.py 2013-05-19 12:09:41.000000000 +0100 @@ -4,10 +4,16 @@ """ +from __future__ import print_function + import os, sys import xlrd, string, shelve import re, glob -from whichdb import whichdb +try: + from dbm import whichdb +except ImportError: + # Python 2 + from whichdb import whichdb import logging log = logging.getLogger(__name__) @@ -43,7 +49,7 @@ def scan_table(mip,dir=CMOR_TABLE_DIR): ll = open( '%s/CMIP5_%s' % (dir, mip), 'r' ).readlines() - lll = map( string.strip, ll ) + lll = list(map( string.strip, ll )) ssss = string.join( lll, ':::' ) vitems = string.split( ssss, ':::variable_entry:' )[1:] @@ -76,8 +82,8 @@ ll = open( os.path.join(dir, '%s.csv' % mip), 'r' ).readlines() ## strip out white space, so that variable can be identified unambiguously by ',var,' - lll = map( lambda x: string.replace( string.strip(x), ' ',''), ll ) - vlist = map( lambda x: x[0], ee ) + lll = [string.replace( string.strip(x), ' ','') for x in ll] + vlist = [x[0] for x in ee] ee2 = [] for x in ee: @@ -160,7 +166,7 @@ version_file = os.path.join(shelve_dir, SHELVE_VERSION_FILE) fh = open(version_file, 'w') - print >>fh, SHELVE_VERSION + print(SHELVE_VERSION, file=fh) fh.close() #--------------------------------------------------------------------------- @@ -169,7 +175,7 @@ def helper_year( val ): if type( val ) == type( 1. ): return int(val) - elif type(val) in [type('x'), type(u'x')]: + elif type(val) in [type('x'), type('x')]: if string.strip(val) == '': return None if val[-1] == '*': @@ -228,7 +234,7 @@ self.status = None def allowed(self,next): - if next not in self.statuses.keys(): + if next not in list(self.statuses.keys()): return False if self.status == None: @@ -308,13 +314,13 @@ key = 'historical%s' % y sh[key] = (93, '7.3',str(this_row[0].value), str(this_row[1].value)) - rl1 = range(46,52) + range(56,97) + range(100,110) + rl1 = list(range(46,52)) + list(range(56,97)) + list(range(100,110)) for r in rl1: this_row = sheet.row(r) key = string.strip( str(this_row[10].value ) ) - if key in mappings.keys(): + if key in list(mappings.keys()): key = mappings[key] - while key in sh.keys(): + while key in list(sh.keys()): key += '+' sh[key] = (r,str(this_row[2].value), str(this_row[0].value), str(this_row[1].value)) sh.close() @@ -329,7 +335,7 @@ def import_other(self,sh,book): oo = book.sheet_by_name('other output') - rl1 = range(39,51) + range(54,95) + range(98,108) + rl1 = list(range(39,51)) + list(range(54,95)) + list(range(98,108)) colh = ['M','N','Q','R'] kkk = 0 for col in [12,13,16,17]: @@ -380,11 +386,11 @@ ll = [this_type] for b in bits: if string.find( b, ':' ) != -1: - st,en,dd = map( int, string.split(b,':') ) + st,en,dd = list(map( int, string.split(b,':') )) for y in range(st,en+1,dd): ll.append( ('year',y) ) elif string.find( b, '-' ) !=-1: - st,en = map( int, string.split(b,'-') ) + st,en = list(map( int, string.split(b,'-') )) ll.append( ('slice',st,en) ) else: log.info('%s %s' % (bits,b) ) @@ -397,7 +403,7 @@ def import_cfmip(self,sh,book): oo = book.sheet_by_name('CFMIP output') - rl1 = range(6,27) + rl1 = list(range(6,27)) ee = {} # # for each row, check for non-zero date ranges, and append to list. NB there may be multiple @@ -481,14 +487,14 @@ if __name__ == '__main__': import sys if len( sys.argv ) != 3: - print 'usage: init.py go ' + print('usage: init.py go ') sys.exit(0) init( sys.argv[2],CMOR_TABLE_DIR, xls_dir=CMIP5_REQUEST_XLS ) sh = shelve.open( sys.argv[2] + '/standard_output_mip', 'r' ) tlist = ['cf3hr', 'cfSites', 'Oyr','Omon','aero','day','6hrPlev','3hr','cfMon'] - print sh.keys(), len(sh.keys()) + print(list(sh.keys()), len(list(sh.keys()))) for t in tlist: l1 = sh[t] ee = scan_table( t ) @@ -517,18 +523,18 @@ if e[3] == 1: kk1 += 1 - print t, len(ee), len(ee2), len(l1), kkk, kk1 + print(t, len(ee), len(ee2), len(l1), kkk, kk1) if kkk > 0: - print '===================================' + print('===================================') for e in ee2: if e[2] == None: - print e - print '===================================' + print(e) + print('===================================') if t == 'day' and kk1 != 10: - print 'error',ee2[0:10] + print('error',ee2[0:10]) if t == 'day': - print ee2[0:10] + print(ee2[0:10]) if len(ee) != len(l1): eevl = [] @@ -540,7 +546,7 @@ rold = l1[i] vold = string.strip(rold[5]) if vold not in eevl: - print t,':: not found: ',vold + print(t,':: not found: ',vold) knf += 1 else: kmd.append( eevl.index( vold ) ) @@ -549,11 +555,11 @@ if i not in kmd: knmd.append( eevl[i] ) knmd.sort() - print knmd + print(knmd) else: for i in range(len(l1)): rold = l1[i] rnew = ee[i] if rnew[0] != rold[5]: - print t,i,rnew[0],rold[5] + print(t,i,rnew[0],rold[5]) Index: drslib-0.3.0a3/drslib/p_cmip5/product.py =================================================================== --- drslib-0.3.0a3.orig/drslib/p_cmip5/product.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/p_cmip5/product.py 2013-05-19 12:02:14.000000000 +0100 @@ -85,7 +85,7 @@ self.mip_sh = shelve.open( mip_table_shelve, flag='r' ) self.tmpl = shelve.open( template, flag='r' ) self.stdo = shelve.open( stdo, flag='r' ) - self.tmpl_keys = self.tmpl.keys() + self.tmpl_keys = list(self.tmpl.keys()) self.tmpl_keys.sort( ddsort(self.tmpl,0).cmp ) self.pos_in_table = 999 self.config = config @@ -126,17 +126,17 @@ fpat = '%s_%s_%s_%s_*.nc' % (self.var,self.table,self.model,self.expt) else: fpat = '*.nc' - fl = map( lambda x: string.split(x, '/')[-1], glob.glob( dir + fpat ) ) + fl = [string.split(x, '/')[-1] for x in glob.glob( dir + fpat )] assert len(fl) != 0,'No files matching %s found in %s' % (fpat,self.path) fl.sort() if self.path_output1 != None or self.path_output2 != None: if self.path_output1 != None: fl1b = glob.glob( self.path_output1 + '/*.nc' ) - fl1 = map( lambda x: string.split(x, '/')[-1], glob.glob( self.path_output1 + '/*.nc' ) ) + fl1 = [string.split(x, '/')[-1] for x in glob.glob( self.path_output1 + '/*.nc' )] else: fl1 = [] if self.path_output2 != None: - fl2 = map( lambda x: string.split(x, '/')[-1], glob.glob( self.path_output2 + '/*.nc' ) ) + fl2 = [string.split(x, '/')[-1] for x in glob.glob( self.path_output2 + '/*.nc' )] else: fl2 = [] if len(fl1) == 0 and len(fl2) == 0: @@ -147,18 +147,18 @@ fld[f] = 'N' for f in fl1: assert f not in fl2, 'Files should not be in output1 and output2: %s, %s' % (self.path_output1,self.path_output2) - if fld.has_key(f): + if f in fld: fld[f] = 'R1' else: fld[f] = 'O1' - keys = fld.keys() + keys = list(fld.keys()) for f in fl2: - if fld.has_key(f): + if f in fld: fld[f] = 'R2' else: fld[f] = 'O2' fli = fl[:] - fl = fld.keys() + fl = list(fld.keys()) fl.sort() self.file_dict = fld @@ -311,7 +311,7 @@ return self.vline[1] def get_cfmip_request_spec(self): - keys = self.stdo['cfmip'].keys() + keys = list(self.stdo['cfmip'].keys()) log.debug( 'get_cfmip_request_spec: %s' % self.rei[1] ) if self.rei[1] not in keys: log.info( '%s not in keys:: %s' % (self.rei[1],str(keys)) ) @@ -341,7 +341,7 @@ def get_request_spec(self): tlist = self.stdo[self.request_col] self.requested_years_list = [] - if self.rei[0]-2 in tlist.keys(): + if self.rei[0]-2 in list(tlist.keys()): tli = self.rei[0]-2 ssp = tlist[tli] self.request_spec = ssp @@ -365,8 +365,8 @@ def load_config(self): assert self.config_exists, 'load_config: need a valid configuration file at this point' if not self.config_loaded: - import ConfigParser - self.cp = ConfigParser.SafeConfigParser() + import configparser + self.cp = configparser.SafeConfigParser() self.cp.read( self.config ) self.config_loaded = True @@ -429,7 +429,7 @@ #################### #################### def find_product_step_one(self,var,table,expt,model,verbose=False): - if table not in self.mip_sh.keys(): + if table not in list(self.mip_sh.keys()): return self.not_ok( 'Bad mip table:: %s ' % table, 'ERR008' ) ## offset_status has 3 levels: -1: not set, 0: set by default, 1: set using info from configuration file. self.offset_status = -1 @@ -517,7 +517,7 @@ if len(self.expt) > 7 and self.expt[0:7] == 'decadal': time_datum = int( self.expt[7:11] ) if time_datum in [1960, 1980, 2005]: - requested_years = map( lambda x: x+time_datum, [10,20,30] ) + requested_years = [x+time_datum for x in [10,20,30]] else: requested_years = (time_datum + 10, ) if self.verbose: @@ -554,18 +554,18 @@ y_ec2eh = int( self.cp.get( self.model, 'branch_year_esmControl_to_esmHistorical' ) ) y_ehist0 = int( self.cp.get( self.model, 'base_year_esmHistorical' ) ) offset = y_ec2h - y_ehist0 - requested_years = map( lambda x: x + offset, self.requested_years_list ) + requested_years = [x + offset for x in self.requested_years_list] return self.select_year_list( requested_years, 'output1', rc='OK300.02' ) elif (self.table == '3hr' and self.expt in ['1pctCO2','abrupt4xCO2'] ): offset = self._get_base_year( self.expt ) assert offset != None, 'offset not found for %s, %s, %s' % (self.model, self.expt, self.table) if self.expt == '1pctCO2': - requested_years = map( lambda x: offset + 110 + x, range(30) ) + requested_years = [offset + 110 + x for x in range(30)] nrq = 30 else: - requested_years = map( lambda x: offset + x, range(5) ) + \ - map( lambda x: offset + 120 + x, range(30) ) + requested_years = [offset + x for x in range(5)] + \ + [offset + 120 + x for x in range(30)] nrq = 35 if self.select_year_list( requested_years, 'output1', force_complete = True, rc='OK300.03', no_except=True ): return True @@ -584,7 +584,7 @@ elif (self.expt == 'piControl' and self.table == '3hr'): offset = self._get_base_year( '1pctCO2' ) if offset != None: - requested_years = map( lambda x: offset - 1 + x, self.requested_years_list ) + requested_years = [offset - 1 + x for x in self.requested_years_list] result = self.select_year_list( requested_years, 'output1', force_complete = True, rc='OK300.04' ) if self.rc != 'ERR005': return result @@ -607,14 +607,14 @@ offset == None if offset != None: - requested_years = map( lambda x: offset - 1 + x, self.requested_years_list ) + requested_years = [offset - 1 + x for x in self.requested_years_list] result = self.select_year_list( requested_years, 'output1', force_complete = True, rc='OK300.05' ) if self.rc != 'ERR005': return result ## return years requested relative to first year if number submitted is greater than 145 if self.nyears_submitted > 145: offset = self.ads_time_period[0] - requested_years = map( lambda x: offset - 1 + x, self.requested_years_list ) + requested_years = [offset - 1 + x for x in self.requested_years_list] result = self.select_year_list( requested_years, 'output1', force_complete = True, rc='OK300.06' ) if self.rc != 'ERR005': return result @@ -625,7 +625,7 @@ y_pic2h = int( self.cp.get( self.model, 'branch_year_picontrol_to_historical' ) ) y_hist0 = int( self.cp.get( self.model, 'base_year_historical' ) ) offset = y_pic2h - y_hist0 - requested_years = map( lambda x: x + offset, self.requested_years_list ) + requested_years = [x + offset for x in self.requested_years_list] result = self.select_year_list( requested_years, 'output1', force_complete = True, rc='OK300.07' ) if self.rc != 'ERR005': return result Index: drslib-0.3.0a3/drslib/p_cmip5/time_slices.py =================================================================== --- drslib-0.3.0a3.orig/drslib/p_cmip5/time_slices.py 2013-05-19 12:02:14.000000000 +0100 +++ drslib-0.3.0a3/drslib/p_cmip5/time_slices.py 2013-05-19 12:02:14.000000000 +0100 @@ -53,8 +53,8 @@ def load_config(self): assert self.parent.config_exists, 'load_config: need a valid configuration file at this point' if not self.parent.config_loaded: - import ConfigParser - self.cp = ConfigParser.SafeConfigParser() + import configparser + self.cp = configparser.SafeConfigParser() self.cp.read( self.parent.config ) self.parent.config_loaded = True debian/rules0000755000000000000000000000133412142701135010243 0ustar #!/usr/bin/make -f # The magic debhelper rule: %: dh $@ --with python2,python3 # export DH_VERBOSE=1 PYTHON2=$(shell pyversions -vr) PYTHON3=$(shell py3versions -vr) build-python%: python$* setup.py build override_dh_auto_build: $(PYTHON3:%=build-python%) dh_auto_build override_dh_auto_clean: dh_auto_clean rm -rf build rm -rf *.egg-info install-python%: python$* setup.py install --root=$(CURDIR)/debian/tmp --install-layout=deb override_dh_auto_install: $(PYTHON3:%=install-python%) dh_auto_install find debian -name '*.xls' -exec chmod -x {} \; find debian -name '*.csv' -exec chmod -x {} \; mkdir -p debian/drslib/etc/bash_completion.d cp debian/drslibCompletion.bash debian/drslib/etc/bash_completion.d debian/python-drslib.install0000644000000000000000000000002212142701740013344 0ustar /usr/lib/python2* debian/compat0000644000000000000000000000000212142712136010363 0ustar 9 debian/drs_checkthredds.10000644000000000000000000000061412146130556012557 0ustar .\" Comment .TH DRS_CHECKTHREDDS "1" "November 2011" "DRSLIB" "User Commands" .SH NAME .SH SYNOPSIS .SH DESCRIPTION .SH USAGE .SH OPTIONS .SH REQUIRES .SH FILES .SH VERSION 3.0.3.a .SH BUGS No Known bugs. Please report bugs using .I reportbug on Debian systems. .SH SEE ALSO .SH AUTHOR DRSLIB was written by Stephen Pascoe, STFC.SC.UK. This manual page was written by Alastair McKinstry. debian/drslib.manpages0000644000000000000000000000010511755217627012173 0ustar debian/drs_tool.1 debian/drs_checkthredds.1 debian/translate_cmip3.1 debian/drslib.install0000644000000000000000000000010511755217627012046 0ustar /usr/bin/drs_tool /usr/bin/translate_cmip3 /usr/bin/drs_checkthredds debian/translate_cmip3.10000644000000000000000000000201612146131751012340 0ustar .TH TRANSLATE_CMIP3 "1" "November 2011" "DRSLIB" "User Commands" .SH NAME translate_cmip3 \- Translate a stream of filepaths from CMIP3 to CMIP5 syntax .SH SYNOPSIS .B translate_cmip3 [\-dryrun] [\-copy] .I cmip3_root cmip5_root .SH DESCRIPTION This is a python helper file to translate CMIP3-formatted files to CMIP5 syntax. .SH OPTIONS .IR -h, --help show short help message and exit .IP \-i INCLUDE .IP \-\-include=INCLUDE Include paths matching .IR INCLUDE regular expression .IP \-e EXCLUDE .IP \-\-exclude=EXCLUDE Include paths matching .IR EXCLUDE regular expression .IP \-c, \-\-copy Copy rather than move files .IP \-d, \-\-dryrun Emit log messages but don't translate anything .IP \-l LOGLEVEL, .IP \-\-loglevel=LOGLEVEL Set logging level .SH VERSION 3.0.3.a .SH BUGS No Known bugs. Please report bugs using .I reportbug on Debian systems. .SH SEE ALSO .BR drs_tool (1), .BR drs_checkthredds (1) .SH AUTHOR DRSLIB was written by Stephen Pascoe, STFC.SC.UK. This manual page was written by Alastair McKinstry. debian/python3-drslib.install0000644000000000000000000000002112142701616013430 0ustar /usr/lib/python3 debian/changelog0000644000000000000000000000244112146131213011033 0ustar drslib (0.3.0a3-5) unstable; urgency=low * Change Depends: on python3-xlrd to a Recommends: as it doesn't exist yet. * Fix man pages (lintian checks; remove bad comments) * Python code is Arch: all * Fix translate_cmip3 manpage -- Alastair McKinstry Mon, 13 May 2013 17:02:30 +0100 drslib (0.3.0a3-4) unstable; urgency=low * Add python3-drslib package; python3 support and patch. * Fixes to debian/rules to fix FTBFS. Closes: #717513; caused by changes in debhelper? package now double-builds cleanly. * Standards-Version: 3.9.4, DH_COMPAT=9; -- Alastair McKinstry Thu, 09 May 2013 13:20:01 +0100 drslib (0.3.0a3-3) unstable; urgency=low * Add python-pkg-resources needed by drslib. Closes: #672908. * Standards-Version: 3.9.3 * Split executables out of python-drslib to drslib package; add man pages. -- Alastair McKinstry Thu, 17 May 2012 15:50:56 +0100 drslib (0.3.0a3-2) unstable; urgency=low * Fix cut-n-paste error for the package description . -- Alastair McKinstry Thu, 24 Nov 2011 16:28:46 +0000 drslib (0.3.0a3-1) unstable; urgency=low * Initial release. (Closes: #649074) -- Alastair McKinstry Thu, 17 Nov 2011 10:09:50 +0000 debian/copyright0000644000000000000000000000310411755217627011134 0ustar This package was debianized by Alastair McKinstry Thursday, 17 Sep 2011 19:12:17 +0100 It was downloaded from http://www.esfg.org/esgf-drslib-site/ Upstream Author: Stephen Pascoe (stephen.pascoe@stfc.ac.uk) Files: * Copyright: 2011, Science & Technology Facilities Council (STFC) License: BSD License: BSD-2-Clause All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. debian/drs_tool.10000644000000000000000000000447012144210137011075 0ustar .TH drslib .SH NAME drs_tool - Command-line interface to DRSLIB .SH SYNPOSIS .SH DESCRIPTION .SH USAGE Usage Usage: drs_tool [command] [options] [drs-pattern] command: list list publication-level datasets todo show file operations pending for the next version upgrade make changes to the selected datasets to upgrade to the next version mapfile make a mapfile of the selected dataset history list all versions of the selected dataset init initialise CMIP5 product detection data .SH OPTIONS .TP \fB\-\-help\fR display this help message and exit .TP \fB\-\-version\fR output version information and exit -h, --help show this help message and exit -R ROOT, --root=ROOT Root directory of the DRS tree -I INCOMING, --incoming=INCOMING Incoming directory for DRS files. Defaults to /output -a ACTIVITY, --activity=ACTIVITY Set DRS attribute activity for dataset discovery -p PRODUCT, --product=PRODUCT Set DRS attribute product for dataset discovery -i INSTITUTE, --institute=INSTITUTE Set DRS attribute institute for dataset discovery -m MODEL, --model=MODEL Set DRS attribute model for dataset discovery -e EXPERIMENT, --experiment=EXPERIMENT Set DRS attribute experiment for dataset discovery -f FREQUENCY, --frequency=FREQUENCY Set DRS attribute frequency for dataset discovery -r REALM, --realm=REALM Set DRS attribute realm for dataset discovery -v VERSION, --version=VERSION Force version upgrades to this version -P FILE, --profile=FILE Profile the script exectuion into FILE --detect-product Automatically detect the DRS product of incoming data .SH REQUIRES .SH .SH FILES .I drs_tool uses .I metaconfig to configure and store configuration information. Its configuration may be found then in a .I metaconfig.conf file, in .B /etc/metaconfig/metaconfig.conf on Debian systems, which may be overriden by .B $HOME/.metaconfig.conf or .B ./metaconfig.conf .SH ENVIRONMENT VARIABLES The variable .B METACONF_CONF .SH VERSION 0.3.0a3 .SH BUGS To query and report bugs in drs_tool, please use .I reportbug on Debian systems. .SH SEE ALSO http://esgf.org/esgf-drslib-site/index.html .SH AUTHOR DRSLIB was written by Stephen Pascoe, STFC.SC.UK. This manual page was written by Alastair McKinstry.