visidata-1.0/0000770000175100017520000000000013232247165013027 5ustar anjaanja00000000000000visidata-1.0/README.md0000660000175100017520000000761113232246526014314 0ustar anjaanja00000000000000# VisiData v1.0 [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) A terminal interface for exploring and arranging tabular data. ## Dependencies - Linux or OS/X - Python 3.4+ - python-dateutil - other modules may be required for opening particular data sources - see [requirements.txt](https://github.com/saulpw/visidata/blob/stable/requirements.txt) or the [supported sources](http://visidata.org/man/#loaders) in the vd manpage ### Install via pip3 Best installation method for users who wish to take advantage of VisiData in their own code, or integrate it into a Python3 virtual environment. To install VisiData, with loaders for the most common data file formats (including csv, tsv, fixed-width text, json, sqlite, http, html and xls): $ pip3 install visidata To install VisiData, plus external dependencies for all available loaders: $ pip3 install "visidata[full]" ### Install via brew Ideal for MacOS users who primarily want to engage with VisiData as an application. This is currently the most reliable way to install VisiData's manpage on MacOS. $ brew install saulpw/vd/visidata Further instructions available [here](https://github.com/saulpw/homebrew-vd). ### Install via apt Packaged for Linux users who do not wish to wrangle with PyPi or python3-pip. Currently, VisiData is undergoing review for integration into the main Debian repository. Until then it is available in our [Debian repo](https://github.com/saulpw/deb-vd). Grab our public key $ wget http://visidata.org/devotees.gpg.key $ apt-key add devotees.gpg.key Add our repository to apt's search list $ sudo apt-get install apt-transport-https $ sudo vim /etc/apt/sources.list deb[arch=amd64] https://raw.githubusercontent.com/saulpw/deb-vd/master sid main $ sudo apt-get update You can then install VisiData by typing: sudo apt-get install visidata Further instructions available [here](https://github.com/saulpw/deb-vd). ## Usage $ vd [] ... $ | vd [] VisiData supports tsv, csv, xlsx, hdf5, sqlite, json and more. Use `-f ` to force a particular filetype. (See the [list of supported sources](http://visidata.org/man#sources)). ## Documentation * Quick reference: `F1` (or `z?`) within `vd` will open the [man page](http://visidata.org/man), which has a list of all commands and options. * [visidata.org/docs](http://visidata.org/docs) has a complete list of links to all official documentation. ## Help and Support For additional information, see the [support page](http://visidata.org/support). ## vdtui The core `vdtui.py` can be used to quickly create efficient terminal workflows. These have been prototyped as proof of this concept: - [vgit](https://github.com/saulpw/vgit): a git interface - [vsh](http://github.com/saulpw/vsh): a collection of utilities like `vping` and `vtop`. - [vdgalcon](https://github.com/saulpw/vdgalcon): a port of the classic game [Galactic Conquest](https://www.galcon.com) Other workflows should also be created as separate apps using vdtui. These apps can be very small and provide a lot of functionality; for example, see the included [viewtsv](bin/viewtsv). ## License The innermost core file, `vdtui.py`, is a single-file stand-alone library that provides a solid framework for building text user interface apps. It is distributed under the MIT free software license, and freely available for inclusion in other projects. Other VisiData components, including the main `vd` application, addons, loaders, and other code in this repository, are available for use and distribution under GPLv3. ## Credits VisiData was created and developed by Saul Pwanson ``. Thanks to all the [contributors](CONTRIBUTING.md#contributors), and to those wonderful users who provide feedback, for making VisiData the awesome tool that it is. visidata-1.0/visidata/0000770000175100017520000000000013232247165014633 5ustar anjaanja00000000000000visidata-1.0/visidata/cmdlog.py0000660000175100017520000002623213232246431016453 0ustar anjaanja00000000000000import time import operator import threading from visidata import * option('replay_wait', 0.0, 'time to wait between replayed commands, in seconds') option('disp_replay_play', '▶', 'status indicator for active replay') option('disp_replay_pause', '‖', 'status indicator for paused replay') option('replay_movement', False, 'insert movements during replay') globalCommand('D', 'vd.push(vd.cmdlog)', 'open CommandLog', 'open-cmdlog') globalCommand('^D', 'saveSheet(vd.cmdlog, inputFilename("save to: ", value=fnSuffix("cmdlog-{0}.vd") or "cmdlog.vd"))', 'save CommandLog to new .vd file', 'save-cmdlog') globalCommand('^U', 'CommandLog.togglePause()', 'pause/resume replay', 'toggle-replay') globalCommand('^I', '(CommandLog.currentReplay or error("no replay to advance")).advance()', 'execute next row in replaying sheet', 'step-replay') globalCommand('^K', '(CommandLog.currentReplay or error("no replay to cancel")).cancel()', 'cancel current replay', 'cancel-replay') #globalCommand('KEY_BACKSPACE', 'vd.cmdlog.undo()', 'remove last action on commandlog and replay') globalCommand('status', 'status(input("status: ", display=False))', 'show given status message') # not necessary to log movements and scrollers nonLogKeys = 'KEY_DOWN KEY_UP KEY_NPAGE KEY_PPAGE kDOWN kUP j k gj gk ^F ^B r < > { } / ? n N gg G g/ g?'.split() nonLogKeys += 'KEY_LEFT KEY_RIGHT h l gh gl c'.split() nonLogKeys += 'zk zj zt zz zb zh zl zKEY_LEFT zKEY_RIGHT'.split() nonLogKeys += '^L ^C ^U ^K ^I ^D KEY_RESIZE'.split() option('rowkey_prefix', 'キ', 'string prefix for rowkey in the cmdlog') def itemsetter(i): def g(obj, v): obj[i] = v return g def namedlist(objname, fieldnames): class NamedListTemplate(list): __name__ = objname _fields = fieldnames def __init__(self, L=None): if L is None: L = [None for f in fieldnames] super().__init__(L) for i, attrname in enumerate(fieldnames): # create property getter/setter for each field setattr(NamedListTemplate, attrname, property(operator.itemgetter(i), itemsetter(i))) return NamedListTemplate CommandLogRow = namedlist('CommandLogRow', 'sheet col row keystrokes input comment'.split()) def fnSuffix(template): for i in range(1, 10): fn = template.format(i) if not Path(fn).exists(): return fn def getColVisibleIdxByFullName(sheet, name): for i, c in enumerate(sheet.visibleCols): if name == c.name: return i def keystr(k): return ','.join(map(str, k)) def getRowIdxByKey(sheet, k): for i, r in enumerate(sheet.rows): if keystr(sheet.rowkey(r)) == k: return i def loggable(keystrokes): if keystrokes in nonLogKeys: return False if keystrokes.startswith('move-'): return False return True def open_vd(p): return CommandLog(p.name, source=p) # rowdef: CommandLog class CommandLog(Sheet): 'Log of commands for current session.' rowtype = 'commands' commands = [ Command('x', 'sheet.replayOne(cursorRow); status("replayed one row")', 'replay command in current row'), Command('gx', 'sheet.replay()', 'replay contents of entire CommandLog'), Command('^C', 'sheet.cursorRowIndex = sheet.nRows', 'abort replay'), ] columns = [ColumnAttr(x) for x in CommandLogRow._fields] paused = False currentReplay = None # CommandLog replaying currently currentReplayRow = None # must be global, to allow replay semaphore = threading.Semaphore(0) filetype = 'vd' def __init__(self, name, source=None, **kwargs): super().__init__(name, source=source, **kwargs) self.currentActiveRow = None self.sheetmap = {} # sheet.name -> vs def newRow(self): return CommandLogRow() def reload(self): reload_tsv_sync(self, header=1) # .vd files always have a header row, regardless of options self.rows = [CommandLogRow(r) for r in self.rows] def undo(self): 'Delete last command, reload sources, and replay entire log.' if len(self.rows) < 2: error('no more to undo') deletedRow = self.rows[-2] # the command to undo del self.rows[-2:] # delete the previous command and the undo command vd().sheets = [self] self.sheetmap = {} for r in self.rows: self.replayOne(r) status('undid "%s"' % deletedRow.keystrokes) def beforeExecHook(self, sheet, keystrokes, args=''): if sheet is self: return # don't record editlog commands if self.currentActiveRow: self.afterExecSheet(sheet, False, '') cmd = sheet.getCommand(keystrokes) sheetname, colname, rowname = '', '', '' if sheet and keystrokes != 'o': contains = lambda s, *substrs: any((a in s) for a in substrs) sheetname = sheet.name if contains(cmd.execstr, 'cursorValue', 'cursorCell', 'cursorRow') and sheet.rows: k = sheet.rowkey(sheet.cursorRow) rowname = (options.rowkey_prefix + keystr(k)) if k else sheet.cursorRowIndex if contains(cmd.execstr, 'cursorValue', 'cursorCell', 'cursorCol', 'cursorVisibleCol'): colname = sheet.cursorCol.name or sheet.visibleCols.index(sheet.cursorCol) self.currentActiveRow = CommandLogRow([sheetname, colname, rowname, keystrokes, args, cmd.helpstr]) def afterExecSheet(self, sheet, escaped, err): 'Records currentActiveRow' if not self.currentActiveRow: # nothing to record return if err: self.currentActiveRow[-1] += ' [%s]' % err if sheet is not self: # don't record jumps to cmdlog # remove user-aborted commands and simple movements if not escaped and loggable(self.currentActiveRow.keystrokes): self.addRow(self.currentActiveRow) self.currentActiveRow = None def openHook(self, vs, src): self.addRow(CommandLogRow(['', '', '', 'o', src, 'open file'])) def getSheet(self, sheetname): vs = self.sheetmap.get(sheetname) if not vs: matchingSheets = [x for x in vd().sheets if x.name == sheetname] if not matchingSheets: status(','.join(x.name for x in vd().sheets)) return None vs = matchingSheets[0] return vs @classmethod def togglePause(self): if not CommandLog.currentReplay: status('no replay to pause') else: if self.paused: CommandLog.currentReplay.advance() self.paused = not self.paused status('paused' if self.paused else 'resumed') def advance(self): CommandLog.semaphore.release() def cancel(self): CommandLog.currentReplayRow = None CommandLog.currentReplay = None self.advance() def moveToReplayContext(self, r): 'set the sheet/row/col to the values in the replay row. return sheet' if not r.sheet: # assert not r.col and not r.row return self # any old sheet should do, row/column don't matter else: vs = self.getSheet(r.sheet) or error('no sheets named %s' % r.sheet) if r.row: try: rowidx = int(r.row) except ValueError: k = r.row[1:] # trim rowkey_prefix rowidx = getRowIdxByKey(vs, k) if rowidx is None: error('no row %s' % r.row) if options.replay_movement: while vs.cursorRowIndex != rowidx: vs.cursorRowIndex += 1 if (rowidx - vs.cursorRowIndex) > 0 else -1 while not self.delay(0.5): pass else: vs.cursorRowIndex = rowidx if r.col: try: vcolidx = int(r.col) except ValueError: vcolidx = getColVisibleIdxByFullName(vs, r.col) if vcolidx is None: error('no column %s' % r.col) if options.replay_movement: while vs.cursorVisibleColIndex != vcolidx: vs.cursorVisibleColIndex += 1 if (vcolidx - vs.cursorVisibleColIndex) > 0 else -1 while not self.delay(0.5): pass assert vs.cursorVisibleColIndex == vcolidx else: vs.cursorVisibleColIndex = vcolidx return vs def delay(self, factor=1): 'returns True if delay satisfied' acquired = CommandLog.semaphore.acquire(timeout=options.replay_wait*factor if not self.paused else None) return acquired or not self.paused def replayOne(self, r): 'Replay the command in one given row.' CommandLog.currentReplayRow = r vs = self.moveToReplayContext(r) vd().keystrokes = r.keystrokes escaped = vs.exec_keystrokes(r.keystrokes) CommandLog.currentReplayRow = None if escaped: # escape during replay aborts replay status('replay aborted') return escaped def replay_sync(self, live=False): 'Replay all commands in log.' self.sheetmap.clear() self.cursorRowIndex = 0 CommandLog.currentReplay = self with Progress(total=len(self.rows)) as prog: while self.cursorRowIndex < len(self.rows): if CommandLog.currentReplay is None: status('replay canceled') return vd().statuses = [] if self.replayOne(self.cursorRow): CommandLog.currentReplay = None return self.cursorRowIndex += 1 prog.addProgress(1) sync(1 if live else 0) # expect this thread also if playing live while not self.delay(): pass status('replay complete') CommandLog.currentReplay = None @async def replay(self): 'Inject commands into live execution with interface.' self.replay_sync(live=True) def getLastArgs(self): 'Get user input for the currently playing command.' if CommandLog.currentReplayRow: return CommandLog.currentReplayRow.input return None def setLastArgs(self, args): 'Set user input on last command, if not already set.' # only set if not already set (second input usually confirmation) if self.currentActiveRow is not None: if not self.currentActiveRow.input: self.currentActiveRow.input = args @property def replayStatus(self): x = options.disp_replay_pause if self.paused else options.disp_replay_play return ' │ %s %s/%s' % (x, self.cursorRowIndex, len(self.rows)) vd().cmdlog = CommandLog('cmdlog') vd().cmdlog.rows = [] # so it can be added to immediately vd().addHook('preexec', vd().cmdlog.beforeExecHook) vd().addHook('postexec', vd().cmdlog.afterExecSheet) vd().addHook('preedit', vd().cmdlog.getLastArgs) vd().addHook('postedit', vd().cmdlog.setLastArgs) vd().addHook('rstatus', lambda sheet: CommandLog.currentReplay and (CommandLog.currentReplay.replayStatus, 'green')) visidata-1.0/visidata/graph.py0000660000175100017520000001243313223031561016301 0ustar anjaanja00000000000000from visidata import * option('color_graph_axis', 'bold', 'color for graph axis labels') globalCommand('.', 'vd.push(GraphSheet(sheet.name+"_graph", sheet, selectedRows or rows, keyCols, [cursorCol]))', 'graph the current column vs key columns Numeric key column is on the x-axis, while categorical key columns determin color') globalCommand('g.', 'vd.push(GraphSheet(sheet.name+"_graph", sheet, selectedRows or rows, keyCols, numericCols(nonKeyVisibleCols)))', 'open a graph of all visible numeric columns vs key column') def numericCols(cols): # isNumeric from describe.py return [c for c in cols if isNumeric(c)] class InvertedCanvas(Canvas): commands = Canvas.commands + [ # swap directions of up/down Command('move-up', 'sheet.cursorBox.ymin += cursorBox.h', 'move cursor up'), Command('move-down', 'sheet.cursorBox.ymin -= cursorBox.h', 'move cursor down'), Command('zj', 'sheet.cursorBox.ymin -= canvasCharHeight', 'move cursor down one line'), Command('zk', 'sheet.cursorBox.ymin += canvasCharHeight', 'move cursor up one line'), Command('J', 'sheet.cursorBox.h -= canvasCharHeight', 'decrease cursor height'), Command('K', 'sheet.cursorBox.h += canvasCharHeight', 'increase cursor height'), ] def zoomTo(self, bbox): super().zoomTo(bbox) self.fixPoint(Point(self.plotviewBox.xmin, self.plotviewBox.ymax), bbox.xymin) def plotpixel(self, x, y, attr, row=None): y = self.plotviewBox.ymax-y+4 self.pixels[y][x][attr].append(row) def scaleY(self, canvasY): 'returns plotter y coordinate, with y-axis inverted' plotterY = super().scaleY(canvasY) return (self.plotviewBox.ymax-plotterY+4) def canvasH(self, plotterY): return (self.plotviewBox.ymax-plotterY)/self.yScaler @property def canvasMouse(self): p = super().canvasMouse p.y = self.visibleBox.ymin + (self.plotviewBox.ymax-self.plotterMouse.y)/self.yScaler return p # provides axis labels, legend class GraphSheet(InvertedCanvas): def __init__(self, name, sheet, rows, xcols, ycols, **kwargs): super().__init__(name, sheet, sourceRows=rows, **kwargs) self.xcols = xcols self.ycols = [ycol for ycol in ycols if isNumeric(ycol)] or error('%s is non-numeric' % '/'.join(yc.name for yc in ycols)) @async def reload(self): nerrors = 0 nplotted = 0 self.reset() status('loading data points') catcols = [c for c in self.xcols if not isNumeric(c)] numcols = numericCols(self.xcols) for ycol in self.ycols: for rownum, row in enumerate(Progress(self.sourceRows)): # rows being plotted from source try: k = tuple(c.getValue(row) for c in catcols) if catcols else (ycol.name,) # convert deliberately to float (to e.g. linearize date) graph_x = float(numcols[0].type(numcols[0].getValue(row))) if numcols else rownum graph_y = ycol.type(ycol.getValue(row)) attr = self.plotColor(k) self.point(graph_x, graph_y, attr, row) nplotted += 1 except Exception: nerrors += 1 if options.debug: raise status('loaded %d points (%d errors)' % (nplotted, nerrors)) self.setZoom(1.0) self.refresh() def setZoom(self, zoomlevel=None): super().setZoom(zoomlevel) self.createLabels() def add_y_axis_label(self, frac): amt = self.visibleBox.ymin + frac*self.visibleBox.h if isinstance(self.canvasBox.ymin, int): txt = '%d' % amt elif isinstance(self.canvasBox.ymin, float): txt = '%.02f' % amt else: txt = str(frac) # plot y-axis labels on the far left of the canvas, but within the plotview height-wise attr = colors[options.color_graph_axis] self.plotlabel(0, self.plotviewBox.ymin + (1.0-frac)*self.plotviewBox.h, txt, attr) def add_x_axis_label(self, frac): amt = self.visibleBox.xmin + frac*self.visibleBox.w txt = ','.join(xcol.format(xcol.type(amt)) for xcol in self.xcols if isNumeric(xcol)) # plot x-axis labels below the plotviewBox.ymax, but within the plotview width-wise attr = colors[options.color_graph_axis] self.plotlabel(self.plotviewBox.xmin+frac*self.plotviewBox.w, self.plotviewBox.ymax+4, txt, attr) def createLabels(self): self.gridlabels = [] # y-axis self.add_y_axis_label(1.00) self.add_y_axis_label(0.75) self.add_y_axis_label(0.50) self.add_y_axis_label(0.25) self.add_y_axis_label(0.00) # x-axis self.add_x_axis_label(1.00) self.add_x_axis_label(0.75) self.add_x_axis_label(0.50) self.add_x_axis_label(0.25) self.add_x_axis_label(0.00) # TODO: if 0 line is within visible bounds, explicitly draw the axis # TODO: grid lines corresponding to axis labels xname = ','.join(xcol.name for xcol in self.xcols if isNumeric(xcol)) or 'row#' self.plotlabel(0, self.plotviewBox.ymax+4, '%*s»' % (int(self.leftMarginPixels/2-2), xname), colors[options.color_graph_axis]) visidata-1.0/visidata/canvas.py0000660000175100017520000005245413226522612016467 0ustar anjaanja00000000000000 from collections import defaultdict, Counter from visidata import * # see www/design/graphics.md option('show_graph_labels', True, 'show axes and legend on graph') option('plot_colors', 'green red yellow cyan magenta white 38 136 168', 'list of distinct colors to use for plotting distinct objects') option('disp_pixel_random', False, 'randomly choose attr from set of pixels instead of most common') option('zoom_incr', 2.0, 'amount to multiply current zoomlevel by when zooming') option('color_graph_hidden', '238 blue', 'color of legend for hidden attribute') class Point: def __init__(self, x, y): self.x = x self.y = y def __str__(self): if isinstance(self.x, int): return '(%d,%d)' % (self.x, self.y) else: return '(%.02f,%.02f)' % (self.x, self.y) @property def xy(self): return (self.x, self.y) class Box: def __init__(self, x, y, w=0, h=0): self.xmin = x self.ymin = y self.w = w self.h = h @property def xymin(self): return Point(self.xmin, self.ymin) @property def xmax(self): return self.xmin + self.w @property def ymax(self): return self.ymin + self.h @property def center(self): return Point(self.xcenter, self.ycenter) @property def xcenter(self): return self.xmin + self.w/2 @property def ycenter(self): return self.ymin + self.h/2 def contains(self, x, y): return x >= self.xmin and \ x < self.xmax and \ y >= self.ymin and \ y < self.ymax def BoundingBox(x1, y1, x2, y2): return Box(min(x1, x2), min(y1, y2), abs(x2-x1), abs(y2-y1)) def clipline(x1, y1, x2, y2, xmin, ymin, xmax, ymax): 'Liang-Barsky algorithm, returns [xn1,yn1,xn2,yn2] of clipped line within given area, or None' dx = x2-x1 dy = y2-y1 pq = [ (-dx, x1-xmin), # left ( dx, xmax-x1), # right (-dy, y1-ymin), # bottom ( dy, ymax-y1), # top ] u1, u2 = 0, 1 for p, q in pq: if p < 0: # from outside to inside u1 = max(u1, q/p) elif p > 0: # from inside to outside u2 = min(u2, q/p) else: # p == 0: # parallel to bbox if q < 0: # completely outside bbox return None if u1 > u2: # completely outside bbox return None xn1 = x1 + dx*u1 yn1 = y1 + dy*u1 xn2 = x1 + dx*u2 yn2 = y1 + dy*u2 return xn1, yn1, xn2, yn2 def iterline(x1, y1, x2, y2): 'Yields (x, y) coords of line from (x1, y1) to (x2, y2)' xdiff = abs(x2-x1) ydiff = abs(y2-y1) xdir = 1 if x1 <= x2 else -1 ydir = 1 if y1 <= y2 else -1 r = max(xdiff, ydiff) if r == 0: # point, not line yield x1, y1 else: x, y = x1, y1 i = 0 while i <= r: x += xdir * xdiff / r y += ydir * ydiff / r yield x, y i += 1 def anySelected(vs, rows): for r in rows: if vs.isSelected(r): return True # - width/height are exactly equal to the number of pixels displayable, and can change at any time. # - needs to refresh from source on resize class Plotter(Sheet): 'pixel-addressable display of entire terminal with (x,y) integer pixel coordinates' columns=[Column('')] # to eliminate errors outside of draw() commands=[ Command('^L', 'refresh()', 'redraw all pixels on canvas'), Command('v', 'options.show_graph_labels = not options.show_graph_labels', 'toggle show_graph_labels'), Command('KEY_RESIZE', 'refresh()', ''), ] def __init__(self, name, **kwargs): super().__init__(name, **kwargs) self.labels = [] # (x, y, text, attr) self.hiddenAttrs = set() self.needsRefresh = False self.resetCanvasDimensions() def resetCanvasDimensions(self): 'sets total available canvas dimensions' self.plotwidth = vd().windowWidth*2 self.plotheight = (vd().windowHeight-1)*4 # exclude status line # pixels[y][x] = { attr: list(rows), ... } self.pixels = [[defaultdict(list) for x in range(self.plotwidth)] for y in range(self.plotheight)] def plotpixel(self, x, y, attr, row=None): self.pixels[y][x][attr].append(row) def plotline(self, x1, y1, x2, y2, attr, row=None): for x, y in iterline(x1, y1, x2, y2): self.plotpixel(round(x), round(y), attr, row) def plotlabel(self, x, y, text, attr): self.labels.append((x, y, text, attr)) def plotlegend(self, i, txt, attr): self.plotlabel(self.plotwidth-30, i*4, txt, attr) @property def plotterCursorBox(self): 'Returns pixel bounds of cursor as a Box. Override to provide a cursor.' return Box(0,0,0,0) @property def plotterMouse(self): return Point(self.mouseX*2, self.mouseY*4) def getPixelAttrRandom(self, x, y): 'weighted-random choice of attr at this pixel.' c = list(attr for attr, rows in self.pixels[y][x].items() for r in rows if attr not in self.hiddenAttrs) return random.choice(c) if c else 0 def getPixelAttrMost(self, x, y): 'most common attr at this pixel.' r = self.pixels[y][x] c = sorted((len(rows), attr, rows) for attr, rows in r.items() if attr not in self.hiddenAttrs) if not c: return 0 _, attr, rows = c[-1] if anySelected(self.source, rows): attr, _ = colors.update(attr, 8, 'bold', 10) return attr def hideAttr(self, attr, hide=True): if hide: self.hiddenAttrs.add(attr) else: self.hiddenAttrs.remove(attr) self.plotlegends() def rowsWithin(self, bbox): 'return list of deduped rows within bbox' ret = {} for y in range(bbox.ymin, bbox.ymax+1): for x in range(bbox.xmin, bbox.xmax+1): for attr, rows in self.pixels[y][x].items(): if attr not in self.hiddenAttrs: for r in rows: ret[id(r)] = r return list(ret.values()) def draw(self, scr): if self.needsRefresh: self.render() scr.erase() if self.pixels: cursorBBox = self.plotterCursorBox getPixelAttr = self.getPixelAttrRandom if options.disp_pixel_random else self.getPixelAttrMost for char_y in range(0, vd().windowHeight-1): # save one line for status for char_x in range(0, vd().windowWidth): block_attrs = [ getPixelAttr(char_x*2 , char_y*4 ), getPixelAttr(char_x*2 , char_y*4+1), getPixelAttr(char_x*2 , char_y*4+2), getPixelAttr(char_x*2+1, char_y*4 ), getPixelAttr(char_x*2+1, char_y*4+1), getPixelAttr(char_x*2+1, char_y*4+2), getPixelAttr(char_x*2 , char_y*4+3), getPixelAttr(char_x*2+1, char_y*4+3), ] pow2 = 1 braille_num = 0 for c in block_attrs: if c: braille_num += pow2 pow2 *= 2 if braille_num != 0: attr = Counter(c for c in block_attrs if c).most_common(1)[0][0] else: attr = 0 if cursorBBox.contains(char_x*2, char_y*4) or \ cursorBBox.contains(char_x*2+1, char_y*4+3): attr, _ = colors.update(attr, 0, options.color_current_row, 10) if attr: scr.addstr(char_y, char_x, chr(0x2800+braille_num), attr) if options.show_graph_labels: for pix_x, pix_y, txt, attr in self.labels: clipdraw(scr, int(pix_y/4), int(pix_x/2), txt, attr, len(txt)) # - has a cursor, of arbitrary position and width/height (not restricted to current zoom) class Canvas(Plotter): 'zoomable/scrollable virtual canvas with (x,y) coordinates in arbitrary units' rowtype = 'plots' aspectRatio = 0.0 leftMarginPixels = 10*2 rightMarginPixels = 6*2 topMarginPixels = 0 bottomMarginPixels = 2*4 # reserve bottom line for x axis commands = Plotter.commands + [ Command('move-left', 'sheet.cursorBox.xmin -= cursorBox.w', ''), Command('move-right', 'sheet.cursorBox.xmin += cursorBox.w', ''), Command('move-down', 'sheet.cursorBox.ymin += cursorBox.h', ''), Command('move-up', 'sheet.cursorBox.ymin -= cursorBox.h', ''), Command('zh', 'sheet.cursorBox.xmin -= canvasCharWidth', ''), Command('zl', 'sheet.cursorBox.xmin += canvasCharWidth', ''), Command('zj', 'sheet.cursorBox.ymin += canvasCharHeight', ''), Command('zk', 'sheet.cursorBox.ymin -= canvasCharHeight', ''), Command('gH', 'sheet.cursorBox.w /= 2', ''), Command('gL', 'sheet.cursorBox.w *= 2', ''), Command('gJ', 'sheet.cursorBox.h /= 2', ''), Command('gK', 'sheet.cursorBox.h *= 2', ''), Command('H', 'sheet.cursorBox.w -= canvasCharWidth', ''), Command('L', 'sheet.cursorBox.w += canvasCharWidth', ''), Command('J', 'sheet.cursorBox.h += canvasCharHeight', ''), Command('K', 'sheet.cursorBox.h -= canvasCharHeight', ''), Command('zz', 'zoomTo(cursorBox)', 'set visible bounds to cursor'), Command('-', 'tmp=cursorBox.center; setZoom(zoomlevel*options.zoom_incr); fixPoint(plotviewBox.center, tmp)', 'zoom out from cursor center'), Command('+', 'tmp=cursorBox.center; setZoom(zoomlevel/options.zoom_incr); fixPoint(plotviewBox.center, tmp)', 'zoom into cursor center'), Command('_', 'sheet.canvasBox = None; sheet.visibleBox = None; setZoom(1.0); refresh()', 'zoom to fit full extent'), Command('z_', 'sheet.aspectRatio = float(input("aspect ratio=", value=aspectRatio)); refresh()', 'set aspect ratio'), # set cursor box with left click Command('BUTTON1_PRESSED', 'sheet.cursorBox = Box(*canvasMouse.xy)', 'start cursor box with left mouse button press'), Command('BUTTON1_RELEASED', 'setCursorSize(canvasMouse)', 'end cursor box with left mouse button release'), Command('BUTTON3_PRESSED', 'sheet.anchorPoint = canvasMouse', 'mark grid point to move'), Command('BUTTON3_RELEASED', 'fixPoint(plotterMouse, anchorPoint)', 'mark canvas anchor point'), Command('BUTTON4_PRESSED', 'tmp=canvasMouse; setZoom(zoomlevel/options.zoom_incr); fixPoint(plotterMouse, tmp)', 'zoom in with scroll wheel'), Command('REPORT_MOUSE_POSITION', 'tmp=canvasMouse; setZoom(zoomlevel*options.zoom_incr); fixPoint(plotterMouse, tmp)', 'zoom out with scroll wheel'), Command('s', 'source.select(list(rowsWithin(plotterCursorBox)))', 'select rows on source sheet contained within canvas cursor'), Command('t', 'source.toggle(list(rowsWithin(plotterCursorBox)))', 'toggle selection of rows on source sheet contained within canvas cursor'), Command('u', 'source.unselect(list(rowsWithin(plotterCursorBox)))', 'unselect rows on source sheet contained within canvas cursor'), Command(ENTER, 'vs=copy(source); vs.rows=list(rowsWithin(plotterCursorBox)); vd.push(vs)', 'Open sheet of source rows contained within canvas cursor'), Command('gs', 'source.select(list(rowsWithin(plotterVisibleBox)))', 'select rows visible on screen'), Command('gt', 'source.toggle(list(rowsWithin(plotterVisibleBox)))', 'toggle selection of rows visible on screen'), Command('gu', 'source.unselect(list(rowsWithin(plotterVisibleBox)))', 'unselect rows visible on screen'), Command('g'+ENTER, 'vs=copy(source); vs.rows=list(rowsWithin(plotterVisibleBox)); vd.push(vs)', 'open sheet of source rows visible on screen'), ] def __init__(self, name, source=None, **kwargs): super().__init__(name, source=source, **kwargs) self.canvasBox = None # bounding box of entire canvas, in canvas units self.visibleBox = None # bounding box of visible canvas, in canvas units self.cursorBox = None # bounding box of cursor, in canvas units self.zoomlevel = 1.0 self.needsRefresh = False self.polylines = [] # list of ([(canvas_x, canvas_y), ...], attr, row) self.gridlabels = [] # list of (grid_x, grid_y, label, attr, row) self.legends = collections.OrderedDict() # txt: attr (visible legends only) self.plotAttrs = {} # key: attr (all keys, for speed) self.reset() def __len__(self): return len(self.polylines) def reset(self): 'clear everything in preparation for a fresh reload()' self.polylines.clear() self.legends.clear() self.plotAttrs.clear() self.unusedAttrs = list(colors[colorname.translate(str.maketrans('_', ' '))] for colorname in options.plot_colors.split())[::-1] def plotColor(self, k): attr = self.plotAttrs.get(k, None) if attr is None: if len(self.unusedAttrs) > 1: attr = self.unusedAttrs.pop(0) legend = ' '.join(str(x) for x in k) else: attr = self.unusedAttrs[0] legend = '[other]' self.legends[legend] = attr self.plotAttrs[k] = attr self.plotlegends() return attr def resetCanvasDimensions(self): super().resetCanvasDimensions() self.plotviewBox = BoundingBox(self.leftMarginPixels, self.topMarginPixels, self.plotwidth-self.rightMarginPixels, self.plotheight-self.bottomMarginPixels) @property def statusLine(self): return 'canvas %s visible %s cursor %s' % (self.canvasBox, self.visibleBox, self.cursorBox) @property def canvasMouse(self): return Point(self.visibleBox.xmin + (self.plotterMouse.x-self.plotviewBox.xmin)/self.xScaler, self.visibleBox.ymin + (self.plotterMouse.y-self.plotviewBox.ymin)/self.yScaler) def setCursorSize(self, p): 'sets width based on diagonal corner p' self.cursorBox = BoundingBox(self.cursorBox.xmin, self.cursorBox.ymin, p.x, p.y) self.cursorBox.w = max(self.cursorBox.w, self.canvasCharWidth) self.cursorBox.h = max(self.cursorBox.h, self.canvasCharHeight) @property def canvasCharWidth(self): 'Width in canvas units of a single char in the terminal' return self.visibleBox.w*2/self.plotviewBox.w @property def canvasCharHeight(self): 'Height in canvas units of a single char in the terminal' return self.visibleBox.h*4/self.plotviewBox.h @property def plotterVisibleBox(self): return BoundingBox(self.scaleX(self.visibleBox.xmin), self.scaleY(self.visibleBox.ymin), self.scaleX(self.visibleBox.xmax), self.scaleY(self.visibleBox.ymax)) @property def plotterCursorBox(self): if self.cursorBox is None: return Box(0,0,0,0) return BoundingBox(self.scaleX(self.cursorBox.xmin), self.scaleY(self.cursorBox.ymin), self.scaleX(self.cursorBox.xmax), self.scaleY(self.cursorBox.ymax)) def point(self, x, y, attr, row=None): self.polylines.append(([(x, y)], attr, row)) def line(self, x1, y1, x2, y2, attr, row=None): self.polylines.append(([(x1, y1), (x2, y2)], attr, row)) def polyline(self, vertexes, attr, row=None): 'adds lines for (x,y) vertexes of a polygon' self.polylines.append((vertexes, attr, row)) def polygon(self, vertexes, attr, row=None): 'adds lines for (x,y) vertexes of a polygon' self.polylines.append((vertexes + [vertexes[0]], attr, row)) def label(self, x, y, text, attr, row=None): self.gridlabels.append((x, y, text, attr, row)) def fixPoint(self, plotterPoint, canvasPoint): 'adjust visibleBox.xymin so that canvasPoint is plotted at plotterPoint' self.visibleBox.xmin = canvasPoint.x - self.canvasW(plotterPoint.x-self.plotviewBox.xmin) self.visibleBox.ymin = canvasPoint.y - self.canvasH(plotterPoint.y-self.plotviewBox.ymin) self.refresh() def zoomTo(self, bbox): 'set visible area to bbox, maintaining aspectRatio if applicable' self.fixPoint(self.plotviewBox.xymin, bbox.xymin) self.zoomlevel=max(bbox.w/self.canvasBox.w, bbox.h/self.canvasBox.h) def setZoom(self, zoomlevel=None): if zoomlevel: self.zoomlevel = zoomlevel self.resetBounds() self.plotlegends() def resetBounds(self): if not self.canvasBox: xmin, ymin, xmax, ymax = None, None, None, None for vertexes, attr, row in self.polylines: for x, y in vertexes: if xmin is None or x < xmin: xmin = x if ymin is None or y < ymin: ymin = y if xmax is None or x > xmax: xmax = x if ymax is None or y > ymax: ymax = y self.canvasBox = BoundingBox(xmin or 0, ymin or 0, xmax or 1, ymax or 1) if not self.visibleBox: # initialize minx/miny, but w/h must be set first to center properly self.visibleBox = Box(0, 0, self.plotviewBox.w/self.xScaler, self.plotviewBox.h/self.yScaler) self.visibleBox.xmin = self.canvasBox.xcenter - self.visibleBox.w/2 self.visibleBox.ymin = self.canvasBox.ycenter - self.visibleBox.h/2 else: self.visibleBox.w = self.plotviewBox.w/self.xScaler self.visibleBox.h = self.plotviewBox.h/self.yScaler if not self.cursorBox: self.cursorBox = Box(self.visibleBox.xmin, self.visibleBox.ymin, self.canvasCharWidth, self.canvasCharHeight) def plotlegends(self): # display labels for i, (legend, attr) in enumerate(self.legends.items()): self._commands[str(i+1)] = Command(str(i+1), 'hideAttr(%s, %s not in hiddenAttrs)' % (attr, attr), 'toggle display of "%s"' % legend) if attr in self.hiddenAttrs: attr = colors[options.color_graph_hidden] self.plotlegend(i, '%s:%s'%(i+1,legend), attr) def checkCursor(self): 'override Sheet.checkCursor' return False @property def xScaler(self): xratio = self.plotviewBox.w/(self.canvasBox.w*self.zoomlevel) if self.aspectRatio: yratio = self.plotviewBox.h/(self.canvasBox.h*self.zoomlevel) return self.aspectRatio*min(xratio, yratio) else: return xratio @property def yScaler(self): yratio = self.plotviewBox.h/(self.canvasBox.h*self.zoomlevel) if self.aspectRatio: xratio = self.plotviewBox.w/(self.canvasBox.w*self.zoomlevel) return min(xratio, yratio) else: return yratio def scaleX(self, x): 'returns plotter x coordinate' return round(self.plotviewBox.xmin+(x-self.visibleBox.xmin)*self.xScaler) def scaleY(self, y): 'returns plotter y coordinate' return round(self.plotviewBox.ymin+(y-self.visibleBox.ymin)*self.yScaler) def canvasW(self, plotter_width): 'plotter X units to canvas units' return plotter_width/self.xScaler def canvasH(self, plotter_height): 'plotter Y units to canvas units' return plotter_height/self.yScaler def refresh(self): 'triggers render() on next draw()' self.needsRefresh = True def render(self): 'resets plotter, cancels previous render threads, spawns a new render' self.needsRefresh = False cancelThread(*(t for t in self.currentThreads if t.name == 'plotAll_async')) self.labels.clear() self.resetCanvasDimensions() self.render_async() @async def render_async(self): 'plots points and lines and text onto the Plotter' self.setZoom() bb = self.visibleBox xmin, ymin, xmax, ymax = bb.xmin, bb.ymin, bb.xmax, bb.ymax xfactor, yfactor = self.xScaler, self.yScaler plotxmin, plotymin = self.plotviewBox.xmin, self.plotviewBox.ymin for vertexes, attr, row in Progress(self.polylines): if len(vertexes) == 1: # single point x1, y1 = vertexes[0] if xmin <= x1 <= xmax and ymin <= y1 <= ymax: x = plotxmin+(x1-xmin)*xfactor y = plotymin+(y1-ymin)*yfactor self.plotpixel(round(x), round(y), attr, row) continue prev_x, prev_y = vertexes[0] for x, y in vertexes[1:]: r = clipline(prev_x, prev_y, x, y, xmin, ymin, xmax, ymax) if r: x1, y1, x2, y2 = r x1 = plotxmin+(x1-xmin)*xfactor y1 = plotymin+(y1-ymin)*yfactor x2 = plotxmin+(x2-xmin)*xfactor y2 = plotymin+(y2-ymin)*yfactor self.plotline(x1, y1, x2, y2, attr, row) prev_x, prev_y = x, y for x, y, text, attr, row in Progress(self.gridlabels): self.plotlabel(self.scaleX(x), self.scaleY(y), text, attr, row) visidata-1.0/visidata/async.py0000660000175100017520000001055513232246431016324 0ustar anjaanja00000000000000import ctypes import threading import pstats import cProfile from .vdtui import * min_thread_time_s = 0.10 # only keep threads that take longer than this number of seconds option('profile_threads', False, 'profile async threads') option('min_memory_mb', 0, 'minimum memory to continue loading and async processing') globalCommand('^C', 'cancelThread(*sheet.currentThreads or error("no active threads on this sheet"))', 'abort all threads on current sheet') globalCommand('g^C', 'cancelThread(*vd.threads or error("no threads"))', 'abort all secondary threads') globalCommand('^T', 'vd.push(vd.threadsSheet)', 'open Threads Sheet') globalCommand('^_', 'toggleProfiling(threading.current_thread())', 'turn profiling on for main process') class ProfileSheet(TextSheet): commands = TextSheet.commands + [ Command('z^S', 'profile.dump_stats(input("save profile to: ", value=name+".prof"))', 'save profile'), ] def __init__(self, name, pr): super().__init__(name, getProfileResults(pr).splitlines()) self.profile = pr def toggleProfiling(t): if not t.profile: t.profile = cProfile.Profile() t.profile.enable() status('profiling of main thread enabled') else: t.profile.disable() status('profiling of main thread disabled') # define @async for potentially long-running functions # when function is called, instead launches a thread # ENTER on that row pushes a profile of the thread class ThreadProfiler: def __init__(self, thread): self.thread = thread if options.profile_threads: self.thread.profile = cProfile.Profile() else: self.thread.profile = None def __enter__(self): if self.thread.profile: self.thread.profile.enable() return self def __exit__(self, exc_type, exc_val, tb): if self.thread.profile: self.thread.profile.disable() # remove very-short-lived async actions if elapsed_s(self.thread) < min_thread_time_s: vd().threads.remove(self.thread) @functools.wraps(vd().toplevelTryFunc) def threadProfileCode(vdself, func, *args, **kwargs): 'Profile @async threads if `options.profile_threads` is set.' with ThreadProfiler(threading.current_thread()) as prof: try: prof.thread.status = threadProfileCode.__wrapped__(vdself, func, *args, **kwargs) except EscapeException as e: prof.thread.status = e def getProfileResults(pr): s = io.StringIO() ps = pstats.Stats(pr, stream=s) ps.strip_dirs() ps.sort_stats('cumulative') ps.print_stats() return s.getvalue() def cancelThread(*threads, exception=EscapeException): 'Raise exception on another thread.' for t in threads: ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(t.ident), ctypes.py_object(exception)) SheetsSheet.commands += [ Command('^C', 'cancelThread(*cursorRow.currentThreads)', 'abort all threads on sheet at cursor'), ] SheetsSheet.columns += [ ColumnAttr('threads', 'currentThreads', type=len), ] # each row is an augmented threading.Thread object class ThreadsSheet(Sheet): rowtype = 'threads' commands = [ Command('d', 'cancelThread(cursorRow)', 'abort thread at current row'), Command('^C', 'd'), Command(ENTER, 'vd.push(ProfileSheet(cursorRow.name+"_profile", cursorRow.profile))', 'push profile sheet for this action'), ] columns = [ ColumnAttr('name'), Column('process_time', type=float, getter=lambda col,row: elapsed_s(row)), ColumnAttr('profile'), ColumnAttr('status'), ] def reload(self): self.rows = vd().threads def elapsed_s(t): return (t.endTime or time.process_time())-t.startTime def checkMemoryUsage(vs): min_mem = options.min_memory_mb if min_mem and vd().unfinishedThreads: tot_m, used_m, free_m = map(int, os.popen('free --total --mega').readlines()[-1].split()[1:]) ret = '[%dMB]' % free_m if free_m < min_mem: attr = 'red' status('%dMB free < %dMB minimum, stopping threads' % (free_m, min_mem)) cancelThread(*vd().unfinishedThreads) curses.flash() else: attr = 'green' return ret, attr vd().threadsSheet = ThreadsSheet('thread_history') vd().toplevelTryFunc = threadProfileCode vd().addHook('rstatus', checkMemoryUsage) visidata-1.0/visidata/freqtbl.py0000660000175100017520000001544013226522612016645 0ustar anjaanja00000000000000import math from visidata import * globalCommand('F', 'vd.push(SheetFreqTable(sheet, cursorCol))', 'open Frequency Table grouped on current column') globalCommand('gF', 'vd.push(SheetFreqTable(sheet, *keyCols))', 'open Frequency Table grouped by all key columns on the source sheet') globalCommand('zF', 'vd.push(SheetFreqTable(sheet, Column("Total", getter=lambda col,row: "Total")))', 'open one-line summary for all selected rows') theme('disp_histogram', '*', 'histogram element character') option('disp_histolen', 80, 'width of histogram column') #option('histogram_bins', 0, 'number of bins for histogram of numeric columns') #option('histogram_even_interval', False, 'if histogram bins should have even distribution of rows') ColumnsSheet.commands += [ Command(ENTER, 'vd.push(SheetFreqTable(source, cursorRow))', 'open a Frequency Table grouped on column referenced in current row') ] def getValueOrError(c, r): try: return c.getValue(r) except Exception as e: return 'error: %s' % e def valueNames(vals): return '-'.join(str(v) for v in vals) # rowdef: ([bin_values], source_rows) class SheetFreqTable(Sheet): 'Generate frequency-table sheet on currently selected column.' rowtype = 'bins' commands = [ # redefine these commands only to change the helpstr Command('t', 'toggle([cursorRow]); cursorDown(1)', 'toggle these entries in source sheet'), Command('s', 'select([cursorRow]); cursorDown(1)', 'select these entries in source sheet'), Command('u', 'unselect([cursorRow]); cursorDown(1)', 'unselect these entries in source sheet'), Command(ENTER, 'vs = copy(source); vs.name += "_"+valueNames(cursorRow[0]); vs.rows=copy(cursorRow[1]); vd.push(vs)', 'open sheet of source rows which are grouped in current cell'), # Command('v', 'options.histogram_even_interval = not options.histogram_even_interval; reload()', 'toggle histogram_even_interval option') ] def __init__(self, sheet, *columns): fqcolname = '%s_%s_freq' % (sheet.name, '-'.join(col.name for col in columns)) super().__init__(fqcolname, source=sheet) self.origCols = columns self.largest = 100 self.nKeys = len(self.origCols) self.columns = [ Column(c.name, type=c.type, width=c.width, getter=lambda col,row,i=i: row[0][i]) for i, c in enumerate(self.origCols) ] self.columns.extend([ Column('count', type=int, getter=lambda col,row: len(row[1])), Column('percent', type=float, getter=lambda col,row: len(row[1])*100/col.sheet.source.nRows), Column('histogram', type=str, getter=lambda col,row: options.disp_histogram*(options.disp_histolen*len(row[1])//col.sheet.largest), width=50), ]) aggregatedCols = [Column(aggregator.__name__+'_'+c.name, type=aggregator.type or c.type, getter=lambda col,row,origcol=c,aggr=aggregator: aggr(origcol, row[1])) for c in self.source.visibleCols for aggregator in getattr(c, 'aggregators', []) ] self.columns.extend(aggregatedCols) if aggregatedCols: # hide percent/histogram if aggregations added for c in self.columns[self.nKeys+1:self.nKeys+3]: c.width = 0 def selectRow(self, row): self.source.select(row[1]) # select all entries in the bin on the source sheet return super().selectRow(row) # then select the bin itself on this sheet def unselectRow(self, row): self.source.unselect(row[1]) return super().unselectRow(row) def numericBinning(self): nbins = options.histogram_bins or int(len(self.source.rows) ** (1./2)) origCol = self.origCols[0] self.columns[0].type = str # separate rows with errors at the column from those without errors errorbin = [] allbin = [] for row in Progress(self.source.rows): v = origCol.getTypedValue(row) if not v: errorbin.append(row) else: allbin.append((v, row)) # find bin pivots from non-error values binPivots = [] sortedValues = sorted(allbin, key=lambda x: x[0]) if options.histogram_even_interval: binsize = len(sortedValues)/nbins pivotIdx = 0 for i in range(math.ceil(len(sortedValues)/binsize)): firstVal = sortedValues[int(pivotIdx)][0] binPivots.append(firstVal) pivotIdx += binsize else: minval, maxval = sortedValues[0][0], sortedValues[-1][0] binWidth = (maxval - minval)/nbins binPivots = list((minval + binWidth*i) for i in range(0, nbins)) binPivots.append(None) # put rows into bins (as self.rows) based on values binMinIdx = 0 binMin = 0 for binMax in binPivots[1:-1]: binrows = [] for i, (v, row) in enumerate(sortedValues[binMinIdx:]): if binMax != binPivots[-2] and v > binMax: break binrows.append(row) binMaxDispVal = origCol.format(binMax) binMinDispVal = origCol.format(binMin) if binMinIdx == 0: binName = '<=%s' % binMaxDispVal elif binMax == binPivots[-2]: binName = '>=%s' % binMinDispVal else: binName = '%s-%s' % (binMinDispVal, binMaxDispVal) self.addRow((binName, binrows)) binMinIdx += i binMin = binMax if errorbin: self.rows.insert(0, ('errors', errorbin)) ntallied = sum(len(x[1]) for x in self.rows) assert ntallied == len(self.source.rows), (ntallied, len(self.source.rows)) def discreteBinning(self): rowidx = {} for r in Progress(self.source.rows): v = tuple(getValueOrError(c, r) for c in self.origCols) histrow = rowidx.get(v) if histrow is None: histrow = (v, []) rowidx[v] = histrow self.addRow(histrow) histrow[1].append(r) self.largest = max(self.largest, len(histrow[1])) self.rows.sort(key=lambda r: len(r[1]), reverse=True) # sort by num reverse @async def reload(self): 'Generate histrow for each row and then reverse-sort by length.' self.rows = [] # if len(self.origCols) == 1 and self.origCols[0].type in (int, float, currency): # self.numericBinning() # else: self.discreteBinning() # automatically add cache to all columns now that everything is binned for c in self.visibleCols: c._cachedValues = collections.OrderedDict() visidata-1.0/visidata/Path.py0000660000175100017520000000777513223031561016111 0ustar anjaanja00000000000000import os import os.path import gzip from .vdtui import * option('skip', 0, 'skip first N lines of text input') class Path: 'File and path-handling class, modeled on `pathlib.Path`.' def __init__(self, fqpn): self.fqpn = fqpn fn = os.path.split(fqpn)[-1] # check if file is gzip-compressed if fn.endswith('.gz'): self.gzip_compressed = True fn = fn[:-3] else: self.gzip_compressed = False self.name, self.ext = os.path.splitext(fn) self.suffix = self.ext[1:] def open_text(self, mode='r'): if self.gzip_compressed: return gzip.open(self.resolve(), mode='rt', encoding=options.encoding, errors=options.encoding_errors) else: return open(self.resolve(), mode=mode, encoding=options.encoding, errors=options.encoding_errors) def __iter__(self): for i, line in enumerate(self.open_text()): if i < options.skip: continue yield line[:-1] def read_text(self): with self.open_text() as fp: return fp.read() def read_bytes(self): with open(self.resolve(), 'rb') as fp: return fp.read() def is_dir(self): return os.path.isdir(self.resolve()) def exists(self): return os.path.exists(self.resolve()) def iterdir(self): return [self.parent] + [Path(os.path.join(self.fqpn, f)) for f in os.listdir(self.resolve())] def stat(self): try: return os.stat(self.resolve()) except Exception: return None def resolve(self): 'Resolve pathname shell variables and ~userdir' return os.path.expandvars(os.path.expanduser(self.fqpn)) def relpath(self, start): return os.path.relpath(os.path.realpath(self.resolve()), start) @property def parent(self): 'Return Path to parent directory.' return Path(self.fqpn + "/..") @property def filesize(self): return self.stat().st_size def __str__(self): return self.fqpn class UrlPath(Path): def __init__(self, url): from urllib.parse import urlparse self.url = url self.obj = urlparse(url) super().__init__(self.obj.path) def __str__(self): return self.url def __getattr__(self, k): return getattr(self.obj, k) class PathFd(Path): 'minimal Path interface to satisfy a tsv loader' def __init__(self, pathname, fp, filesize=0): super().__init__(pathname) self.fp = fp self.alreadyRead = [] # shared among all RepeatFile instances self._filesize = filesize def read_text(self): return self.fp.read() def open_text(self): return RepeatFile(self) @property def filesize(self): return self._filesize class RepeatFile: def __init__(self, pathfd): self.pathfd = pathfd self.iter = None def __enter__(self): self.iter = RepeatFileIter(self) return self def __exit__(self, a,b,c): pass def read(self, n=None): r = '' while len(r) < n: try: s = next(self.iter) r += s + '\n' n += len(r) except StopIteration: break # end of file return r def seek(self, n): assert n == 0, 'RepeatFile can only seek to beginning' self.iter = RepeatFileIter(self) def __iter__(self): return RepeatFileIter(self) def __next__(self): return next(self.iter) class RepeatFileIter: def __init__(self, rf): self.rf = rf self.nextIndex = 0 def __iter__(self): return RepeatFileIter(self.rf) def __next__(self): if self.nextIndex < len(self.rf.pathfd.alreadyRead): r = self.rf.pathfd.alreadyRead[self.nextIndex] else: r = next(self.rf.pathfd.fp) self.rf.pathfd.alreadyRead.append(r) self.nextIndex += 1 return r visidata-1.0/visidata/zscroll.py0000660000175100017520000000277613223031561016701 0ustar anjaanja00000000000000from .vdtui import * # vim-style scrolling with the 'z' prefix globalCommand('zt', 'sheet.topRowIndex = cursorRowIndex', 'scroll current row to top of screen') globalCommand('zz', 'sheet.topRowIndex = cursorRowIndex-int(nVisibleRows/2)', 'scroll current row to center of screen') globalCommand('zb', 'sheet.topRowIndex = cursorRowIndex-nVisibleRows+1', 'scroll current row to bottom of screen') #globalCommand(['zL', 'kRIT5'], 'sheet.cursorVisibleColIndex = sheet.leftVisibleColIndex = rightVisibleColIndex', 'move cursor one page right') #globalCommand(['zH', 'kLFT5'], 'pageLeft()', 'move cursor one page left') globalCommand(['zh', 'zKEY_LEFT'], 'sheet.leftVisibleColIndex -= 1', 'scroll one column left') globalCommand('zk', 'sheet.topRowIndex -= 1', 'scroll one row up') globalCommand('zj', 'sheet.topRowIndex += 1', 'scroll one row down') globalCommand(['zl', 'zKEY_RIGHT'], 'sheet.leftVisibleColIndex += 1', 'scroll one column right') #globalCommand('zs', 'sheet.leftVisibleColIndex = cursorVisibleColIndex', 'scroll sheet to leftmost column') #globalCommand('ze', 'tmp = cursorVisibleColIndex; pageLeft(); sheet.cursorVisibleColIndex = tmp', 'scroll sheet to rightmost column') #globalCommand('zKEY_END', 'sheet.cursorRowIndex = len(rows)-1; sheet.cursorVisibleColIndex = len(visibleCols)-1', 'go to last row and last column') #globalCommand('zKEY_HOME', 'sheet.topRowIndex = sheet.cursorRowIndex = 0; sheet.leftVisibleColIndex = sheet.cursorVisibleColIndex = 0', 'go to top row and top column') #globalCommand('0', 'gh') visidata-1.0/visidata/pyobj.py0000660000175100017520000001542313226522612016332 0ustar anjaanja00000000000000from visidata import * option('pyobj_show_hidden', False, 'show methods and _private properties') globalCommand('^X', 'expr = input("eval: ", "expr", completer=CompleteExpr()); push_pyobj(expr, eval(expr))', 'evaluate Python expression and open result as Python object') globalCommand('g^X', 'expr = input("exec: ", "expr", completer=CompleteExpr()); exec(expr, getGlobals())', 'execute Python statement in the global scope') globalCommand('z^X', 'status(evalexpr(inputExpr("status="), cursorRow))', 'evaluate Python expression on current row and show result on status line') globalCommand('^Y', 'status(type(cursorRow)); push_pyobj("%s[%s]" % (sheet.name, cursorRowIndex), cursorRow)', 'open current row as Python object') globalCommand('z^Y', 'status(type(cursorValue)); push_pyobj("%s[%s].%s" % (sheet.name, cursorRowIndex, cursorCol.name), cursorValue)', 'open current cell as Python object') globalCommand('g^Y', 'status(type(sheet)); push_pyobj(sheet.name+"_sheet", sheet)', 'open current sheet as Python object') # used as ENTER in several pyobj sheets globalCommand('pyobj-dive', 'push_pyobj("%s[%s]" % (name, cursorRowIndex), cursorRow).cursorRowIndex = cursorColIndex', 'dive further into Python object') #### generic list/dict/object browsing def push_pyobj(name, pyobj): vs = load_pyobj(name, pyobj) if vs: return vd().push(vs) else: status('unknown type ' + type(pyobj)) def load_pyobj(name, pyobj): 'Return Sheet object of appropriate type for given sources in `args`.' if isinstance(pyobj, list) or isinstance(pyobj, tuple): if getattr(pyobj, '_fields', None): # list of namedtuple return SheetNamedTuple(name, pyobj) else: return SheetList(name, pyobj) elif isinstance(pyobj, dict): return SheetDict(name, pyobj) elif isinstance(pyobj, object): return SheetObject(name, pyobj) else: status('unknown type ' + type(pyobj)) def open_pyobj(path): 'Provide wrapper for `load_pyobj`.' return load_pyobj(path.name, eval(path.read_text())) def getPublicAttrs(obj): 'Return all public attributes (not methods or `_`-prefixed) on object.' return [k for k in dir(obj) if not k.startswith('_') and not callable(getattr(obj, k))] def PyobjColumns(obj): 'Return columns for each public attribute on an object.' return [ColumnAttr(k, type(getattr(obj, k))) for k in getPublicAttrs(obj)] def AttrColumns(attrnames): 'Return column names for all elements of list `attrnames`.' return [ColumnAttr(name) for name in attrnames] def DictKeyColumns(d): 'Return a list of Column objects from dictionary keys.' return [ColumnItem(k, k) for k in d.keys()] def SheetList(name, src, **kwargs): 'Creates a Sheet from a list of homogenous dicts or namedtuples.' if not src: status('no content in ' + name) return if isinstance(src[0], dict): return ListOfDictSheet(name, source=src, **kwargs) elif isinstance(src[0], tuple): if getattr(src[0], '_fields', None): # looks like a namedtuple return ListOfNamedTupleSheet(name, source=src, **kwargs) # simple list return ListOfPyobjSheet(name, source=src, **kwargs) class ListOfPyobjSheet(Sheet): rowtype = 'python objects' commands = [Command(ENTER, 'pyobj-dive')] def reload(self): self.rows = self.source self.columns = [Column(self.name, getter=lambda col,row: row, setter=lambda col,row,val: setitem(col.sheet.source, col.sheet.source.index(row), val))] # rowdef: dict class ListOfDictSheet(Sheet): rowtype = 'dicts' commands = [Command(ENTER, 'pyobj-dive')] def reload(self): self.columns = DictKeyColumns(self.source[0]) self.rows = self.source # rowdef: namedtuple class ListOfNamedTupleSheet(Sheet): rowtype = 'namedtuples' commands = [Command(ENTER, 'pyobj-dive')] def reload(self): self.columns = [ColumnItem(k, i) for i, k in enumerate(self.source[0]._fields)] self.rows = self.source # rowdef: PyObj class SheetNamedTuple(Sheet): rowtype = 'values' 'a single namedtuple, with key and value columns' commands = [Command(ENTER, 'dive()', 'dive further into Python object')] columns = [ColumnItem('name', 0), ColumnItem('value', 1)] def __init__(self, name, src, **kwargs): super().__init__(name, source=src, **kwargs) def reload(self): self.rows = list(zip(self.source._fields, self.source)) def dive(self): push_pyobj(joinSheetnames(self.name, self.cursorRow[0]), self.cursorRow[1]) class SheetDict(Sheet): rowtype = 'items' commands = [ Command('e', 'edit()', 'edit contents of current cell'), Command(ENTER, 'dive()', 'dive further into Python object') ] def __init__(self, name, src, **kwargs): super().__init__(name, source=src, **kwargs) def reload(self): self.columns = [ColumnItem('key', 0)] self.rows = list(list(x) for x in self.source.items()) self.columns.append(ColumnItem('value', 1)) def edit(self): self.source[self.cursorRow[0]][1] = self.editCell(1) self.cursorRowIndex += 1 self.reload() def dive(self): push_pyobj(joinSheetnames(self.name, self.cursorRow[0]), self.cursorRow[1]) class ColumnSourceAttr(Column): 'Use row as attribute name on sheet source' def calcValue(self, attrname): return getattr(self.sheet.source, attrname) def setValue(self, attrname, value): return setattr(self.sheet.source, attrname, value) # rowdef: attrname class SheetObject(Sheet): rowtype = 'attributes' commands = [ Command(ENTER, 'v = getattr(source, cursorRow); push_pyobj(joinSheetnames(name, cursorRow), v() if callable(v) else v)', 'dive further into Python object'), Command('e', 'setattr(source, cursorRow, type(getattr(source, cursorRow))(editCell(1))); sheet.cursorRowIndex += 1; reload()', 'edit contents of current cell'), Command('v', 'options.pyobj_show_hidden = not options.pyobj_show_hidden; reload()', 'toggle whether methods and hidden properties are shown') ] def __init__(self, name, obj, **kwargs): super().__init__(name, source=obj, **kwargs) def reload(self): self.rows = [] for r in dir(self.source): if not options.pyobj_show_hidden: try: if r.startswith('_') or callable(getattr(self.source, r)): continue self.addRow(r) except Exception: pass self.columns = [ Column(type(self.source).__name__ + '_attr'), ColumnSourceAttr('value'), ColumnExpr('docstring', 'value.__doc__') ] self.recalc() self.nKeys = 1 visidata-1.0/visidata/regex.py0000660000175100017520000000403113223031561016305 0ustar anjaanja00000000000000from visidata import * globalCommand(':', 'addRegexColumns(makeRegexSplitter, sheet, cursorColIndex, cursorCol, cursorRow, input("split regex: ", type="regex"))', 'add new columns from regex split; # columns determined by example row at cursor') globalCommand(';', 'addRegexColumns(makeRegexMatcher, sheet, cursorColIndex, cursorCol, cursorRow, input("match regex: ", type="regex"))', 'add new column from capture groups of regex; also requires example row') globalCommand('*', 'addColumn(regexTransform(cursorCol, input("transform column by regex: ", type="regex")), cursorColIndex+1)', 'regex/subst - replace regex with subst, which may include backreferences (\\1 etc)') option('regex_maxsplit', 0, 'maxsplit to pass to regex.split') def makeRegexSplitter(regex, origcol): return lambda row, regex=regex, origcol=origcol, maxsplit=options.regex_maxsplit: regex.split(origcol.getDisplayValue(row), maxsplit=maxsplit) def makeRegexMatcher(regex, origcol): return lambda row, regex=regex, origcol=origcol: regex.search(origcol.getDisplayValue(row)).groups() def addRegexColumns(regexMaker, vs, colIndex, origcol, exampleRow, regexstr): regex = re.compile(regexstr, regex_flags()) func = regexMaker(regex, origcol) result = func(exampleRow) for i, g in enumerate(result): c = Column(origcol.name+'_re'+str(i), getter=lambda col,row,i=i,func=func: func(row)[i]) vs.addColumn(c, index=colIndex+i+1) def regexTransform(origcol, instr): i = indexWithEscape(instr, '/') if i is None: before = instr after = '' else: before = instr[:i] after = instr[i+1:] newCol = Column(origcol.name + '_re', getter=lambda col,row,origcol=origcol, before=before, after=after: re.sub(before, after, origcol.getDisplayValue(row), flags=regex_flags())) return newCol def indexWithEscape(s, char, escape_char='\\'): i=0 while i < len(s): if s[i] == escape_char: i += 1 elif s[i] == char: return i i += 1 return None visidata-1.0/visidata/man/0000770000175100017520000000000013232247165015406 5ustar anjaanja00000000000000visidata-1.0/visidata/man/vd.10000660000175100017520000006454013232246526016113 0ustar anjaanja00000000000000.Dd January 24, 2017 .Dt vd \&1 "Quick Reference Guide" .Os Linux/MacOS . .\" Long option with arg: .Lo f filetype format .\" Long flag: .Lo f filetype .de Lo .It Cm -\\$1 Ns , Cm --\\$2 Ns = Ns Ar \\$3 .. .de Lf .It Cm -\\$1 Ns , Cm --\\$2 .. .Sh NAME . .Nm VisiData .Nd a terminal utility for exploring and arranging tabular data . .Sh SYNOPSIS . .Nm vd .Op Ar options .Op Ar input No ... . .Nm vd .Op Ar options .Cm --play Ar cmdlog .Op Cm -w Ar waitsecs .Op Cm --batch .Op Cm -o Ar output .Op Ar field Ns Cm = Ns Ar value No ... . .Sh DESCRIPTION .Nm VisiData is a multipurpose tool for exploring, cleaning, editing, and restructuring data. Rows can be selected, filtered, and grouped; columns can be rearranged, transformed, and derived via regex or Python expressions; workflows can be saved, documented, and replayed. .Nm VisiData No is built on the Sy vdtui No platform. . .Ss REPLAY MODE .Bl -tag -width XXXXXXXXXXXXXXXXXXXXXX -compact .Lo p play cmdlog .No replays a saved Ar cmdlog No within the interface . .Lo w replay-wait seconds .No waits Ar seconds No between commands . .Lf b batch replays in batch mode (with no interface) . .Lo o output file .No saves final visible sheet to Ar file No as .tsv . .It Sy --replay-movement= Ns Ar bool .No toggles Sy --play No to move cursor cell-by-cell .It Ar field Ns Cm = Ns Ar value .No replaces \&"{ Ns Ar field Ns }\&" in Ar cmdlog No contents with Ar value .El . .Ss Commands During Replay .Bl -tag -width XXXXXXXXXXXXXXXXXXX -compact -offset XXX .It Sy ^U pauses/resumes replay .It Sy Tab executes next row in replaying sheet .It Sy ^K cancels current replay .El . .Ss GLOBAL COMMANDS In most cases, commands that affect 'selected rows' will affect all rows if no rows are selected. .Pp .Ss Keystrokes For The Cautious .Bl -tag -width XXXXXXXXXXXXXXX -compact .It Ic " F1 z?" views this man page .It Ic "gF1 gz?" views all commands available on current sheet .It Ic ^Q aborts program immediately .It Ic ^C cancels user input or aborts all async threads on current sheet .It Ic " q" quits current sheet .It Ic "gq" quits all sheets (clean exit) .El .Ss "Cursor Movement" . .Bl -tag -width XXXXXXXXXXXXXXX -compact . .It Ic "Arrow PgUp Home" moves as expected .It Ic " h j k l" moves left/down/up/right .It Ic "gh gj gk gl" moves all the way to the left/bottom/top/right .It Ic " G gg" moves all the way to the bottom/top .It Ic "^B ^F" scrolls one page back/forward .Pp .It Ic " / ?" Ar regex .No searches for Ar regex No forward/backward in current column .It Ic "g/ g?" Ar regex .No searches for Ar regex No forward/backward over all visible columns .It Ic " n N" moves to next/previous match from last search .Pp .It Ic " < >" moves up/down the current column to the next value .It Ic " { }" moves up/down the current column to the next selected row . .El .Pp .Bl -tag -width XXXXXXXXXXXXXXX -compact .Pp .It Ic " c" Ar regex .No moves to the next column with name matching Ar regex .It Ic " r" Ar regex .No moves to the next row with key matching Ar regex .It Ic "zc zr" Ar number .No moves to given column/row Ar number .Pp .It Ic " H J K L" slides current row/column left/down/up/right .It Ic "gH gJ gK gL" slides current row/column all the way to the left/bottom/top/right of sheet .Pp .It Ic "zh zj zk zl" scrolls one left/down/up/right .It Ic " zt zz zb " scrolls current row to top/center/bottom of screen .El . .Ss Column Manipulation . .Bl -tag -width XXXXXXXXXXXXXXX -compact . .It Ic " _" Ns " (underscore)" adjusts width of current column .It Ic "g_" adjusts width of all visible columns .It Ic "z_" Ar number .No adjusts width of current column to Ar number .Pp .It Ic " -" Ns " (hyphen)" hides current column (to unhide, go to .Sy C Ns olumns sheet and Sy e Ns dit its width) .It Ic "z-" Ns reduces width of current column by half .Pp .It Ic \&! Ns pins current column on the left as a key column .It Ic "~ # % $ @" sets type of current column to untyped/int/float/currency/date .It Ic " ^" edits name of current column .It Ic " g^" sets names of all unnamed visible columns to contents of selected rows (or current row) .It Ic " z^" sets name of current column to contents of current cell .It Ic "gz^" sets name of current column to combined contents of current column for selected rows (or current row) .Pp .It Ic " =" Ar expr .No creates new column from Python Ar expr Ns , with column names as variables .It Ic " g=" Ar expr .No sets current column for selected rows to result of Python Ar expr .It Ic "gz=" Ar expr .No sets current column for selected rows to the items in result of Python sequence Ar expr .It Ic " z=" Ar expr .No sets current cell to result of evaluated Python Ar expr No on current row .El .Bl -tag -width XXXXXXXXXXXXXXX -compact .It Ic " '" Ns " (tick)" adds a frozen copy of current column with all cells evaluated .It Ic "g'" opens a frozen copy of current sheet with all visible columns evaluated .It Ic "z' gz'" adds/resets cache for current/all visible column(s) .Pp .It Ic "\&:" Ar regex .No adds new columns from Ar regex No split; number of columns determined by example row at cursor .It Ic "\&;" Ar regex .No adds new columns from capture groups of Ar regex Ns ; also requires example row .It Ic "*" Ar regex Ns Sy / Ns Ar subst .No replaces Ar regex No with Ar subst Ns , which may include backreferences ( Ns Sy \e1 No etc) .El .Ss Row Selection . .Bl -tag -width XXXXXXXXXXXXXXX -compact . .It Ic " s t u" selects/toggles/unselects current row .It Ic "gs gt gu" selects/toggles/unselects all rows .It Ic " | \e\ " Ns Ar regex .No selects/unselects rows matching Ar regex No in current column .It Ic "g| g\e\ " Ns Ar regex .No selects/unselects rows matching Ar regex No in any visible column .It Ic " \&," Ns " (comma)" selects rows matching current cell in current column .It Ic "g\&," selects rows matching this row in all visible columns . .El . . .Ss Row Sorting/Filtering . .Bl -tag -width XXXXXXXXXXXXXXX -compact . .It Ic " [ ]" sorts ascending/descending by current column .It Ic "g[ g]" sorts ascending/descending by all key columns .It Ic " \&"" opens duplicate sheet with only selected rows .It Ic "g\&"" opens duplicate sheet with all rows .It Ic "gz\&"" opens duplicate sheet with deepcopy of selected rows .El .Ss Editing Rows and Cells . .Bl -tag -width XXXXXXXXXXXXXXX -compact .It Ic " a" appends a blank row .It Ic " ga" Ar number .No appends Ar number No blank rows .It Ic " d gd" deletes current/all selected row(s) and moves to clipboard .It Ic " y gy" yanks (copies) current/all selected row(s) to clipboard .It Ic " zy" yanks (copies) current cell to clipboard .It Ic " p P" pastes clipboard rows after/before current row .It Ic " zp gzp" sets contents of current column for current/all selected row(s) to last clipboard value .It Ic " f" fills null cells in current column with contents of non-null cells up the current column . . .It Ic " e" Ar text edits contents of current cell .It Ic " ge" Ar text .No sets contents of current column for selected rows to Ar text .It Ic " zd Del" .No sets contents of current cell to Sy None .It Ic "gzd gDel" .No sets contents of cells in current column to Sy None No for selected rows . .El . .Ss " Commands While Editing Input" .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic "Enter ^C" accepts/aborts input .It Ic ^O opens external $EDITOR to edit contents .It Ic ^R reloads initial value .It Ic "^A ^E" moves to beginning/end of line .It Ic "^B ^F" moves back/forward one character .It Ic "^H ^D" deletes previous/current character .It Ic ^T transposes previous and current characters .It Ic "^U ^K" clears from cursor to beginning/end of line .It Ic "Backspace Del" deletes previous/current character .It Ic Insert toggles insert mode .It Ic "Up Down" sets contents to previous/next in history .It Ic "Tab Shift-Tab" autocompletes input (when available) . .El . .Ss Data Toolkit .Bl -tag -width XXXXXXXXXXXXXXX -compact .It Ic " o" Ar input opens .Ar input No in Sy VisiData .It Ic "^S" Ar filename .No saves current sheet to Ar filename No in format determined by extension (default .tsv) .It Ic "^D" Ar filename.vd .No saves CommandLog to Ar filename.vd No file .It Ic "A" Ar number .No opens new blank sheet with Ar number No columns .It Ic "R" Ar number opens duplicate sheet with a random population subset of .Ar number No rows .Pp .It Ic " +" Ar aggregator .No adds Ar aggregator No to current column (see Sy "Frequency Table" Ns ) .It Ic "z+" Ar aggregator .No displays result of Ar aggregator No over values in selected rows for current column .Pp .El .Ss Data Visualization .Bl -tag -width XXXXXXXXXXXXX -compact .It Ic " ." No (dot) .No plots current numeric column vs key columns. Numeric key column is used for the x-axis; categorical key column values determine color. .It Ic "g." .No plots a graph of all visible numeric columns vs key columns. .Pp .El .No If rows on the current sheet represent plottable coordinates (as in .shp or vector .mbtiles sources), .Ic " ." No plots the current row, and Ic "g." No plots all selected rows (or all rows if none selected). .Ss " Canvas-specific Commands" .Bl -tag -width XXXXXXXXXXXXXXXXXX -compact -offset XXX .It Ic " + -" increases/decreases zoomlevel, centered on cursor .It Ic " _" No (underscore) zooms to fit full extent .It Ic " s t u" selects/toggles/unselects rows on source sheet contained within canvas cursor .It Ic "gs gt gu" selects/toggles/unselects rows visible on screen .It Ic " Enter" opens sheet of source rows contained within canvas cursor .It Ic "gEnter" opens sheet of source rows visible on screen .It Ic " 1" No - Ic "9" toggles display of layers .It Ic "^L" redraws all pixels on canvas .It Ic " w" .No toggles Ic show_graph_labels No option .It Ic "mouse scrollwheel" zooms in/out of canvas .It Ic "left click-drag" sets canvas cursor .It Ic "right click-drag" scrolls canvas .El .Ss Other Commands . .Bl -tag -width XXXXXXXXXXXXXXX -compact . .It Ic V views contents of current cell in a new TextSheet .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset xxx .It Ic "v" toggles visibility (text wrap on TextSheet, legends/axes on Graph) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact .It Ic " ^^" No (Ctrl-^) jumps to previous sheet (swaps with current sheet) .It Ic " ^E" views traceback for most recent error .It Ic "g^E" views traceback for most recent errors .It Ic "z^E" views traceback for error in current cell .It Ic " ^L" refreshes screen .It Ic " ^G" shows cursor position and bounds of current sheet on status line .It Ic " ^V" shows version information on status line .It Ic " ^R" reloads current sheet .It Ic "z^R" clears cache for current column .It Ic " ^Z" suspends VisiData process .It Ic " ^P" .No opens Sy Status History . .El .Pp .Bl -tag -width XXXXXXXXXXXXXXX -compact .It Ic " ^Y z^Y g^Y" opens current row/cell/sheet as Python object .It Ic " ^X" Ar expr .No evaluates Python Ar expr No and opens result as python object .It Ic "z^X" Ar expr .No evaluates Python Ar expr No on current row and shows result on status line .It Ic "g^X" Ar stmt .No executes Python Ar stmt No in the global scope .El . .Ss Internal Sheets List .Bl -tag -width Xx -compact .It " " .It Sy Metasheets .It Sy " \&." .Sy Columns Sheet No (Shift-C) " edit column properties" .It Sy " \&." .Sy Sheets Sheet No (Shift-S) " jump between sheets or join them together" .It Sy " \&." .Sy Options Sheet No (Shift-O) " edit all configurable options" .It Sy " \&." .Sy Commandlog No (Shift-D) " modify and save commands for replay" .It Sy " \&." .Sy Error Sheet No (^E) " view last error" .It Sy " \&." .Sy Status History No (^P) " view history of status messages" .It Sy " \&." .Sy Threads Sheet No (^T) " view, cancel, and profile asynchronous threads" .Pp .It Sy Derived Sheets .It Sy " \&." .Sy Frequency Table No (Shift-F) " group rows by column value, with aggregations of other columns" .It Sy " \&." .Sy Describe Sheet No (Shift-I) " view summary statistics for each column" .It Sy " \&." .Sy Pivot Table No (Shift-W) " group rows by key and summarize current column" .It Sy " \&." .Sy Melted Sheet No (Shift-M) " unpivot non-key columns into variable/value columns" .El .Ss METASHEETS .Ss Columns Sheet (Shift-C) .Bl -inset -compact .It Properties of columns on the source sheet can be changed with standard editing commands ( Ns Sy e ge g= Del Ns ) on the Sy Columns Sheet Ns . Multiple aggregators can be set by listing them (separated by spaces) in the aggregators column. The 'g' commands affect the selected rows, which are actually the literal columns on the source sheet. .It (global commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic gC .No opens Sy Columns Sheet No with all columns from all sheets .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic " &" adds column from concatenating selected source columns .It Ic " !" pins current column on the left as a key column on source sheet .It Ic "g!" pins selected columns on the left as key columns on source sheet .It Ic "g+" adds aggregator to selected source columns .It Ic "g_" No (underscore) adjusts widths of selected columns on source sheet .It Ic "g-" No (hyphen) hides selected columns on source sheet .It Ic " ~ # % $ @" sets type of current column on source sheet to str/int/float/currency/date .It Ic "g~ g# g% g$ g@" sets type of selected columns on source sheet to str/int/float/currency/date .It Ic "z~ gz~" sets type of current/selected column(s) on source sheet to anytype .It Ic " Enter" .No opens a Sy Frequency Table No sheet grouped on column referenced in current row .El . .Ss Sheets Sheet (Shift-S) .Bl -inset -compact .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic "Enter" jumps to sheet referenced in current row .It Ic "gC" .No opens Sy Columns Sheet No with all columns from selected sheets .It Ic "&" Ar jointype .No merges selected sheets with visible columns from all, keeping rows according to Ar jointype Ns : .El .Bl -tag -width x -compact -offset XXXXXXXXXXXXXXXXXXXX .It Sy "\&." .Sy inner No " keeps only rows which match keys on all sheets" .It Sy "\&." .Sy left No " keeps all rows from first selected sheet" .It Sy "\&." .Sy full No " keeps all rows from all sheets (union)" .It Sy "\&." .Sy diff No " keeps only rows NOT in all sheets" .It Sy "\&." .Sy append No "keeps all rows from all sheets (concatenation)" .El . .Ss Options Sheet (Shift-O) .Bl -inset -compact .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic "Enter" edits option .El . .Ss CommandLog (Shift-D) .Bl -inset -compact .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic " x" replays command in current row .It Ic "gx" replays contents of entire CommandLog .It Ic " ^C" aborts replay .El . .Ss DERIVED SHEETS .Ss Frequency Table (Shift-F) .Bl -inset -compact .It A Sy Frequency Table No groups rows by one or more columns, and includes summary columns for those with aggregators. .It (global commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic gF opens Frequency Table, grouped by all key columns on source sheet .It Ic zF opens one-line summary for selected rows .It (sheet-specific commands) .It Ic " s t u" selects/toggles/unselects these entries in source sheet .It Ic " Enter" opens sheet of source rows which are grouped in current cell .El . .Ss Describe Sheet (Shift-I) .Bl -inset -compact .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic "zs zu" selects/unselects rows on source sheet which are being described in current cell .It Ic " !" pins current column on the left as a key column on source sheet .It Ic " Enter" .No opens a Sy Frequency Table No sheet grouped on column referenced in current row .It Ic "zEnter" opens copy of source sheet with rows described in current cell .El . .Ss Pivot Table (Shift-W) .Bl -inset -compact .It Set key column(s) and aggregators on column(s) before pressing Sy Shift-W No on the column to pivot. .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic " Enter" opens sheet of source rows aggregated in current pivot row .It Ic "zEnter" opens sheet of source rows aggregated in current pivot cell .El .Ss Melted Sheet (Shift-M) .Bl -inset -compact .It Opens melted sheet (unpivot), with all non-key columns reduced to Variable-Value rows. .El .Ss Python Object Sheet (^X ^Y g^Y z^Y) .Bl -inset -compact .It (sheet-specific commands) .El .Bl -tag -width XXXXXXXXXXXXXXX -compact -offset XXX .It Ic "Enter" dives further into Python object .It Ic "e" edits contents of current cell .It Ic "v" toggles whether methods and hidden properties are shown .El . .Sh COMMANDLINE OPTIONS .Bl -tag -width XXXXXXXXXXXXXXXXXXXXXXXXXXX -compact . .Lo f filetype filetype .No "tsv " uses loader for .Ar filetype instead of file extension . .Lo y confirm-overwrite F .No "True " overwrites existing files without confirmation . .It Cm --diff Ns = Ns Ar base .No "None " .No adds colorizer for all sheets against Ar base . .El .Bl -tag -width XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -compact .It Sy --encoding Ns = Ns Ar "str " No "utf-8" as passed to codecs.open .It Sy --encoding-errors Ns = Ns Ar "str " No "surrogateescape" as passed to codecs.open .It Sy --regex-flags Ns = Ns Ar "str " No "I" flags to pass to re.compile() [AILMSUX] .It Sy --default-width Ns = Ns Ar "int " No "20" default column width .It Sy --wrap Ns = Ns Ar "bool " No "False" wrap text to fit window width on TextSheet .It Sy --cmd-after-edit Ns = Ns Ar "str " No "j" command keystroke to execute after successful edit .It Sy --cmdlog-longname Ns = Ns Ar "bool " No "False" Use command longname in cmdlog if available .It Sy --none-is-null Ns = Ns Ar "bool " No "True" if Python None counts as null .It Sy --empty-is-null Ns = Ns Ar "bool " No "False" if empty string counts as null .It Sy --false-is-null Ns = Ns Ar "bool " No "False" if Python False counts as null .It Sy --zero-is-null Ns = Ns Ar "bool " No "False" if integer 0 counts as null .It Sy --force-valid-colnames Ns = Ns Ar "bool " No "False" clean column names to be valid Python identifiers .It Sy --debug Ns = Ns Ar "bool " No "False" exit on error and display stacktrace .It Sy --curses-timeout Ns = Ns Ar "int " No "100" curses timeout in ms .It Sy --force-256-colors Ns = Ns Ar "bool " No "False" use 256 colors even if curses reports fewer .It Sy --use-default-colors Ns = Ns Ar "bool " No "False" set curses to use default terminal colors .It Sy --note-pending Ns = Ns Ar "str " No "\[u231B]" note to display for pending cells .It Sy --note-format-exc Ns = Ns Ar "str " No "?" cell note for an exception during type conversion or formatting .It Sy --note-getter-exc Ns = Ns Ar "str " No "!" cell note for an exception during computation .It Sy --scroll-incr Ns = Ns Ar "int " No "3" amount to scroll with scrollwheel .It Sy --skip Ns = Ns Ar "int " No "0" skip first N lines of text input .It Sy --profile-threads Ns = Ns Ar "bool " No "True" profile async threads .It Sy --min-memory-mb Ns = Ns Ar "int " No "0" minimum memory to continue loading and async processing .It Sy --confirm-overwrite Ns = Ns Ar "bool " No "True" whether to prompt for overwrite confirmation on save .It Sy --header Ns = Ns Ar "int " No "1" parse first N rows of .csv/.tsv as column names .It Sy --delimiter Ns = Ns Ar "str " No " " delimiter to use for tsv filetype .It Sy --filetype Ns = Ns Ar "str " No "" specify file type .It Sy --pyobj-show-hidden Ns = Ns Ar "bool " No "False" show methods and _private properties .It Sy --replay-wait Ns = Ns Ar "float " No "0.0" time to wait between replayed commands, in seconds .It Sy --replay-movement Ns = Ns Ar "bool " No "False" insert movements during replay .It Sy --rowkey-prefix Ns = Ns Ar "str " No "\[u30AD]" string prefix for rowkey in the cmdlog .It Sy --regex-maxsplit Ns = Ns Ar "int " No "0" maxsplit to pass to regex.split .It Sy --show-graph-labels Ns = Ns Ar "bool " No "True" show axes and legend on graph .It Sy --plot-colors Ns = Ns Ar "str " No "" list of distinct colors to use for plotting distinct objects .It Sy --zoom-incr Ns = Ns Ar "float " No "2.0" amount to multiply current zoomlevel by when zooming .It Sy --csv-dialect Ns = Ns Ar "str " No "excel" dialect passed to csv.reader .It Sy --csv-delimiter Ns = Ns Ar "str " No "," delimiter passed to csv.reader .It Sy --csv-quotechar Ns = Ns Ar "str " No """ quotechar passed to csv.reader .It Sy --csv-skipinitialspace Ns = Ns Ar "bool " No "True" skipinitialspace passed to csv.reader .It Sy --fixed-rows Ns = Ns Ar "int " No "1000" number of rows to check for fixed width columns .El . .Ss DISPLAY OPTIONS .No Display options can only be set via the Sx Options Sheet No or a config file (see Sx FILES Ns ). .Pp . .Bl -tag -width XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -compact .It Sy "disp_none " No "" visible contents of a cell whose value is None .It Sy "disp_date_fmt " No "%Y-%m-%d" default fmtstr to strftime for date values .It Sy "disp_truncator " No "\[u2026]" indicator that the contents are only partially visible .It Sy "disp_oddspace " No "\[u00B7]" displayable character for odd whitespace .It Sy "disp_unprintable " No "." substitute character for unprintables .It Sy "disp_column_sep " No "|" separator between columns .It Sy "disp_keycol_sep " No "\[u2016]" separator between keys and rest of columns .It Sy "disp_status_fmt " No "{sheet.name}| " status line prefix .It Sy "disp_status_sep " No " | " separator between statuses .It Sy "disp_edit_fill " No "_" edit field fill character .It Sy "disp_more_left " No "<" header note indicating more columns to the left .It Sy "disp_more_right " No ">" header note indicating more columns to the right .It Sy "disp_error_val " No "\[u00BF]" displayed contents for computation exception .It Sy "disp_ambig_width " No "1" width to use for unicode chars marked ambiguous .It Sy "color_default " No "normal" the default color .It Sy "color_default_hdr " No "bold underline" color of the column headers .It Sy "color_current_row " No "reverse" color of the cursor row .It Sy "color_current_col " No "bold" color of the cursor column .It Sy "color_current_hdr " No "reverse underline" color of the header for the cursor column .It Sy "color_column_sep " No "246 blue" color of column separators .It Sy "color_key_col " No "81 cyan" color of key columns .It Sy "color_selected_row " No "215 yellow" color of selected rows .It Sy "color_status " No "bold" status line color .It Sy "color_edit_cell " No "normal" edit cell color .It Sy "disp_pending " No "" string to display in pending cells .It Sy "color_note_pending " No "bold magenta" color of note of pending cells .It Sy "color_note_type " No "226 green" cell note for numeric types in anytype columns .It Sy "color_format_exc " No "48 bold yellow" color of formatting exception note .It Sy "color_getter_exc " No "red bold" color of computation exception note .It Sy "disp_histogram " No "*" histogram element character .It Sy "disp_histolen " No "80" width of histogram column .It Sy "disp_replay_play " No "\[u25B6]" status indicator for active replay .It Sy "disp_replay_pause " No "\[u2016]" status indicator for paused replay .It Sy "disp_pixel_random " No "False" randomly choose attr from set of pixels instead of most common .It Sy "color_graph_hidden " No "238 blue" color of legend for hidden attribute .It Sy "color_graph_axis " No "bold" color for graph axis labels .El . .Sh EXAMPLES .Dl Nm vd Cm foo.tsv .Pp .Dl Nm vd Cm -f sqlite bar.db .Pp .Dl Nm vd Cm postgres:// Ns Ar username Ns Sy "\&:" Ns Ar password Ns Sy @ Ns Ar hostname Ns Sy "\&:" Ns Ar port Ns Sy / Ns Ar database .Pp .Dl Nm vd Cm --play tests/pivot.vd --replay-wait 1 --output tests/pivot.tsv .Pp .Dl Ic ls -l | Nm vd Cm -f fixed --skip 1 --header 0 .Pp .Dl Nm vd --diff foo.tsv bar.tsv .Pp .Sh FILES At the start of every session, .Sy VisiData No looks for Pa $HOME/.visidatarc Ns , and calls Python exec() on its contents if it exists. For example: .Bd -literal options.min_memory_mb=100 # stop processing without 100MB free globalCommand('0', 'gh') # alias '0' to move to first column, like vim def median(values): L = sorted(values) return L[len(L)//2] aggregator('median', median) .Ed . .Sh SUPPORTED SOURCES These are the supported sources: .Pp .Bl -inset -compact -offset xxx .It Sy tsv No (tab-separated value) .Bl -inset -compact -offset xxx .It Plain and simple. Nm VisiData No writes tsv format by default. See the Sy --delimiter No option. .El .El .Pp .Bl -inset -compact -offset xxx .It Sy csv No (comma-separated value) .Bl -inset -compact -offset xxx .It .csv files are a scourge upon the earth, and still regrettably common. .It See the Sy --csv-dialect Ns , Sy --csv-delimiter Ns , Sy --csv-quotechar Ns , and Sy --csv-skipinitialspace No options. .It Accepted dialects are Ic excel-tab Ns , Ic unix Ns , and Ic excel Ns . .El .El .Pp .Bl -inset -compact -offset xxx .It Sy fixed No (fixed width text) .Bl -inset -compact -offset xxx .It Columns are autodetected from the first 1000 rows (adjustable with Sy --fixed-rows Ns ). .El .El .Pp .Bl -inset -compact -offset xxx .It Sy json No (single object) and Sy jsonl No (one object per line). .Bl -inset -compact -offset xxx .It Cells containing lists (e.g. Sy [3] Ns ) or dicts ( Ns Sy {3} Ns ) can be opened into a new sheet with with Sy z^Y . .El .El . .Pp The following URL schemes are supported: .Bl -inset -compact -offset xxx .It Sy http No (requires Sy requests Ns ); can be used as transport for with another filetype .It Sy postgres No (requires Sy psycopg2 Ns ) .El . .Pp .Bl -inset -compact .It The following sources may include multiple tables. The initial sheet is the table directory; .Sy Enter No loads the entire table into memory. .El . .Pp .Bl -inset -compact -offset xxx .It Sy sqlite .It Sy xlsx No (requires Sy openpyxl Ns ) .It Sy xls No (requires Sy xlrd Ns ) .It Sy hdf5 No (requires Sy h5py Ns ) .It Sy shp No (requires Sy pyshp Ns ) .It Sy mbtiles No (requires Sy mapbox-vector-tile Ns ) .It Sy html No (requires Sy lxml Ns ) .El . In addition, .zip and .gz files are decompressed on-the-fly. . .Sh AUTHOR .Nm VisiData was made by .An Saul Pwanson Aq Mt vd@saul.pw Ns . visidata-1.0/visidata/data.py0000660000175100017520000002531113232246431016114 0ustar anjaanja00000000000000import random from .vdtui import * option('confirm_overwrite', True, 'whether to prompt for overwrite confirmation on save') option('header', 1, 'parse first N rows of .csv/.tsv as column names') option('delimiter', '\t', 'delimiter to use for tsv filetype') option('filetype', '', 'specify file type') # slide rows/columns around globalCommand('H', 'moveVisibleCol(cursorVisibleColIndex, max(cursorVisibleColIndex-1, 0)); sheet.cursorVisibleColIndex -= 1', 'slide current column left') globalCommand('J', 'sheet.cursorRowIndex = moveListItem(rows, cursorRowIndex, min(cursorRowIndex+1, nRows-1))', 'move current row down') globalCommand('K', 'sheet.cursorRowIndex = moveListItem(rows, cursorRowIndex, max(cursorRowIndex-1, 0))', 'move current row up') globalCommand('L', 'moveVisibleCol(cursorVisibleColIndex, min(cursorVisibleColIndex+1, nVisibleCols-1)); sheet.cursorVisibleColIndex += 1', 'move current column right') globalCommand('gH', 'moveListItem(columns, cursorColIndex, nKeys)', 'slide current column all the way to the left of sheet') globalCommand('gJ', 'moveListItem(rows, cursorRowIndex, nRows)', 'slide current row to the bottom of sheet') globalCommand('gK', 'moveListItem(rows, cursorRowIndex, 0)', 'slide current row all the way to the top of sheet') globalCommand('gL', 'moveListItem(columns, cursorColIndex, nCols)', 'slide current column all the way to the right of sheet') globalCommand('c', 'searchColumnNameRegex(input("column name regex: ", "regex"), moveCursor=True)', 'move to the next column with name matching regex') globalCommand('r', 'moveRegex(sheet, regex=input("row key regex: ", "regex"), columns=keyCols or [visibleCols[0]])', 'move to the next row with key matching regex') globalCommand('zc', 'sheet.cursorVisibleColIndex = int(input("column number: "))', 'move to the given column number') globalCommand('zr', 'sheet.cursorRowIndex = int(input("row number: "))', 'move to the given row number') globalCommand('R', 'nrows=int(input("random population size: ")); vs=vd.push(copy(sheet)); vs.name+="_sample"; vs.rows=random.sample(rows, nrows)', 'open duplicate sheet with a random population subset of # rows') globalCommand('a', 'rows.insert(cursorRowIndex+1, newRow()); cursorDown(1)', 'append a blank row') globalCommand('ga', 'for r in range(int(input("add rows: "))): addRow(newRow())', 'add N blank rows') globalCommand('f', 'fillNullValues(cursorCol, selectedRows or rows)', 'fills null cells in current column with contents of non-null cells up the current column') def fillNullValues(col, rows): 'Fill null cells in col with the previous non-null value' lastval = None nullfunc = isNullFunc() n = 0 for r in rows: val = col.getValue(r) if nullfunc(val): if lastval: col.setValue(r, lastval) n += 1 else: lastval = val status("filled %d values" % n) def updateColNames(sheet): for c in sheet.visibleCols: if not c._name: c.name = "_".join(c.getDisplayValue(r) for r in sheet.selectedRows or [sheet.cursorRow]) globalCommand('z^', 'sheet.cursorCol.name = cursorDisplay', 'set name of current column to contents of current cell') globalCommand('g^', 'updateColNames(sheet)', 'set names of all visible columns to contents of selected rows (or current row)') globalCommand('gz^', 'sheet.cursorCol.name = "_".join(sheet.cursorCol.getDisplayValue(r) for r in selectedRows or [cursorRow]) ', 'set current column name to combined contents of current cell in selected rows (or current row)') # gz^ with no selectedRows is same as z^ globalCommand('o', 'vd.push(openSource(inputFilename("open: ")))', 'open input in VisiData') globalCommand('^S', 'saveSheet(sheet, inputFilename("save to: ", value=getDefaultSaveName(sheet)), options.confirm_overwrite)', 'save current sheet to filename in format determined by extension (default .tsv)') globalCommand('z=', 'cursorCol.setValue(cursorRow, evalexpr(inputExpr("set cell="), cursorRow))', 'set current cell to result of evaluated Python expression on current row') globalCommand('gz=', 'for r, v in zip(selectedRows or rows, eval(input("set column= ", "expr", completer=CompleteExpr()))): cursorCol.setValue(r, v)', 'set current column for selected rows to the items in result of Python sequence expression') globalCommand('A', 'vd.push(newSheet(int(input("num columns for new sheet: "))))', 'open new blank sheet with N columns') globalCommand('gKEY_F(1)', 'help-commands') # vdtui generic commands sheet globalCommand('gz?', 'help-commands') # vdtui generic commands sheet # in VisiData, F1/z? refer to the man page globalCommand('z?', 'openManPage()', 'launch VisiData manpage') globalCommand('KEY_F(1)', 'z?') def openManPage(): from pkg_resources import resource_filename with SuspendCurses(): os.system(' '.join(['man', resource_filename(__name__, 'man/vd.1')])) def newSheet(ncols): return Sheet('unnamed', columns=[ColumnItem('', i, width=8) for i in range(ncols)]) def inputFilename(prompt, *args, **kwargs): return input(prompt, "filename", *args, completer=completeFilename, **kwargs) def completeFilename(val, state): i = val.rfind('/') if i < 0: # no / base = '' partial = val elif i == 0: # root / base = '/' partial = val[1:] else: base = val[:i] partial = val[i+1:] files = [] for f in os.listdir(Path(base or '.').resolve()): if f.startswith(partial): files.append(os.path.join(base, f)) files.sort() return files[state%len(files)] def getDefaultSaveName(sheet): src = getattr(sheet, 'source', None) if isinstance(src, Path): return str(src) else: return sheet.name+'.'+getattr(sheet, 'filetype', 'tsv') def saveSheet(vs, fn, confirm_overwrite=False): 'Save sheet `vs` with given filename `fn`.' if Path(fn).exists(): if confirm_overwrite: confirm('%s already exists. overwrite? ' % fn) basename, ext = os.path.splitext(fn) funcname = 'save_' + ext[1:] if funcname not in getGlobals(): funcname = 'save_tsv' getGlobals().get(funcname)(vs, Path(fn).resolve()) status('saving to ' + fn) class DirSheet(Sheet): 'Sheet displaying directory, using ENTER to open a particular file.' rowtype = 'files' commands = [ Command(ENTER, 'vd.push(openSource(cursorRow[0]))', 'open file') # path, filename ] columns = [ Column('filename', getter=lambda col,row: row[0].name + row[0].ext), Column('type', getter=lambda col,row: row[0].is_dir() and '/' or row[0].suffix), Column('size', type=int, getter=lambda col,row: row[1].st_size), Column('mtime', type=date, getter=lambda col,row: row[1].st_mtime) ] def reload(self): self.rows = [(p, p.stat()) for p in self.source.iterdir()] # if not p.name.startswith('.')] def openSource(p, filetype=None): 'calls open_ext(Path) or openurl_scheme(UrlPath, filetype)' if isinstance(p, str): if '://' in p: return openSource(UrlPath(p), filetype) # convert to Path and recurse else: return openSource(Path(p), filetype) # convert to Path and recurse elif isinstance(p, UrlPath): openfunc = 'openurl_' + p.scheme return getGlobals()[openfunc](p, filetype=filetype) elif isinstance(p, Path): if not filetype: filetype = options.filetype or p.suffix if os.path.isdir(p.resolve()): vs = DirSheet(p.name, source=p) filetype = 'dir' else: openfunc = 'open_' + filetype.lower() if openfunc not in getGlobals(): status('no %s function' % openfunc) filetype = 'txt' openfunc = 'open_txt' vs = getGlobals()[openfunc](p) else: # some other object status('unknown object type %s' % type(p)) vs = None if vs: status('opening %s as %s' % (p.name, filetype)) return vs #### enable external addons def open_vd(p): 'Opens a .vd file as a .tsv file' vs = open_tsv(p) vs.reload() return vs def open_txt(p): 'Create sheet from `.txt` file at Path `p`, checking whether it is TSV.' with p.open_text() as fp: if options.delimiter in next(fp): # peek at the first line return open_tsv(p) # TSV often have .txt extension return TextSheet(p.name, p) def _getTsvHeaders(fp, nlines): headers = [] i = 0 while i < nlines: L = next(fp) L = L.rstrip('\n') if L: headers.append(L.split(options.delimiter)) i += 1 return headers def open_tsv(p, vs=None): 'Parse contents of Path `p` and populate columns.' if vs is None: vs = Sheet(p.name, source=p) vs.loader = lambda vs=vs: reload_tsv(vs) header_lines = int(options.header) with vs.source.open_text() as fp: headers = _getTsvHeaders(fp, header_lines or 1) # get one data line if no headers if header_lines == 0: vs.columns = ArrayColumns(len(headers[0])) else: # columns ideally reflect the max number of fields over all rows # but that's a lot of work for a large dataset vs.columns = ArrayNamedColumns('\\n'.join(x) for x in zip(*headers[:header_lines])) vs.recalc() return vs @async def reload_tsv(vs, **kwargs): 'Asynchronous wrapper for `reload_tsv_sync`.' reload_tsv_sync(vs) def reload_tsv_sync(vs, **kwargs): 'Perform synchronous loading of TSV file, discarding header lines.' header_lines = kwargs.get('header', options.header) delim = options.delimiter vs.rows = [] with vs.source.open_text() as fp: _getTsvHeaders(fp, header_lines) # discard header lines with Progress(total=vs.source.filesize) as prog: while True: try: L = next(fp) except StopIteration: break L = L.rstrip('\n') if L: vs.addRow(L.split(delim)) prog.addProgress(len(L)) status('loaded %s' % vs.name) @async def save_tsv(vs, fn): 'Write sheet to file `fn` as TSV.' # replace tabs and newlines delim = options.delimiter replch = options.disp_oddspace trdict = {ord(delim): replch, 10: replch, 13: replch} with open(fn, 'w', encoding=options.encoding, errors=options.encoding_errors) as fp: colhdr = delim.join(col.name.translate(trdict) for col in vs.visibleCols) + '\n' if colhdr.strip(): # is anything but whitespace fp.write(colhdr) for r in Progress(vs.rows): fp.write(delim.join(col.getDisplayValue(r).translate(trdict) for col in vs.visibleCols) + '\n') status('%s save finished' % fn) visidata-1.0/visidata/vdtui.py0000770000175100017520000026323013232246526016351 0ustar anjaanja00000000000000#!/usr/bin/env python3 # # Copyright 2017 Saul Pwanson http://saul.pw/vdtui # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE. # 'vdtui: a curses framework for columnar data' # Just include this whole file in your project as-is. If you do make # modifications, please keep the base vdtui version and append your own id and # version. __version__ = 'saul.pw/vdtui v1.0' __author__ = 'Saul Pwanson ' __license__ = 'MIT' __status__ = 'Beta' from builtins import * import sys import os import collections from copy import copy, deepcopy from contextlib import suppress import curses import datetime import functools import io import itertools import string import re import textwrap import threading import traceback import time class EscapeException(BaseException): 'Inherits from BaseException to avoid "except Exception" clauses. Do not use a blanket "except:" or the task will be uncancelable.' pass class ExpectedException(Exception): 'an expected exception' pass baseCommands = collections.OrderedDict() # [cmd.name] -> Command baseOptions = collections.OrderedDict() # [opt.name] -> opt class Command: def __init__(self, name, execstr, helpstr=''): self.name = name self.execstr = execstr self.helpstr = helpstr def globalCommand(keystrokes, execstr, helpstr='', longname=None): if isinstance(keystrokes, str): keystrokes = [keystrokes] if longname: cmd = Command(longname, execstr, helpstr) baseCommands[longname] = cmd assert helpstr or (execstr in baseCommands), 'unknown longname ' + execstr helpstr = '' for ks in keystrokes: baseCommands[ks] = Command(ks, longname or execstr, helpstr) def option(name, default, helpstr=''): baseOptions[name] = [name, default, default, helpstr] class OptionsObject: 'minimalist options framework' def __init__(self, d): object.__setattr__(self, '_opts', d) def __getattr__(self, k): # options.foo name, value, default, helpstr = self._opts[k] return value def __setattr__(self, k, v): # options.foo = v self.__setitem__(k, v) def __setitem__(self, k, v): # options[k] = v if k not in self._opts: raise Exception('no such option "%s"' % k) curval = self._opts[k][1] t = type(curval) if isinstance(v, str) and t is bool: # special case for bool options v = v and (v[0] not in "0fFnN") # ''/0/false/no are false, everything else is true elif curval is not None: # if None, do not apply type conversion v = t(v) self._opts[k][1] = v options = OptionsObject(baseOptions) alias = globalCommand theme = option option('encoding', 'utf-8', 'as passed to codecs.open') option('encoding_errors', 'surrogateescape', 'as passed to codecs.open') option('regex_flags', 'I', 'flags to pass to re.compile() [AILMSUX]') option('default_width', 20, 'default column width') option('wrap', False, 'wrap text to fit window width on TextSheet') option('cmd_after_edit', 'j', 'command keystroke to execute after successful edit') option('cmdlog_longname', False, 'Use command longname in cmdlog if available') option('none_is_null', True, 'if Python None counts as null') option('empty_is_null', False, 'if empty string counts as null') option('false_is_null', False, 'if Python False counts as null') option('zero_is_null', False, 'if integer 0 counts as null') option('force_valid_colnames', False, 'clean column names to be valid Python identifiers') option('debug', False, 'exit on error and display stacktrace') option('curses_timeout', 100, 'curses timeout in ms') theme('force_256_colors', False, 'use 256 colors even if curses reports fewer') theme('use_default_colors', False, 'set curses to use default terminal colors') disp_column_fill = ' ' # pad chars after column value theme('disp_none', '', 'visible contents of a cell whose value is None') theme('disp_date_fmt','%Y-%m-%d', 'default fmtstr to strftime for date values') theme('disp_truncator', '…', 'indicator that the contents are only partially visible') theme('disp_oddspace', '\u00b7', 'displayable character for odd whitespace') theme('disp_unprintable', '.', 'substitute character for unprintables') theme('disp_column_sep', '|', 'separator between columns') theme('disp_keycol_sep', '\u2016', 'separator between keys and rest of columns') theme('disp_status_fmt', '{sheet.name}| ', 'status line prefix') theme('disp_status_sep', ' | ', 'separator between statuses') theme('disp_edit_fill', '_', 'edit field fill character') theme('disp_more_left', '<', 'header note indicating more columns to the left') theme('disp_more_right', '>', 'header note indicating more columns to the right') theme('disp_error_val', '¿', 'displayed contents for computation exception') theme('disp_ambig_width', 1, 'width to use for unicode chars marked ambiguous') theme('color_default', 'normal', 'the default color') theme('color_default_hdr', 'bold underline', 'color of the column headers') theme('color_current_row', 'reverse', 'color of the cursor row') theme('color_current_col', 'bold', 'color of the cursor column') theme('color_current_hdr', 'reverse underline', 'color of the header for the cursor column') theme('color_column_sep', '246 blue', 'color of column separators') theme('color_key_col', '81 cyan', 'color of key columns') theme('color_selected_row', '215 yellow', 'color of selected rows') theme('color_status', 'bold', 'status line color') theme('color_edit_cell', 'normal', 'edit cell color') theme('disp_pending', '', 'string to display in pending cells') theme('note_pending', '⌛', 'note to display for pending cells') theme('note_format_exc', '?', 'cell note for an exception during type conversion or formatting') theme('note_getter_exc', '!', 'cell note for an exception during computation') theme('color_note_pending', 'bold magenta', 'color of note of pending cells') theme('color_note_type', '226 green', 'cell note for numeric types in anytype columns') theme('color_format_exc', '48 bold yellow', 'color of formatting exception note') theme('color_getter_exc', 'red bold', 'color of computation exception note') theme('scroll_incr', 3, 'amount to scroll with scrollwheel') ENTER='^J' ESC='^[' globalCommand('KEY_RESIZE', '', 'no-op by default') globalCommand('q', 'vd.sheets.pop(0)', 'quit current sheet') globalCommand('KEY_LEFT', 'cursorRight(-1)', 'move one column left', 'move-left') globalCommand('KEY_DOWN', 'cursorDown(+1)', 'move one row down', 'move-down') globalCommand('KEY_UP', 'cursorDown(-1)', 'move one row up', 'move-up') globalCommand('KEY_RIGHT', 'cursorRight(+1)', 'move one column right', 'move-right') globalCommand('KEY_NPAGE', 'cursorDown(nVisibleRows); sheet.topRowIndex += nVisibleRows', 'move one page forward', 'move-page-down') globalCommand('KEY_PPAGE', 'cursorDown(-nVisibleRows); sheet.topRowIndex -= nVisibleRows', 'move one page backward', 'move-page-up') globalCommand('gq', 'vd.sheets.clear()', 'quit all sheets (clean exit)', 'quit-all') globalCommand('gh', 'sheet.cursorVisibleColIndex = sheet.leftVisibleColIndex = 0', 'move all the way to the left', 'move-far-left') globalCommand('KEY_HOME', 'sheet.cursorRowIndex = sheet.topRowIndex = 0', 'move all the way to the top', 'move-top') globalCommand('KEY_END', 'sheet.cursorRowIndex = len(rows); sheet.topRowIndex = cursorRowIndex-nVisibleRows', 'move all the way to the bottom', 'move-bottom') globalCommand('gl', 'sheet.leftVisibleColIndex = len(visibleCols)-1; pageLeft(); sheet.cursorVisibleColIndex = len(visibleCols)-1', 'move all the way to the right', 'move-far-right') globalCommand('h', 'move-left') globalCommand('j', 'move-down') globalCommand('k', 'move-up') globalCommand('l', 'move-right') globalCommand('gKEY_LEFT', 'move-far-left') globalCommand('gKEY_RIGHT', 'move-far-right') globalCommand('gKEY_UP', 'move-top') globalCommand('gKEY_DOWN', 'move-bottom') globalCommand(['^F', 'kDOWN'], 'move-page-down') globalCommand(['^B', 'kUP'], 'move-page-up') globalCommand(['gg', 'gk'], 'move-top') globalCommand(['G', 'gj'], 'move-bottom') globalCommand('BUTTON1_PRESSED', 'sheet.cursorRowIndex=topRowIndex+mouseY-1', 'move-mouse-row') globalCommand('BUTTON1_RELEASED', 'sheet.topRowIndex=cursorRowIndex-mouseY+1', 'scroll-mouse-row') globalCommand('BUTTON4_PRESSED', 'cursorDown(options.scroll_incr); sheet.topRowIndex += options.scroll_incr', 'move scroll_incr forward', 'scroll-up') globalCommand('REPORT_MOUSE_POSITION', 'cursorDown(-options.scroll_incr); sheet.topRowIndex -= options.scroll_incr', 'move scroll_incr backward', 'scroll-down') globalCommand('^L', 'vd.scr.clear()', 'refresh screen') globalCommand('^G', 'status(statusLine)', 'show cursor position and bounds of current sheet on status line') globalCommand('^V', 'status(__version__)', 'show version information on status line') globalCommand('^P', 'vd.push(TextSheet("statusHistory", vd.statusHistory, rowtype="statuses"))', 'open Status History') globalCommand('<', 'moveToNextRow(lambda row,sheet=sheet,col=cursorCol,val=cursorValue: col.getValue(row) != val, reverse=True) or status("no different value up this column")', 'move up the current column to the next value') globalCommand('>', 'moveToNextRow(lambda row,sheet=sheet,col=cursorCol,val=cursorValue: col.getValue(row) != val) or status("no different value down this column")', 'move down the current column to the next value') globalCommand('{', 'moveToNextRow(lambda row,sheet=sheet: sheet.isSelected(row), reverse=True) or status("no previous selected row")', 'move up the current column to the previous selected row') globalCommand('}', 'moveToNextRow(lambda row,sheet=sheet: sheet.isSelected(row)) or status("no next selected row")', 'move down the current column to the next selected row') globalCommand('_', 'cursorCol.toggleWidth(cursorCol.getMaxWidth(visibleRows))', 'adjust width of current column', 'width-curcol-max') globalCommand('z_', 'cursorCol.width = int(input("set width= ", value=cursorCol.width))', 'adjust current column width to given number', 'width-curcol-input') globalCommand('-', 'cursorCol.width = 0', 'hide current column', 'width-curcol-zero') globalCommand('z-', 'cursorCol.width = cursorCol.width//2', 'reduce width of current column by half', 'width-curcol-half') globalCommand('!', 'toggleKeyColumn(cursorColIndex); cursorRight(+1)', 'pin current column on the left as a key column', 'toggle-curcol-key') globalCommand('z~', 'cursorCol.type = anytype', 'set type of current column to anytype', 'type-curcol-any') globalCommand('~', 'cursorCol.type = str', 'set type of current column to str', 'type-curcol-str') globalCommand('@', 'cursorCol.type = date', 'set type of current column to date', 'type-curcol-date') globalCommand('#', 'cursorCol.type = int', 'set type of current column to int', 'type-curcol-int') globalCommand('$', 'cursorCol.type = currency', 'set type of current column to currency', 'type-curcol-currency') globalCommand('%', 'cursorCol.type = float', 'set type of current column to float', 'type-curcol-float') globalCommand('^', 'cursorCol.name = editCell(cursorVisibleColIndex, -1)', 'edit name of current column', 'edit-curcol-name') globalCommand('g_', 'for c in visibleCols: c.width = c.getMaxWidth(visibleRows)', 'adjust width of all visible columns', 'width-cols-max') globalCommand('[', 'orderBy(cursorCol)', 'sort ascending by current column', 'sort-curcol-asc') globalCommand(']', 'orderBy(cursorCol, reverse=True)', 'sort descending by current column', 'sort-curcol-desc') globalCommand('g[', 'orderBy(*keyCols)', 'sort ascending by all key columns', 'sort-keycols-asc') globalCommand('g]', 'orderBy(*keyCols, reverse=True)', 'sort descending by all key columns', 'sort-keycols-desc') globalCommand('^E', 'vd.lastErrors and vd.push(TextSheet("last_error", vd.lastErrors[-1])) or status("no error")', 'view traceback for most recent error') globalCommand('z^E', 'vd.push(TextSheet("cell_error", getattr(cursorCell, "error", None) or error("no error this cell")))', 'view traceback for error in current cell') globalCommand('^^', 'vd.sheets[1:] or error("no previous sheet"); vd.sheets[0], vd.sheets[1] = vd.sheets[1], vd.sheets[0]', 'jump to previous sheet (swap with current sheet)') globalCommand('g^E', 'vd.push(TextSheet("last_errors", sum(vd.lastErrors[-10:], [])))', 'view traceback for most recent errors') globalCommand('^R', 'reload(); recalc(); status("reloaded")', 'reload current sheet') globalCommand('z^R', 'cursorCol._cachedValues.clear()', 'clear cache for current column') globalCommand('/', 'moveRegex(sheet, regex=input("/", type="regex"), columns="cursorCol", backward=False)', 'search for regex forwards in current column') globalCommand('?', 'moveRegex(sheet, regex=input("?", type="regex"), columns="cursorCol", backward=True)', 'search for regex backwards in current column') globalCommand('n', 'moveRegex(sheet, reverse=False)', 'move to next match from last search') globalCommand('N', 'moveRegex(sheet, reverse=True)', 'move to previous match from last search') globalCommand('g/', 'moveRegex(sheet, regex=input("g/", type="regex"), backward=False, columns="visibleCols")', 'search for regex forwards over all visible columns') globalCommand('g?', 'moveRegex(sheet, regex=input("g?", type="regex"), backward=True, columns="visibleCols")', 'search for regex backwards over all visible columns') globalCommand('e', 'cursorCol.setValues([cursorRow], editCell(cursorVisibleColIndex)); sheet.exec_keystrokes(options.cmd_after_edit)', 'edit contents of current cell') globalCommand('ge', 'cursorCol.setValues(selectedRows or rows, input("set selected to: ", value=cursorValue))', 'set contents of current column for selected rows to input') globalCommand('zd', 'cursorCol.setValues([cursorRow], None)', 'set contents of current cell to None', 'set-curcell-none') globalCommand('gzd', 'cursorCol.setValues(selectedRows, None)', 'set contents of cells in current column to None for selected rows', 'set-selected-curcol-none') globalCommand('KEY_DC', 'set-curcell-none') globalCommand('gKEY_DC', 'set-selected-curcol-none') globalCommand('t', 'toggle([cursorRow]); cursorDown(1)', 'toggle selection of current row') globalCommand('s', 'select([cursorRow]); cursorDown(1)', 'select current row') globalCommand('u', 'unselect([cursorRow]); cursorDown(1)', 'unselect current row') globalCommand('|', 'selectByIdx(vd.searchRegex(sheet, regex=input("|", type="regex"), columns="cursorCol"))', 'select rows matching regex in current column') globalCommand('\\', 'unselectByIdx(vd.searchRegex(sheet, regex=input("\\\\", type="regex"), columns="cursorCol"))', 'unselect rows matching regex in current column') globalCommand('gt', 'toggle(rows)', 'toggle selection of all rows') globalCommand('gs', 'select(rows)', 'select all rows') globalCommand('gu', '_selectedRows.clear()', 'unselect all rows') globalCommand('g|', 'selectByIdx(vd.searchRegex(sheet, regex=input("g|", type="regex"), columns="visibleCols"))', 'select rows matching regex in any visible column') globalCommand('g\\', 'unselectByIdx(vd.searchRegex(sheet, regex=input("g\\\\", type="regex"), columns="visibleCols"))', 'unselect rows matching regex in any visible column') globalCommand(',', 'select(gatherBy(lambda r,c=cursorCol,v=cursorValue: c.getValue(r) == v), progress=False)', 'select rows matching current cell in current column') globalCommand('g,', 'select(gatherBy(lambda r,v=cursorRow: r == v), progress=False)', 'select rows matching current cell in all visible columns') globalCommand('"', 'vs = copy(sheet); vs.name += "_selectedref"; vs.rows = list(selectedRows or rows); vs.select(selectedRows); vd.push(vs)', 'open duplicate sheet with only selected rows') globalCommand('g"', 'vs = copy(sheet); vs.name += "_copy"; vs.rows = list(rows); vs.select(selectedRows); vd.push(vs)', 'open duplicate sheet with all rows') globalCommand('gz"', 'vs = deepcopy(sheet); vs.name += "_selectedcopy"; vs.rows = async_deepcopy(vs, selectedRows or rows); vd.push(vs); status("pushed sheet with async deepcopy of all rows")', 'open duplicate sheet with deepcopy of selected rows') globalCommand('=', 'addColumn(ColumnExpr(inputExpr("new column expr=")), index=cursorColIndex+1)', 'create new column from Python expression, with column names as variables') globalCommand('g=', 'cursorCol.setValuesFromExpr(selectedRows or rows, inputExpr("set selected="))', 'set current column for selected rows to result of Python expression') globalCommand('V', 'vd.push(TextSheet("%s[%s].%s" % (name, cursorRowIndex, cursorCol.name), cursorDisplay.splitlines()))', 'view contents of current cell in a new sheet') globalCommand('S', 'vd.push(SheetsSheet("sheets"))', 'open Sheets Sheet') globalCommand('C', 'vd.push(ColumnsSheet(sheet.name+"_columns", source=sheet))', 'open Columns Sheet') globalCommand('gC', 'vd.push(ColumnsSheet("all_columns", source=vd.sheets))', 'open Columns Sheet with all columns from all sheets') globalCommand('O', 'vd.push(vd.optionsSheet)', 'open Options') globalCommand(['KEY_F(1)', 'z?'], 'vd.push(HelpSheet(name + "_commands", source=sheet))', 'view VisiData man page', 'help-commands') globalCommand('^Z', 'suspend()', 'suspend VisiData process') # VisiData uses Python native int, float, str, and adds simple date, currency, and anytype. # # A type T is used internally in these ways: # o = T(val) # for interpreting raw value # o = T(str) # for conversion from string (when setting) # o = T() # for default value to be used when conversion fails # # The resulting object o must be orderable and convertible to a string for display and certain outputs (like csv). def anytype(r=None): 'minimalist "any" type' return r anytype.__name__ = '' floatchars='+-0123456789.eE' def currency(s=''): 'dirty float (strip non-numeric characters)' if isinstance(s, str): s = ''.join(ch for ch in s if ch in floatchars) return float(s) if s else float() try: import dateutil.parser except ImportError: pass @functools.total_ordering class date: 'datetime wrapper, constructed from time_t or from str with dateutil.parse' def __init__(self, s=None): if s is None: self.dt = datetime.datetime.now() elif isinstance(s, int) or isinstance(s, float): self.dt = datetime.datetime.fromtimestamp(s) elif isinstance(s, str): self.dt = dateutil.parser.parse(s) elif isinstance(s, date): self.dt = s.dt else: assert isinstance(s, datetime.datetime), (type(s), s) self.dt = s def to_string(self, fmtstr=None): 'Convert datetime object to string, using options.disp_date_fmt.' if not fmtstr: fmtstr = options.disp_date_fmt return self.dt.strftime(fmtstr) def __getattr__(self, k): 'Forward unknown attributes to inner datetime object' return getattr(self.dt, k) def __str__(self): return self.to_string() def __hash__(self): return hash(self.dt) def __float__(self): return self.dt.timestamp() def __lt__(self, a): return self.dt < a.dt def __eq__(self, a): return self.dt == a.dt def __sub__(self, a): return self.dt - a.dt def __add__(self, a): return date(self.dt + a) typemap = { None: 'Ø', str: '', date: '@', int: '#', len: '#', currency: '$', float: '%', anytype: '', } def joinSheetnames(*sheetnames): 'Concatenate sheet names in a standard way' return '_'.join(str(x) for x in sheetnames) def error(s): 'Raise an expection.' status(s) raise ExpectedException(s) def status(*args): 'Return status property via function call.' return vd().status(*args) def input(*args, **kwargs): return vd().input(*args, **kwargs) def moveListItem(L, fromidx, toidx): "Move element within list `L` and return element's new index." r = L.pop(fromidx) L.insert(toidx, r) return toidx def enumPivot(L, pivotIdx): '''Model Python `enumerate()` but starting midway through sequence `L`. Begin at index following `pivotIdx`, traverse through end. At sequence-end, begin at sequence-head, continuing through `pivotIdx`.''' rng = range(pivotIdx+1, len(L)) rng2 = range(0, pivotIdx+1) for i in itertools.chain(rng, rng2): yield i, L[i] def clean_to_id(s): # [Nas Banov] https://stackoverflow.com/a/3305731 return re.sub(r'\W|^(?=\d)', '_', str(s)) @functools.lru_cache() def vd(): 'Return VisiData singleton, which contains all global context' return VisiData() def exceptionCaught(e, **kwargs): return vd().exceptionCaught(e, **kwargs) def stacktrace(): return traceback.format_exc().strip().splitlines() def chooseOne(choices): 'Return one of `choices` elements (if list) or values (if dict).' def choiceCompleter(v, i): opts = [x for x in choices if x.startswith(v)] return opts[i%len(opts)] if isinstance(choices, dict): return choices[input('/'.join(choices.keys()) + ': ', completer=choiceCompleter)] else: return input('/'.join(str(x) for x in choices) + ': ', completer=choiceCompleter) def regex_flags(): 'Return flags to pass to regex functions from options' return sum(getattr(re, f.upper()) for f in options.regex_flags) def moveRegex(sheet, *args, **kwargs): list(vd().searchRegex(sheet, *args, moveCursor=True, **kwargs)) def sync(expectedThreads=0): vd().sync(expectedThreads) def async(func): 'Function decorator, to make calls to `func()` spawn a separate thread if available.' def _execAsync(*args, **kwargs): return vd().execAsync(func, *args, **kwargs) return _execAsync class Progress: def __init__(self, iterable=None, total=None, sheet=None): self.iterable = iterable self.total = total if total is not None else len(iterable) self.sheet = sheet if sheet else getattr(threading.current_thread(), 'sheet', None) self.made = 0 def __enter__(self): if self.sheet: self.sheet.progresses.append(self) return self def addProgress(self, n): self.made += n def __exit__(self, exc_type, exc_val, tb): if self.sheet: self.sheet.progresses.remove(self) def __iter__(self): with self as prog: for item in self.iterable: yield item self.made += 1 @async def _async_deepcopy(vs, newlist, oldlist): for r in Progress(oldlist): newlist.append(deepcopy(r)) def async_deepcopy(vs, rowlist): ret = [] _async_deepcopy(vs, ret, rowlist) return ret class VisiData: allPrefixes = 'gz' # embig'g'en, 'z'mallify def __init__(self): self.sheets = [] self.statuses = [] # statuses shown until next action self.lastErrors = [] self.searchContext = {} self.statusHistory = [] self.lastInputs = collections.defaultdict(collections.OrderedDict) # [input_type] -> prevInputs self.keystrokes = '' self.inInput = False self.prefixWaiting = False self.scr = None # curses scr self.hooks = collections.defaultdict(list) # [hookname] -> list(hooks) self.threads = [] # all long-running threads, including main and finished self.addThread(threading.current_thread(), endTime=0) self.addHook('rstatus', lambda sheet,self=self: (self.keystrokes, 'white')) self.addHook('rstatus', self.rightStatus) def status(self, *args): 'Add status message to be shown until next action.' s = '; '.join(str(x) for x in args) self.statuses.append(s) self.statusHistory.insert(0, args[0] if len(args) == 1 else args) return s def addHook(self, hookname, hookfunc): 'Add hookfunc by hookname, to be called by corresponding `callHook`.' self.hooks[hookname].insert(0, hookfunc) def callHook(self, hookname, *args, **kwargs): 'Call all functions registered with `addHook` for the given hookname.' r = [] for f in self.hooks[hookname]: try: r.append(f(*args, **kwargs)) except Exception as e: exceptionCaught(e) return r def addThread(self, t, endTime=None): t.startTime = time.process_time() t.endTime = endTime t.status = '' t.profile = None self.threads.append(t) def execAsync(self, func, *args, **kwargs): 'Execute `func(*args, **kwargs)` in a separate thread.' currentSheet = self.sheets[0] thread = threading.Thread(target=self.toplevelTryFunc, daemon=True, args=(func,)+args, kwargs=kwargs) self.addThread(thread) currentSheet.currentThreads.append(thread) thread.sheet = currentSheet thread.start() return thread def toplevelTryFunc(self, func, *args, **kwargs): 'Thread entry-point for `func(*args, **kwargs)` with try/except wrapper' t = threading.current_thread() t.name = func.__name__ ret = None try: ret = func(*args, **kwargs) except EscapeException as e: # user aborted t.status += 'aborted by user' status('%s aborted' % t.name) except Exception as e: exceptionCaught(e) t.sheet.currentThreads.remove(t) return ret @property def unfinishedThreads(self): 'A list of unfinished threads (those without a recorded `endTime`).' return [t for t in self.threads if getattr(t, 'endTime', None) is None] def checkForFinishedThreads(self): 'Mark terminated threads with endTime.' for t in self.unfinishedThreads: if not t.is_alive(): t.endTime = time.process_time() if not getattr(t, 'status', None): t.status = 'ended' def sync(self, expectedThreads=0): 'Wait for all but expectedThreads async threads to finish.' while len(self.unfinishedThreads) > expectedThreads: time.sleep(.3) self.checkForFinishedThreads() def refresh(self): Sheet.visibleCols.fget.cache_clear() def editText(self, y, x, w, **kwargs): 'Wrap global editText with `preedit` and `postedit` hooks.' v = self.callHook('preedit') if not v or v[0] is None: with EnableCursor(): v = editText(self.scr, y, x, w, **kwargs) else: v = v[0] if kwargs.get('display', True): self.status('"%s"' % v) self.callHook('postedit', v) return v def input(self, prompt, type='', **kwargs): 'Compose input prompt.' if type: ret = self._inputLine(prompt, history=list(self.lastInputs[type].keys()), **kwargs) self.lastInputs[type][ret] = ret else: ret = self._inputLine(prompt, **kwargs) return ret def _inputLine(self, prompt, **kwargs): 'Add prompt to bottom of screen and get line of input from user.' scr = self.scr if scr: scr.addstr(self.windowHeight-1, 0, prompt) self.inInput = True rstatus, _ = self.rightStatus(self.sheets[0]) ret = self.editText(self.windowHeight-1, len(prompt), self.windowWidth-len(prompt)-len(rstatus), attr=colors[options.color_edit_cell], unprintablechar=options.disp_unprintable, **kwargs) self.inInput = False return ret def getkeystroke(self, scr, vs=None): 'Get keystroke and display it on status bar.' k = None try: k = scr.get_wch() self.drawRightStatus(scr, vs or self.sheets[0]) # continue to display progress % except curses.error: return '' # curses timeout if isinstance(k, str): if ord(k) >= 32 and ord(k) != 127: # 127 == DEL or ^? return k k = ord(k) return curses.keyname(k).decode('utf-8') # kwargs: regex=None, columns=None, backward=False def searchRegex(self, sheet, moveCursor=False, reverse=False, **kwargs): 'Set row index if moveCursor, otherwise return list of row indexes.' def findMatchingColumn(sheet, row, columns, func): 'Find column for which func matches the displayed value in this row' for c in columns: if func(c.getDisplayValue(row)): return c self.searchContext.update(kwargs) regex = kwargs.get("regex") if regex: self.searchContext["regex"] = re.compile(regex, regex_flags()) or error('invalid regex: %s' % regex) regex = self.searchContext.get("regex") or error("no regex") columns = self.searchContext.get("columns") if columns == "cursorCol": columns = [sheet.cursorCol] elif columns == "visibleCols": columns = tuple(sheet.visibleCols) elif isinstance(columns, Column): columns = [columns] if not columns: error('bad columns') searchBackward = self.searchContext.get("backward") if reverse: searchBackward = not searchBackward if searchBackward: rng = range(sheet.cursorRowIndex-1, -1, -1) rng2 = range(sheet.nRows-1, sheet.cursorRowIndex-1, -1) else: rng = range(sheet.cursorRowIndex+1, sheet.nRows) rng2 = range(0, sheet.cursorRowIndex+1) matchingRowIndexes = 0 with Progress(total=sheet.nRows) as prog: for r in itertools.chain(rng, rng2): prog.addProgress(1) c = findMatchingColumn(sheet, sheet.rows[r], columns, regex.search) if c: if moveCursor: sheet.cursorRowIndex = r sheet.cursorVisibleColIndex = sheet.visibleCols.index(c) if r in rng2: status('search wrapped') return else: matchingRowIndexes += 1 yield r status('%s matches for /%s/' % (matchingRowIndexes, regex.pattern)) def exceptionCaught(self, exc=None, status=True): 'Maintain list of most recent errors and return most recent one.' if isinstance(exc, ExpectedException): # already reported, don't log return self.lastErrors.append(stacktrace()) if status: return self.status(self.lastErrors[-1][-1]) # last line of latest error if options.debug: raise def drawLeftStatus(self, scr, vs): 'Draw left side of status bar.' try: lstatus = self.leftStatus(vs) attr = colors[options.color_status] _clipdraw(scr, self.windowHeight-1, 0, lstatus, attr, self.windowWidth) except Exception as e: self.exceptionCaught(e) def drawRightStatus(self, scr, vs): 'Draw right side of status bar.' rightx = self.windowWidth-1 for rstatcolor in self.callHook('rstatus', vs): if rstatcolor: try: rstatus, color = rstatcolor rstatus = ' '+rstatus rightx -= len(rstatus) attr = colors[color] _clipdraw(scr, self.windowHeight-1, rightx, rstatus, attr, len(rstatus)) except Exception as e: self.exceptionCaught(e) curses.doupdate() def leftStatus(self, vs): 'Compose left side of status bar and add status messages.' s = vs.leftStatus() s += options.disp_status_sep.join(self.statuses) return s def rightStatus(self, sheet): 'Compose right side of status bar.' if sheet.currentThreads: status = '%9d %2d%%' % (len(sheet), sheet.progressPct) else: status = '%9d %s' % (len(sheet), sheet.rowtype) return status, options.color_status @property def windowHeight(self): return self.scr.getmaxyx()[0] if self.scr else 25 @property def windowWidth(self): return self.scr.getmaxyx()[1] if self.scr else 80 def run(self, scr): 'Manage execution of keystrokes and subsequent redrawing of screen.' global sheet scr.timeout(int(options.curses_timeout)) with suppress(curses.error): curses.curs_set(0) self.scr = scr self.keystrokes = '' while True: if not self.sheets: # if no more sheets, exit return sheet = self.sheets[0] threading.current_thread().sheet = sheet try: sheet.draw(scr) except Exception as e: self.exceptionCaught(e) self.drawLeftStatus(scr, sheet) self.drawRightStatus(scr, sheet) # visible during this getkeystroke keystroke = self.getkeystroke(scr, sheet) if keystroke: # wait until next keystroke to clear statuses and previous keystrokes if not self.prefixWaiting: self.keystrokes = '' self.statuses = [] if keystroke == 'KEY_MOUSE': try: devid, x, y, z, bstate = curses.getmouse() if bstate & curses.BUTTON_CTRL: self.keystrokes += "CTRL-" bstate &= ~curses.BUTTON_CTRL if bstate & curses.BUTTON_ALT: self.keystrokes += "ALT-" bstate &= ~curses.BUTTON_ALT if bstate & curses.BUTTON_SHIFT: self.keystrokes += "SHIFT-" bstate &= ~curses.BUTTON_SHIFT keystroke = curses.mouseEvents.get(bstate, str(bstate)) sheet.mouseX, sheet.mouseY = x, y except curses.error: keystroke = '' self.keystrokes += keystroke self.drawRightStatus(scr, sheet) # visible for commands that wait for input if not keystroke: # timeout instead of keypress pass elif keystroke == '^Q': return self.lastErrors and '\n'.join(self.lastErrors[-1]) elif self.keystrokes in sheet._commands: sheet.exec_keystrokes(self.keystrokes) self.prefixWaiting = False elif keystroke in self.allPrefixes: self.keystrokes = ''.join(sorted(set(self.keystrokes))) # prefix order/quantity does not matter self.prefixWaiting = True else: status('no command for "%s"' % (self.keystrokes)) self.prefixWaiting = False self.checkForFinishedThreads() self.callHook('predraw') try: sheet.checkCursor() except Exception as e: exceptionCaught(e) def replace(self, vs): 'Replace top sheet with the given sheet `vs`.' self.sheets.pop(0) return self.push(vs) def remove(self, vs): if vs in self.sheets: self.sheets.remove(vs) else: error('sheet not on stack') def push(self, vs): 'Move given sheet `vs` to index 0 of list `sheets`.' if vs: vs.vd = self if vs in self.sheets: self.sheets.remove(vs) self.sheets.insert(0, vs) elif vs.rows == tuple(): # empty tuple = first time sentinel vs.rows = list(vs.rows) self.sheets.insert(0, vs) vs.reload() vs.recalc() # set up Columns else: self.sheets.insert(0, vs) return vs # end VisiData class class LazyMap: 'provides a lazy mapping to obj attributes. useful when some attributes are expensive properties.' def __init__(self, obj): self.obj = obj def keys(self): return dir(self.obj) def __getitem__(self, k): if k not in dir(self.obj): raise KeyError(k) return getattr(self.obj, k) def __setitem__(self, k, v): setattr(self.obj, k, v) class CompleteExpr: def __init__(self, sheet=None): self.sheet = sheet def __call__(self, val, state): i = len(val)-1 while val[i:].isidentifier() and i >= 0: i -= 1 if i < 0: base = '' partial = val elif val[i] == '.': # no completion of attributes return None else: base = val[:i+1] partial = val[i+1:] varnames = [] varnames.extend(sorted((base+col.name) for col in self.sheet.columns if col.name.startswith(partial))) varnames.extend(sorted((base+x) for x in globals() if x.startswith(partial))) return varnames[state%len(varnames)] class Colorizer: def __init__(self, colorizerType, precedence, colorfunc): self.type = colorizerType self.precedence = precedence self.func = colorfunc class Sheet: columns = [] # list of Column # commands = [] # list of Command colorizers = [ # list of Colorizer Colorizer('hdr', 0, lambda s,c,r,v: options.color_default_hdr), Colorizer('hdr', 9, lambda s,c,r,v: options.color_current_hdr if c is s.cursorCol else None), Colorizer('hdr', 8, lambda s,c,r,v: options.color_key_col if c in s.keyCols else None), Colorizer('col', 5, lambda s,c,r,v: options.color_current_col if c is s.cursorCol else None), Colorizer('col', 7, lambda s,c,r,v: options.color_key_col if c in s.keyCols else None), Colorizer('cell', 2, lambda s,c,r,v: options.color_default), Colorizer('row', 8, lambda s,c,r,v: options.color_selected_row if s.isSelected(r) else None), ] nKeys = 0 # self.columns[:nKeys] are all pinned to the left and matched on join rowtype = 'rows' def __init__(self, name, **kwargs): self.name = name self.rows = tuple() # list of opaque row objects (tuple until first reload) self.cursorRowIndex = 0 # absolute index of cursor into self.rows self.cursorVisibleColIndex = 0 # index of cursor into self.visibleCols self.topRowIndex = 0 # cursorRowIndex of topmost row self.leftVisibleColIndex = 0 # cursorVisibleColIndex of leftmost column self.rightVisibleColIndex = 0 self.loader = None # as computed during draw() self.rowLayout = {} # [rowidx] -> y self.visibleColLayout = {} # [vcolidx] -> (x, w) # list of all columns in display order self.columns = kwargs.get('columns') or [copy(c) for c in self.columns] or [Column('')] self.recalc() # commands specific to this sheet sheetcmds = collections.OrderedDict() if hasattr(self, 'commands'): for cmd in self.commands: sheetcmds[cmd.name] = cmd self._commands = collections.ChainMap(sheetcmds, baseCommands) self._selectedRows = {} # id(row) -> row # for progress bar self.progresses = [] # list of Progress objects # track all async threads from sheet self.currentThreads = [] self._colorizers = {'row': [], 'col': [], 'hdr': [], 'cell': []} for b in [self] + list(self.__class__.__bases__): for c in getattr(b, 'colorizers', []): self.addColorizer(c) self.__dict__.update(kwargs) def __bool__(self): 'an instantiated Sheet always tests true' return True def __len__(self): return self.nRows def addColorizer(self, c): self._colorizers[c.type].append(c) def colorizeRow(self, row): return self.colorize(['row'], None, row) def colorizeColumn(self, col): return self.colorize(['col'], col, None) def colorizeHdr(self, col): return self.colorize(['hdr'], col, None) def colorizeCell(self, col, row, value): return self.colorize(['col', 'row', 'cell'], col, row, value) def colorize(self, colorizerTypes, col, row, value=None): 'Returns curses attribute for the given col/row/value' attr = 0 attrpre = 0 for colorizerType in colorizerTypes: for colorizer in sorted(self._colorizers[colorizerType], key=lambda x: x.precedence): color = colorizer.func(self, col, row, value) if color: attr, attrpre = colors.update(attr, attrpre, color, colorizer.precedence) return attr def leftStatus(self): 'Compose left side of status bar for this sheet (overridable).' return options.disp_status_fmt.format(sheet=self) def newRow(self): return list((None for c in self.columns)) def addRow(self, row, index=None): if index is None: self.rows.append(row) else: self.rows.insert(index, row) return row def searchColumnNameRegex(self, colregex, moveCursor=False): 'Select visible column matching `colregex`, if found.' for i, c in enumPivot(self.visibleCols, self.cursorVisibleColIndex): if re.search(colregex, c.name, regex_flags()): if moveCursor: self.cursorVisibleColIndex = i return c def recalc(self): for c in self.columns: if c._cachedValues: c._cachedValues.clear() c.sheet = self c.name = c._name def reload(self): 'Default reloader wraps provided `loader` function' if self.loader: self.loader() else: status('no reloader') def __copy__(self): 'copy sheet design (no rows). deepcopy columns so their attributes (width, type, name) may be adjusted independently.' cls = self.__class__ ret = cls.__new__(cls) ret.__dict__.update(self.__dict__) ret.rows = [] # a fresh list without incurring any overhead ret.columns = deepcopy(self.columns) # deepcopy columns even for shallow copy of sheet ret.recalc() # set .sheet on columns ret._selectedRows = {} ret.topRowIndex = ret.cursorRowIndex = 0 ret.progresses = [] ret.currentThreads = [] return ret def __deepcopy__(self, memo): 'same as Sheet.__copy__' ret = self.__copy__() memo[id(self)] = ret return ret @async def deleteSelected(self): 'Delete all selected rows.' oldrows = copy(self.rows) oldidx = self.cursorRowIndex ndeleted = 0 row = None # row to re-place cursor after while oldidx < len(oldrows): if not self.isSelected(oldrows[oldidx]): row = self.rows[oldidx] break oldidx += 1 self.rows.clear() for r in Progress(oldrows): if not self.isSelected(r): self.rows.append(r) if r is row: self.cursorRowIndex = len(self.rows)-1 else: ndeleted += 1 nselected = len(self._selectedRows) self._selectedRows.clear() status('deleted %s rows' % ndeleted) if ndeleted != nselected: error('expected %s' % nselected) def __repr__(self): return self.name def evalexpr(self, expr, row): return eval(expr, getGlobals(), LazyMapRow(self, row)) def inputExpr(self, prompt, *args, **kwargs): return input(prompt, "expr", *args, completer=CompleteExpr(self), **kwargs) def getCommand(self, keystrokes, default=None): k = keystrokes cmd = None while k in self._commands: cmd = self._commands.get(k, default) k = cmd.execstr # see if execstr is actually just an alias for another keystroke return cmd def exec_keystrokes(self, keystrokes, vdglobals=None): # handle multiple commands concatenated? return self.exec_command(self.getCommand(keystrokes), vdglobals, keystrokes=keystrokes) def exec_command(self, cmd, args='', vdglobals=None, keystrokes=None): "Execute `cmd` tuple with `vdglobals` as globals and this sheet's attributes as locals. Returns True if user cancelled." escaped = False err = '' if vdglobals is None: vdglobals = getGlobals() self.sheet = self try: self.vd.callHook('preexec', self, cmd.name if options.cmdlog_longname else keystrokes) exec(cmd.execstr, vdglobals, LazyMap(self)) except EscapeException as e: # user aborted self.vd.status('aborted') escaped = True except Exception as e: err = self.vd.exceptionCaught(e) try: self.vd.callHook('postexec', self.vd.sheets[0] if self.vd.sheets else None, escaped, err) except Exception: self.vd.exceptionCaught(e) self.vd.refresh() return escaped @property def name(self): return self._name or '' @name.setter def name(self, name): 'Set name without spaces.' self._name = name.strip().replace(' ', '_') @property def progressMade(self): return sum(prog.made for prog in self.progresses) @property def progressTotal(self): return sum(prog.total for prog in self.progresses) @property def progressPct(self): 'Percent complete as indicated by async actions.' if self.progressTotal != 0: return int(self.progressMade*100/self.progressTotal) return 0 @property def nVisibleRows(self): 'Number of visible rows at the current window height.' return self.vd.windowHeight-2 @property def cursorCol(self): 'Current Column object.' return self.visibleCols[self.cursorVisibleColIndex] @property def cursorRow(self): 'The row object at the row cursor.' return self.rows[self.cursorRowIndex] @property def visibleRows(self): # onscreen rows 'List of rows onscreen. ' return self.rows[self.topRowIndex:self.topRowIndex+self.nVisibleRows] @property @functools.lru_cache() def visibleCols(self): # non-hidden cols 'List of `Column` which are not hidden.' return [c for c in self.columns if not c.hidden] @property def cursorColIndex(self): 'Index of current column into `columns`. Linear search; prefer `cursorCol` or `cursorVisibleColIndex`.' return self.columns.index(self.cursorCol) @property def keyCols(self): 'List of the key columns (the first `nKeys` columns).' return self.columns[:self.nKeys] @property def nonKeyVisibleCols(self): 'All columns which are not keysList of unhidden non-key columns.' return [c for c in self.columns[self.nKeys:] if not c.hidden] @property def keyColNames(self): 'String of key column names, for SheetsSheet convenience.' return ' '.join(c.name for c in self.keyCols) @property def cursorCell(self): 'Displayed value (DisplayWrapper) at current row and column.' return self.cursorCol.getCell(self.cursorRow) @property def cursorDisplay(self): 'Displayed value (DisplayWrapper.display) at current row and column.' return self.cursorCol.getDisplayValue(self.cursorRow) @property def cursorTypedValue(self): 'Typed value at current row and column.' return self.cursorCol.getTypedValue(self.cursorRow) @property def cursorValue(self): 'Raw value at current row and column.' return self.cursorCol.getValue(self.cursorRow) @property def statusLine(self): 'String of row and column stats.' rowinfo = 'row %d/%d (%d selected)' % (self.cursorRowIndex, self.nRows, len(self._selectedRows)) colinfo = 'col %d/%d (%d visible)' % (self.cursorColIndex, self.nCols, len(self.visibleCols)) return '%s %s' % (rowinfo, colinfo) @property def nRows(self): 'Number of rows on this sheet.' return len(self.rows) @property def nCols(self): 'Number of columns on this sheet.' return len(self.columns) @property def nVisibleCols(self): 'Number of visible columns on this sheet.' return len(self.visibleCols) ## selection code def isSelected(self, row): 'True if given row is selected. O(log n).' return id(row) in self._selectedRows @async def toggle(self, rows): 'Toggle selection of given `rows`.' for r in Progress(rows, len(self.rows)): if not self.unselectRow(r): self.selectRow(r) def selectRow(self, row): 'Select given row. O(log n)' self._selectedRows[id(row)] = row def unselectRow(self, row): 'Unselect given row, return True if selected; else return False. O(log n)' if id(row) in self._selectedRows: del self._selectedRows[id(row)] return True else: return False @async def select(self, rows, status=True, progress=True): "Select given rows. Don't show progress if progress=False; don't show status if status=False." before = len(self._selectedRows) for r in (Progress(rows) if progress else rows): self.selectRow(r) if status: vd().status('selected %s%s rows' % (len(self._selectedRows)-before, ' more' if before > 0 else '')) @async def unselect(self, rows, status=True, progress=True): "Unselect given rows. Don't show progress if progress=False; don't show status if status=False." before = len(self._selectedRows) for r in (Progress(rows) if progress else rows): self.unselectRow(r) if status: vd().status('unselected %s/%s rows' % (before-len(self._selectedRows), before)) def selectByIdx(self, rowIdxs): 'Select given row indexes, without progress bar.' self.select((self.rows[i] for i in rowIdxs), progress=False) def unselectByIdx(self, rowIdxs): 'Unselect given row indexes, without progress bar.' self.unselect((self.rows[i] for i in rowIdxs), progress=False) def gatherBy(self, func): 'Generate only rows for which the given func returns True.' for r in Progress(self.rows): try: if func(r): yield r except Exception: pass def orderBy(self, *cols, **kwargs): self.rows.sort(key=lambda r,cols=cols: tuple(c.getTypedValue(r) for c in cols), **kwargs) @property def selectedRows(self): 'List of selected rows in sheet order. [O(nRows*log(nSelected))]' if len(self._selectedRows) <= 1: return list(self._selectedRows.values()) return [r for r in self.rows if id(r) in self._selectedRows] ## end selection code def moveVisibleCol(self, fromVisColIdx, toVisColIdx): 'Move column to another position in sheet.' fromColIdx = self.columns.index(self.visibleCols[fromVisColIdx]) toColIdx = self.columns.index(self.visibleCols[toVisColIdx]) moveListItem(self.columns, fromColIdx, toColIdx) return toVisColIdx def cursorDown(self, n=1): 'Move cursor down `n` rows (or up if `n` is negative).' self.cursorRowIndex += n def cursorRight(self, n=1): 'Move cursor right `n` visible columns (or left if `n` is negative).' self.cursorVisibleColIndex += n self.calcColLayout() def pageLeft(self): '''Redraw page one screen to the left. Note: keep the column cursor in the same general relative position: - if it is on the furthest right column, then it should stay on the furthest right column if possible - likewise on the left or in the middle So really both the `leftIndex` and the `cursorIndex` should move in tandem until things are correct.''' targetIdx = self.leftVisibleColIndex # for rightmost column firstNonKeyVisibleColIndex = self.visibleCols.index(self.nonKeyVisibleCols[0]) while self.rightVisibleColIndex != targetIdx and self.leftVisibleColIndex > firstNonKeyVisibleColIndex: self.cursorVisibleColIndex -= 1 self.leftVisibleColIndex -= 1 self.calcColLayout() # recompute rightVisibleColIndex # in case that rightmost column is last column, try to squeeze maximum real estate from screen if self.rightVisibleColIndex == self.nVisibleCols-1: # try to move further left while right column is still full width while self.leftVisibleColIndex > 0: rightcol = self.visibleCols[self.rightVisibleColIndex] if rightcol.width > self.visibleColLayout[self.rightVisibleColIndex][1]: # went too far self.cursorVisibleColIndex += 1 self.leftVisibleColIndex += 1 break else: self.cursorVisibleColIndex -= 1 self.leftVisibleColIndex -= 1 self.calcColLayout() # recompute rightVisibleColIndex def addColumn(self, col, index=None): 'Insert column at given index or after all columns.' if col: if index is None: index = len(self.columns) col.sheet = self self.columns.insert(index, col) return col def toggleKeyColumn(self, colidx): 'Toggle column at given index as key column.' if colidx >= self.nKeys: # if not a key, add it moveListItem(self.columns, colidx, self.nKeys) self.nKeys += 1 return 1 else: # otherwise move it after the last key self.nKeys -= 1 moveListItem(self.columns, colidx, self.nKeys) return 0 def rowkey(self, row): 'returns a tuple of the key for the given row' return tuple(c.getValue(row) for c in self.keyCols) def moveToNextRow(self, func, reverse=False): 'Move cursor to next (prev if reverse) row for which func returns True. Returns False if no row meets the criteria.' rng = range(self.cursorRowIndex-1, -1, -1) if reverse else range(self.cursorRowIndex+1, self.nRows) for i in rng: try: if func(self.rows[i]): self.cursorRowIndex = i return True except Exception: pass return False def checkCursor(self): 'Keep cursor in bounds of data and screen.' # keep cursor within actual available rowset if self.nRows == 0 or self.cursorRowIndex <= 0: self.cursorRowIndex = 0 elif self.cursorRowIndex >= self.nRows: self.cursorRowIndex = self.nRows-1 if self.cursorVisibleColIndex <= 0: self.cursorVisibleColIndex = 0 elif self.cursorVisibleColIndex >= self.nVisibleCols: self.cursorVisibleColIndex = self.nVisibleCols-1 if self.topRowIndex <= 0: self.topRowIndex = 0 elif self.topRowIndex > self.nRows-1: self.topRowIndex = self.nRows-1 # (x,y) is relative cell within screen viewport x = self.cursorVisibleColIndex - self.leftVisibleColIndex y = self.cursorRowIndex - self.topRowIndex + 1 # header # check bounds, scroll if necessary if y < 1: self.topRowIndex = self.cursorRowIndex elif y > self.nVisibleRows: self.topRowIndex = self.cursorRowIndex-self.nVisibleRows+1 if x <= 0: self.leftVisibleColIndex = self.cursorVisibleColIndex else: while True: if self.leftVisibleColIndex == self.cursorVisibleColIndex: # not much more we can do break self.calcColLayout() if self.cursorVisibleColIndex < min(self.visibleColLayout.keys()): self.leftVisibleColIndex -= 1 continue elif self.cursorVisibleColIndex > max(self.visibleColLayout.keys()): self.leftVisibleColIndex += 1 continue cur_x, cur_w = self.visibleColLayout[self.cursorVisibleColIndex] if cur_x+cur_w < self.vd.windowWidth: # current columns fit entirely on screen break self.leftVisibleColIndex += 1 def calcColLayout(self): 'Set right-most visible column, based on calculation.' minColWidth = len(options.disp_more_left)+len(options.disp_more_right) sepColWidth = len(options.disp_column_sep) winWidth = self.vd.windowWidth self.visibleColLayout = {} x = 0 vcolidx = 0 for vcolidx in range(0, self.nVisibleCols): col = self.visibleCols[vcolidx] if col.width is None and self.visibleRows: # handle delayed column width-finding col.width = col.getMaxWidth(self.visibleRows)+minColWidth if vcolidx != self.nVisibleCols-1: # let last column fill up the max width col.width = min(col.width, options.default_width) width = col.width if col.width is not None else options.default_width if col in self.keyCols or vcolidx >= self.leftVisibleColIndex: # visible columns self.visibleColLayout[vcolidx] = [x, min(width, winWidth-x)] x += width+sepColWidth if x > winWidth-1: break self.rightVisibleColIndex = vcolidx def drawColHeader(self, scr, y, vcolidx): 'Compose and draw column header for given vcolidx.' col = self.visibleCols[vcolidx] # hdrattr highlights whole column header # sepattr is for header separators and indicators sepattr = colors[options.color_column_sep] hdrattr = self.colorizeHdr(col) C = options.disp_column_sep if (self.keyCols and col is self.keyCols[-1]) or vcolidx == self.rightVisibleColIndex: C = options.disp_keycol_sep x, colwidth = self.visibleColLayout[vcolidx] # ANameTC T = typemap.get(col.type, '?') N = ' ' + col.name # save room at front for LeftMore if len(N) > colwidth-1: N = N[:colwidth-len(options.disp_truncator)] + options.disp_truncator _clipdraw(scr, y, x, N, hdrattr, colwidth) _clipdraw(scr, y, x+colwidth-len(T), T, hdrattr, len(T)) if vcolidx == self.leftVisibleColIndex and col not in self.keyCols and self.nonKeyVisibleCols.index(col) > 0: A = options.disp_more_left scr.addstr(y, x, A, sepattr) if C and x+colwidth+len(C) < self.vd.windowWidth: scr.addstr(y, x+colwidth, C, sepattr) def isVisibleIdxKey(self, vcolidx): 'Return boolean: is given column index a key column?' return self.visibleCols[vcolidx] in self.keyCols def draw(self, scr): 'Draw entire screen onto the `scr` curses object.' numHeaderRows = 1 scr.erase() # clear screen before every re-draw vd().refresh() if not self.columns: return self.rowLayout = {} self.calcColLayout() vcolidx = 0 for vcolidx, colinfo in sorted(self.visibleColLayout.items()): x, colwidth = colinfo col = self.visibleCols[vcolidx] if x < self.vd.windowWidth: # only draw inside window headerRow = 0 self.drawColHeader(scr, headerRow, vcolidx) y = headerRow + numHeaderRows for rowidx in range(0, self.nVisibleRows): dispRowIdx = self.topRowIndex + rowidx if dispRowIdx >= self.nRows: break self.rowLayout[dispRowIdx] = y row = self.rows[dispRowIdx] cellval = col.getCell(row, colwidth-1) attr = self.colorizeCell(col, row, cellval) attrpre = 0 sepattr = self.colorizeRow(row) # must apply current row here, because this colorization requires cursorRowIndex if dispRowIdx == self.cursorRowIndex: attr, attrpre = colors.update(attr, 0, options.color_current_row, 10) sepattr, _ = colors.update(sepattr, 0, options.color_current_row, 10) sepattr = sepattr or colors[options.color_column_sep] _clipdraw(scr, y, x, disp_column_fill+cellval.display, attr, colwidth) note = getattr(cellval, 'note', None) if note: noteattr, _ = colors.update(attr, attrpre, cellval.notecolor, 8) _clipdraw(scr, y, x+colwidth-len(note), note, noteattr, len(note)) sepchars = options.disp_column_sep if (self.keyCols and col is self.keyCols[-1]) or vcolidx == self.rightVisibleColIndex: sepchars = options.disp_keycol_sep if x+colwidth+len(sepchars) <= self.vd.windowWidth: scr.addstr(y, x+colwidth, sepchars, sepattr) y += 1 if vcolidx+1 < self.nVisibleCols: scr.addstr(headerRow, self.vd.windowWidth-2, options.disp_more_right, colors[options.color_column_sep]) def editCell(self, vcolidx=None, rowidx=None): 'Call `editText` at its place on the screen. Returns the new value, properly typed' if vcolidx is None: vcolidx = self.cursorVisibleColIndex x, w = self.visibleColLayout.get(vcolidx, (0, 0)) col = self.visibleCols[vcolidx] if rowidx is None: rowidx = self.cursorRowIndex if rowidx < 0: # header y = 0 currentValue = col.name else: y = self.rowLayout.get(rowidx, 0) currentValue = col.getValue(self.rows[self.cursorRowIndex]) r = self.vd.editText(y, x, w, value=currentValue, fillchar=options.disp_edit_fill, truncchar=options.disp_truncator) if rowidx >= 0: r = col.type(r) # convert input to column type return r def isNullFunc(): 'Returns isNull function according to current options.' nullset = [] if options.none_is_null: nullset.append(None) if options.empty_is_null: nullset.append('') if options.false_is_null: nullset.append(False) if options.zero_is_null: nullset.append(0) return lambda v,nullset=nullset: v in nullset class Column: def __init__(self, name, type=anytype, cache=False, **kwargs): self.sheet = None # owning sheet, set in Sheet.addColumn self.name = name # display visible name self.fmtstr = '' # by default, use str() self.type = type # anytype/str/int/float/date/func self.getter = lambda col, row: row self.setter = None # setter(col,row,value) self.width = None # == 0 if hidden, None if auto-compute next time self._cachedValues = collections.OrderedDict() if cache else None for k, v in kwargs.items(): setattr(self, k, v) # instead of __dict__.update(kwargs) to invoke property.setters def __copy__(self): cls = self.__class__ ret = cls.__new__(cls) ret.__dict__.update(self.__dict__) if ret._cachedValues: ret._cachedValues = collections.OrderedDict() # a fresh cache return ret def __deepcopy__(self, memo): return self.__copy__() # no separate deepcopy @property def name(self): return self._name or '' @name.setter def name(self, name): if isinstance(name, str): name = name.strip() if options.force_valid_colnames: name = clean_to_id(name) self._name = name @property def fmtstr(self): if self._fmtstr: return self._fmtstr t = self.type if t is int: return '{:d}' elif t is float: return '{:.02f}' elif t is currency: return '{:,.02f}' @fmtstr.setter def fmtstr(self, v): self._fmtstr = v def format(self, cellval): 'Return displayable string of `cellval` according to our `Column.type` and `Column.fmtstr`' if cellval is None: return options.disp_none # complex objects can be arbitrarily large (like sheet.rows) # this shortcut must be before self.type(cellval) (anytype will completely stringify) if isinstance(cellval, list): return '[%s]' % len(cellval) if isinstance(cellval, dict): return '{%s}' % len(cellval) t = self.type typedval = t(cellval) if t is date: return typedval.to_string(self.fmtstr) elif self.fmtstr: return self.fmtstr.format(typedval) else: return str(typedval) @property def hidden(self): 'A column is hidden if its width == 0.' return self.width == 0 def getValueRows(self, rows): 'Generate (val, row) for the given `rows` at this Column, excluding errors and nulls.' f = isNullFunc() for r in rows: try: v = self.type(self.getValue(r)) if not f(v): yield v, r except Exception: pass def getValues(self, rows): for v, r in self.getValueRows(rows): yield v def calcValue(self, row): return (self.getter)(self, row) def getTypedValue(self, row): '''Returns the properly-typed value for the given row at this column. Returns the type's default value if either the getter or the type conversion fails.''' try: return self.type(self.getValue(row)) except Exception as e: # exceptionCaught(e, status=False) return self.type() def getValue(self, row): 'Memoize calcValue with key id(row)' if self._cachedValues is None: return self.calcValue(row) k = id(row) if k in self._cachedValues: return self._cachedValues[k] ret = self.calcValue(row) self._cachedValues[k] = ret if len(self._cachedValues) > 256: # max number of entries self._cachedValues.popitem(last=False) return ret def getCell(self, row, width=None): 'Return DisplayWrapper for displayable cell value.' try: cellval = self.getValue(row) except Exception as e: return DisplayWrapper(None, error=stacktrace(), display=options.disp_error_val, note=options.note_getter_exc, notecolor=options.color_getter_exc) if isinstance(cellval, threading.Thread): return DisplayWrapper(None, display=options.disp_pending, note=options.note_pending, notecolor=options.color_note_pending) if isinstance(cellval, bytes): cellval = cellval.decode(options.encoding, options.encoding_errors) dw = DisplayWrapper(cellval) try: dispval = self.format(cellval) if width and self.type in (int, float, currency, len): dispval = dispval.rjust(width-1) dw.display = dispval # annotate cells with raw value type in anytype columns if self.type is anytype and options.color_note_type: dw.note = typemap.get(type(cellval), None) dw.notecolor = options.color_note_type except Exception as e: # type conversion or formatting failed dw.error = stacktrace() dw.display = str(cellval) dw.note = options.note_format_exc dw.notecolor = options.color_format_exc return dw def getDisplayValue(self, row): return self.getCell(row).display def setValue(self, row, value): 'Set our column value for given row to `value`.' self.setter or error(self.name+' column cannot be changed') self.setter(self, row, value) def setValues(self, rows, value): 'Set our column value for given list of rows to `value`.' value = self.type(value) for r in rows: self.setValue(r, value) status('set %d values = %s' % (len(rows), value)) @async def setValuesFromExpr(self, rows, expr): compiledExpr = compile(expr, '', 'eval') for row in Progress(rows): self.setValue(row, self.sheet.evalexpr(compiledExpr, row)) status('set %d values = %s' % (len(rows), expr)) def getMaxWidth(self, rows): 'Return the maximum length of any cell in column or its header.' w = 0 if len(rows) > 0: w = max(max(len(self.getDisplayValue(r)) for r in rows), len(self.name))+2 return max(w, len(self.name)) def toggleWidth(self, width): 'Change column width to either given `width` or default value.' if self.width != width: self.width = width else: self.width = int(options.default_width) # ---- Column makers def setitem(r, i, v): # function needed for use in lambda r[i] = v def ColumnAttr(name, attr=None, **kwargs): 'Column using getattr/setattr of given attr.' if attr is None: attr = name return Column(name, getter=lambda col,row,attr=attr: getattr(row, attr, None), setter=lambda col,row,val,attr=attr: setattr(row, attr, val), **kwargs) def ColumnItem(name, key=None, **kwargs): 'Column using getitem/setitem of given key.' if key is None: key = name return Column(name, getter=lambda col,row,key=key: row[key], setter=lambda col,row,val,key=key: setitem(row, key, val), **kwargs) def ArrayNamedColumns(columns): 'Return list of ColumnItems from given list of column names.' return [ColumnItem(colname, i) for i, colname in enumerate(columns)] def ArrayColumns(ncols): 'Return list of ColumnItems for given row length.' return [ColumnItem('', i, width=8) for i in range(ncols)] class SubrowColumn(Column): def __init__(self, origcol, subrowidx, **kwargs): super().__init__(origcol.name, type=origcol.type, width=origcol.width, **kwargs) self.origcol = origcol self.subrowidx = subrowidx def getValue(self, row): subrow = row[self.subrowidx] if subrow is not None: return self.origcol.getValue(subrow) def setValue(self, row, value): subrow = row[self.subrowidx] if subrow is not None: self.origcol.setValue(subrow, value) class DisplayWrapper: def __init__(self, value, **kwargs): self.value = value self.__dict__.update(kwargs) class ColumnEnum(Column): 'types and aggregators. row. should be kept to the values in the mapping m, and can be set by the a string key into the mapping.' def __init__(self, name, m, default=None): super().__init__(name) self.mapping = m self.default = default def getValue(self, row): v = getattr(row, self.name, None) return v.__name__ if v else None def setValue(self, row, value): if isinstance(value, str): # first try to get the actual value from the mapping value = self.mapping.get(value, value) setattr(row, self.name, value or self.default) class LazyMapRow: 'Calculate column values as needed.' def __init__(self, sheet, row): self.row = row self.sheet = sheet self._keys = [c.name for c in self.sheet.columns] def keys(self): return self._keys def __getitem__(self, colid): try: i = self._keys.index(colid) return self.sheet.columns[i].getTypedValue(self.row) except ValueError: if colid in ['row', '__row__']: return self.row elif colid in ['sheet', '__sheet__']: return self.sheet raise KeyError(colid) class ColumnExpr(Column): def __init__(self, name, expr=None): super().__init__(name) self.expr = expr or name def calcValue(self, row): return self.sheet.evalexpr(self.compiledExpr, row) @property def expr(self): return self._expr @expr.setter def expr(self, expr): self._expr = expr self.compiledExpr = compile(expr, '', 'eval') ### def confirm(prompt): yn = input(prompt, value='n')[:1] if not yn or yn not in 'Yy': error('disconfirmed') import unicodedata @functools.lru_cache(maxsize=8192) def clipstr(s, dispw): '''Return clipped string and width in terminal display characters. Note: width may differ from len(s) if East Asian chars are 'fullwidth'.''' w = 0 ret = '' ambig_width = options.disp_ambig_width for c in s: if c != ' ' and unicodedata.category(c) in ('Cc', 'Zs', 'Zl'): # control char, space, line sep ret += options.disp_oddspace w += len(options.disp_oddspace) else: ret += c eaw = unicodedata.east_asian_width(c) if eaw == 'A': # ambiguous w += ambig_width elif eaw in 'WF': # wide/full w += 2 elif not unicodedata.combining(c): w += 1 if w > dispw-len(options.disp_truncator)+1: ret = ret[:-2] + options.disp_truncator # replace final char with ellipsis w += len(options.disp_truncator) break return ret, w ## text viewer and dir browser # rowdef: (linenum, str) class TextSheet(Sheet): 'Displays any iterable source, with linewrap if wrap set in init kwargs or options.' rowtype = 'lines' commands = [ Command('v', 'sheet.wrap = not getattr(sheet, "wrap", options.wrap); status("text%s wrapped" % ("" if wrap else " NOT")); reload()', 'toggle text wrap for this sheet') ] filetype = 'txt' def __init__(self, name, source, **kwargs): super().__init__(name, source=source, **kwargs) @async def reload(self): self.columns = [Column(self.name, getter=lambda col,row: row[1])] self.rows = [] winWidth = vd().windowWidth for text in self.source: if getattr(self, 'wrap', options.wrap): startingLine = len(self.rows) for i, L in enumerate(textwrap.wrap(str(text), width=winWidth-2)): self.addRow((startingLine+i, L)) else: self.addRow((len(self.rows), text)) class ColumnsSheet(Sheet): rowtype = 'columns' class ValueColumn(Column): 'passthrough to the value on the source cursorRow' def calcValue(self, srcCol): return srcCol.getDisplayValue(srcCol.sheet.cursorRow) def setValue(self, srcCol, val): srcCol.setValue(self.sheet.source.cursorRow, val) columns = [ ColumnAttr('sheet'), ColumnAttr('name'), ColumnAttr('width', type=int), ColumnEnum('type', globals(), default=anytype), ColumnAttr('fmtstr'), ValueColumn('value') ] nKeys = 2 colorizers = [ Colorizer('row', 7, lambda self,c,r,v: options.color_key_col if r in r.sheet.keyCols else None), ] commands = [] def reload(self): if isinstance(self.source, Sheet): self.rows = self.source.columns self.cursorRowIndex = self.source.cursorColIndex self.columns[0].width = 0 # hide 'sheet' column if only one sheet else: # lists of Columns self.rows = [] for src in self.source: if src is not self: self.rows.extend(src.columns) class SheetsSheet(Sheet): rowtype = 'sheets' commands = [Command(ENTER, 'jumpTo(cursorRowIndex)', 'jump to sheet referenced in current row')] columns = [ ColumnAttr('name'), ColumnAttr('nRows', type=int), ColumnAttr('nCols', type=int), ColumnAttr('nVisibleCols', type=int), ColumnAttr('cursorDisplay'), ColumnAttr('keyColNames'), ColumnAttr('source'), ] def reload(self): self.rows = vd().sheets def jumpTo(self, sheetnum): if sheetnum != 0: moveListItem(self.rows, sheetnum, 0) self.rows.pop(1) class HelpSheet(Sheet): 'Show all commands available to the source sheet.' rowtype = 'commands' class HelpColumn(Column): def calcValue(self, r): cmd = self.sheet.source.getCommand(self.prefix+r.name, None) return cmd.helpstr if cmd else '-' columns = [ ColumnAttr('keystrokes', 'name'), ColumnAttr('helpstr'), HelpColumn('with_g_prefix', prefix='g'), HelpColumn('with_z_prefix', prefix='z'), ColumnAttr('execstr', width=0), ] nKeys = 1 def reload(self): self.rows = [] for src in self.source._commands.maps: self.rows.extend(src.values()) class OptionsSheet(Sheet): rowtype = 'options' commands = [ Command(ENTER, 'editOption(cursorRow)', 'edit option'), Command('e', ENTER) ] columns = [ColumnItem('option', 0), Column('value', getter=lambda col,row: row[1], setter=lambda col,row,val: setattr(options, row[0], val)), ColumnItem('default', 2), ColumnItem('description', 3)] colorizers = [] nKeys = 1 def editOption(self, row): if isinstance(row[2], bool): self.source[row[0]] = not row[1] else: self.source[row[0]] = self.editCell(1) def reload(self): self.rows = list(self.source._opts.values()) vd().optionsSheet = OptionsSheet('options', source=options) ### Curses helpers def clipdraw(scr, y, x, s, attr, w=None): 'Draw string `s` at (y,x)-(y,x+w), clipping with ellipsis char.' _, windowWidth = scr.getmaxyx() dispw = 0 try: if w is None: w = windowWidth-1 w = min(w, windowWidth-x-1) if w == 0: # no room anyway return # convert to string just before drawing s, dispw = clipstr(str(s), w) scr.addstr(y, x, disp_column_fill*w, attr) scr.addstr(y, x, s, attr) except Exception as e: # raise type(e)('%s [clip_draw y=%s x=%s dispw=%s w=%s]' % (e, y, x, dispw, w) # ).with_traceback(sys.exc_info()[2]) pass _clipdraw = clipdraw # https://stackoverflow.com/questions/19833315/running-system-commands-in-python-using-curses-and-panel-and-come-back-to-previ class SuspendCurses: 'Context Manager to temporarily leave curses mode' def __enter__(self): curses.endwin() def __exit__(self, exc_type, exc_val, tb): newscr = curses.initscr() newscr.refresh() curses.doupdate() class EnableCursor: def __enter__(self): with suppress(curses.error): curses.mousemask(0) curses.curs_set(1) def __exit__(self, exc_type, exc_val, tb): with suppress(curses.error): curses.curs_set(0) curses.mousemask(-1) def launchExternalEditor(v, linenum=0): editor = os.environ.get('EDITOR') or error('$EDITOR not set') import tempfile fd, fqpn = tempfile.mkstemp(text=True) with open(fd, 'w') as fp: fp.write(v) with SuspendCurses(): cmd = '%s %s' % (editor, fqpn) if linenum: cmd += ' +%s' % linenum os.system(cmd) with open(fqpn, 'r') as fp: return fp.read() def suspend(): import signal with SuspendCurses(): os.kill(os.getpid(), signal.SIGSTOP) # history: earliest entry first def editText(scr, y, x, w, attr=curses.A_NORMAL, value='', fillchar=' ', truncchar='-', unprintablechar='.', completer=lambda text,idx: None, history=[], display=True): 'A better curses line editing widget.' ESC='^[' ENTER='^J' def until_get_wch(): 'Ignores get_wch timeouts' ret = None while not ret: try: ret = scr.get_wch() except curses.error: pass return ret def splice(v, i, s): 'Insert `s` into string `v` at `i` (such that v[i] == s[0]).' return v if i < 0 else v[:i] + s + v[i:] def clean_printable(s): 'Escape unprintable characters.' return ''.join(c if c.isprintable() else ('<%04X>' % ord(c)) for c in str(s)) def delchar(s, i, remove=1): 'Delete `remove` characters from str `s` beginning at position `i`.' return s if i < 0 else s[:i] + s[i+remove:] class CompleteState: def __init__(self, completer_func): self.comps_idx = -1 self.completer_func = completer_func self.former_i = None self.just_completed = False def complete(self, v, i, state_incr): self.just_completed = True self.comps_idx += state_incr if self.former_i is None: self.former_i = i try: r = self.completer_func(v[:self.former_i], self.comps_idx) except Exception as e: # beep/flash; how to report exception? return v, i if not r: # beep/flash to indicate no matches? return v, i v = r + v[i:] return v, len(v) def reset(self): if self.just_completed: self.just_completed = False else: self.former_i = None self.comps_idx = -1 class HistoryState: def __init__(self, history): self.history = history self.hist_idx = None self.prev_val = None def up(self, v, i): if self.hist_idx is None: self.hist_idx = len(self.history) self.prev_val = v if self.hist_idx > 0: self.hist_idx -= 1 v = self.history[self.hist_idx] i = len(v) return v, i def down(self, v, i): if self.hist_idx is None: return v, i elif self.hist_idx < len(self.history)-1: self.hist_idx += 1 v = self.history[self.hist_idx] else: v = self.prev_val self.hist_idx = None i = len(v) return v, i history_state = HistoryState(history) complete_state = CompleteState(completer) insert_mode = True first_action = True v = str(value) # value under edit i = 0 # index into v left_truncchar = right_truncchar = truncchar def rfind_nonword(s, a, b): while not s[b].isalnum() and b >= a: # first skip non-word chars b -= 1 while s[b].isalnum() and b >= a: b -= 1 return b while True: if display: dispval = clean_printable(v) else: dispval = '*' * len(v) dispi = i # the onscreen offset within the field where v[i] is displayed if len(dispval) < w: # entire value fits dispval += fillchar*(w-len(dispval)) elif i == len(dispval): # cursor after value (will append) dispi = w-1 dispval = left_truncchar + dispval[len(dispval)-w+2:] + fillchar elif i >= len(dispval)-w//2: # cursor within halfwidth of end dispi = w-(len(dispval)-i) dispval = left_truncchar + dispval[len(dispval)-w+1:] elif i <= w//2: # cursor within halfwidth of beginning dispval = dispval[:w-1] + right_truncchar else: dispi = w//2 # visual cursor stays right in the middle k = 1 if w%2==0 else 0 # odd widths have one character more dispval = left_truncchar + dispval[i-w//2+1:i+w//2-k] + right_truncchar scr.addstr(y, x, dispval, attr) scr.move(y, x+dispi) ch = vd().getkeystroke(scr) if ch == '': continue elif ch == 'KEY_IC': insert_mode = not insert_mode elif ch == '^A' or ch == 'KEY_HOME': i = 0 elif ch == '^B' or ch == 'KEY_LEFT': i -= 1 elif ch == '^C' or ch == ESC: raise EscapeException(ch) elif ch == '^D' or ch == 'KEY_DC': v = delchar(v, i) elif ch == '^E' or ch == 'KEY_END': i = len(v) elif ch == '^F' or ch == 'KEY_RIGHT': i += 1 elif ch in ('^H', 'KEY_BACKSPACE', '^?'): i -= 1; v = delchar(v, i) elif ch == '^I': v, i = complete_state.complete(v, i, +1) elif ch == 'KEY_BTAB': v, i = complete_state.complete(v, i, -1) elif ch == ENTER: break elif ch == '^K': v = v[:i] # ^Kill to end-of-line elif ch == '^O': v = launchExternalEditor(v) elif ch == '^R': v = str(value) # ^Reload initial value elif ch == '^T': v = delchar(splice(v, i-2, v[i-1]), i) # swap chars elif ch == '^U': v = v[i:]; i = 0 # clear to beginning elif ch == '^V': v = splice(v, i, until_get_wch()); i += 1 # literal character elif ch == '^W': j = rfind_nonword(v, 0, i-1); v = v[:j+1] + v[i:]; i = j+1 elif ch == '^Z': v = suspend() elif history and ch == 'KEY_UP': v, i = history_state.up(v, i) elif history and ch == 'KEY_DOWN': v, i = history_state.down(v, i) elif ch.startswith('KEY_'): pass else: if first_action: v = '' if insert_mode: v = splice(v, i, ch) else: v = v[:i] + ch + v[i+1:] i += 1 if i < 0: i = 0 if i > len(v): i = len(v) first_action = False complete_state.reset() return v class ColorMaker: def __init__(self): self.attrs = {} self.color_attrs = {} def setup(self): if options.use_default_colors: curses.use_default_colors() default_bg = -1 else: default_bg = curses.COLOR_BLACK self.color_attrs['black'] = curses.color_pair(0) for c in range(0, options.force_256_colors and 256 or curses.COLORS): curses.init_pair(c+1, c, default_bg) self.color_attrs[str(c)] = curses.color_pair(c+1) for c in 'red green yellow blue magenta cyan white'.split(): colornum = getattr(curses, 'COLOR_' + c.upper()) self.color_attrs[c] = curses.color_pair(colornum+1) for a in 'normal blink bold dim reverse standout underline'.split(): self.attrs[a] = getattr(curses, 'A_' + a.upper()) def keys(self): return list(self.attrs.keys()) + list(self.color_attrs.keys()) def __getitem__(self, colornamestr): color, prec = self.update(0, 0, colornamestr, 10) return color def update(self, attr, attr_prec, colornamestr, newcolor_prec): attr = attr or 0 if isinstance(colornamestr, str): for colorname in colornamestr.split(' '): if colorname in self.color_attrs: if newcolor_prec > attr_prec: attr &= ~2047 attr |= self.color_attrs[colorname.lower()] attr_prec = newcolor_prec elif colorname in self.attrs: attr |= self.attrs[colorname.lower()] return attr, attr_prec colors = ColorMaker() def setupcolors(stdscr, f, *args): curses.raw() # get control keys instead of signals curses.meta(1) # allow "8-bit chars" curses.mousemask(-1) # even more than curses.ALL_MOUSE_EVENTS curses.mouseinterval(0) # very snappy but does not allow for [multi]click curses.mouseEvents = {} for k in dir(curses): if k.startswith('BUTTON') or k == 'REPORT_MOUSE_POSITION': curses.mouseEvents[getattr(curses, k)] = k return f(stdscr, *args) def wrapper(f, *args): return curses.wrapper(setupcolors, f, *args) ### external interface def run(*sheetlist): 'Main entry point; launches vdtui with the given sheets already pushed (last one is visible)' # reduce ESC timeout to 25ms. http://en.chys.info/2009/09/esdelay-ncurses/ os.putenv('ESCDELAY', '25') ret = wrapper(cursesMain, sheetlist) if ret: print(ret) def cursesMain(_scr, sheetlist): 'Populate VisiData object with sheets from a given list.' colors.setup() for vs in sheetlist: vd().push(vs) # first push does a reload status(' or z? opens help') return vd().run(_scr) def addGlobals(g): 'importers can call `addGlobals(globals())` to have their globals accessible to execstrings' globals().update(g) def getGlobals(): return globals() if __name__ == '__main__': run(*(TextSheet('contents', open(src)) for src in sys.argv[1:])) visidata-1.0/visidata/__init__.py0000660000175100017520000000156313223031561016741 0ustar anjaanja00000000000000 'VisiData: a curses interface for exploring and arranging tabular data' from .vdtui import __version__ from .vdtui import * from .Path import * from .async import * from .zscroll import * from .aggregators import * from .data import * from .clipboard import * from .freqtbl import * from .describe import * from .pyobj import * from .metasheets import * from .pivot import * from .tidydata import * from .cmdlog import * from .freeze import * from .regex import * from .canvas import * from .graph import * from .loaders.csv import * from .loaders.json import * from .loaders.zip import * from .loaders.xlsx import * from .loaders.hdf5 import * from .loaders.sqlite import * from .loaders.fixed_width import * from .loaders.postgres import * from .loaders.shp import * from .loaders.mbtiles import * from .loaders.http import * from .loaders.html import * addGlobals(globals()) visidata-1.0/visidata/freeze.py0000660000175100017520000000337513226522612016472 0ustar anjaanja00000000000000from visidata import * from copy import deepcopy globalCommand("'", 'addColumn(StaticColumn(sheet.rows, cursorCol), cursorColIndex+1)', 'add a frozen copy of current column with all cells evaluated') globalCommand("g'", 'vd.push(StaticSheet(sheet)); status("pushed frozen copy of "+name)', 'open a frozen copy of current sheet with all visible columns evaluated') globalCommand("z'", 'resetCache(cursorCol)', 'add/reset cache for current column') globalCommand("gz'", 'resetCache(*visibleCols)', 'add/reset cache for all visible columns') globalCommand("zg'", "gz'") def resetCache(self, *cols): for col in cols: col._cachedValues = collections.OrderedDict() status("reset cache for " + (cols[0].name if len(cols) == 1 else str(len(cols))+" columns")) Sheet.resetCache = resetCache def StaticColumn(rows, col): c = deepcopy(col) frozenData = {} @async def _calcRows(sheet): for r in Progress(rows): try: frozenData[id(r)] = col.getValue(r) except Exception as e: frozenData[id(r)] = e _calcRows(col.sheet) c.calcValue=lambda row,d=frozenData: d[id(row)] c.setter=lambda col,row,val,d=frozenData: setitem(d, id(row), val) c.name = c.name + '_frozen' return c class StaticSheet(Sheet): 'A copy of the source sheet with all cells frozen.' def __init__(self, source): super().__init__(source.name + "'", source=source) self.columns = [ColumnItem(col.name, i, width=col.width, type=col.type) for i,col in enumerate(self.source.columns)] self.nKeys = source.nKeys @async def reload(self): self.rows = [] for r in Progress(self.source.rows): self.addRow([col.getValue(r) for col in self.source.columns]) visidata-1.0/visidata/loaders/0000770000175100017520000000000013232247165016264 5ustar anjaanja00000000000000visidata-1.0/visidata/loaders/zip.py0000660000175100017520000000207213223031561017431 0ustar anjaanja00000000000000import codecs from visidata import * class open_zip(Sheet): 'Provide wrapper around `zipfile` library for opening ZIP files.' rowtype = 'files' commands = [ Command(ENTER, 'vd.push(openZipFileEntry(cursorRow))', 'open this file'), Command('g'+ENTER, 'for r in selectedRows or rows: vd.push(openZipFileEntry(r))', 'open all selected files') ] columns = [ ColumnAttr('filename'), ColumnAttr('file_size', type=int), ColumnAttr('date_time'), ColumnAttr('compress_size', type=int) ] def __init__(self, p): super().__init__(p.name, source=p) def reload(self): import zipfile with zipfile.ZipFile(self.source.resolve(), 'r') as zfp: self.rows = zfp.infolist() def openZipFileEntry(self, zi): import zipfile zfp = zipfile.ZipFile(self.source.resolve(), 'r') decodedfp = codecs.iterdecode(zfp.open(zi), encoding=options.encoding, errors=options.encoding_errors) return openSource(PathFd(zi.filename, decodedfp, filesize=zi.file_size)) visidata-1.0/visidata/loaders/fixed_width.py0000660000175100017520000000313513223031561021126 0ustar anjaanja00000000000000 from visidata import * option('fixed_rows', 1000, 'number of rows to check for fixed width columns') def open_fixed(p): return FixedWidthColumnsSheet(p.name, source=p) class FixedWidthColumn(Column): def __init__(self, name, i, j, **kwargs): super().__init__(name, **kwargs) self.i, self.j = i, j def getValue(self, row): return row[0][self.i:self.j] def setValue(self, row, value): row[0] = row[:self.i] + '%*s' % (self.j-self.i, value) + row[self.j:] def columnize(rows): 'Generate (i,j) indexes for fixed-width columns found in rows' ## find all character columns that are not spaces allNonspaces = set() allNonspaces.add(max(len(r) for r in rows)+1) for r in rows: for i, ch in enumerate(r): if not ch.isspace(): allNonspaces.add(i) colstart = 0 prev = 0 # collapse fields for i in allNonspaces: if i > prev+1: yield colstart, prev+1 colstart = i prev = i class FixedWidthColumnsSheet(Sheet): rowtype = 'lines' columns = [ColumnItem('line', 0)] @async def reload(self): self.rows = [] for line in self.source: self.addRow([line]) self.columns = [] # compute fixed width columns for i, j in columnize(list(r[0] for r in self.rows[:options.fixed_rows])): c = FixedWidthColumn('', i, j) c.name = '_'.join(c.getValue(r) for r in self.rows[:options.header]) self.addColumn(c) # discard header rows self.rows = self.rows[options.header:] visidata-1.0/visidata/loaders/html.py0000660000175100017520000000425713223031561017602 0ustar anjaanja00000000000000import html from visidata import * def open_html(p): return HtmlTablesSheet(p.name, source=p) # rowdef: lxml.html.HtmlElement class HtmlTablesSheet(Sheet): rowtype = 'tables' columns = [ Column('tag', getter=lambda col,row: row.tag), Column('id', getter=lambda col,row: row.attrib.get('id')), Column('classes', getter=lambda col,row: row.attrib.get('class')), ] commands = [ Command(ENTER, 'vd.push(HtmlTableSheet(name=cursorRow.attrib.get("id", "table_" + str(cursorRowIndex)), source=cursorRow))', 'open this table') ] @async def reload(self): import lxml.html from lxml import etree utf8_parser = etree.HTMLParser(encoding='utf-8') with self.source.open_text() as fp: html = lxml.html.etree.parse(fp, parser=utf8_parser) self.rows = [] for e in html.iter(): if e.tag == 'table': self.addRow(e) class HtmlTableSheet(Sheet): rowtype = 'rows' columns = [] @async def reload(self): self.rows = [] for r in self.source.iter(): if r.tag == 'tr': row = [(c.text or '').strip() for c in r.getchildren()] if any(row): self.addRow(row) for i, name in enumerate(self.rows[0]): self.addColumn(ColumnItem(name, i)) self.rows = self.rows[1:] @async def save_html(sheet, fn): 'Save sheet as
in an HTML file.' with open(fn, 'w', encoding='ascii', errors='xmlcharrefreplace') as fp: fp.write('\n'.format(sheetname=sheet.name)) # headers fp.write('') for col in sheet.visibleCols: contents = html.escape(col.name) fp.write(''.format(colname=contents)) fp.write('\n') # rows for r in Progress(sheet.rows): fp.write('') for col in sheet.visibleCols: fp.write('') fp.write('\n') fp.write('
{colname}
') fp.write(html.escape(col.getDisplayValue(r))) fp.write('
') status('%s save finished' % fn) visidata-1.0/visidata/loaders/hdf5.py0000660000175100017520000000306513223031561017460 0ustar anjaanja00000000000000from visidata import * class SheetH5Obj(Sheet): 'Support sheets in HDF5 format.' commands = [ Command(ENTER, 'vd.push(SheetH5Obj(joinSheetnames(name,cursorRow.name), source=cursorRow))', 'open this group or dataset'), Command('A', 'vd.push(SheetDict(cursorRow.name + "_attrs", cursorRow.attrs))', 'open metadata sheet for this object') ] def reload(self): import h5py if isinstance(self.source, h5py.Group): self.rowtype = 'objects' self.columns = [ Column(self.source.name, str, lambda r: r.name.split('/')[-1]), Column('type', str, lambda r: type(r).__name__), Column('nItems', int, lambda r: len(r)), ] self.rows = [ self.source[objname] for objname in self.source.keys() ] elif isinstance(self.source, h5py.Dataset): if len(self.source.shape) == 1: self.columns = [ColumnItem(colname, colname) for colname in self.source.dtype.names or [0]] self.rows = self.source[:] # copy elif len(self.source.shape) == 2: # matrix self.columns = ArrayColumns(self.source.shape[1]) self.rows = self.source[:] # copy else: status('too many dimensions in shape %s' % str(self.source.shape)) else: status('unknown h5 object type %s' % type(self.source)) class open_hdf5(SheetH5Obj): def __init__(self, p): import h5py super().__init__(p.name, h5py.File(str(p), 'r')) open_h5 = open_hdf5 visidata-1.0/visidata/loaders/sqlite.py0000660000175100017520000000334413223031561020133 0ustar anjaanja00000000000000from visidata import * def open_sqlite(path): vs = SqliteSheet(path.name + '_tables', path, 'sqlite_master') vs.columns = vs.getColumns('sqlite_master') return vs class SqliteSheet(Sheet): 'Provide functionality for importing SQLite databases.' commands = [ Command(ENTER, 'vd.push(SqliteSheet(joinSheetnames(source.name, cursorRow[1]), sheet, cursorRow[1]))', 'load the entire table into memory') ] def __init__(self, name, pathOrSheet, tableName): super().__init__(name, source=pathOrSheet, tableName=tableName) if isinstance(pathOrSheet, Sheet): self.conn = pathOrSheet.conn elif isinstance(pathOrSheet, Path): import sqlite3 self.conn = sqlite3.connect(pathOrSheet.resolve()) # must not be @async due to sqlite lib being single-threaded def reload(self): tblname = self.tableName self.columns = self.getColumns(tblname) r = self.conn.execute('SELECT COUNT(*) FROM %s' % tblname).fetchall() self.rows = [] for r in Progress(self.conn.execute("SELECT * FROM %s" % tblname), total=r[0][0]-1): self.addRow(r) def getColumns(self, tableName): cols = [] for i, r in enumerate(self.conn.execute('PRAGMA TABLE_INFO(%s)' % tableName)): c = ColumnItem(r[1], i) t = r[2].lower() if t == 'integer': c.type = int elif t == 'text': c.type = str elif t == 'blob': c.type = str c.width = 0 elif t == 'real': c.type = float else: status('unknown sqlite type "%s"' % t) cols.append(c) return cols visidata-1.0/visidata/loaders/xlsx.py0000660000175100017520000000431013232246431017626 0ustar anjaanja00000000000000 from visidata import * class open_xlsx(Sheet): 'Load XLSX file (in Excel Open XML format).' commands = [ Command(ENTER, 'vd.push(sheet.getSheet(cursorRow))', 'load the entire table into memory') ] def __init__(self, path): super().__init__(path.name, source=path) self.workbook = None @async def reload(self): import openpyxl self.columns = [Column('name')] self.workbook = openpyxl.load_workbook(self.source.resolve(), data_only=True, read_only=True) self.rows = list(self.workbook.sheetnames) def getSheet(self, sheetname): worksheet = self.workbook.get_sheet_by_name(sheetname) return xlsxSheet(joinSheetnames(self.name, sheetname), source=worksheet) class xlsxSheet(Sheet): @async def reload(self): worksheet = self.source self.columns = [] self.rows = [] for row in Progress(worksheet.iter_rows(), worksheet.max_row or 0): L = list(cell.value for cell in row) for i in range(len(self.columns), len(L)): # no-op if already done self.addColumn(ColumnItem(None, i, width=8)) self.addRow(L) class open_xls(Sheet): 'Load XLS file (in Excel format).' commands = [ Command(ENTER, 'vd.push(sheet.getSheet(cursorRow))', 'load the entire table into memory') ] def __init__(self, path): super().__init__(path.name, source=path) self.workbook = None @async def reload(self): import xlrd self.columns = [Column('name')] self.workbook = xlrd.open_workbook(self.source.resolve()) self.rows = list(self.workbook.sheet_names()) def getSheet(self, sheetname): worksheet = self.workbook.sheet_by_name(sheetname) return xlsSheet(joinSheetnames(self.name, sheetname), source=worksheet) class xlsSheet(Sheet): @async def reload(self): worksheet = self.source self.columns = [] for i in range(worksheet.ncols): self.addColumn(ColumnItem(None, i, width=8)) for rownum in Progress(range(worksheet.nrows)): self.addRow(list(worksheet.cell(rownum, colnum).value for colnum in range(worksheet.ncols))) visidata-1.0/visidata/loaders/shp.py0000660000175100017520000000367013226522612017433 0ustar anjaanja00000000000000from visidata import * def open_shp(p): return ShapeSheet(p.name, source=p) # pyshp doesn't care about file extensions open_dbf = open_shp shptypes = { 'C': str, 'N': float, 'L': float, 'F': float, 'D': date, 'M': str, } def shptype(ftype, declen): t = shptypes[ftype[:1]] if t is float and declen == 0: return int return t # rowdef: shaperec class ShapeSheet(Sheet): rowtype = 'shapes' columns = [ Column('shapeType', width=0, getter=lambda col,row: row.shape.shapeType) ] commands = [ Command('.', 'vd.push(ShapeMap(name+"_map", sheet, sourceRows=[cursorRow]))', ''), Command('g.', 'vd.push(ShapeMap(name+"_map", sheet, sourceRows=selectedRows or rows))', ''), ] @async def reload(self): import shapefile sf = shapefile.Reader(self.source.resolve()) self.columns += [ Column(fname, getter=lambda col,row,i=i: row.record[i], type=shptype(ftype, declen)) for i, (fname, ftype, fieldlen, declen) in enumerate(sf.fields[1:]) # skip DeletionFlag ] self.rows = [] for shaperec in Progress(sf.iterShapeRecords(), total=sf.numRecords): self.addRow(shaperec) class ShapeMap(Canvas): aspectRatio = 1.0 @async def reload(self): self.reset() for row in Progress(self.sourceRows): # color according to key k = tuple(col.getValue(row) for col in self.source.keyCols) if row.shape.shapeType == 5: self.polygon(row.shape.points, self.plotColor(k), row) elif row.shape.shapeType == 3: self.polyline(row.shape.points, self.plotColor(k), row) elif row.shape.shapeType == 1: x, y = row.shape.points[0] self.point(x, y, self.plotColor(k), row) else: status('notimpl shapeType %s' % row.shape.shapeType) self.refresh() visidata-1.0/visidata/loaders/__init__.py0000660000175100017520000000000013223031561020353 0ustar anjaanja00000000000000visidata-1.0/visidata/loaders/http.py0000660000175100017520000000066513223031561017614 0ustar anjaanja00000000000000from visidata import * class HttpPath(PathFd): def __init__(self, url, req): from urllib.parse import urlparse obj = urlparse(url) super().__init__(obj.path, req) self.req = req def openurl_http(p, filetype=None): import requests r = requests.get(p.url, stream=True) return openSource(HttpPath(p.url, r.iter_lines(decode_unicode=True)), filetype=filetype) openurl_https = openurl_http visidata-1.0/visidata/loaders/mbtiles.py0000660000175100017520000001006313226522612020272 0ustar anjaanja00000000000000from visidata import * import json import gzip import sqlite3 def open_pbf(p): return PbfSheet(p.name, tile_data=p.read_bytes()) def open_mbtiles(p): return MbtilesSheet(p.name, source=p) def getListDepth(L): if not isinstance(L, list): return 0 if len(L) == 0: return 0 return getListDepth(L[0]) + 1 def getFeatures(tile_data): for layername, layer in tile_data.items(): for feat in layer['features']: yield layername, feat class MbtilesSheet(Sheet): columns = [ ColumnItem('zoom_level', 0), ColumnItem('tile_column', 1), ColumnItem('tile_row', 2), ] commands = [ Command(ENTER, 'vd.push(PbfSheet(tilename(cursorRow), source=sheet, sourceRow=cursorRow))', 'open this tile'), Command('.', 'tn=tilename(cursorRow); vd.push(PbfCanvas(tn+"_map", source=PbfSheet(tn, sourceRows=list(getFeatures(getTile(*cursorRow)))))', 'plot this tile'), # Command('g.', 'tn=tilename(cursorRow); vd.push(PbfCanvas(tn+"_map", source=PbfSheet(tn), sourceRows=sum((list(getFeatures(getTile(*r))) for r in selectedRows or rows), [])))', 'plot selected tiles'), ] def tilename(self, row): return ",".join(str(x) for x in row) def getTile(self, zoom_level, tile_col, tile_row): import mapbox_vector_tile con = sqlite3.connect(self.source.resolve()) tile_data = con.execute(''' SELECT tile_data FROM tiles WHERE zoom_level = ? AND tile_column = ? AND tile_row = ?''', (zoom_level, tile_col, tile_row)).fetchone()[0] return mapbox_vector_tile.decode(gzip.decompress(tile_data)) @async def reload(self): con = sqlite3.connect(self.source.resolve()) self.metadata = dict(con.execute('SELECT name, value FROM metadata').fetchall()) tiles = con.execute('SELECT zoom_level, tile_column, tile_row FROM tiles') for r in Progress(tiles.fetchall()): self.addRow(r) class PbfSheet(Sheet): columns = [ ColumnItem('layer', 0), Column('geometry_type', getter=lambda col,row: row[1]['geometry']['type']), Column('geometry_coords', getter=lambda col,row: row[1]['geometry']['coordinates'], width=0), Column('geometry_coords_depth', getter=lambda col,row: getListDepth(row[1]['geometry']['coordinates']), width=0), ] nKeys = 1 # layer commands = [ Command('.', 'vd.push(PbfCanvas(name+"_map", source=sheet, sourceRows=[cursorRow]))', 'plot this row only'), Command('g.', 'vd.push(PbfCanvas(name+"_map", source=sheet, sourceRows=selectedRows or rows))', 'plot as map'), ] @async def reload(self): props = set() # property names self.rows = [] for r in getFeatures(self.source.getTile(*self.sourceRow)): self.rows.append(r) props.update(r[1]['properties'].keys()) for key in props: self.addColumn(Column(key, getter=lambda col,row,key=key: row[1]['properties'][key])) class PbfCanvas(InvertedCanvas): aspectRatio = 1.0 def iterpolylines(self, r): layername, feat = r geom = feat['geometry'] t = geom['type'] coords = geom['coordinates'] key = self.source.rowkey(r) if t == 'LineString': yield coords, self.plotColor(key), r elif t == 'Point': yield [coords], self.plotColor(key), r elif t == 'Polygon': for poly in coords: yield poly+[poly[0]], self.plotColor(key), r elif t == 'MultiLineString': for line in coords: yield line, self.plotColor(key), r elif t == 'MultiPolygon': for mpoly in coords: for poly in mpoly: yield poly+[poly[0]], self.plotColor(key), r else: assert False, t @async def reload(self): self.reset() for r in Progress(self.sourceRows): for vertexes, attr, row in self.iterpolylines(r): self.polyline(vertexes, attr, row) self.refresh() visidata-1.0/visidata/loaders/csv.py0000660000175100017520000000472713226522612017440 0ustar anjaanja00000000000000 from visidata import * import csv option('csv_dialect', 'excel', 'dialect passed to csv.reader') option('csv_delimiter', ',', 'delimiter passed to csv.reader') option('csv_quotechar', '"', 'quotechar passed to csv.reader') option('csv_skipinitialspace', True, 'skipinitialspace passed to csv.reader') csv.field_size_limit(sys.maxsize) def open_csv(p): vs = Sheet(p.name, source=p) vs.loader = lambda vs=vs: load_csv(vs) return vs def wrappedNext(rdr): try: return next(rdr) except csv.Error as e: return ['[csv.Error: %s]' % e] @async def load_csv(vs): 'Convert from CSV, first handling header row specially.' with vs.source.open_text() as fp: samplelen = min(len(wrappedNext(fp)) for i in range(10)) # for progress only fp.seek(0) for i in range(options.skip): wrappedNext(fp) # discard initial lines rdr = csv.reader(fp, dialect=options.csv_dialect, quotechar=options.csv_quotechar, delimiter=options.csv_delimiter, skipinitialspace=options.csv_skipinitialspace) vs.rows = [] # headers first, to setup columns before adding rows headers = [wrappedNext(rdr) for i in range(int(options.header))] if headers: # columns ideally reflect the max number of fields over all rows vs.columns = ArrayNamedColumns('\\n'.join(x) for x in zip(*headers)) else: r = wrappedNext(rdr) vs.addRow(wrappedNext(rdr)) vs.columns = ArrayColumns(len(vs.rows[0])) vs.recalc() # make columns usable with Progress(total=vs.source.filesize) as prog: try: while True: vs.addRow(wrappedNext(rdr)) prog.addProgress(samplelen) except StopIteration: pass vs.recalc() return vs def save_csv(sheet, fn): 'Save as single CSV file, handling column names as first line.' with open(fn, 'w', newline='', encoding=options.encoding, errors=options.encoding_errors) as fp: cw = csv.writer(fp, dialect=options.csv_dialect, delimiter=options.csv_delimiter, quotechar=options.csv_quotechar) colnames = [col.name for col in sheet.visibleCols] if ''.join(colnames): cw.writerow(colnames) for r in sheet.rows: cw.writerow([col.getDisplayValue(r) for col in sheet.visibleCols]) visidata-1.0/visidata/loaders/json.py0000660000175100017520000000312113223031561017574 0ustar anjaanja00000000000000import json from visidata import * def open_json(p): return JSONSheet(p.name, source=p, jsonlines=False) def open_jsonl(p): return JSONSheet(p.name, source=p, jsonlines=True) class JSONSheet(Sheet): commands = [Command(ENTER, 'pyobj-dive')] @async def reload(self): if not self.jsonlines: try: self.reload_json() except json.decoder.JSONDecodeError: status('trying jsonl') self.jsonlines = True if self.jsonlines: self.reload_jsonl() def reload_json(self): self.rows = [] with self.source.open_text() as fp: r = json.load(fp, object_hook=self.addRow) self.rows = [r] if isinstance(r, dict) else r self.columns = DictKeyColumns(self.rows[0]) self.recalc() def reload_jsonl(self): with self.source.open_text() as fp: self.rows = [] for L in fp: self.addRow(json.loads(L)) def addRow(self, row): if not self.rows: self.columns = DictKeyColumns(row) self.recalc() self.rows.append(row) return row @async def save_json(vs, fn): def rowdict(cols, row): d = {} for col in cols: try: d[col.name] = col.getValue(row) except Exception: pass return d with open(fn, 'w') as fp: vcols = vs.visibleCols for chunk in json.JSONEncoder().iterencode([rowdict(vcols, r) for r in Progress(vs.rows)]): fp.write(chunk) visidata-1.0/visidata/loaders/postgres.py0000660000175100017520000000604413226522612020505 0ustar anjaanja00000000000000from visidata import * def codeToType(type_code, colname): import psycopg2 try: tname = psycopg2._psycopg.string_types[type_code].name if 'INTEGER' in tname: return int if 'STRING' in tname: return str except KeyError: status('unknown postgres type_code %s for %s' % (type_code, colname)) return anytype def openurl_postgres(url, filetype=None): import psycopg2 dbname = url.path[1:] conn = psycopg2.connect( user=url.username, dbname=dbname, host=url.hostname, port=url.port, password=url.password) return PgTablesSheet(dbname+"_tables", sql=SQL(conn)) class SQL: def __init__(self, conn): self.conn = conn def cur(self, qstr): randomname = ''.join(random.choice(string.ascii_uppercase) for _ in range(6)) cur = self.conn.cursor(randomname) cur.execute(qstr) return cur @async def query_async(self, qstr, callback=None): with self.cur(qstr) as cur: callback(cur) cur.close() def cursorToColumns(cur): cols = [] for i, coldesc in enumerate(cur.description): c = ColumnItem(coldesc.name, i, type=codeToType(coldesc.type_code, coldesc.name)) cols.append(c) return cols # rowdef: (table_name, ncols) class PgTablesSheet(Sheet): rowtype = 'tables' nKeys = 1 commands = [ Command(ENTER, 'vd.push(PgTable(name+"."+cursorRow[0], source=cursorRow[0], sql=sql))', 'open this table'), ] def reload(self): qstr = "SELECT table_name, COUNT(column_name) AS ncols FROM information_schema.columns WHERE table_schema = 'public' GROUP BY table_name" with self.sql.cur(qstr) as cur: self.nrowsPerTable = {} self.rows = [] # try to get first row to make cur.description available r = cur.fetchone() if r: self.addRow(r) self.columns = cursorToColumns(cur) self.addColumn(Column('nrows', type=int, getter=lambda col,row: col.sheet.getRowCount(row[0]))) for r in cur: self.addRow(r) def setRowCount(self, cur): result = cur.fetchall() tablename = result[0][0] self.nrowsPerTable[tablename] = result[0][1] def getRowCount(self, tablename): if tablename not in self.nrowsPerTable: thread = self.sql.query_async("SELECT '%s', COUNT(*) FROM %s" % (tablename, tablename), callback=self.setRowCount) self.nrowsPerTable[tablename] = thread return self.nrowsPerTable[tablename] # rowdef: tuple of values as returned by fetchone() class PgTable(Sheet): @async def reload(self): with self.sql.cur("SELECT * FROM " + self.source) as cur: self.rows = [] r = cur.fetchone() if r: self.addRow(r) self.columns = cursorToColumns(cur) for r in cur: self.addRow(r) addGlobals(globals()) visidata-1.0/visidata/aggregators.py0000660000175100017520000000405613223031561017507 0ustar anjaanja00000000000000import collections from visidata import * globalCommand('+', 'addAggregator([cursorCol], chooseOne(aggregators))', 'add aggregator to current column') globalCommand('z+', 'status(chooseOne(aggregators)(cursorCol, selectedRows or rows))', 'display result of aggregator over values in selected rows for current column') aggregators = collections.OrderedDict() def aggregator(name, func, type=None): 'Define simple aggregator `name` that calls func(values)' def _func(col, rows): # wrap builtins so they can have a .type return func(col.getValues(rows)) _func.type = type _func.__name__ = name aggregators[name] = _func def fullAggregator(name, type, func): 'Define aggregator `name` that calls func(col, rows)' func.type=type func.__name__ = name aggregators[name] = func def mean(vals): vals = list(vals) if vals: return float(sum(vals))/len(vals) def median(values): L = sorted(values) return L[len(L)//2] aggregator('min', min) aggregator('max', max) aggregator('avg', mean, float) aggregator('mean', mean, float) aggregator('median', median) aggregator('sum', sum) aggregator('distinct', lambda values: len(set(values)), int) aggregator('count', lambda values: sum(1 for v in values), int) # returns keys of the row with the max value fullAggregator('keymax', anytype, lambda col, rows: col.sheet.rowkey(max(col.getValueRows(rows))[1])) ColumnsSheet.commands += [ Command('g+', 'addAggregator(selectedRows or source.nonKeyVisibleCols, chooseOne(aggregators))', 'add aggregator to selected source columns'), ] ColumnsSheet.columns += [ Column('aggregators', getter=lambda col,row: ' '.join(x.__name__ for x in getattr(row, 'aggregators', [])), setter=lambda col,row,val: setattr(row, 'aggregators', list(aggregators[k] for k in (val or '').split()))) ] def addAggregator(cols, aggr): for c in cols: if not hasattr(c, 'aggregators'): c.aggregators = [] if aggr not in c.aggregators: c.aggregators += [aggr] addGlobals(globals()) visidata-1.0/visidata/clipboard.py0000660000175100017520000000247413226522612017150 0ustar anjaanja00000000000000 from visidata import * # adds vd.clipvalue and vd.cliprows # vd.cliprows = [(source_sheet, source_row_idx, source_row)] globalCommand('y', 'vd.cliprows = [(sheet, cursorRowIndex, cursorRow)]', 'copy current row to clipboard') globalCommand('d', 'vd.cliprows = [(sheet, cursorRowIndex, rows.pop(cursorRowIndex))]', 'delete current row and move it to clipboard') globalCommand('p', 'rows[cursorRowIndex+1:cursorRowIndex+1] = list(deepcopy(r) for s,i,r in vd.cliprows)', 'paste clipboard rows after current row') globalCommand('P', 'rows[cursorRowIndex:cursorRowIndex] = list(deepcopy(r) for s,i,r in vd.cliprows)', 'paste clipboard rows after current row') globalCommand('gd', 'vd.cliprows = list((None, i, r) for i, r in enumerate(selectedRows)); deleteSelected()', 'delete all selected rows and move them to clipboard') globalCommand('gy', 'vd.cliprows = list((None, i, r) for i, r in enumerate(selectedRows))', 'copy all selected rows to clipboard') globalCommand('zy', 'vd.clipvalue = cursorDisplay', 'copy current cell to clipboard') globalCommand('gzp', 'cursorCol.setValues(selectedRows or rows, vd.clipvalue)', 'set contents of current column for selected rows to last clipboard value') globalCommand('zp', 'cursorCol.setValue(cursorRow, vd.clipvalue)', 'set contents of current column for current row to last clipboard value') visidata-1.0/visidata/describe.py0000660000175100017520000000713413226522612016767 0ustar anjaanja00000000000000from statistics import mode, median, mean, stdev from visidata import * globalCommand('I', 'vd.push(DescribeSheet(sheet.name+"_describe", source=sheet, sourceRows=selectedRows or rows))', 'open Describe Sheet') def isNumeric(col): return col.type in (int,float,currency,date) def isError(col, row): try: v = col.getValue(row) if v is not None: col.type(v) return False except Exception as e: return True def returnException(f, *args, **kwargs): try: return f(*args, **kwargs) except Exception as e: return e class DescribeColumn(Column): def __init__(self, name, **kwargs): super().__init__(name, getter=lambda col,srccol: col.sheet.describeData[srccol].get(col.name), **kwargs) # rowdef: Column from source sheet class DescribeSheet(Sheet): rowtype = 'columns' columns = [ Column('column', type=str, getter=lambda col,row: row.name), DescribeColumn('errors', type=len), DescribeColumn('nulls', type=len), DescribeColumn('distinct',type=len), DescribeColumn('mode', type=anytype), DescribeColumn('min', type=anytype), DescribeColumn('max', type=anytype), DescribeColumn('median', type=anytype), DescribeColumn('mean', type=float), DescribeColumn('stdev', type=float), ] commands = [ Command('zs', 'source.select(cursorValue)', 'select rows on source sheet which are being described in current cell'), Command('zu', 'source.unselect(cursorValue)', 'unselect rows on source sheet which are being described in current cell'), Command('z'+ENTER, 'vs=copy(source); vs.rows=cursorValue; vs.name+="_%s_%s"%(cursorRow.name,cursorCol.name); vd.push(vs)', 'open copy of source sheet with rows described in current cell'), Command(ENTER, 'vd.push(SheetFreqTable(source, cursorRow))', 'open a Frequency Table sheet grouped on column referenced in current row'), Command('!', 'source.toggleKeyColumn(source.columns.index(cursorRow))', 'pin current column on left as a key column on source sheet') ] colorizers = [ Colorizer('row', 7, lambda self,c,r,v: options.color_key_col if r in self.source.keyCols else None), ] @async def reload(self): self.rows = list(self.source.columns) # column deleting/reordering here does not affect actual columns self.describeData = { col: {} for col in self.source.columns } for srccol in Progress(self.source.columns): self.reloadColumn(srccol) @async def reloadColumn(self, srccol): d = self.describeData[srccol] isNull = isNullFunc() vals = list() d['errors'] = list() d['nulls'] = list() d['distinct'] = set() for sr in Progress(self.sourceRows): try: v = srccol.getValue(sr) if isNull(v): d['nulls'].append(sr) else: v = srccol.type(v) vals.append(v) d['distinct'].add(v) except Exception as e: d['errors'].append(sr) d['mode'] = self.calcStatistic(d, mode, vals) if isNumeric(srccol): for func in [min, max, median, mean, stdev]: d[func.__name__] = self.calcStatistic(d, func, vals) def calcStatistic(self, d, func, *args, **kwargs): r = returnException(func, *args, **kwargs) d[func.__name__] = r return r visidata-1.0/visidata/pivot.py0000660000175100017520000000727213232246431016352 0ustar anjaanja00000000000000from visidata import * globalCommand('W', 'vd.push(SheetPivot(sheet, [cursorCol]))', 'Pivot the current column into a new sheet') # rowdef: (tuple(keyvalues), dict(variable_value -> list(rows))) class SheetPivot(Sheet): 'Summarize key columns in pivot table and display as new sheet.' rowtype = 'aggregated rows' commands = [ Command('z'+ENTER, 'vs=copy(source); vs.name+="_%s"%cursorCol.aggvalue; vs.rows=cursorRow[1].get(cursorCol.aggvalue, []); vd.push(vs)', 'open sheet of source rows aggregated in current cell'), Command(ENTER, 'vs=copy(source); vs.name+="_%s"%"+".join(cursorRow[0]); vs.rows=sum(cursorRow[1].values(), []); vd.push(vs)', 'open sheet of source rows aggregated in current cell') ] def __init__(self, srcsheet, variableCols): super().__init__(srcsheet.name+'_pivot_'+''.join(c.name for c in variableCols), source=srcsheet) self.nonpivotKeyCols = [] self.variableCols = variableCols for colnum, col in enumerate(srcsheet.keyCols): if col not in variableCols: newcol = Column(col.name, getter=lambda col,row,colnum=colnum: row[0][colnum]) newcol.srccol = col self.nonpivotKeyCols.append(newcol) def reload(self): # two different threads for better interactive display self.reloadCols() self.reloadRows() @async def reloadCols(self): self.columns = copy(self.nonpivotKeyCols) self.nKeys = len(self.nonpivotKeyCols) aggcols = [(c, aggregator) for c in self.source.visibleCols for aggregator in getattr(c, 'aggregators', [])] if not aggcols: aggcols = [(c, aggregators["count"]) for c in self.variableCols] for col in self.variableCols: for aggcol, aggregator in aggcols: aggname = '%s_%s' % (aggcol.name, aggregator.__name__) if aggregator.__name__ != 'count': # already have count above c = Column('Total_' + aggname, type=aggregator.type or aggcol.type, getter=lambda col,row,aggcol=aggcol,agg=aggregator: agg(aggcol, sum(row[1].values(), []))) self.addColumn(c) allValues = set() for value in Progress(col.getValues(self.source.rows), total=len(self.source.rows)): if value not in allValues: allValues.add(value) c = Column(value+'_'+aggname, type=aggregator.type or aggcol.type, getter=lambda col,row,aggcol=aggcol,aggvalue=value,agg=aggregator: agg(aggcol, row[1].get(aggvalue, []))) c.aggvalue = value self.addColumn(c) c = Column('Total_count', type=int, getter=lambda col,row: len(sum(row[1].values(), []))) self.addColumn(c) @async def reloadRows(self): rowidx = {} self.rows = [] for r in Progress(self.source.rows): keys = tuple(keycol.srccol.getTypedValue(r) for keycol in self.nonpivotKeyCols) pivotrow = rowidx.get(keys) if pivotrow is None: pivotrow = (keys, {}) rowidx[keys] = pivotrow self.addRow(pivotrow) for col in self.variableCols: varval = col.getTypedValue(r) matchingRows = pivotrow[1].get(varval) if matchingRows is None: pivotrow[1][varval] = [r] else: matchingRows.append(r) visidata-1.0/visidata/tidydata.py0000660000175100017520000000213413226522612017005 0ustar anjaanja00000000000000from visidata import * globalCommand('M', 'vd.push(MeltedSheet(sheet))', 'open melted sheet (unpivot)') melt_var_colname = 'Variable' # column name to use for the melted variable name melt_value_colname = 'Value' # column name to use for the melted value # rowdef: (sourceRow, sourceCol) class MeltedSheet(Sheet): "Perform 'melt', the reverse of 'pivot', on input sheet." rowtype = 'melted values' def __init__(self, sheet): super().__init__(sheet.name + '_melted', source=sheet) @async def reload(self): sheet = self.source self.columns = [SubrowColumn(c, 0) for c in sheet.keyCols] self.nKeys = sheet.nKeys self.columns.extend([Column(melt_var_colname, getter=lambda col,row: row[1].name), Column(melt_value_colname, getter=lambda col,row: row[1].getValue(row[0]))]) colsToMelt = [copy(c) for c in sheet.nonKeyVisibleCols] self.rows = [] for r in Progress(self.source.rows): for c in colsToMelt: if c.getValue(r) is not None: self.addRow((r, c)) visidata-1.0/visidata/metasheets.py0000660000175100017520000002040413226522612017344 0ustar anjaanja00000000000000from visidata import * OptionsSheet.colorizers += [ Colorizer('cell', 9, lambda s,c,r,v: v if c.name in ['value', 'default'] and r[0].startswith('color_') else None) ] def combineColumns(cols): 'Return Column object formed by joining fields in given columns.' return Column("+".join(c.name for c in cols), getter=lambda col,row,cols=cols,ch=' ': ch.join(c.getDisplayValue(row) for c in cols)) def createJoinedSheet(sheets, jointype=''): if jointype == 'append': return SheetConcat('&'.join(vs.name for vs in sheets), sources=sheets) else: return SheetJoin('+'.join(vs.name for vs in sheets), sources=sheets, jointype=jointype) jointypes = ["inner", "outer", "full", "diff", "append"] SheetsSheet.commands += [ Command('&', 'vd.replace(createJoinedSheet(selectedRows, jointype=chooseOne(jointypes)))', 'merge the selected sheets with visible columns from all, keeping rows according to jointype'), Command('gC', 'vd.push(ColumnsSheet("all_columns", source=selectedRows or rows[1:]))', 'open Columns Sheet with all columns from selected sheets'), ] SheetsSheet.columns.insert(1, ColumnAttr('progressPct')) # used ColumnsSheet, affecting the 'row' (source column) columnCommands = [ Command('_', 'cursorRow.width = cursorRow.getMaxWidth(source.visibleRows)', 'adjust width of source column'), Command('-', 'cursorRow.width = 0', 'hide source column on source sheet'), Command('%', 'cursorRow.type = float', 'set type of source column to float'), Command('#', 'cursorRow.type = int', 'set type of source column to int'), Command('@', 'cursorRow.type = date', 'set type of source column to date'), Command('$', 'cursorRow.type = currency', 'set type of source column to currency'), Command('~', 'cursorRow.type = str', 'set type of current column to str'), Command('z~', 'cursorRow.type = anytype', 'set type of current column to anytype'), Command('g!', 'for c in selectedRows or [cursorRow]: source.toggleKeyColumn(source.columns.index(c))', 'pin selected columns on the left as key columns on source sheet'), Command('g-', 'for c in selectedRows or source.nonKeyVisibleCols: c.width = 0', 'hide selected source columns on source sheet'), Command('g_', 'for c in selectedRows or source.nonKeyVisibleCols: c.width = c.getMaxWidth(source.visibleRows)', 'adjust widths of selected source columns'), Command('g%', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = float', 'set type of selected source columns to float'), Command('g#', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = int', 'set type of selected source columns to int'), Command('g@', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = date', 'set type of selected source columns to date'), Command('g$', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = currency', 'set type of selected columns to currency'), Command('g~', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = str', 'set type of selected columns to str'), Command('gz~', 'for c in selectedRows or source.nonKeyVisibleCols: c.type = anytype', 'set type of selected columns to anytype'), ] ColumnsSheet.commands += columnCommands + [ Command('!', 'source.toggleKeyColumn(cursorRowIndex)', 'pin current column on the left as a key column on source sheet'), Command('&', 'rows.insert(cursorRowIndex, combineColumns(selectedRows))', 'add column from concatenating selected source columns'), ] DescribeSheet.commands += columnCommands ColumnsSheet.columns += [ ColumnAttr('expr'), ] #### slicing and dicing # rowdef: [(key, ...), sheet1_row, sheet2_row, ...] # if a sheet does not have this key, sheet#_row is None class SheetJoin(Sheet): 'Column-wise join/merge. `jointype` constructor arg should be one of jointypes.' @async def reload(self): sheets = self.sources # first item in joined row is the key tuple from the first sheet. # first columns are the key columns from the first sheet, using its row (0) self.columns = [] for i, c in enumerate(sheets[0].keyCols): self.addColumn(SubrowColumn(ColumnItem(c.name, i), 0)) self.nKeys = sheets[0].nKeys rowsBySheetKey = {} rowsByKey = {} with Progress(total=sum(len(vs.rows) for vs in sheets)*2) as prog: for vs in sheets: # tally rows by keys for each sheet rowsBySheetKey[vs] = collections.defaultdict(list) for r in vs.rows: prog.addProgress(1) key = tuple(c.getTypedValue(r) for c in vs.keyCols) rowsBySheetKey[vs][key].append(r) for sheetnum, vs in enumerate(sheets): # subsequent elements are the rows from each source, in order of the source sheets for c in vs.nonKeyVisibleCols: self.addColumn(SubrowColumn(c, sheetnum+1)) for r in vs.rows: prog.addProgress(1) key = tuple(c.getTypedValue(r) for c in vs.keyCols) if key not in rowsByKey: # gather for this key has not been done yet # multiplicative for non-unique keys rowsByKey[key] = [] for crow in itertools.product(*[rowsBySheetKey[vs2].get(key, [None]) for vs2 in sheets]): rowsByKey[key].append([key] + list(crow)) self.rows = [] with Progress(total=len(rowsByKey)) as prog: for k, combinedRows in rowsByKey.items(): prog.addProgress(1) if self.jointype == 'full': # keep all rows from all sheets for combinedRow in combinedRows: self.addRow(combinedRow) elif self.jointype == 'inner': # only rows with matching key on all sheets for combinedRow in combinedRows: if all(combinedRow): self.addRow(combinedRow) elif self.jointype == 'outer': # all rows from first sheet for combinedRow in combinedRows: if combinedRow[1]: self.addRow(combinedRow) elif self.jointype == 'diff': # only rows without matching key on all sheets for combinedRow in combinedRows: if not all(combinedRow): self.addRow(combinedRow) class ColumnConcat(Column): def __init__(self, name, colsBySheet, **kwargs): super().__init__(name, **kwargs) self.colsBySheet = colsBySheet def calcValue(self, row): srcSheet, srcRow = row srcCol = self.colsBySheet.get(srcSheet, None) if srcCol: return srcCol.calcValue(srcRow) def setValue(self, row, v): srcSheet, srcRow = row srcCol = self.colsBySheet.get(srcSheet, None) if srcCol: srcCol.setValue(srcRow, v) else: error('column not on source sheet') # rowdef: (Sheet, row) class SheetConcat(Sheet): 'combination of multiple sheets by row concatenation' def reload(self): self.rows = [] for sheet in self.sources: for r in sheet.rows: self.addRow((sheet, r)) self.columns = [] allColumns = {} for srcsheet in self.sources: for srccol in srcsheet.visibleCols: colsBySheet = allColumns.get(srccol.name, None) if colsBySheet is None: colsBySheet = {} # dict of [Sheet] -> Column allColumns[srccol.name] = colsBySheet if isinstance(srccol, ColumnExpr): combinedCol = copy(srccol) else: combinedCol = ColumnConcat(srccol.name, colsBySheet, type=srccol.type) self.addColumn(combinedCol) if srcsheet in colsBySheet: status('%s has multiple columns named "%s"' % (srcsheet.name, srccol.name)) colsBySheet[srcsheet] = srccol self.recalc() # to set .sheet on columns, needed if this reload becomes async visidata-1.0/MANIFEST.in0000660000175100017520000000005413223031561014554 0ustar anjaanja00000000000000include README.md include visidata/man/vd.1 visidata-1.0/visidata.egg-info/0000770000175100017520000000000013232247165016325 5ustar anjaanja00000000000000visidata-1.0/visidata.egg-info/requires.txt0000660000175100017520000000004713232247165020727 0ustar anjaanja00000000000000python-dateutil openpyxl xlrd requests visidata-1.0/visidata.egg-info/PKG-INFO0000660000175100017520000001345113232247165017427 0ustar anjaanja00000000000000Metadata-Version: 1.2 Name: visidata Version: 1.0 Summary: curses interface for exploring and arranging tabular data Home-page: http://visidata.org Author: Saul Pwanson Author-email: visidata@saul.pw License: GPLv3 Download-URL: https://github.com/saulpw/visidata/tarball/1.0 Description-Content-Type: UNKNOWN Description: # VisiData v1.0 [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) A terminal interface for exploring and arranging tabular data. ## Dependencies - Linux or OS/X - Python 3.4+ - python-dateutil - other modules may be required for opening particular data sources - see [requirements.txt](https://github.com/saulpw/visidata/blob/stable/requirements.txt) or the [supported sources](http://visidata.org/man/#loaders) in the vd manpage ### Install via pip3 Best installation method for users who wish to take advantage of VisiData in their own code, or integrate it into a Python3 virtual environment. To install VisiData, with loaders for the most common data file formats (including csv, tsv, fixed-width text, json, sqlite, http, html and xls): $ pip3 install visidata To install VisiData, plus external dependencies for all available loaders: $ pip3 install "visidata[full]" ### Install via brew Ideal for MacOS users who primarily want to engage with VisiData as an application. This is currently the most reliable way to install VisiData's manpage on MacOS. $ brew install saulpw/vd/visidata Further instructions available [here](https://github.com/saulpw/homebrew-vd). ### Install via apt Packaged for Linux users who do not wish to wrangle with PyPi or python3-pip. Currently, VisiData is undergoing review for integration into the main Debian repository. Until then it is available in our [Debian repo](https://github.com/saulpw/deb-vd). Grab our public key $ wget http://visidata.org/devotees.gpg.key $ apt-key add devotees.gpg.key Add our repository to apt's search list $ sudo apt-get install apt-transport-https $ sudo vim /etc/apt/sources.list deb[arch=amd64] https://raw.githubusercontent.com/saulpw/deb-vd/master sid main $ sudo apt-get update You can then install VisiData by typing: sudo apt-get install visidata Further instructions available [here](https://github.com/saulpw/deb-vd). ## Usage $ vd [] ... $ | vd [] VisiData supports tsv, csv, xlsx, hdf5, sqlite, json and more. Use `-f ` to force a particular filetype. (See the [list of supported sources](http://visidata.org/man#sources)). ## Documentation * Quick reference: `F1` (or `z?`) within `vd` will open the [man page](http://visidata.org/man), which has a list of all commands and options. * [visidata.org/docs](http://visidata.org/docs) has a complete list of links to all official documentation. ## Help and Support For additional information, see the [support page](http://visidata.org/support). ## vdtui The core `vdtui.py` can be used to quickly create efficient terminal workflows. These have been prototyped as proof of this concept: - [vgit](https://github.com/saulpw/vgit): a git interface - [vsh](http://github.com/saulpw/vsh): a collection of utilities like `vping` and `vtop`. - [vdgalcon](https://github.com/saulpw/vdgalcon): a port of the classic game [Galactic Conquest](https://www.galcon.com) Other workflows should also be created as separate apps using vdtui. These apps can be very small and provide a lot of functionality; for example, see the included [viewtsv](bin/viewtsv). ## License The innermost core file, `vdtui.py`, is a single-file stand-alone library that provides a solid framework for building text user interface apps. It is distributed under the MIT free software license, and freely available for inclusion in other projects. Other VisiData components, including the main `vd` application, addons, loaders, and other code in this repository, are available for use and distribution under GPLv3. ## Credits VisiData was created and developed by Saul Pwanson ``. Thanks to all the [contributors](CONTRIBUTING.md#contributors), and to those wonderful users who provide feedback, for making VisiData the awesome tool that it is. Keywords: console tabular data spreadsheet terminal viewer textpunkcurses csv hdf5 h5 xlsx excel tsv Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: Environment :: Console :: Curses Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Database :: Front-Ends Classifier: Topic :: Scientific/Engineering Classifier: Topic :: Office/Business :: Financial :: Spreadsheet Classifier: Topic :: Scientific/Engineering :: Visualization Classifier: Topic :: Utilities Requires-Python: >=3.4 visidata-1.0/visidata.egg-info/SOURCES.txt0000660000175100017520000000163713232247165020221 0ustar anjaanja00000000000000MANIFEST.in README.md setup.py bin/vd visidata/Path.py visidata/__init__.py visidata/aggregators.py visidata/async.py visidata/canvas.py visidata/clipboard.py visidata/cmdlog.py visidata/data.py visidata/describe.py visidata/freeze.py visidata/freqtbl.py visidata/graph.py visidata/metasheets.py visidata/pivot.py visidata/pyobj.py visidata/regex.py visidata/tidydata.py visidata/vdtui.py visidata/zscroll.py visidata.egg-info/PKG-INFO visidata.egg-info/SOURCES.txt visidata.egg-info/dependency_links.txt visidata.egg-info/requires.txt visidata.egg-info/top_level.txt visidata/loaders/__init__.py visidata/loaders/csv.py visidata/loaders/fixed_width.py visidata/loaders/hdf5.py visidata/loaders/html.py visidata/loaders/http.py visidata/loaders/json.py visidata/loaders/mbtiles.py visidata/loaders/postgres.py visidata/loaders/shp.py visidata/loaders/sqlite.py visidata/loaders/xlsx.py visidata/loaders/zip.py visidata/man/vd.1visidata-1.0/visidata.egg-info/top_level.txt0000660000175100017520000000001113232247165021050 0ustar anjaanja00000000000000visidata visidata-1.0/visidata.egg-info/dependency_links.txt0000660000175100017520000000000113232247165022374 0ustar anjaanja00000000000000 visidata-1.0/PKG-INFO0000660000175100017520000001345113232247165014131 0ustar anjaanja00000000000000Metadata-Version: 1.2 Name: visidata Version: 1.0 Summary: curses interface for exploring and arranging tabular data Home-page: http://visidata.org Author: Saul Pwanson Author-email: visidata@saul.pw License: GPLv3 Download-URL: https://github.com/saulpw/visidata/tarball/1.0 Description-Content-Type: UNKNOWN Description: # VisiData v1.0 [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) A terminal interface for exploring and arranging tabular data. ## Dependencies - Linux or OS/X - Python 3.4+ - python-dateutil - other modules may be required for opening particular data sources - see [requirements.txt](https://github.com/saulpw/visidata/blob/stable/requirements.txt) or the [supported sources](http://visidata.org/man/#loaders) in the vd manpage ### Install via pip3 Best installation method for users who wish to take advantage of VisiData in their own code, or integrate it into a Python3 virtual environment. To install VisiData, with loaders for the most common data file formats (including csv, tsv, fixed-width text, json, sqlite, http, html and xls): $ pip3 install visidata To install VisiData, plus external dependencies for all available loaders: $ pip3 install "visidata[full]" ### Install via brew Ideal for MacOS users who primarily want to engage with VisiData as an application. This is currently the most reliable way to install VisiData's manpage on MacOS. $ brew install saulpw/vd/visidata Further instructions available [here](https://github.com/saulpw/homebrew-vd). ### Install via apt Packaged for Linux users who do not wish to wrangle with PyPi or python3-pip. Currently, VisiData is undergoing review for integration into the main Debian repository. Until then it is available in our [Debian repo](https://github.com/saulpw/deb-vd). Grab our public key $ wget http://visidata.org/devotees.gpg.key $ apt-key add devotees.gpg.key Add our repository to apt's search list $ sudo apt-get install apt-transport-https $ sudo vim /etc/apt/sources.list deb[arch=amd64] https://raw.githubusercontent.com/saulpw/deb-vd/master sid main $ sudo apt-get update You can then install VisiData by typing: sudo apt-get install visidata Further instructions available [here](https://github.com/saulpw/deb-vd). ## Usage $ vd [] ... $ | vd [] VisiData supports tsv, csv, xlsx, hdf5, sqlite, json and more. Use `-f ` to force a particular filetype. (See the [list of supported sources](http://visidata.org/man#sources)). ## Documentation * Quick reference: `F1` (or `z?`) within `vd` will open the [man page](http://visidata.org/man), which has a list of all commands and options. * [visidata.org/docs](http://visidata.org/docs) has a complete list of links to all official documentation. ## Help and Support For additional information, see the [support page](http://visidata.org/support). ## vdtui The core `vdtui.py` can be used to quickly create efficient terminal workflows. These have been prototyped as proof of this concept: - [vgit](https://github.com/saulpw/vgit): a git interface - [vsh](http://github.com/saulpw/vsh): a collection of utilities like `vping` and `vtop`. - [vdgalcon](https://github.com/saulpw/vdgalcon): a port of the classic game [Galactic Conquest](https://www.galcon.com) Other workflows should also be created as separate apps using vdtui. These apps can be very small and provide a lot of functionality; for example, see the included [viewtsv](bin/viewtsv). ## License The innermost core file, `vdtui.py`, is a single-file stand-alone library that provides a solid framework for building text user interface apps. It is distributed under the MIT free software license, and freely available for inclusion in other projects. Other VisiData components, including the main `vd` application, addons, loaders, and other code in this repository, are available for use and distribution under GPLv3. ## Credits VisiData was created and developed by Saul Pwanson ``. Thanks to all the [contributors](CONTRIBUTING.md#contributors), and to those wonderful users who provide feedback, for making VisiData the awesome tool that it is. Keywords: console tabular data spreadsheet terminal viewer textpunkcurses csv hdf5 h5 xlsx excel tsv Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: Environment :: Console :: Curses Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Database :: Front-Ends Classifier: Topic :: Scientific/Engineering Classifier: Topic :: Office/Business :: Financial :: Spreadsheet Classifier: Topic :: Scientific/Engineering :: Visualization Classifier: Topic :: Utilities Requires-Python: >=3.4 visidata-1.0/setup.cfg0000660000175100017520000000004613232247165014651 0ustar anjaanja00000000000000[egg_info] tag_build = tag_date = 0 visidata-1.0/setup.py0000770000175100017520000000342213232246526014545 0ustar anjaanja00000000000000#!/usr/bin/env python3 from setuptools import setup # tox can't actually run python3 setup.py: https://github.com/tox-dev/tox/issues/96 #from visidata import __version__ __version__ = '1.0' setup(name='visidata', version=__version__, install_requires=['python-dateutil', 'openpyxl', 'xlrd', 'requests'], description='curses interface for exploring and arranging tabular data', long_description=open('README.md').read(), author='Saul Pwanson', python_requires='>=3.4', author_email='visidata@saul.pw', url='http://visidata.org', download_url='https://github.com/saulpw/visidata/tarball/' + __version__, scripts=['bin/vd'], py_modules = ['visidata'], packages=['visidata', 'visidata.loaders'], include_package_data=True, data_files = [('share/man/man1', ['visidata/man/vd.1'])], package_data={'': ['man/vd.1']}, license='GPLv3', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Console', 'Environment :: Console :: Curses', 'Intended Audience :: Developers', 'Intended Audience :: Science/Research', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: GNU General Public License v3 (GPLv3)', 'Operating System :: OS Independent', 'Programming Language :: Python :: 3', 'Topic :: Database :: Front-Ends', 'Topic :: Scientific/Engineering', 'Topic :: Office/Business :: Financial :: Spreadsheet', 'Topic :: Scientific/Engineering :: Visualization', 'Topic :: Utilities', ], keywords=('console tabular data spreadsheet terminal viewer textpunk' 'curses csv hdf5 h5 xlsx excel tsv'), ) visidata-1.0/bin/0000770000175100017520000000000013232247165013577 5ustar anjaanja00000000000000visidata-1.0/bin/vd0000770000175100017520000001315413232246526014142 0ustar anjaanja00000000000000#!/usr/bin/env python3 # # Usage: $0 [] [ ...] # $0 [] --play [--batch] [-w ] [-o ] [field=value ...] __version__ = 'saul.pw/VisiData v1.0' import os from visidata import * option('color_diff', 'red', 'color of values different from --diff source') option('color_diff_add', 'yellow', 'color of rows/columns added to --diff source') # for --play def eval_vd(logpath, *args, **kwargs): 'Instantiate logpath with args/kwargs replaced and replay all commands.' log = logpath.read_text().format(*args, **kwargs) src = PathFd(logpath.fqpn, io.StringIO(log), filesize=len(log)) vs = openSource(src, filetype='vd') vd().push(vs) vs.vd = vd() return vs # for --diff def makeDiffColorizer(othersheet): def colorizeDiffs(sheet, col, row, cellval): vcolidx = sheet.visibleCols.index(col) rowidx = sheet.rows.index(row) if vcolidx < len(othersheet.visibleCols) and rowidx < len(othersheet.rows): otherval = othersheet.visibleCols[vcolidx].getValue(othersheet.rows[rowidx]) if cellval.value != otherval: return options.color_diff else: return options.color_diff_add return colorizeDiffs def main(): 'Open the given sources using the VisiData interface.' import argparse parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('inputs', nargs='*', help='initial sources') parser.add_argument('-f', dest='filetype', default='', help='uses loader for filetype instead of file extension') parser.add_argument('-y', dest='confirm_overwrite', default=True, action='store_false', help='overwrites existing files without confirmation') parser.add_argument('-p', '--play', dest='play', default=None, help='replays a saved .vd file within the interface') parser.add_argument('-b', '--batch', dest='batch', action='store_true', default=False, help='replays in batch mode (with no interface and all status sent to stdout)') parser.add_argument('-o', '--output', dest='output', default=None, help='saves the final visible sheet to output (as .tsv) at the end of replay') parser.add_argument('-w', dest='replay_wait', default=0, help='time to wait between replayed commands, in seconds') parser.add_argument('-d', dest='delimiter', help='delimiter to use for tsv filetype') parser.add_argument('--diff', dest='diff', default=None, help='show diffs from all sheets against this source') parser.add_argument('-v', '--version', action='version', version=__version__) for optname, v in vdtui.baseOptions.items(): name, optval, defaultval, helpstr = v if name.startswith('color_') or name.startswith('disp_'): continue action = 'store_true' if optval is False else 'store' parser.add_argument('--' + optname.replace('_', '-'), action=action, dest=optname, default=None, help=helpstr) args = parser.parse_args() # user customisations in config file in standard location fnrc = '~/.visidatarc' p = Path(fnrc) if p.exists(): code = compile(open(p.resolve()).read(), p.resolve(), 'exec') exec(code, globals()) vdtui.addGlobals(globals()) # command-line overrides for optname, optval in vars(args).items(): if optval is not None and optname not in ['inputs', 'play', 'batch', 'output', 'diff']: vdtui.options[optname] = optval stdinSource = None if not sys.stdin.isatty(): # duplicate stdin for input and reopen tty as stdin stdinSource = PathFd('stdin', open(os.dup(0))) f = open('/dev/tty') os.dup2(f.fileno(), 0) startrow, startcol = None, None fmtargs = [] fmtkwargs = {} inputs = [] for arg in args.inputs: if arg.startswith('+'): # position cursor at start if ':' in arg: startrow, startcol = arg[1:].split(':') else: startrow = arg[1:] elif '=' in arg: # parse 'key=value' pairs for formatting cmdlog template in replay mode k, v = arg.split('=') fmtkwargs[k] = v elif arg == '-': inputs.append(stdinSource) else: inputs.append(arg) fmtargs.append(arg) if stdinSource and stdinSource not in inputs: # '|vd' without explicit '-' inputs.append(stdinSource) if args.diff: vs = openSource(args.diff) vd().push(vs) Sheet.colorizers.append(Colorizer("cell", 8, makeDiffColorizer(vs))) if not args.play: if not inputs: inputs = ['.'] sources = [] for src in inputs: vs = openSource(src) vdtui.vd().cmdlog.openHook(vs, src) sources.append(vs) if startrow is not None: vs.cursorRowIndex = int(startrow)-1 if startcol is not None: vs.cursorVisibleColIndex = int(startcol)-1 vdtui.run(*sources) return if args.play == '-': vdfile = stdinSource assert isinstance(vdfile, PathFd) vdfile.name = 'stdin.vd' else: vdfile = Path(args.play) vs = eval_vd(vdfile, *fmtargs, **fmtkwargs) if args.batch: vd().status = print vd().execAsync = lambda func, *args, **kwargs: func(*args, **kwargs) # disable async vs.replay_sync() if args.output: saveSheet(vd().sheets[0], args.output, confirm_overwrite=False) sync() else: vs.replay() run() if __name__ == '__main__': vdtui.status(__version__) main() os._exit(0) # cleanup can be expensive with large datasets