syslog-summary-1.14/0000775000175000017500000000000011254406525012604 5ustar neoneosyslog-summary-1.14/NEWS0000644000175000017500000000000011254406525013267 0ustar neoneosyslog-summary-1.14/README0000644000175000017500000000055211254406525013464 0ustar neoneosyslog-summary ============== This program summarizes the contents of a log file written by syslog, by displaying each unique (except for the time) line once, and also the number of times such a line occurs in the input. The lines are displayed in the order they occur in the input. It is also possible to define some "ignore rules" using regular expressions. syslog-summary-1.14/syslog-summary.10000644000175000017500000000731711254406525015707 0ustar neoneo.\" Title: SYSLOG-SUMMARY .\" Author: Lars Wirzenius .\" Generator: DocBook XSL Stylesheets v1.73.2 .\" Date: 02/07/2008 .\" Manual: Syslog-Summary User Manual .\" Source: syslog-summary .\" .TH "SYSLOG\-SUMMARY" "1" "02/07/2008" "syslog-summary" "Syslog-Summary User Manual" .\" disable hyphenation .nh .\" disable justification (adjust text to left margin only) .ad l .SH "NAME" syslog-summary - summarize the contents of a syslog log file .SH "SYNOPSIS" .HP 15 \fBsyslog\-summary\fR [\fB\-s\fR\ |\ \fB\-\-state=\fR\ \fIfilename\fR] [\fB\-i\fR\ |\ \fB\-\-ignore=\fR\ \fIfilename\fR] [\fB\-d\fR\ |\ \fB\-\-debug\fR] [\fB\-r\fR\ |\ \fB\-\-repeat\fR] \fIlogfile\fR... .HP 15 \fBsyslog\-summary\fR [\fB\-h\fR\ |\ \fB\-\-help\fR] .SH "DESCRIPTION" .PP This manual page documents briefly the \fBsyslog\-summary\fR command\. .PP \fBsyslog\-summary\fR summarizes the contents of log files via the \fBsyslog\fR(3) service, by displaying each unique (except for the time) line once, and also the number of times such a line occurs in the input\. The lines are displayed in the order they occur in the input\. .SH "OPTIONS" .PP \fB\-i \fR\fB\fIfilename\fR\fR, \fB\-\-ignore=\fR\fB\fIfilename\fR\fR .RS 4 Read regular expressions from \fIfilename\fR and ignore lines in the logfiles that match them\. .RE .PP \fB\-s \fR\fB\fIfilename\fR\fR, \fB\-\-state=\fR\fB\fIfilename\fR\fR .RS 4 Read state information from \fIfilename\fR\. The state contains information about the already reported parts of a log file, and prevents \fBsyslog\-summary\fR from reporting the same things many times\. This is useful when \fBsyslog\-summary\fR is run from \fBcrontab\fR every hour\. The file is created, if it doesn\'t exist already\. .RE .PP \fB\-r\fR, \fB\-\-repeat\fR .RS 4 Merge "last message repeated * times" lines with the repeated event\. .RE .PP \fB\-d\fR, \fB\-\-debug\fR .RS 4 Enable verbose messages when errors occur (i\.e\. "debug mode")\. .RE .PP \fB\-h\fR, \fB\-\-help\fR .RS 4 Show summary of options\. .RE .SH "FILES" .PP \fI/etc/syslog\-summary/ignore\.rules\fR .RS 4 Default ignore\.rules file with default regular expressions\. .RE .SH "DIAGNOSTICS" .PP When reporting a bug, please run \fBsyslog\-summary\fR with the \fB\-d\fR (or \fB\-\-debug\fR) flag enabled\. .PP \fBsyslog\-summary\fR provides some return codes, that can be used in scripts: .\" line length increase to cope w/ tbl weirdness .ll +(\n(LLu * 62u / 100u) .TS ll. \fICode\fR \fIDiagnostic\fR T{ \fB0\fR T} T{ Program exited successfully\. T} T{ \fB1\fR T} T{ Something went wrong, please run the program with the debug messages enabled\. T} .TE .\" line length decrease back to previous value .ll -(\n(LLu * 62u / 100u) .sp .SH "AUTHORS" .PP \fBLars Wirzenius\fR <\&liw@iki\.fi\&> .sp -1n .IP "" 4 Wrote this manpage for the Debian system\. .sp -1n .IP "" 4 Wrote the first version of syslog\-summary\. .sp -1n .IP "" 4 Maintained the Debian package from 1998 to 2000\. .PP \fBTommi Virtanen\fR <\&tv@debian\.org\&> .sp -1n .IP "" 4 Maintained the package from 2001 to early 2008\. .PP \fBDavid Paleino\fR <\&d\.paleino@gmail\.com\&> .sp -1n .IP "" 4 Maintains the package since early 2008\. .SH "COPYRIGHT" Copyright \(co 2008 David Paleino .br Copyright \(co 2001-2007 Tommi Virtanen .br Copyright \(co 1998-2000 Lars Wirzenius .br .PP This manual page was written for the Debian system (but may be used by others)\. .PP Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 3 or (at your option) any later version published by the Free Software Foundation\. .PP On Debian systems, the complete text of the GNU General Public License can be found in \fI/usr/share/common\-licenses/GPL\fR\. .sp syslog-summary-1.14/syslog-summary0000755000175000017500000002012711254406525015545 0ustar neoneo#!/usr/bin/env python2.5 # -*- coding: utf-8 -*- # Copyright © 2008-2009, David Paleino # © 2001-2008, Tommi Virtanen # © 1998-2000, Lars Wirzenius # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, # MA 02110-1301, USA. """Summarize the contents of a syslog log file. The syslog(3) service writes system log messages in a certain format: Jan 17 19:21:50 zeus kernel: klogd 1.3-3, log source = /proc/kmsg started. This program summarizes the contents of such a file, by displaying each unique (except for the time) line once, and also the number of times such a line occurs in the input. The lines are displayed in the order they occur in the input. Lars Wirzenius Tommi Virtanen David Paleino """ version = "1.14" import sys, re, getopt, string from gzip import open as gzopen from hashlib import sha1 from optparse import OptionParser datepats = [ re.compile(r"^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9] [ 0-9][0-9]:[0-9][0-9]:[0-9][0-9] "), re.compile(r"^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9][0-9][0-9]:[0-9][0-9] "), re.compile(r"^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9] "), ] pidpat = re.compile(r"^([^ ]* [^ ]*)\[[0-9][0-9]*\]: ") repeatpat = re.compile(r"^[^ ]* last message repeated (\d+) times$") ignore_pats = [] def io_error(err, filename, die=True): """Prints a nice error message, i.e. Tracebacks are ugly to end users""" import os, errno, traceback num = err.errno # DEBUG && die ensures that if it's a non-fatal exception, we don't # show all the traceback mess... if DEBUG: if die: traceback.print_exc(file=sys.stderr) else: print "[E] %s [%s(%s) - %s]" % (os.strerror(num), errno.errorcode[num], num, filename) if die: sys.exit(1) def read_patterns(filename): """Reads patterns to ignore from file specified by -i | --ignore=""" pats = [] try: f = open(filename, "r") except IOError, e: io_error(e, filename, False) return [] for line in f: rule = line.strip() if rule[0:1] == "#": continue else: pats.append(re.compile(rule)) f.close() return pats def read_states(filename): """Reads the previous state saved into the argument of -s | --state=""" states = {} if not filename: return states try: f = open(filename, "r") except IOError, e: io_error(e, filename, False) return states for line in f: fields = string.split(line) states[fields[0]] = (string.atoi(fields[1]), fields[2]) f.close() return states def save_states(filename, states): if not filename: return try: f = open(filename, "w") except IOError, e: io_error(e, filename, True) for filename in states.keys(): value = states[filename] f.write("%s %d %s\n" % (filename, value[0], value[1])) f.close() def should_be_ignored(line): for pat in ignore_pats: if pat.search(line): return 1 return 0 def split_date(line): for pat in datepats: m = pat.match(line) if m: return line[:m.end()], line[m.end():] print "line has bad date", "<" + string.rstrip(line) + ">" return None, line def is_gzipped(filename): """Returns True if the filename is a gzipped compressed file""" try: import magic ms = magic.open(magic.MAGIC_NONE) ms.load() if re.search("^gzip compressed data.*", ms.file(filename)): return True else: return False except: from os.path import splitext if not QUIET: print "Using fallback detection... please install python-magic for better gzip detection." if splitext(filename)[1] == ".gz": return True else: return False def summarize(filename, states): counts = {} order = [] ignored_count = 0 if not QUIET: print "Summarizing %s" % filename # If the file is a gzipped log, open it # using the proper function from the gzip # module. try: if is_gzipped(filename): file = gzopen(filename, "rb") else: file = open(filename, "r") except IOError, e: io_error(e, filename, True) linecount = 0 shaobj = sha1() if filename in states: oldlines, oldsha = states[filename] for i in xrange(oldlines): line = file.readline() shaobj.update(line) # print "OLD-new: %s" % shaobj.hexdigest() # print "OLD-file: %s" % oldsha if shaobj.hexdigest() != oldsha: #file.seek(0, 0) file.seek(0) shaobj = sha1() else: linecount = oldlines if not QUIET: print "%8d Lines skipped (already processed)" % linecount line = file.readline() previous = None # print "BEFORE-while: %s" % shaobj.hexdigest() foo=0 while line: # foo+=1 shaobj.update(line) linecount += 1 if should_be_ignored(line): ignored_count += 1 if DEBUG: print "Ignoring: %s" % line line = file.readline() date, rest = split_date(line) if date: found = pidpat.search(rest) if found: rest = found.group(1) + ": " + rest[found.end():] count = 1 repeated = None if REPEAT: repeated = repeatpat.search(rest) if repeated and previous: count = int(repeated.group(1)) rest = previous if counts.has_key(rest): counts[rest] = counts[rest] + count else: assert count == 1 counts[rest] = count order.append(rest) if not repeated: previous = rest line = file.readline() file.close() # print "TOT-lines: %d" % linecount # print "TOT-ignor: %d" % ignored_count # print "AFTER-while: %s" % shaobj.hexdigest() # print foo states[filename] = (linecount + ignored_count, shaobj.hexdigest()) # print states if QUIET and order: print "Summarizing %s" % filename if not QUIET or order: print "%8d Patterns to ignore" % len(ignore_pats) print "%8d Ignored lines" % ignored_count for rest in order: print "%8d %s" % (counts[rest], rest), if not QUIET or order: print def main(): global ignore_pats, IGNORE_FILENAME, STATE_FILENAME, REPEAT, QUIET, DEBUG parser = OptionParser(usage="%prog [options] [ ...]", version="%%prog %s" % version, description="Summarize the contents of a syslog log file") parser.add_option("-i", "--ignore", dest="ignorefile", default="/etc/syslog-summary/ignore.rules", help="read regular expressions from , and ignore lines in the that match them", metavar="") parser.add_option("-s", "--state", dest="statefile", help="read state information from (see the man page)", metavar="") parser.add_option("-r", "--repeat", action="store_true", dest="repeat", default=False, help="merge \"last message repeated x times\" with the event repeated") parser.add_option("-q", "--quiet", action="store_true", dest="quiet", default=False, help="don't output anything, unless there were unmatched lines") parser.add_option("-d", "--debug", action="store_true", dest="debug", default=False, help="shows additional messages in case of error") (options, args) = parser.parse_args() if len(sys.argv) == 1: parser.error("no logfile specified") IGNORE_FILENAME = options.ignorefile STATE_FILENAME = options.statefile REPEAT = options.repeat QUIET = options.quiet DEBUG = options.debug ignore_pats = read_patterns(IGNORE_FILENAME) states = read_states(STATE_FILENAME) for filename in args: summarize(filename, states) save_states(STATE_FILENAME, states) if __name__ == "__main__": main() syslog-summary-1.14/ignore.rules0000644000175000017500000000021511254406525015137 0ustar neoneo# # Use this file to define regular expressions to ignore. # #last message repeated [0-9]* times #kernel: Checking 'hlt' instruction... Ok. syslog-summary-1.14/AUTHORS0000644000175000017500000000032711254406525013654 0ustar neoneoFormer maintainers ------------------ 1998-2000, Lars Wirzenius 2001-2007, Tommi Virtanen Current maintainer ------------------ 2008-now, David Paleino syslog-summary-1.14/ChangeLog0000644000175000017500000000124711254406525014360 0ustar neoneosyslog-summary (1.14) * Added ability to read gzip compressed files, using python-magic or, if unavailable, os.path.splitext as fallback. * Added this ChangeLog file to track changes made to the sourcecode. * Moved regexp matching (ignore rules) earlier in the code (i.e. right after reading the line). * Using hashlib (Python 2.5) instead of the deprecated md5 module * printable_md5() removed: using hexdigest() method of the hashlib object. * Using SHA-1 instead of MD5 now. * Added rule to make a tarball to release :) * Code moved to GitHub, project management to Launchpad. -- David Paleino Thu, 17 Sep 2009 12:18:28 +0200 syslog-summary-1.14/Makefile0000644000175000017500000000166611254406525014253 0ustar neoneoXP = xsltproc --nonet \ --param man.charmap.use.subset "0" \ --param make.year.ranges "1" \ --param make.single.year.ranges "1" version = $(shell grep ^version syslog-summary | cut -d\" -f2) install: install -m 755 syslog-summary $(DESTDIR)/usr/bin/syslog-summary install -m 644 ignore.rules $(DESTDIR)/etc/syslog-summary/ignore.rules uninstall: [ ! -f $(DESTDIR)/usr/bin/syslog-summary ] || rm -v $(DESTDIR)/usr/bin/syslog-summary [ ! -d $(DESTDIR)/etc/syslog-summary ] || rm -vrf $(DESTDIR)/etc/syslog-summary/ syslog-summary.1: syslog-summary.1.xml $(XP) $< dist: clean mkdir syslog-summary-$(version)/ find . -maxdepth 1 -type f | xargs cp -t syslog-summary-$(version)/ @rm -rf syslog-summary-$(version)/.git/ tar zcf syslog-summary-$(version).tar.gz syslog-summary-$(version)/ rm -rf syslog-summary-$(version)/ clean: @find . -type d -name "syslog-summary-*" | xargs rm -rf @find . -type f -name "*.tar.gz" -delete syslog-summary-1.14/syslog-summary.1.xml0000644000175000017500000002036611254406525016505 0ustar neoneo ]> &dhtitle; &dhpackage; Lars Wirzenius Wrote this manpage for the Debian system. Wrote the first version of &dhpackage;. Maintained the Debian package from 1998 to 2000.
liw@iki.fi
Tommi Virtanen Maintained the package from 2001 to early 2008.
tv@debian.org
David Paleino Maintains the package since early 2008.
d.paleino@gmail.com
2008 David Paleino 2001 2002 2003 2004 2005 2006 2007 Tommi Virtanen 1998 1999 2000 Lars Wirzenius This manual page was written for the Debian system (but may be used by others). Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 3 or (at your option) any later version published by the Free Software Foundation. On Debian systems, the complete text of the GNU General Public License can be found in /usr/share/common-licenses/GPL. &dhucpackage; &dhsection; &dhpackage; summarize the contents of a syslog log file &dhpackage; filename filename logfile &dhpackage; DESCRIPTION This manual page documents briefly the &dhpackage; command. &dhpackage; summarizes the contents of log files via the syslog 3 service, by displaying each unique (except for the time) line once, and also the number of times such a line occurs in the input. The lines are displayed in the order they occur in the input. OPTIONS Read regular expressions from filename and ignore lines in the logfiles that match them. Read state information from filename. The state contains information about the already reported parts of a log file, and prevents syslog-summary from reporting the same things many times. This is useful when syslog-summary is run from crontab every hour. The file is created, if it doesn't exist already. Merge "last message repeated * times" lines with the repeated event. Enable verbose messages when errors occur (i.e. "debug mode"). Show summary of options. FILES /etc/syslog-summary/ignore.rules Default ignore.rules file with default regular expressions. DIAGNOSTICS When reporting a bug, please run syslog-summary with the (or ) flag enabled. &dhpackage; provides some return codes, that can be used in scripts: Code Diagnostic 0 Program exited successfully. 1 Something went wrong, please run the program with the debug messages enabled.