pflogsumm-1.1.5/0000750000175000017500000000000011713601765014045 5ustar jseymourjseymourpflogsumm-1.1.5/pflogsumm.pl0000755000175000017500000016460111713600725016426 0ustar jseymourjseymour#!/usr/bin/perl -w eval 'exec perl -S $0 "$@"' if 0; =head1 NAME pflogsumm.pl - Produce Postfix MTA logfile summary Copyright (C) 1998-2010 by James S. Seymour, Release 1.1.5 =head1 SYNOPSIS pflogsumm.pl -[eq] [-d ] [--detail ] [--bounce-detail ] [--deferral-detail ] [-h ] [-i|--ignore-case] [--iso-date-time] [--mailq] [-m|--uucp-mung] [--no-no-msg-size] [--problems-first] [--rej-add-from] [--reject-detail ] [--smtp-detail ] [--smtpd-stats] [--smtpd-warning-detail ] [--syslog-name=string] [-u ] [--verbose-msg-detail] [--verp-mung[=]] [--zero-fill] [file1 [filen]] pflogsumm.pl -[help|version] If no file(s) specified, reads from stdin. Output is to stdout. =head1 DESCRIPTION Pflogsumm is a log analyzer/summarizer for the Postfix MTA. It is designed to provide an over-view of Postfix activity, with just enough detail to give the administrator a "heads up" for potential trouble spots. Pflogsumm generates summaries and, in some cases, detailed reports of mail server traffic volumes, rejected and bounced email, and server warnings, errors and panics. =head1 OPTIONS --bounce-detail Limit detailed bounce reports to the top . 0 to suppress entirely. -d today generate report for just today -d yesterday generate report for just "yesterday" --deferral-detail Limit detailed deferral reports to the top . 0 to suppress entirely. --detail Sets all --*-detail, -h and -u to . Is over-ridden by individual settings. --detail 0 suppresses *all* detail. -e extended (extreme? excessive?) detail Emit detailed reports. At present, this includes only a per-message report, sorted by sender domain, then user-in-domain, then by queue i.d. WARNING: the data built to generate this report can quickly consume very large amounts of memory if a lot of log entries are processed! -h top to display in host/domain reports. 0 = none. See also: "-u" and "--*-detail" options for further report-limiting options. --help Emit short usage message and bail out. (By happy coincidence, "-h" alone does much the same, being as it requires a numeric argument :-). Yeah, I know: lame.) -i --ignore-case Handle complete email address in a case-insensitive manner. Normally pflogsumm lower-cases only the host and domain parts, leaving the user part alone. This option causes the entire email address to be lower- cased. --iso-date-time For summaries that contain date or time information, use ISO 8601 standard formats (CCYY-MM-DD and HH:MM), rather than "Mon DD CCYY" and "HHMM". -m modify (mung?) UUCP-style bang-paths --uucp-mung This is for use when you have a mix of Internet-style domain addresses and UUCP-style bang-paths in the log. Upstream UUCP feeds sometimes mung Internet domain style address into bang-paths. This option can sometimes undo the "damage". For example: "somehost.dom!username@foo" (where "foo" is the next host upstream and "somehost.dom" was whence the email originated) will get converted to "foo!username@somehost.dom". This also affects the extended detail report (-e), to help ensure that by- domain-by-name sorting is more accurate. --mailq Run "mailq" command at end of report. Merely a convenience feature. (Assumes that "mailq" is in $PATH. See "$mailqCmd" variable to path thisi if desired.) --no_bounce_detail --no_deferral_detail --no_reject_detail These switches are deprecated in favour of --bounce-detail, --deferral-detail and --reject-detail, respectively. Suppresses the printing of the following detailed reports, respectively: message bounce detail (by relay) message deferral detail message reject detail See also: "-u" and "-h" for further report-limiting options. --no-no-msg-size Do not emit report on "Messages with no size data". Message size is reported only by the queue manager. The message may be delivered long-enough after the (last) qmgr log entry that the information is not in the log(s) processed by a particular run of pflogsumm.pl. This throws off "Recipients by message size" and the total for "bytes delivered." These are normally reported by pflogsumm as "Messages with no size data." --no-smtpd-warnings This switch is deprecated in favour of smtpd-warning-detail On a busy mail server, say at an ISP, SMTPD warnings can result in a rather sizeable report. This option turns reporting them off. --problems-first Emit "problems" reports (bounces, defers, warnings, etc.) before "normal" stats. --rej-add-from For those reject reports that list IP addresses or host/domain names: append the email from address to each listing. (Does not apply to "Improper use of SMTP command pipelining" report.) -q quiet - don't print headings for empty reports note: headings for warning, fatal, and "master" messages will always be printed. --reject-detail Limit detailed smtpd reject, warn, hold and discard reports to the top . 0 to suppress entirely. --smtp-detail Limit detailed smtp delivery reports to the top . 0 to suppress entirely. --smtpd-stats Generate smtpd connection statistics. The "per-day" report is not generated for single-day reports. For multiple-day reports: "per-hour" numbers are daily averages (reflected in the report heading). --smtpd-warning-detail Limit detailed smtpd warnings reports to the top . 0 to suppress entirely. --syslog-name=name Set syslog-name to look for for Postfix log entries. By default, pflogsumm looks for entries in logfiles with a syslog name of "postfix," the default. If you've set a non-default "syslog_name" parameter in your Postfix configuration, use this option to tell pflogsumm what that is. See the discussion about the use of this option under "NOTES," below. -u top to display in user reports. 0 == none. See also: "-h" and "--*-detail" options for further report-limiting options. --verbose-msg-detail For the message deferral, bounce and reject summaries: display the full "reason", rather than a truncated one. Note: this can result in quite long lines in the report. --verp-mung do "VERP" generated address (?) munging. Convert --verp-mung=2 sender addresses of the form "list-return-NN-someuser=some.dom@host.sender.dom" to "list-return-ID-someuser=some.dom@host.sender.dom" In other words: replace the numeric value with "ID". By specifying the optional "=2" (second form), the munging is more "aggressive", converting the address to something like: "list-return@host.sender.dom" Actually: specifying anything less than 2 does the "simple" munging and anything greater than 1 results in the more "aggressive" hack being applied. See "NOTES" regarding this option. --version Print program name and version and bail out. --zero-fill "Zero-fill" certain arrays so reports come out with data in columns that that might otherwise be blank. =head1 RETURN VALUE Pflogsumm doesn't return anything of interest to the shell. =head1 ERRORS Error messages are emitted to stderr. =head1 EXAMPLES Produce a report of previous day's activities: pflogsumm.pl -d yesterday /var/log/maillog A report of prior week's activities (after logs rotated): pflogsumm.pl /var/log/maillog.0 What's happened so far today: pflogsumm.pl -d today /var/log/maillog Crontab entry to generate a report of the previous day's activity at 10 minutes after midnight. 10 0 * * * /usr/local/sbin/pflogsumm -d yesterday /var/log/maillog 2>&1 |/usr/bin/mailx -s "`uname -n` daily mail stats" postmaster Crontab entry to generate a report for the prior week's activity. (This example assumes one rotates ones mail logs weekly, some time before 4:10 a.m. on Sunday.) 10 4 * * 0 /usr/local/sbin/pflogsumm /var/log/maillog.0 2>&1 |/usr/bin/mailx -s "`uname -n` weekly mail stats" postmaster The two crontab examples, above, must actually be a single line each. They're broken-up into two-or-more lines due to page formatting issues. =head1 SEE ALSO The pflogsumm FAQ: pflogsumm-faq.txt. =head1 NOTES Pflogsumm makes no attempt to catch/parse non-Postfix log entries. Unless it has "postfix/" in the log entry, it will be ignored. It's important that the logs are presented to pflogsumm in chronological order so that message sizes are available when needed. For display purposes: integer values are munged into "kilo" and "mega" notation as they exceed certain values. I chose the admittedly arbitrary boundaries of 512k and 512m as the points at which to do this--my thinking being 512x was the largest number (of digits) that most folks can comfortably grok at-a-glance. These are "computer" "k" and "m", not 1000 and 1,000,000. You can easily change all of this with some constants near the beginning of the program. "Items-per-day" reports are not generated for single-day reports. For multiple-day reports: "Items-per-hour" numbers are daily averages (reflected in the report headings). Message rejects, reject warnings, holds and discards are all reported under the "rejects" column for the Per-Hour and Per-Day traffic summaries. Verp munging may not always result in correct address and address-count reduction. Verp munging is always in a state of experimentation. The use of this option may result in inaccurate statistics with regards to the "senders" count. UUCP-style bang-path handling needs more work. Particularly if Postfix is not being run with "swap_bangpath = yes" and/or *is* being run with "append_dot_mydomain = yes", the detailed by-message report may not be sorted correctly by-domain-by-user. (Also depends on upstream MTA, I suspect.) The "percent rejected" and "percent discarded" figures are only approximations. They are calculated as follows (example is for "percent rejected"): percent rejected = (rejected / (delivered + rejected + discarded)) * 100 There are some issues with the use of --syslog-name. The problem is that, even with Postfix' $syslog_name set, it will sometimes still log things with "postfix" as the syslog_name. This is noted in /etc/postfix/sample-misc.cf: # Beware: a non-default syslog_name setting takes effect only # after process initialization. Some initialization errors will be # logged with the default name, especially errors while parsing # the command line and errors while accessing the Postfix main.cf # configuration file. As a consequence, pflogsumm must always look for "postfix," in logs, as well as whatever is supplied for syslog_name. Where this becomes an issue is where people are running two or more instances of Postfix, logging to the same file. In such a case: . Neither instance may use the default "postfix" syslog name and... . Log entries that fall victim to what's described in sample-misc.cf will be reported under "postfix", so that if you're running pflogsumm twice, once for each syslog_name, such log entries will show up in each report. The Pflogsumm Home Page is at: http://jimsun.LinxNet.com/postfix_contrib.html =head1 REQUIREMENTS For certain options (e.g.: --smtpd-stats), Pflogsumm requires the Date::Calc module, which can be obtained from CPAN at http://www.perl.com. Pflogsumm is currently written and tested under Perl 5.8.3. As of version 19990413-02, pflogsumm worked with Perl 5.003, but future compatibility is not guaranteed. =head1 LICENSE This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You may have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. An on-line copy of the GNU General Public License can be found http://www.fsf.org/copyleft/gpl.html. =cut use strict; use locale; use Getopt::Long; eval { require Date::Calc }; my $hasDateCalc = $@ ? 0 : 1; my $mailqCmd = "mailq"; my $release = "1.1.5"; # Variables and constants used throughout pflogsumm use vars qw( $progName $usageMsg %opts $divByOneKAt $divByOneMegAt $oneK $oneMeg @monthNames %monthNums $thisYr $thisMon $msgCntI $msgSizeI $msgDfrsI $msgDlyAvgI $msgDlyMaxI $isoDateTime ); # Some constants used by display routines. I arbitrarily chose to # display in kilobytes and megabytes at the 512k and 512m boundaries, # respectively. Season to taste. $divByOneKAt = 524288; # 512k $divByOneMegAt = 536870912; # 512m $oneK = 1024; # 1k $oneMeg = 1048576; # 1m # Constants used throughout pflogsumm @monthNames = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec); %monthNums = qw( Jan 0 Feb 1 Mar 2 Apr 3 May 4 Jun 5 Jul 6 Aug 7 Sep 8 Oct 9 Nov 10 Dec 11); ($thisMon, $thisYr) = (localtime(time()))[4,5]; $thisYr += 1900; # # Variables used only in main loop # # Per-user data my (%recipUser, $recipUserCnt); my (%sendgUser, $sendgUserCnt); # Per-domain data my (%recipDom, $recipDomCnt); # recipient domain data my (%sendgDom, $sendgDomCnt); # sending domain data # Indexes for arrays in above $msgCntI = 0; # message count $msgSizeI = 1; # total messages size $msgDfrsI = 2; # number of defers $msgDlyAvgI = 3; # total of delays (used for averaging) $msgDlyMaxI = 4; # max delay my ( $cmd, $qid, $addr, $size, $relay, $status, $delay, $dateStr, $dateStrRFC3339, %panics, %fatals, %warnings, %masterMsgs, %msgSizes, %deferred, %bounced, %noMsgSize, %msgDetail, $msgsRcvd, $msgsDlvrd, $sizeRcvd, $sizeDlvrd, $msgMonStr, $msgMon, $msgDay, $msgTimeStr, $msgHr, $msgMin, $msgSec, $msgYr, $revMsgDateStr, $dayCnt, %msgsPerDay, %rejects, $msgsRjctd, %warns, $msgsWrnd, %discards, $msgsDscrdd, %holds, $msgsHld, %rcvdMsg, $msgsFwdd, $msgsBncd, $msgsDfrdCnt, $msgsDfrd, %msgDfrdFlgs, %connTime, %smtpdPerDay, %smtpdPerDom, $smtpdConnCnt, $smtpdTotTime, %smtpMsgs ); $dayCnt = $smtpdConnCnt = $smtpdTotTime = 0; # Init total messages delivered, rejected, and discarded $msgsDlvrd = $msgsRjctd = $msgsDscrdd = 0; # Init messages received and delivered per hour my @rcvPerHr = (0) x 24; my @dlvPerHr = @rcvPerHr; my @dfrPerHr = @rcvPerHr; # defers per hour my @bncPerHr = @rcvPerHr; # bounces per hour my @rejPerHr = @rcvPerHr; # rejects per hour my $lastMsgDay = 0; # Init "doubly-sub-scripted array": cnt, total and max time per-hour my @smtpdPerHr; for (0 .. 23) { $smtpdPerHr[$_] = [0,0,0]; } ($progName = $0) =~ s/^.*\///; $usageMsg = "usage: $progName -[eq] [-d ] [--detail ] [--bounce-detail ] [--deferral-detail ] [-h ] [-i|--ignore-case] [--iso-date-time] [--mailq] [-m|--uucp-mung] [--no-no-msg-size] [--problems-first] [--rej-add-from] [--reject-detail ] [--smtp-detail ] [--smtpd-stats] [--smtpd-warning-detail ] [--syslog-name=string] [-u ] [--verbose-msg-detail] [--verp-mung[=]] [--zero-fill] [file1 [filen]] $progName --[version|help]"; # Accept either "_"s or "-"s in --switches foreach (@ARGV) { last if($_ eq "--"); tr/_/-/ if(/^--\w/); } # Some pre-inits for convenience $isoDateTime = 0; # Don't use ISO date/time formats GetOptions( "bounce-detail=i" => \$opts{'bounceDetail'}, "d=s" => \$opts{'d'}, "deferral-detail=i" => \$opts{'deferralDetail'}, "detail=i" => \$opts{'detail'}, "e" => \$opts{'e'}, "help" => \$opts{'help'}, "h=i" => \$opts{'h'}, "ignore-case" => \$opts{'i'}, "i" => \$opts{'i'}, "iso-date-time" => \$isoDateTime, "mailq" => \$opts{'mailq'}, "m" => \$opts{'m'}, "no-bounce-detail" => \$opts{'noBounceDetail'}, "no-deferral-detail" => \$opts{'noDeferralDetail'}, "no-no-msg-size" => \$opts{'noNoMsgSize'}, "no-reject-detail" => \$opts{'noRejectDetail'}, "no-smtpd-warnings" => \$opts{'noSMTPDWarnings'}, "problems-first" => \$opts{'pf'}, "q" => \$opts{'q'}, "rej-add-from" => \$opts{'rejAddFrom'}, "reject-detail=i" => \$opts{'rejectDetail'}, "smtp-detail=i" => \$opts{'smtpDetail'}, "smtpd-stats" => \$opts{'smtpdStats'}, "smtpd-warning-detail=i" => \$opts{'smtpdWarnDetail'}, "syslog-name=s" => \$opts{'syslogName'}, "u=i" => \$opts{'u'}, "uucp-mung" => \$opts{'m'}, "verbose-msg-detail" => \$opts{'verbMsgDetail'}, "verp-mung:i" => \$opts{'verpMung'}, "version" => \$opts{'version'}, "zero-fill" => \$opts{'zeroFill'} ) || die "$usageMsg\n"; # internally: 0 == none, undefined == -1 == all $opts{'h'} = -1 unless(defined($opts{'h'})); $opts{'u'} = -1 unless(defined($opts{'u'})); $opts{'bounceDetail'} = -1 unless(defined($opts{'bounceDetail'})); $opts{'deferralDetail'} = -1 unless(defined($opts{'deferralDetail'})); $opts{'smtpDetail'} = -1 unless(defined($opts{'smtpDetail'})); $opts{'smtpdWarnDetail'} = -1 unless(defined($opts{'smtpdWarnDetail'})); $opts{'rejectDetail'} = -1 unless(defined($opts{'rejectDetail'})); # These go away eventually if(defined($opts{'noBounceDetail'})) { $opts{'bounceDetail'} = 0; warn "$progName: \"no_bounce_detail\" is deprecated, use \"bounce-detail=0\" instead\n" } if(defined($opts{'noDeferralDetail'})) { $opts{'deferralDetail'} = 0; warn "$progName: \"no_deferral_detail\" is deprecated, use \"deferral-detail=0\" instead\n" } if(defined($opts{'noRejectDetail'})) { $opts{'rejectDetail'} = 0; warn "$progName: \"no_reject_detail\" is deprecated, use \"reject-detail=0\" instead\n" } if(defined($opts{'noSMTPDWarnings'})) { $opts{'smtpdWarnDetail'} = 0; warn "$progName: \"no_smtpd_warnings\" is deprecated, use \"smtpd-warning-detail=0\" instead\n" } # If --detail was specified, set anything that's not enumerated to it if(defined($opts{'detail'})) { foreach my $optName (qw (h u bounceDetail deferralDetail smtpDetail smtpdWarnDetail rejectDetail)) { $opts{$optName} = $opts{'detail'} unless($opts{"$optName"} != -1); } } my $syslogName = $opts{'syslogName'}? $opts{'syslogName'} : "postfix"; if(defined($opts{'help'})) { print "$usageMsg\n"; exit 0; } if(defined($opts{'version'})) { print "$progName $release\n"; exit 0; } if($hasDateCalc) { # manually import the Date::Calc routine we want # # This looks stupid, but it's the only way to shut Perl up about # "Date::Calc::Delta_DHMS" used only once" if -w is on. (No, # $^W = 0 doesn't work in this context.) *Delta_DHMS = *Date::Calc::Delta_DHMS; *Delta_DHMS = *Date::Calc::Delta_DHMS; } elsif(defined($opts{'smtpdStats'})) { # If user specified --smtpd-stats but doesn't have Date::Calc # installed, die with friendly help message. die < unprocessed") || # die "couldn't open \"unprocessed\": $!\n"; while(<>) { next if(defined($dateStr) && ! (/^${dateStr} / || /^${dateStrRFC3339}T/)); s/: \[ID \d+ [^\]]+\] /: /; # lose "[ID nnnnnn some.thing]" stuff my $logRmdr; # "Traditional" timestamp format? if((($msgMonStr, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = /^(...) {1,2}(\d{1,2}) (\d{2}):(\d{2}):(\d{2}) \S+ (.+)$/) == 6) { # Convert string to numeric value for later "month rollover" check $msgMon = $monthNums{$msgMonStr}; } else { # RFC 3339 timestamp format? next unless((($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d+)?(?:[\+\-](?:\d{2}):(?:\d{2})|Z) \S+ (.+)$/) == 7); # RFC 3339 months start at "1", we index from 0 --$msgMon; } unless((($cmd, $qid) = $logRmdr =~ m#^(?:postfix|$syslogName)(?:/(?:smtps|submission))?/([^\[:]*).*?: ([^:\s]+)#o) == 2 || (($cmd, $qid) = $logRmdr =~ m#^((?:postfix)(?:-script)?)(?:\[\d+\])?: ([^:\s]+)#o) == 2) { #print UNPROCD "$_"; next; } chomp; # If the log line's month is greater than our current month, # we've probably had a year rollover # FIXME: For processing old logfiles: This is a broken test! $msgYr = ($msgMon > $thisMon? $thisYr - 1 : $thisYr); # the following test depends on one getting more than one message a # month--or at least that successive messages don't arrive on the # same month-day in successive months :-) unless($msgDay == $lastMsgDay) { $lastMsgDay = $msgDay; $revMsgDateStr = sprintf "%d%02d%02d", $msgYr, $msgMon, $msgDay; ++$dayCnt; if(defined($opts{'zeroFill'})) { ${$msgsPerDay{$revMsgDateStr}}[4] = 0; } } # regexp rejects happen in "cleanup" if($cmd eq "cleanup" && (my($rejSubTyp, $rejReas, $rejRmdr) = $logRmdr =~ /\/cleanup\[\d+\]: .*?\b(reject|warning|hold|discard): (header|body) (.*)$/) == 3) { $rejRmdr =~ s/( from \S+?)?; from=<.*$// unless($opts{'verbMsgDetail'}); $rejRmdr = string_trimmer($rejRmdr, 64, $opts{'verbMsgDetail'}); if($rejSubTyp eq "reject") { ++$rejects{$cmd}{$rejReas}{$rejRmdr} unless($opts{'rejectDetail'} == 0); ++$msgsRjctd; } elsif($rejSubTyp eq "warning") { ++$warns{$cmd}{$rejReas}{$rejRmdr} unless($opts{'rejectDetail'} == 0); ++$msgsWrnd; } elsif($rejSubTyp eq "hold") { ++$holds{$cmd}{$rejReas}{$rejRmdr} unless($opts{'rejectDetail'} == 0); ++$msgsHld; } elsif($rejSubTyp eq "discard") { ++$discards{$cmd}{$rejReas}{$rejRmdr} unless($opts{'rejectDetail'} == 0); ++$msgsDscrdd; } ++$rejPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[4]; } elsif($qid eq 'warning') { (my $warnReas = $logRmdr) =~ s/^.*warning: //; $warnReas = string_trimmer($warnReas, 66, $opts{'verbMsgDetail'}); unless($cmd eq "smtpd" && $opts{'noSMTPDWarnings'}) { ++$warnings{$cmd}{$warnReas}; } } elsif($qid eq 'fatal') { (my $fatalReas = $logRmdr) =~ s/^.*fatal: //; $fatalReas = string_trimmer($fatalReas, 66, $opts{'verbMsgDetail'}); ++$fatals{$cmd}{$fatalReas}; } elsif($qid eq 'panic') { (my $panicReas = $logRmdr) =~ s/^.*panic: //; $panicReas = string_trimmer($panicReas, 66, $opts{'verbMsgDetail'}); ++$panics{$cmd}{$panicReas}; } elsif($qid eq 'reject') { proc_smtpd_reject($logRmdr, \%rejects, \$msgsRjctd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($qid eq 'reject_warning') { proc_smtpd_reject($logRmdr, \%warns, \$msgsWrnd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($qid eq 'hold') { proc_smtpd_reject($logRmdr, \%holds, \$msgsHld, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($qid eq 'discard') { proc_smtpd_reject($logRmdr, \%discards, \$msgsDscrdd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($cmd eq 'master') { ++$masterMsgs{(split(/^.*master.*: /, $logRmdr))[1]}; } elsif($cmd eq 'smtpd') { if($logRmdr =~ /\[\d+\]: \w+: client=(.+?)(,|$)/) { # # Warning: this code in two places! # ++$rcvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[0]; ++$msgsRcvd; $rcvdMsg{$qid} = gimme_domain($1); # Whence it came # DEBUG DEBUG DEBUG #print STDERR "Received: $qid\n"; } elsif(my($rejSubTyp) = $logRmdr =~ /\[\d+\]: \w+: (reject(?:_warning)?|hold|discard): /) { if($rejSubTyp eq 'reject') { proc_smtpd_reject($logRmdr, \%rejects, \$msgsRjctd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($rejSubTyp eq 'reject_warning') { proc_smtpd_reject($logRmdr, \%warns, \$msgsWrnd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($rejSubTyp eq 'hold') { proc_smtpd_reject($logRmdr, \%holds, \$msgsHld, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($rejSubTyp eq 'discard') { proc_smtpd_reject($logRmdr, \%discards, \$msgsDscrdd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } } else { next unless(defined($opts{'smtpdStats'})); if($logRmdr =~ /: connect from /) { $logRmdr =~ /\/smtpd\[(\d+)\]: /; @{$connTime{$1}} = ($msgYr, $msgMon + 1, $msgDay, $msgHr, $msgMin, $msgSec); } elsif($logRmdr =~ /: disconnect from /) { my ($pid, $hostID) = $logRmdr =~ /\/smtpd\[(\d+)\]: disconnect from (.+)$/; if(exists($connTime{$pid})) { $hostID = gimme_domain($hostID); my($d, $h, $m, $s) = Delta_DHMS(@{$connTime{$pid}}, $msgYr, $msgMon + 1, $msgDay, $msgHr, $msgMin, $msgSec); delete($connTime{$pid}); # dispose of no-longer-needed item my $tSecs = (86400 * $d) + (3600 * $h) + (60 * $m) + $s; ++$smtpdPerHr[$msgHr][0]; $smtpdPerHr[$msgHr][1] += $tSecs; $smtpdPerHr[$msgHr][2] = $tSecs if($tSecs > $smtpdPerHr[$msgHr][2]); unless(${$smtpdPerDay{$revMsgDateStr}}[0]++) { ${$smtpdPerDay{$revMsgDateStr}}[1] = 0; ${$smtpdPerDay{$revMsgDateStr}}[2] = 0; } ${$smtpdPerDay{$revMsgDateStr}}[1] += $tSecs; ${$smtpdPerDay{$revMsgDateStr}}[2] = $tSecs if($tSecs > ${$smtpdPerDay{$revMsgDateStr}}[2]); unless(${$smtpdPerDom{$hostID}}[0]++) { ${$smtpdPerDom{$hostID}}[1] = 0; ${$smtpdPerDom{$hostID}}[2] = 0; } ${$smtpdPerDom{$hostID}}[1] += $tSecs; ${$smtpdPerDom{$hostID}}[2] = $tSecs if($tSecs > ${$smtpdPerDom{$hostID}}[2]); ++$smtpdConnCnt; $smtpdTotTime += $tSecs; } } } } else { my $toRmdr; if((($addr, $size) = $logRmdr =~ /from=<([^>]*)>, size=(\d+)/) == 2) { next if($msgSizes{$qid}); # avoid double-counting! if($addr) { if($opts{'m'} && $addr =~ /^(.*!)*([^!]+)!([^!@]+)@([^\.]+)$/) { $addr = "$4!" . ($1? "$1" : "") . $3 . "\@$2"; } $addr =~ s/(@.+)/\L$1/ unless($opts{'i'}); $addr = lc($addr) if($opts{'i'}); $addr = verp_mung($addr); } else { $addr = "from=<>" } $msgSizes{$qid} = $size; push(@{$msgDetail{$qid}}, $addr) if($opts{'e'}); # Avoid counting forwards if($rcvdMsg{$qid}) { # Get the domain out of the sender's address. If there is # none: Use the client hostname/IP-address my $domAddr; unless((($domAddr = $addr) =~ s/^[^@]+\@(.+)$/$1/) == 1) { $domAddr = $rcvdMsg{$qid} eq "pickup"? $addr : $rcvdMsg{$qid}; } ++$sendgDomCnt unless(${$sendgDom{$domAddr}}[$msgCntI]); ++${$sendgDom{$domAddr}}[$msgCntI]; ${$sendgDom{$domAddr}}[$msgSizeI] += $size; ++$sendgUserCnt unless(${$sendgUser{$addr}}[$msgCntI]); ++${$sendgUser{$addr}}[$msgCntI]; ${$sendgUser{$addr}}[$msgSizeI] += $size; $sizeRcvd += $size; delete($rcvdMsg{$qid}); # limit hash size } } elsif((($addr, $relay, $delay, $status, $toRmdr) = $logRmdr =~ /to=<([^>]*)>, (?:orig_to=<[^>]*>, )?relay=([^,]+), (?:conn_use=[^,]+, )?delay=([^,]+), (?:delays=[^,]+, )?(?:dsn=[^,]+, )?status=(\S+)(.*)$/) >= 4) { if($opts{'m'} && $addr =~ /^(.*!)*([^!]+)!([^!@]+)@([^\.]+)$/) { $addr = "$4!" . ($1? "$1" : "") . $3 . "\@$2"; } $addr =~ s/(@.+)/\L$1/ unless($opts{'i'}); $addr = lc($addr) if($opts{'i'}); $relay = lc($relay) if($opts{'i'}); (my $domAddr = $addr) =~ s/^[^@]+\@//; # get domain only if($status eq 'sent') { # was it actually forwarded, rather than delivered? if($toRmdr =~ /forwarded as /) { ++$msgsFwdd; next; } ++$recipDomCnt unless(${$recipDom{$domAddr}}[$msgCntI]); ++${$recipDom{$domAddr}}[$msgCntI]; ${$recipDom{$domAddr}}[$msgDlyAvgI] += $delay; if(! ${$recipDom{$domAddr}}[$msgDlyMaxI] || $delay > ${$recipDom{$domAddr}}[$msgDlyMaxI]) { ${$recipDom{$domAddr}}[$msgDlyMaxI] = $delay } ++$recipUserCnt unless(${$recipUser{$addr}}[$msgCntI]); ++${$recipUser{$addr}}[$msgCntI]; ++$dlvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[1]; ++$msgsDlvrd; # DEBUG DEBUG DEBUG #print STDERR "Delivered: $qid\n"; if($msgSizes{$qid}) { ${$recipDom{$domAddr}}[$msgSizeI] += $msgSizes{$qid}; ${$recipUser{$addr}}[$msgSizeI] += $msgSizes{$qid}; $sizeDlvrd += $msgSizes{$qid}; } else { ${$recipDom{$domAddr}}[$msgSizeI] += 0; ${$recipUser{$addr}}[$msgSizeI] += 0; $noMsgSize{$qid} = $addr unless($opts{'noNoMsgSize'}); push(@{$msgDetail{$qid}}, "(sender not in log)") if($opts{'e'}); # put this back later? mebbe with -v? # msg_warn("no message size for qid: $qid"); } push(@{$msgDetail{$qid}}, $addr) if($opts{'e'}); } elsif($status eq 'deferred') { unless($opts{'deferralDetail'} == 0) { my ($deferredReas) = $logRmdr =~ /, status=deferred \(([^\)]+)/; unless(defined($opts{'verbMsgDetail'})) { $deferredReas = said_string_trimmer($deferredReas, 65); $deferredReas =~ s/^\d{3} //; $deferredReas =~ s/^connect to //; } ++$deferred{$cmd}{$deferredReas}; } ++$dfrPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[2]; ++$msgsDfrdCnt; ++$msgsDfrd unless($msgDfrdFlgs{$qid}++); ++${$recipDom{$domAddr}}[$msgDfrsI]; if(! ${$recipDom{$domAddr}}[$msgDlyMaxI] || $delay > ${$recipDom{$domAddr}}[$msgDlyMaxI]) { ${$recipDom{$domAddr}}[$msgDlyMaxI] = $delay } } elsif($status eq 'bounced') { unless($opts{'bounceDetail'} == 0) { my ($bounceReas) = $logRmdr =~ /, status=bounced \((.+)\)/; unless(defined($opts{'verbMsgDetail'})) { $bounceReas = said_string_trimmer($bounceReas, 66); $bounceReas =~ s/^\d{3} //; } ++$bounced{$relay}{$bounceReas}; } ++$bncPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[3]; ++$msgsBncd; } else { # print UNPROCD "$_\n"; } } elsif($cmd eq 'pickup' && $logRmdr =~ /: (sender|uid)=/) { # # Warning: this code in two places! # ++$rcvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[0]; ++$msgsRcvd; $rcvdMsg{$qid} = "pickup"; # Whence it came } elsif($cmd eq 'smtp' && $opts{'smtpDetail'} != 0) { # Was an IPv6 problem here if($logRmdr =~ /.* connect to (\S+?): ([^;]+); address \S+ port.*$/) { ++$smtpMsgs{lc($2)}{$1}; } elsif($logRmdr =~ /.* connect to ([^[]+)\[\S+?\]: (.+?) \(port \d+\)$/) { ++$smtpMsgs{lc($2)}{$1}; } else { # print UNPROCD "$_\n"; } } else { # print UNPROCD "$_\n"; } } } # debugging #close(UNPROCD) || # die "problem closing \"unprocessed\": $!\n"; # Calculate percentage of messages rejected and discarded my $msgsRjctdPct = 0; my $msgsDscrddPct = 0; if(my $msgsTotal = $msgsDlvrd + $msgsRjctd + $msgsDscrdd) { $msgsRjctdPct = int(($msgsRjctd/$msgsTotal) * 100); $msgsDscrddPct = int(($msgsDscrdd/$msgsTotal) * 100); } if(defined($dateStr)) { print "Postfix log summaries for $dateStr\n"; } print_subsect_title("Grand Totals"); print "messages\n\n"; printf " %6d%s received\n", adj_int_units($msgsRcvd); printf " %6d%s delivered\n", adj_int_units($msgsDlvrd); printf " %6d%s forwarded\n", adj_int_units($msgsFwdd); printf " %6d%s deferred", adj_int_units($msgsDfrd); printf " (%d%s deferrals)", adj_int_units($msgsDfrdCnt) if($msgsDfrdCnt); print "\n"; printf " %6d%s bounced\n", adj_int_units($msgsBncd); printf " %6d%s rejected (%d%%)\n", adj_int_units($msgsRjctd), $msgsRjctdPct; printf " %6d%s reject warnings\n", adj_int_units($msgsWrnd); printf " %6d%s held\n", adj_int_units($msgsHld); printf " %6d%s discarded (%d%%)\n", adj_int_units($msgsDscrdd), $msgsDscrddPct; print "\n"; printf " %6d%s bytes received\n", adj_int_units($sizeRcvd); printf " %6d%s bytes delivered\n", adj_int_units($sizeDlvrd); printf " %6d%s senders\n", adj_int_units($sendgUserCnt); printf " %6d%s sending hosts/domains\n", adj_int_units($sendgDomCnt); printf " %6d%s recipients\n", adj_int_units($recipUserCnt); printf " %6d%s recipient hosts/domains\n", adj_int_units($recipDomCnt); if(defined($opts{'smtpdStats'})) { print "\nsmtpd\n\n"; printf " %6d%s connections\n", adj_int_units($smtpdConnCnt); printf " %6d%s hosts/domains\n", adj_int_units(int(keys %smtpdPerDom)); printf " %6d avg. connect time (seconds)\n", $smtpdConnCnt > 0? ($smtpdTotTime / $smtpdConnCnt) + .5 : 0; { my ($sec, $min, $hr) = get_smh($smtpdTotTime); printf " %2d:%02d:%02d total connect time\n", $hr, $min, $sec; } } print "\n"; print_problems_reports() if(defined($opts{'pf'})); print_per_day_summary(\%msgsPerDay) if($dayCnt > 1); print_per_hour_summary(\@rcvPerHr, \@dlvPerHr, \@dfrPerHr, \@bncPerHr, \@rejPerHr, $dayCnt); print_recip_domain_summary(\%recipDom, $opts{'h'}); print_sending_domain_summary(\%sendgDom, $opts{'h'}); if(defined($opts{'smtpdStats'})) { print_per_day_smtpd(\%smtpdPerDay, $dayCnt) if($dayCnt > 1); print_per_hour_smtpd(\@smtpdPerHr, $dayCnt); print_domain_smtpd_summary(\%smtpdPerDom, $opts{'h'}); } print_user_data(\%sendgUser, "Senders by message count", $msgCntI, $opts{'u'}, $opts{'q'}); print_user_data(\%recipUser, "Recipients by message count", $msgCntI, $opts{'u'}, $opts{'q'}); print_user_data(\%sendgUser, "Senders by message size", $msgSizeI, $opts{'u'}, $opts{'q'}); print_user_data(\%recipUser, "Recipients by message size", $msgSizeI, $opts{'u'}, $opts{'q'}); print_hash_by_key(\%noMsgSize, "Messages with no size data", 0, 1); print_problems_reports() unless(defined($opts{'pf'})); print_detailed_msg_data(\%msgDetail, "Message detail", $opts{'q'}) if($opts{'e'}); # Print "problems" reports sub print_problems_reports { unless($opts{'deferralDetail'} == 0) { print_nested_hash(\%deferred, "message deferral detail", $opts{'deferralDetail'}, $opts{'q'}); } unless($opts{'bounceDetail'} == 0) { print_nested_hash(\%bounced, "message bounce detail (by relay)", $opts{'bounceDetail'}, $opts{'q'}); } unless($opts{'rejectDetail'} == 0) { print_nested_hash(\%rejects, "message reject detail", $opts{'rejectDetail'}, $opts{'q'}); print_nested_hash(\%warns, "message reject warning detail", $opts{'rejectDetail'}, $opts{'q'}); print_nested_hash(\%holds, "message hold detail", $opts{'rejectDetail'}, $opts{'q'}); print_nested_hash(\%discards, "message discard detail", $opts{'rejectDetail'}, $opts{'q'}); } unless($opts{'smtpDetail'} == 0) { print_nested_hash(\%smtpMsgs, "smtp delivery failures", $opts{'smtpDetail'}, $opts{'q'}); } unless($opts{'smtpdWarnDetail'} == 0) { print_nested_hash(\%warnings, "Warnings", $opts{'smtpdWarnDetail'}, $opts{'q'}); } print_nested_hash(\%fatals, "Fatal Errors", 0, $opts{'q'}); print_nested_hash(\%panics, "Panics", 0, $opts{'q'}); print_hash_by_cnt_vals(\%masterMsgs,"Master daemon messages", 0, $opts{'q'}); } if($opts{'mailq'}) { # flush stdout first cuz of asynchronousity $| = 1; print_subsect_title("Current Mail Queue"); system($mailqCmd); } # print "per-day" traffic summary # (done in a subroutine only to keep main-line code clean) sub print_per_day_summary { my($msgsPerDay) = @_; my $value; print_subsect_title("Per-Day Traffic Summary"); print < $b } keys(%$msgsPerDay)) { my ($msgYr, $msgMon, $msgDay) = unpack("A4 A2 A2", $_); if($isoDateTime) { printf " %04d-%02d-%02d ", $msgYr, $msgMon + 1, $msgDay } else { my $msgMonStr = $monthNames[$msgMon]; printf " $msgMonStr %2d $msgYr", $msgDay; } foreach $value (@{$msgsPerDay->{$_}}) { my $value2 = $value? $value : 0; printf " %6d%s", adj_int_units($value2); } print "\n"; } } # print "per-hour" traffic summary # (done in a subroutine only to keep main-line code clean) sub print_per_hour_summary { my ($rcvPerHr, $dlvPerHr, $dfrPerHr, $bncPerHr, $rejPerHr, $dayCnt) = @_; my $reportType = $dayCnt > 1? 'Daily Average' : 'Summary'; my ($hour, $value); print_subsect_title("Per-Hour Traffic $reportType"); print < 0? "(top $cnt)" : ""; my $avgDly; print_subsect_title("Host/Domain Summary: Message Delivery $topCnt"); print <{$_}}[$msgCntI]) { $avgDly = (${$hashRef->{$_}}[$msgDlyAvgI] / ${$hashRef->{$_}}[$msgCntI]); } else { $avgDly = 0; } printf " %6d%s %6d%s %6d%s %5.1f %s %5.1f %s %s\n", adj_int_units(${$hashRef->{$_}}[$msgCntI]), adj_int_units(${$hashRef->{$_}}[$msgSizeI]), adj_int_units(${$hashRef->{$_}}[$msgDfrsI]), adj_time_units($avgDly), adj_time_units(${$hashRef->{$_}}[$msgDlyMaxI]), $_; last if --$cnt == 0; } } # print "per-sender-domain" traffic summary # (done in a subroutine only to keep main-line code clean) sub print_sending_domain_summary { use vars '$hashRef'; local($hashRef) = $_[0]; my($cnt) = $_[1]; return if($cnt == 0); my $topCnt = $cnt > 0? "(top $cnt)" : ""; print_subsect_title("Host/Domain Summary: Messages Received $topCnt"); print <{$_}}[$msgCntI]), adj_int_units(${$hashRef->{$_}}[$msgSizeI]), $_; last if --$cnt == 0; } } # print "per-user" data sorted in descending order # order (i.e.: highest first) sub print_user_data { my($hashRef, $title, $index, $cnt, $quiet) = @_; my $dottedLine; return if($cnt == 0); $title = sprintf "%s%s", $cnt > 0? "top $cnt " : "", $title; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; foreach (map { $_->[0] } sort { $b->[1] <=> $a->[1] || $a->[2] cmp $b->[2] } map { [ $_, $hashRef->{$_}[$index], normalize_host($_) ] } (keys(%$hashRef))) { printf " %6d%s %s\n", adj_int_units(${$hashRef->{$_}}[$index]), $_; last if --$cnt == 0; } } # print "per-hour" smtpd connection summary # (done in a subroutine only to keep main-line code clean) sub print_per_hour_smtpd { my ($smtpdPerHr, $dayCnt) = @_; my ($hour, $value); if($dayCnt > 1) { print_subsect_title("Per-Hour SMTPD Connection Daily Average"); print <[0] || next; my $avg = int($smtpdPerHr[$hour]->[0]? ($smtpdPerHr[$hour]->[1]/$smtpdPerHr[$hour]->[0]) + .5 : 0); if($dayCnt > 1) { $smtpdPerHr[$hour]->[0] /= $dayCnt; $smtpdPerHr[$hour]->[1] /= $dayCnt; $smtpdPerHr[$hour]->[0] += .5; $smtpdPerHr[$hour]->[1] += .5; } my($sec, $min, $hr) = get_smh($smtpdPerHr[$hour]->[1]); if($isoDateTime) { printf " %02d:00-%02d:00", $hour, $hour + 1; } else { printf " %02d00-%02d00 ", $hour, $hour + 1; } printf " %6d%s %2d:%02d:%02d", adj_int_units($smtpdPerHr[$hour]->[0]), $hr, $min, $sec; if($dayCnt < 2) { printf " %6ds %6ds", $avg, $smtpdPerHr[$hour]->[2]; } print "\n"; } } # print "per-day" smtpd connection summary # (done in a subroutine only to keep main-line code clean) sub print_per_day_smtpd { my ($smtpdPerDay, $dayCnt) = @_; print_subsect_title("Per-Day SMTPD Connection Summary"); print < $b } keys(%$smtpdPerDay)) { my ($msgYr, $msgMon, $msgDay) = unpack("A4 A2 A2", $_); if($isoDateTime) { printf " %04d-%02d-%02d ", $msgYr, $msgMon + 1, $msgDay } else { my $msgMonStr = $monthNames[$msgMon]; printf " $msgMonStr %2d $msgYr", $msgDay; } my $avg = (${$smtpdPerDay{$_}}[1]/${$smtpdPerDay{$_}}[0]) + .5; my($sec, $min, $hr) = get_smh(${$smtpdPerDay{$_}}[1]); printf " %6d%s %2d:%02d:%02d %6ds %6ds\n", adj_int_units(${$smtpdPerDay{$_}}[0]), $hr, $min, $sec, $avg, ${$smtpdPerDay{$_}}[2]; } } # print "per-domain-smtpd" connection summary # (done in a subroutine only to keep main-line code clean) sub print_domain_smtpd_summary { use vars '$hashRef'; local($hashRef) = $_[0]; my($cnt) = $_[1]; return if($cnt == 0); my $topCnt = $cnt > 0? "(top $cnt)" : ""; my $avgDly; print_subsect_title("Host/Domain Summary: SMTPD Connections $topCnt"); print <{$_}}[1]/${$hashRef->{$_}}[0]) + .5; my ($sec, $min, $hr) = get_smh(${$hashRef->{$_}}[1]); printf " %6d%s %2d:%02d:%02d %6ds %6ds %s\n", adj_int_units(${$hashRef->{$_}}[0]), $hr, $min, $sec, $avg, ${$hashRef->{$_}}[2], $_; last if --$cnt == 0; } } # print hash contents sorted by numeric values in descending # order (i.e.: highest first) sub print_hash_by_cnt_vals { my($hashRef, $title, $cnt, $quiet) = @_; my $dottedLine; $title = sprintf "%s%s", $cnt? "top $cnt " : "", $title; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; really_print_hash_by_cnt_vals($hashRef, $cnt, ' '); } # print hash contents sorted by key in ascending order sub print_hash_by_key { my($hashRef, $title, $cnt, $quiet) = @_; my $dottedLine; $title = sprintf "%s%s", $cnt? "first $cnt " : "", $title; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; foreach (sort keys(%$hashRef)) { printf " %s %s\n", $_, $hashRef->{$_}; last if --$cnt == 0; } } # print "nested" hashes sub print_nested_hash { my($hashRef, $title, $cnt, $quiet) = @_; my $dottedLine; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; walk_nested_hash($hashRef, $cnt, 0); } # "walk" a "nested" hash sub walk_nested_hash { my ($hashRef, $cnt, $level) = @_; $level += 2; my $indents = ' ' x $level; my ($keyName, $hashVal) = each(%$hashRef); if(ref($hashVal) eq 'HASH') { foreach (sort keys %$hashRef) { print "$indents$_"; # If the next hash is finally the data, total the # counts for the report and print my $hashVal2 = (each(%{$hashRef->{$_}}))[1]; keys(%{$hashRef->{$_}}); # "reset" hash iterator unless(ref($hashVal2) eq 'HASH') { print " (top $cnt)" if($cnt > 0); my $rptCnt = 0; $rptCnt += $_ foreach (values %{$hashRef->{$_}}); print " (total: $rptCnt)"; } print "\n"; walk_nested_hash($hashRef->{$_}, $cnt, $level); } } else { really_print_hash_by_cnt_vals($hashRef, $cnt, $indents); } } # print per-message info in excruciating detail :-) sub print_detailed_msg_data { use vars '$hashRef'; local($hashRef) = $_[0]; my($title, $quiet) = @_[1,2]; my $dottedLine; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; foreach (sort by_domain_then_user keys(%$hashRef)) { printf " %s %s\n", $_, shift(@{$hashRef->{$_}}); foreach (@{$hashRef->{$_}}) { print " $_\n"; } print "\n"; } } # *really* print hash contents sorted by numeric values in descending # order (i.e.: highest first), then by IP/addr, in ascending order. sub really_print_hash_by_cnt_vals { my($hashRef, $cnt, $indents) = @_; foreach (map { $_->[0] } sort { $b->[1] <=> $a->[1] || $a->[2] cmp $b->[2] } map { [ $_, $hashRef->{$_}, normalize_host($_) ] } (keys(%$hashRef))) { printf "$indents%6d%s %s\n", adj_int_units($hashRef->{$_}), $_; last if --$cnt == 0; } } # Print a sub-section title with properly-sized underline sub print_subsect_title { my $title = $_[0]; print "\n$title\n" . "-" x length($title) . "\n"; } # Normalize IP addr or hostname # (Note: Makes no effort to normalize IPv6 addrs. Just returns them # as they're passed-in.) sub normalize_host { # For IP addrs and hostnames: lop off possible " (user@dom.ain)" bit my $norm1 = (split(/\s/, $_[0]))[0]; if((my @octets = ($norm1 =~ /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/)) == 4) { # Dotted-quad IP address return(pack('U4', @octets)); } else { # Possibly hostname or user@dom.ain return(join( '', map { lc $_ } reverse split /[.@]/, $norm1 )); } } # subroutine to sort by domain, then user in domain, then by queue i.d. # Note: mixing Internet-style domain names and UUCP-style bang-paths # may confuse this thing. An attempt is made to use the first host # preceding the username in the bang-path as the "domain" if none is # found otherwise. sub by_domain_then_user { # first see if we can get "user@somedomain" my($userNameA, $domainA) = split(/\@/, ${$hashRef->{$a}}[0]); my($userNameB, $domainB) = split(/\@/, ${$hashRef->{$b}}[0]); # try "somedomain!user"? ($userNameA, $domainA) = (split(/!/, ${$hashRef->{$a}}[0]))[-1,-2] unless($domainA); ($userNameB, $domainB) = (split(/!/, ${$hashRef->{$b}}[0]))[-1,-2] unless($domainB); # now re-order "mach.host.dom"/"mach.host.do.co" to # "host.dom.mach"/"host.do.co.mach" $domainA =~ s/^(.*)\.([^\.]+)\.([^\.]{3}|[^\.]{2,3}\.[^\.]{2})$/$2.$3.$1/ if($domainA); $domainB =~ s/^(.*)\.([^\.]+)\.([^\.]{3}|[^\.]{2,3}\.[^\.]{2})$/$2.$3.$1/ if($domainB); # oddly enough, doing this here is marginally faster than doing # an "if-else", above. go figure. $domainA = "" unless($domainA); $domainB = "" unless($domainB); if($domainA lt $domainB) { return -1; } elsif($domainA gt $domainB) { return 1; } else { # disregard leading bang-path $userNameA =~ s/^.*!//; $userNameB =~ s/^.*!//; if($userNameA lt $userNameB) { return -1; } elsif($userNameA gt $userNameB) { return 1; } else { if($a lt $b) { return -1; } elsif($a gt $b) { return 1; } } } return 0; } # Subroutine used by host/domain reports to sort by count, then size. # We "fix" un-initialized values here as well. Very ugly and un- # structured to do this here - but it's either that or the callers # must run through the hashes twice :-(. sub by_count_then_size { ${$hashRef->{$a}}[$msgCntI] = 0 unless(${$hashRef->{$a}}[$msgCntI]); ${$hashRef->{$b}}[$msgCntI] = 0 unless(${$hashRef->{$b}}[$msgCntI]); if(${$hashRef->{$a}}[$msgCntI] == ${$hashRef->{$b}}[$msgCntI]) { ${$hashRef->{$a}}[$msgSizeI] = 0 unless(${$hashRef->{$a}}[$msgSizeI]); ${$hashRef->{$b}}[$msgSizeI] = 0 unless(${$hashRef->{$b}}[$msgSizeI]); return(${$hashRef->{$a}}[$msgSizeI] <=> ${$hashRef->{$b}}[$msgSizeI]); } else { return(${$hashRef->{$a}}[$msgCntI] <=> ${$hashRef->{$b}}[$msgCntI]); } } # return traditional and RFC3339 date strings to match in log sub get_datestrs { my ($dateOpt) = $_[0]; my $time = time(); if($dateOpt eq "yesterday") { # Back up to yesterday $time -= ((localtime($time))[2] + 2) * 3600; } elsif($dateOpt ne "today") { die "$usageMsg\n"; } my ($t_mday, $t_mon, $t_year) = (localtime($time))[3,4,5]; return sprintf("%s %2d", $monthNames[$t_mon], $t_mday), sprintf("%04d-%02d-%02d", $t_year+1900, $t_mon+1, $t_mday); } # if there's a real domain: uses that. Otherwise uses the IP addr. # Lower-cases returned domain name. # # Optional bit of code elides the last octet of an IPv4 address. # (In case one wants to assume an IPv4 addr. is a dialup or other # dynamic IP address in a /24.) # Does nothing interesting with IPv6 addresses. # FIXME: I think the IPv6 address parsing may be weak sub gimme_domain { $_ = $_[0]; my($domain, $ipAddr); # split domain/ipaddr into separates # newer versions of Postfix have them "dom.ain[i.p.add.ress]" # older versions of Postfix have them "dom.ain/i.p.add.ress" unless((($domain, $ipAddr) = /^([^\[]+)\[((?:\d{1,3}\.){3}\d{1,3})\]/) == 2 || (($domain, $ipAddr) = /^([^\/]+)\/([0-9a-f.:]+)/i) == 2) { # more exhaustive method ($domain, $ipAddr) = /^([^\[\(\/]+)[\[\(\/]([^\]\)]+)[\]\)]?:?\s*$/; } # "mach.host.dom"/"mach.host.do.co" to "host.dom"/"host.do.co" if($domain eq 'unknown') { $domain = $ipAddr; # For identifying the host part on a Class C network (commonly # seen with dial-ups) the following is handy. # $domain =~ s/\.\d+$//; } else { $domain =~ s/^(.*)\.([^\.]+)\.([^\.]{3}|[^\.]{2,3}\.[^\.]{2})$/\L$2.$3/; } return $domain; } # Return (value, units) for integer sub adj_int_units { my $value = $_[0]; my $units = ' '; $value = 0 unless($value); if($value > $divByOneMegAt) { $value /= $oneMeg; $units = 'm' } elsif($value > $divByOneKAt) { $value /= $oneK; $units = 'k' } return($value, $units); } # Return (value, units) for time sub adj_time_units { my $value = $_[0]; my $units = 's'; $value = 0 unless($value); if($value > 3600) { $value /= 3600; $units = 'h' } elsif($value > 60) { $value /= 60; $units = 'm' } return($value, $units); } # Trim a "said:" string, if necessary. Add elipses to show it. # FIXME: This sometimes elides The Wrong Bits, yielding # summaries that are less useful than they could be. sub said_string_trimmer { my($trimmedString, $maxLen) = @_; while(length($trimmedString) > $maxLen) { if($trimmedString =~ /^.* said: /) { $trimmedString =~ s/^.* said: //; } elsif($trimmedString =~ /^.*: */) { $trimmedString =~ s/^.*?: *//; } else { $trimmedString = substr($trimmedString, 0, $maxLen - 3) . "..."; last; } } return $trimmedString; } # Trim a string, if necessary. Add elipses to show it. sub string_trimmer { my($trimmedString, $maxLen, $doNotTrim) = @_; $trimmedString = substr($trimmedString, 0, $maxLen - 3) . "..." if(! $doNotTrim && (length($trimmedString) > $maxLen)); return $trimmedString; } # Get seconds, minutes and hours from seconds sub get_smh { my $sec = shift @_; my $hr = int($sec / 3600); $sec -= $hr * 3600; my $min = int($sec / 60); $sec -= $min * 60; return($sec, $min, $hr); } # Process smtpd rejects sub proc_smtpd_reject { my ($logLine, $rejects, $msgsRjctd, $rejPerHr, $msgsPerDay) = @_; my ($rejTyp, $rejFrom, $rejRmdr, $rejReas); my ($from, $to); my $rejAddFrom = 0; ++$$msgsRjctd; ++$$rejPerHr; ++$$msgsPerDay; # Hate the sub-calling overhead if we're not doing reject details # anyway, but this is the only place we can do this. return if($opts{'rejectDetail'} == 0); # This could get real ugly! # First: get everything following the "reject: ", etc. token # Was an IPv6 problem here ($rejTyp, $rejFrom, $rejRmdr) = ($logLine =~ /^.* \b(?:reject(?:_warning)?|hold|discard): (\S+) from (\S+?): (.*)$/); # Next: get the reject "reason" $rejReas = $rejRmdr; unless(defined($opts{'verbMsgDetail'})) { if($rejTyp eq "RCPT" || $rejTyp eq "DATA" || $rejTyp eq "CONNECT") { # special treatment :-( # If there are "<>"s immediately following the reject code, that's # an email address or HELO string. There can be *anything* in # those--incl. stuff that'll screw up subsequent parsing. So just # get rid of it right off. $rejReas =~ s/^(\d{3} <).*?(>:)/$1$2/; $rejReas =~ s/^(?:.*?[:;] )(?:\[[^\]]+\] )?([^;,]+)[;,].*$/$1/; $rejReas =~ s/^((?:Sender|Recipient) address rejected: [^:]+):.*$/$1/; $rejReas =~ s/(Client host|Sender address) .+? blocked/blocked/; } elsif($rejTyp eq "MAIL") { # *more* special treatment :-( grrrr... $rejReas =~ s/^\d{3} (?:<.+>: )?([^;:]+)[;:]?.*$/$1/; } else { $rejReas =~ s/^(?:.*[:;] )?([^,]+).*$/$1/; } } # Snag recipient address # Second expression is for unknown recipient--where there is no # "to=" field, third for pathological case where recipient # field is unterminated, forth when all else fails. (($to) = $rejRmdr =~ /to=<([^>]+)>/) || (($to) = $rejRmdr =~ /\d{3} <([^>]+)>: User unknown /) || (($to) = $rejRmdr =~ /to=<(.*?)(?:[, ]|$)/) || ($to = "<>"); $to = lc($to) if($opts{'i'}); # Snag sender address (($from) = $rejRmdr =~ /from=<([^>]+)>/) || ($from = "<>"); if(defined($from)) { $rejAddFrom = $opts{'rejAddFrom'}; $from = verp_mung($from); $from = lc($from) if($opts{'i'}); } # stash in "triple-subscripted-array" if($rejReas =~ m/^Sender address rejected:/) { # Sender address rejected: Domain not found # Sender address rejected: need fully-qualified address ++$rejects->{$rejTyp}{$rejReas}{$from}; } elsif($rejReas =~ m/^(Recipient address rejected:|User unknown( |$))/) { # Recipient address rejected: Domain not found # Recipient address rejected: need fully-qualified address # User unknown (in local/relay recipient table) #++$rejects->{$rejTyp}{$rejReas}{$to}; my $rejData = $to; if($rejAddFrom) { $rejData .= " (" . ($from? $from : gimme_domain($rejFrom)) . ")"; } ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } elsif($rejReas =~ s/^.*?\d{3} (Improper use of SMTP command pipelining);.*$/$1/) { # Was an IPv6 problem here my ($src) = $logLine =~ /^.+? from (\S+?):.*$/; ++$rejects->{$rejTyp}{$rejReas}{$src}; } elsif($rejReas =~ s/^.*?\d{3} (Message size exceeds fixed limit);.*$/$1/) { my $rejData = gimme_domain($rejFrom); $rejData .= " ($from)" if($rejAddFrom); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } elsif($rejReas =~ s/^.*?\d{3} (Server configuration (?:error|problem));.*$/(Local) $1/) { my $rejData = gimme_domain($rejFrom); $rejData .= " ($from)" if($rejAddFrom); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } else { # print STDERR "dbg: unknown reject reason $rejReas !\n\n"; my $rejData = gimme_domain($rejFrom); $rejData .= " ($from)" if($rejAddFrom); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } } # Hack for VERP (?) - convert address from somthing like # "list-return-36-someuser=someplace.com@lists.domain.com" # to "list-return-ID-someuser=someplace.com@lists.domain.com" # to prevent per-user listing "pollution." More aggressive # munging converts to something like # "list-return@lists.domain.com" (Instead of "return," there # may be numeric list name/id, "warn", "error", etc.?) sub verp_mung { my $addr = $_[0]; if(defined($opts{'verpMung'})) { $addr =~ s/((?:bounce[ds]?|no(?:list|reply|response)|return|sentto|\d+).*?)(?:[\+_\.\*-]\d+\b)+/$1-ID/i; if($opts{'verpMung'} > 1) { $addr =~ s/[\*-](\d+[\*-])?[^=\*-]+[=\*][^\@]+\@/\@/; } } return $addr; } ### ### Warning and Error Routines ### # Emit warning message to stderr sub msg_warn { warn "warning: $progName: $_[0]\n"; } pflogsumm-1.1.5/ToDo0000644000175000017500000000522011712255732014637 0ustar jseymourjseymour To Be Done (Maybe) Fix parsing for "451 4.3.5 Server configuration error;" Rename pflogsumm.pl -> pflogsumm. See http://docs.freebsd.org/info/cvs/cvs.info.Moving_files.html date ranges, "lastweek", etc.? (options for?) break-down by local vs. non-local?, further "drill-downs" to sender/recipient domains? Separate reports by-domain? (Would require that pflogsumm write files instead of emitting to stdout? Or maybe that would be an option?) Separate options for sender domain and receiver domain? Or perhaps sender address and receiver address regex "filtering" options? lower-case domains (Done) don't k-ize msg counts? (Or do so at a higher boundary?) (Done) implement proper version numbering (interim version numbering in place) (use SCCS or RCS?) (Done) add changelog (Done) Expand docs (in code?) Re-do docs to POD format? (Done) Improve UUCP-style bang-path handling Add option to use disk-based hash files instead of in-memory so processing logfiles with lots of postfix log entries doesn't run the machine out of memory? (This would really whack performance!) (Maybe only needed with "-e" option.) Add another "step" to integer processing: "g"? (Display formatting will presently mess up at values exceeding 999999m.) Internationalization? Necessary? There's English-language month abbreviations hard-coded in pflogsumm. I'm admittedly real weak in this area. (Covered by ISO 8601 option [below]?) Add option to lower-case entire addresses, rather than only the domains. (Or make that the default, and add an option to make it just domains?) (The RFCs say username parts shouldn't be touched!) (Done) Add percentage-of-total to all (?) reports? Add ability to handle compressed files within pflogsumm? (Unlikely) SMTP logging by host (msgs, connects) (add to new "by-domain" reports?) (Done) UUCP logging of some sort? (At least what is sent to what host/gateway. Since rmail receives incoming, don't know as I can do anything there.) Option for ISO 8601 standard date & time formats. (Done) Option for specifying "host" (for multi-host logfiles)? Add POSIX regexp body_checks "bypass" docs to FAQ. (Liviu Daia, Noel Jones) Add "helper" program to distribution based on Wolfgang Zeikat's MIME::Lite suggestion for the body_checks issue? (Will want to make it read from stdin, securely create tempfile and clean up afterward.) Expand on SAV reporting? (Len Conrad) Add "reject details" on "too many errors after " log lines? (Len Conrad) pflogsumm-1.1.5/README0000644000175000017500000000237211351216731014727 0ustar jseymourjseymour Pflogsumm README There's not much to installing pflogsumm, so it's all manual. 1. Unpack the distribution (if you're reading this, you've already gotten that far) 2. Copy or move pflogsumm.pl to some directory from which you'll want to execute it. Maybe rename it to just "pflogsumm." Watch the ownerships and permissions. Make sure it's executable. E.g.: cp pflogsumm.pl /usr/local/bin/pflogsumm chown bin:bin /usr/local/bin/pflogsumm chmod 755 /usr/local/bin/pflogsumm 3. If there's a manual page available (pflogsumm.1), copy that to /usr/local/man/man1 or wherever you stash local manpages. Make sure it's world-readable. E.g.: cp pflogsumm.1 /usr/local/man/man1/pflogsumm.1 chown bin:bin /usr/local/man/man1/pflogsumm.1 chmod 644 /usr/local/man/man1/pflogsumm.1 4. Read the man page (or the top of pflogsumm itself) for usage. 5. Check the FAQ (pflogsumm-faq.txt) 6. Configure your cron jobs if you're going to run pflogsumm on an automatic, scheduled basis. There are tips in the manpage and the FAQ. That's about it. As the manpage and FAQ both note: pflogsumm requires the Date::Calc Perl module if you want to use --smtpd_stats. pflogsumm-1.1.5/ChangeLog0000644000175000017500000006657211713601700015631 0ustar jseymourjseymourChangeLog for pflogsumm.pl [Note: Let me know if you would like to be notified as new versions are released. The latest released version can always be found at http://jimsun.LinxNet.com/postfix_contrib.html.] rel-1.1.5 20120205 Fixed RFC 3339 support. Releases 1.1.3 and 1.1.4 were badly broken in this respect. Thanks and a tip o' the hat to Sven Hoexter (sven-at-timegate-dot-de) for the help. rel-1.1.4 20120201 Modified for compatibility with -o syslog_name=blurfl/submission and -o syslog_name=blurfl/smtps set in master.cf. (These are the defaults in Postfix 2.9 and beyond.) N.B.: This doesn't mean you'll get submission and smtps broken-out separately from plain old smtp, it simply means the presence of the new sub-strings won't break Pflogsumm. Changed "_"s (underscores) in option switches to "-"s (dashes). (Underscores are still accepted.) Thanks and a tip o' the hat to David Landgren (david-at-landgren-dot-net) for the suggestion. Removed switches deprecated in 1.1.3 from the docs. Improved ISO timestamp parsing to account for optional fractional seconds part. (This is thrown-away by Pflogsumm.) Minor updates to the FAQ. Replaced "depreciated" with "deprecated" throughout. Thanks and a tip o' the hat to Rob Arista for the heads-up. Fixed bug in host normalization function that was broken for IPv4 addresses. rel-1.1.3 20100320 Added long-awaited switches to optionally reduce detail reporting: --bounce_detail=N, --deferral_detail=N, --reject_detail=N, --smtp_detail=N, smtpd_warning_detail=N, and --detail=N. Setting any of them to 0 suppresses that detail entirely. --detail=N sets the default for all of them, as well as for -u=N and -h=N. With the above enhancements, the following switches are deprecated, and will eventually be removed: --no_bounce_detail, --no_deferral_detail, --no_reject_detail and --no_smtpd_warnings. They are replaced by setting the desired --*_detail=0. They still work, but using them generates a warning. Added support for parsing logs with RFC 3339 timestamps. Thanks and a tip o' the hat to sftf-at-yandrex-dot-ru for the heads-up and the code contribution. (N.B.: My code does not require a command-line switch. The format is detected automatically.) Fixed some --ignore-case inconsistincies. Thanks and a tip o' the hat to Richard Blanchet (richard-dot-blanchet-at-free-dot-fr) for the heads-up and the diff. Fixed parsing bug that resulted in attempts to treat kind-of-IPv4-looking strings as IPv4 addresses. (I really need to improve reject/defer/etc. "reason" parsing to fix this properly.) Thanks to Joseph Vit (jvit-at-certicon-dot-cz) for the bug report. rel-1.1.2 20080629 Fixed bug with calculating yesterday's date in vicinity of DST changes. (Thanks and a tip o' the hat to Wieland Chmielewski for bringing the problem to my attention.) Added missing "underlining" to some (sub-)section titles for consistency. rel-1.1.1 20070406 Fixed to parse Postfix-2.3 (and beyond) logfiles. Thanks to whomever contributed to http://bugs.gentoo.org/show_bug.cgi?id=144236 Removed support for vmailer. Removed "SMTPD_STATS_SUPPORT" "fences" in code in favour of code to automatically detect the availability of Date::Calc. If --smtpd_stats is specified and Date::Calc is not installed, now bails-out with friendly message. (Adapted from suggestion and examples provided by David Landgren . Thanks!) Removed rem_smtpd_stats_supp.pl utility from distribution. (No longer needed.) Memory footprint improvement: Pflogsumm no longer stores data for reports that are supressed via --no_ switches. Removed extraneous arguments in two calls to print_nested_hash that would result in the "quiet" flag being ignored. Thanks to Pavel Urban (pupu-at-pupu-dot-cz) for bringing that to my attention. Added notes to FAQ about translations and i18n, about mismatching "received"/"delivered" counts, about bug in calculating "yesterday," and about John Fawcett's "prepflog." rel-1.1.0 20031212 Promoted 1.0.18 (Beta) to "production/stable" version release. rel-1.0.18 20031204 Fixed reject parsing for "DATA" smtpd rejects. rel-1.0.17 20031129 Fixed reject parsing to properly recognize bare "User unknown". (Thanks to J.D. Bronson" for the bug-report and sample logfile lines.) rel-1.0.16 20031128 Re-worked "to" and "from" field parsing in reject report handling to make it more robust in pathological cases. (Thanks to Paul Brooks and Lars Hecking for the bug-reports and sample logfile lines.) Fixed warnings resulting from non-standard, extraneous syslog input. (Thanks to Mathias Behrle for the report.) Fixed reject parsing to account for really atrocious garbage in HELO strings, sender addresses and recipient addresses. (Thanks to Lars Hecking for the bug-report and sample logfile lines.) Fixed reject parsing to properly recognize "CONNECT" smtpd rejects. (Thanks to Mike Vanecek for the bug-report.) Fixed reject parsing to properly recognize "User unknown in relay recipient table." (Thanks to Lars Hecking for the bug-report and sample logfile lines.) Some code optimization resulting in 3-5% performance improvement. rel-1.0.15 20030914 Pflogsumm *should* now properly parse and handle log entries with IPv6 addresses in them. (Adapted from idea and code submitted by Stefan `Sec` Zehl .) Fixed "User unknown in local recipient table" reject reports to show target recipient address, rather then sending domain, to be consistent with other "recipient" reports. (Thanks to WC Jones for the suggestion.) Fixed parsing of "Recipient address rejected" for recipient address verification (RAV) rejects. (Thanks to Len Conrad for the suggestion.) FAQ additions regarding recommendations on how to format custom reject reports for "best" results in Pflogsumm's output and note regarding "non-standard" syslogd's. rel-1.0.14 20030909 Fixed bug in parsing for "Host/Domain Summary: Messages Received" report improvement (rel-1.0.13) that resulted from (unexpected, to me) lines such as ... postfix/smtpd[31430]: E02DDB04E: client=blurfl[1.2.3.4], sasl_method=LOGIN, sasl_username=phred rel-1.0.13 20030907 The "Host/Domain Summary: Messages Received" report would show simply "from=<>", for the host/domain, for postmaster bounces. Pflogsumm now substitutes the client hostname or IP address for these, unless it's from the pickup daemon, in which case "from=<>" is retained. (Note that "Senders by message count/size" reports are unaffected by this change.) "Senders by message count" and "Recipients by message count" reports are now secondarily sorted by domain, host and user parts. (As a side-effect: So are "Senders by message size" and "Recipients by message size" but, being as the odds are against numerous senders and recipients having the same total message sizes, this change hasn't much effect with those.) rel-1.0.12 20030902 Rejects, warns, etc. now print sub-category totals. E.g.: message reject detail --------------------- RCPT Relay access denied (total: 6) (Adapted from idea and code submitted by blake7-at-blake7-dot-org.) Reject, warning, etc. reports are now sorted by 2nd column (e.g.: IP address, domain, etc.) within count. (Adapted from idea and code submitted by David Landgren .) Added --no_smtpd_warnings (report) option. Added --no_no_msg_size (report) option. A couple of minor improvements to reject parsing/reporting. rel-1.0.11 20030617 This is a bug-fix release. There was a problem in the way pflogsumm-1.0.8 through 1.0.10 handled the --syslog_name option: When --syslog_name was specified, some log entries with the default "postfix" name would be missed. This revision may introduce incompatibilities if you're logging two or more instances of Postfix to the same log. See the docs included in the tarball for details. rel-1.0.10 20030219 Re-worked "% rejected" calculation to include messages discarded and added "% discarded" calculation/display. rel-1.0.9 20030217 Bugfix: If Perl's -w is specified at run-time and there were no messages delivered or rejected, uninitialized variable warnings would be issued in the percent rejected calculation code. Thanks for Larry Hansford (and many others since!) for the bug report. rel-1.0.8 20030216 Bugfix: Fixed problem with "orig_to=" being parsed as "to=". This resulted in *very* wrong output. Thanks to Bjorn Swift for the report. Added "% rejected" to Grand Totals "rejected" figure. This is calculated as: rejected / (delivered + rejected). (I did this purely because it amuses me.) Bugfix: Fix, in reject processing, for truncated overly-long "to" fields. Thanks to Rick Troxel for reporting the problem. Added --syslog_name option. Thanks to Ben Rosengart for the suggestion. rel-1.0.7 20021231 Corrected and improved message reject/reject warn/hold/discard parsing. Again. (Thanks to Peter Santiago for reporting the problem that initiated these improvements.) rel-1.0.6 20021230 Added support for reporting message reject warnings, holds and discards. Note: Message rejects, reject warnings, holds and discards are all reported under the "rejects" column for the Per-Hour and Per-Day traffic summaries. More aggressive verp munging (again). (Prompted, in part, by a suggestion from Casey Peel. Thanks!) Verp munging now applied to sender addresses in smtpd reject reports. WARNING: Please note that verp munging is highly experimental! Pflogsumm distribution changed to gzip'd tarball format. Tightened-up parsing. Thanks for Ralf Hildebrandt for noting and reporting the problem. Docs at the top of pflogsumm.pl changed to POD format for automated manpage generation. README added. Automatically-generated manpage added. "To Do" moved out of ChangeLog into separate file. Package now includes convenience Perl script for removing smtpd stats support for those who don't have Date::Calc, don't want to install it and don't care about smtpd stats reporting. Belated thanks to Len Conrad in regards to the Sender Address Verification work in 1.0.5. rel-1.0.5 20021222 Fixed to parse smtpd rejects for Postfix versions as of 20021026. (Retained compatibility with older versions of Postfix.) Note: smtpd and header-/body-checks warn, hold and discard messages are *not* currently parsed/reported. I'll need to get some logfile entries. Fixed parsing to handle the new "sender address verification" lines. Added "--zero_fill" option to put zeros in columns that might otherwise be blank in some reports. (Suggestion by Matthias Andree). Fixed "Message size exceeds fixed limit" parsing for reject reporting. rel-1.0.4 20020224 Added "--no_*_detail" options. (Suppresses some of the "detail" reports.) Added "--version" option. (Thanks to "Guillaume") Improved handling of "[ID nnnnnn some.thing]" stuff (Thanks to Blake Dunmire) Repaired and optimized some of the "reject" parsing. Added processing and report of smtp delivery failures. Added --rej_add_from option: For those reject reports that list IP addresses or host/domain names: append the email from address to each listing. (Note: does not apply to "Improper use of SMTP command pipelining" report.) rel-1.0.3 20010520 Minor re-work of "reject: RCPT" parsing to account for Yet Another Change in the logfile format. (Semi-colon changed to a comma in "blocked using rbl.maps.vix.com,".) rel-1.0.2 20010519 Took another whack at "verp" munging. *sigh* Added code to summarize "Improper use of SMTP command pipelining" rejects by client. rel-1.0.1 20010518 Modified to catch "reject: header" log entries changed as of postfix release-20010228 (?). Prior versions of postfix had the string "warning: " (where the qid normally is). Thanks to Glen Eustace , Len Conrad , Daniel Roesen , Milivoj Ivkovic and j_zuilkowski@hotmail.com (Jon Zuilkowski) for reports and/or patches. Fixed a couple of "uninitialized variable" problems. Committed (actually starting with 20000925-01beta) to CVS. 20000925-01 Added a line to compensate for (new?) "[ID nnnnnn some.thing]" sub-strings that appear in logfile entries under Sun Solaris 8. (At least. Others?) Note: Upon being committed to CVS, this became rel-0.9.0. 20000916-01 Forgot to add "--problems_first" to the "usage" output and in the synopsis at the top of the comments. 20000908-01beta Re-did what 20000907-02beta was *supposed* to be! To wit: replaced missing "--ignore_case" bugfix, "panic" entry processing, improvements to "fatal" and "warning" message reporting and missing "--mailq" option. (Obviously: 20000907-02beta was derived from the wrong code base.) 20000907-02beta Fixed bug in ISO date formatting that caused the month to be off by one. Thanks to Kurt Andersen for the report and the patch. Fixed overflow of connect time reporting into days. (Can happen during weekly summaries for sites with large volumes of email.) Thanks again to Kurt Andersen for the report and the fix. Improved "rejects" reporting *again*. Thanks to Thomas Parmelan for the patch. Added "--problems_first" option to print "problem" reports such as bounces, defers, warnings, fatal errors, etc. before "normal" reports. 19991219-02 Fixed bug in code that prevented "--ignore_case" from actually doing anything. Thanks to Nadeem Hasan for reporting this and supplying the fix. 19991219-01beta Added the following caveat to the "Notes" section of Pflogsumm: ------------------------------------------------------------- IMPORTANT: Pflogsumm makes no attempt to catch/parse non- postfix/vmailer daemon log entries. (I.e.: Unless it has "postfix/" or "vmailer/" in the log entry, it will be ignored.) ------------------------------------------------------------- Added reporting of "panic" log messages. This was missed until now! Increased reporting detail of "fatal" and "warning" entries. (Actually, "warning" detail was increased in 19991120-01beta. Neglected to note it then.) 19991123-01 (unreleased) Added "--mailq" option. (Convenience factor.) Runs Postfix's "mailq" command at the end of the other reports. ------------------------------------------------------- NOTE: If Postfix's "mailq" command isn't in your $PATH, you'll have to edit the "$mailqCmd" variable located near the top of pflogsumm to path it explicitly. ------------------------------------------------------- 19991120-01 Tried once again to improve parsing of reject log entries. Specifically: those associated with "RCPT" rejects. 19991016-01 (not generally released) Added --smtpd_stats. Generates smtpd connection statistics. --------------------------------------------------------------- NOTE: Support for --smtpd_stats requires the Date::Calc module (available from CPAN). If you don't want to go to the trouble of fetching & installing that module, and you don't want smtpd stats anyway, *carefully* identify all of the code sections delimited by "# ---Begin: SMTPD_STATS_SUPPORT---" and "# ---End: SMTPD_STATS_SUPPORT---" and remove them. --------------------------------------------------------------- 19990909-01 (not generally released) Added -i and --ignore_case options. Causes entire email address to be lower-cased instead of just the host/domain part. Added "use locale". (This means that the sorting order within reports may be different from before--depending on how you have your machine's locale set.) 19990904-03 Improved "reason" parsing and reporting for bounced and deferred messages. Added parsing of "cleanup" reject lines to catch PCRE/regexp rejects. Added "reject" stats to per-hour and (on multi-day reports) per- day reports. Improved "warnings" report to show details. A single message deferred multiple times showed up as multiple deferrals--implying that multiple messages were deferred. Now shows "how many messages were deferred" and "how many deferrals" as separate stats. Changed display of "Grand Totals" to make it a bit more readable (IMO). Added "automatic perl finder" line for those systems that don't support the "#!" notation. By popular demand: added note to comments as to where pflogsumm home page could be found :-). 19990413-02 Fixed problem with last octet of IP address getting truncated in reports when IP address used in place of unknown hosts. Changed the way a few internal variables were handled to be compatible with Perl 5.003. Don't run it under Perl 5.003 with the "-w" perl switch, tho! It will issue lots of warnings. All tests I performed indicated that it produces the correct output, however. ------------------------------------------------------------ NOTE: While this version was tested to work with Perl 5.003, I recommend that you upgrade to 5.004 or later. I will not guarantee that I'll remember to do the full regression- testing that I usually do with 5.003 as well. ------------------------------------------------------------ 19990411-01 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is GONE! (As per the previous notice.) Added "--help" option to emit short usage message and bail out. Added "--iso_date_time" switch to change displays of dates and times to ISO 8601 standard formats (CCYY-MM-DD and HH:MM), rather than "Month-name Day-number CCYY" and "HHMM" formats (the default). Added "--verbose_msg_detail" switch. This causes the full "reason" to be displayed for the message deferral, bounce and reject summaries. (Note: this can result in quite long lines in the report. Also note that there have been a couple of subtle changes in the "reason" parsing/reporting in the default mode (no "--verbose_msg_detail".) Added "--verp_mung" option. The problem this addresses is "VERP" generated (??? so far as I can tell!) addresses (?) of the form: "list-return-NN-someuser=some.dom@host.sender.dom" These result in mail from the same "user" and site to look like it originated from different users, when in fact it originates from the same "user." There are presently two "levels" of address munging available. With no numeric argument (or any value less than 2), the above address will be converted to: "list-return-ID-someuser=some.dom@host.sender.dom" In other words: the numeric value will be replaced with "ID". By specifying "--verp_mung=2", the munging is more "aggressive", converting the above address to something like: "list@host.sender.dom" Which looks more "normal." (Actually: specifying anything less than 2 does the "simple" munging and anything greater than 1 results in the more "aggressive" hack being applied.) Added "--uucp_mung" switch for consistence with "--verp_mung". 19990321-01 NOTICE: As of this version of pflogsumm.pl, versions of VMailer prior to 19981023 are no longer supported. Sorry. Pflogsumm-19990121-01.pl will be made permanently available from now on for those with out-of-date versions of VMailer prior to 19981023. NOTICE: As of this version of pflogsumm.pl, the "-c" switch is DEPRECATED. This version is transitional and retains it. The next version will not have it. Subsequent versions may re-use it for another purpose. Use the "-h" and "-u" switches instead. Added "-h" and "-u" switches to provide finer-grained control over report output. Deprecated "-c". Added "deferred" and "bounced" to "Grand Totals", "by-day" and "by- hour" reports. Added "by-host/domain" reports. For sent (delivered) and received messages: lists message count, total size of messages and host/domain. For delivered messages: also lists number of deferred messages and average and maximum delivery time. Both reports sorted by message count in descending order. Grand totals also now list number of recipient and sender hosts/domains. Re-wrote "by-user" data collection storage to reduce memory consumption by in-memory hashes. Moved "credits" from pflogsumm.pl to this file. 19990121-01 Now accounts for forwarded messages. Side-effects of the above: . Total messages size now broken-out into total bytes received and total bytes delivered. . Count of forwarded messages now reported. . Postfix-internally-generated messages (e.g.: Postmaster notifications of bounces) are no longer counted as "received". (They do, however, show up as "delivered".) . Forwarded addresses no longer show up as "recipients" (just as with aliases and mailing lists). Note that "delivered" will exceed "received" when messages are forwarded because of additional header lines. 19990116-01 Added processing for "reject" log entries. Expanded detail of "deferred" and "bounced" log entries to include "reason". 19990110-05 Added "messages received/delivered by hour" and "messages received/delivered by day" reports. See the "Notes" section in the documentation for details on how these behave. Broke-out total message count to "messages received" and "messages delivered". (For the above two enhancements: "postfix/pickup" and "postfix/smtpd" lines are now processed. They used to be discarded.) Renamed "summary" report to "Grand Totals". Added code to parse date & time stamps from log entries. This was needed, in part, for the "messages per-hour/day" reports. It would have been necessary for future enhancements in the way of date- & time-based processing anyway. Added "Notes" section to docs at top of code. 19990109-01 Improved display of large integer values. 19990107-01 Bugfix only. Data for "extended detail" listing was being built even if "-e" not specified. This resulted in unexpected excessive memory consumption when crunching large amounts of data. Added warning about memory consumption when "-e" option specified. 19990103-01 Further improvement to "accuracy" of by-domain-then-logname sort. (Presently used only by "extended detail" listing). For comparison purposes: mungs "machine(s).host.dom" into "host.dom.machine(s)" so sort is keyed on "base" domain name before machines within the domain. Does *not* attempt to reverse the order of the "machine(s)" - so within a particular "base" domain, may not come out in quite the right order. ("foo.bar.some.dom" will come out before "sales.acme.some.dom", for example.) Also works for 2x2-style domain names. (I.e.: "some.do.co") 19990102-01 (never released) Added "mung UUCP-style bang-paths" switch (-m). Improved performance and "accuracy" of by-domain-then-logname sort used by (only at present) "extended detail" listing. 19990101-02 Added "extended detail" option (-e). At present this includes only a per-message detail listing, which lists per-message detail sorted by sender domain, then sender username, then by queue i.d. Improved docs a bit. 19990101-01 Replaced warning message when message size unavailable in favor of producing a report of these, sorted by queue i.d. Unlike the other reports, this report header is not emitted at all if there are none of these. (Always acts as if the -q switch had been specified). 19981231.01 Added experimental code to lower-case all domain names so that "user@foo.dom" and "user@FOO.DOM" will come out the same. Added test for existence of message size value when "to=" records are being processed. This was necessary for cases in which the logfile entry containing the "status=sent" record is not processed at the same time as the logfile containing the "size=nnnn" record. Note that this will produce a summary that will show recipient counts without matching recipient sizes. The only way to cure this would be to create a separate disk file to "memorize" message sizes. (Which would introduce a whole new raft of problems.) Added warning message (emitted to stderr) when the situation above is detected. Fixed "usage" message to indicate you can specify files on command line Wrapped a couple of long lines in the comments and code. Added (temporary) version numbering scheme. Started this log. Other changes/enhancements since previous un-version-numbered versions: deals with log entries for VMailer as well as Postfix, more robust parsing of "to=" and "from=" fields (now handles spaces in these), eliminated double-counting of message sizes (happened when delivery was deferred), re-structured parsing to be more robust (not- to-mention correct!), added "grand summary" report at top (total messages, total size, number of senders and recipients). Credits [Note: The credits reflect suggestions and code contributions that have actually been added. If your contribution doesn't appear here, it may simply mean that it hasn't been added yet. (In which case it should be on the list above.) On the other hand: if I failed to credit you for something that *has* been added, please let me know!] Paul D. Robertson For much testing and patience and many good suggestions on how pflogsumm could be improved. Simon J Mudd For the following code contributions: Add "deferred" and "bounced" to "by hour" reports. (I also added these to "by day" reports and "Grand Totals".) "VERP" (?) address munger (less-agressive version) Suggestion for "by domain" delivery delay report. For the --smtpd_stats suggestion. Anders Arnholm For pointing out the problem with forwarded messages. Walcir Fontanini For pointers to changes to make for Perl 5.003 compatibility. (Added to 19990413-02beta.) (Which I will *try* to keep in mind!) Eric Cholet For the --ignore_case patch. Kurt Andersen For the ISO date formatting month-off-by-one patch and the connect time overflow fix. Thomas Parmelan For improved "rejects" reporting patch. Glen Eustace Patch to fix "reject: header" matching after Wietse changed the logfile format. pflogsumm-1.1.5/pflogsumm-faq.txt0000644000175000017500000007254611712256151017401 0ustar jseymourjseymour FAQ for Pflogsumm.pl - A Log Summarizer/Analyzer for the Postfix MTA Introduction I wouldn't have believed it. What started out mostly as a light- hearted exercise in improving my facility with Perl--with the hope that something useful would come out of it as well--has turned out to be a somewhat popular utility. And as more Admins find out about postfix, and more end up trying pflogsumm.pl, many of the questions, suggestions, and enhancement requests are becoming "frequently asked". So odd as it seems (to me, at any rate), it looks like it's time for a FAQ. Index of pflogsumm.pl Frequently Asked Questions (in no particular order) 1. Project Status 2. "Could You Make" or "Here's A Patch To Make" Pflogsumm Do ... 3. Requires Date:Calc Module 4. Built-In Support for Compressed Logs 5. Processing Multiple Log Files 6. Time-Based Reporting and Statistics 7. By-domain Listings 8. Reject, Deferred and Bounced Detail Info 9. "Orphaned" (no size) Messages 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 11. Pflogsumm is generating lots of "uninitialized value" warnings 12. Pflogsumm just doesn't work or doesn't report anything 13. Postfix Rejects Pflogsumm Reports Because Of Body Checks 14. Pflogsumm Reports Double Traffic When Anti-Virus Scanner Used 15. Pflogsumm's numbers don't add up 16. Hourly stats for reports run without "-d" option are halved 17. How Do I Get Pflogsumm To Email Reports To Me Daily/Weekly/etc.? 18. How Can I View Pflogsumm's Reports In My Web Browser? 19. New Red Hat install - Pflogsumm no longer works! 20. How can I best format my custom reject messages for display in Pflogsumm's output? 21. Pflogsumm doesn't understand my log file format 22. Why Isn't There Any Mention Of "Monkey Butler" In The FAQ? 23. Translating Pflogsumm (Support for Internationalization) 24. Pflogsumm may sometimes calculate "yesterday" incorrectly 25. Sending Logfile Samples 26. From Where Can I Obtain Pflogsumm? 1. Project Status New work on Pflogsumm is sporadic. It pretty much does everything I need it to do and, so far as I can tell, pretty much what most other people need it to do. And my time is limited. I'll still take bug reports. I'll still fix bugs. (But I promise no time-line.) I'll still answer questions (as time allows). And I *may* add the occasional enhancement or whatever--as the mood strikes--but Pflogsumm is pretty much a "finished work" as far as I'm concerned. 2. "Could You Make" or "Here's A Patch To Make" Pflogsumm Do ... Unless it's a *bug* fix, please see: "1. Project Status" To the argument "But it's a patch, all you have to do is...," the answer is: "Not quite." Every time I make a change to Pflogsumm I have to run it through a series of regression checks to make sure the change didn't break something. Then there's the commit, documentation, web page update, etc. cycle. I'm particularly unlikely to add code to Pflogsumm to account for non-standard Postfix log entries. "Non-standard" being defined as "other than what Wietse's code does." Or additional stats gathering that nobody else has requested and strikes *me* as of limited interest or use. In addition to the development cycle, there's the issue of "code bloat." Pflogsumm already takes enough (too much?) time and memory on busy machines with large logs. I'm not prone to make this worse for the sake of these things. See Also: 21. Pflogsumm doesn't understand my log file format 3. Requires Date::Calc Module Pflogsumm requires the Date::Calc module. You can download and install the Date::Calc module from CPAN. It can be found at: http://search.cpan.org/search?module=Date::Calc Or you can remove the code that's dependent on the Date::Calc module. For the convenience of folks that would prefer to take this approach, I've "fenced" all such code like this: # ---Begin: SMTPD_STATS_SUPPORT--- . . . . . . # ---End: SMTPD_STATS_SUPPORT--- However, if you do this you will lose support for --smtpd-stats. Later versions of the Pflogsumm distribution include a script to semi-automate removing smtpd stats support, if you so-desire. As of Pflogsumm-1.1.1, the presence of Date::Calc is optional. If you don't want to use the Pflogsumm options that depend upon it, you neither need Date::Calc, nor is it necessary to manually remove the code that depends upon it. 4. Built-In Support for Compressed Logs I took a look at this. There is a Perl module (which I downloaded, built, and installed here) to interface to libz, but after considering the changes that would be necessary--and the fact that those changes would require that potential users have to download/build/install libz (and of the correct version) and the additional Perl module, I decided to forego this enhancement. I could just open a pipe within Pflogsumm and use zcat/gunzip/gzip. That would depend upon a) them being there [probably a safe bet-- considering the logs somehow got into that format :-), but...] and b) one of these either being in the path or having an environment variable or a script variable or... The thing is, in the latter case there's really no "savings" over simply piping into Pflogsumm in the first place. Multiple processes get spawned, pipes opened, etc. either way. It would add a little convenience, is all. So I could do it. And there are a couple of ways I could do it. And my mind is certainly still open on the issue. I'm just not convinced there's a good reason to do it, is all. And I'd like to avoid "creeping over-feature-itis" if I can. My position is *not* set in stone on this issue. In the mean-time: zcat /var/log/maillog.0.gz |pflogsumm.pl or gunzip should do the trick quite nicely for you. If you've a complex situation, for example: your logs aren't rotated exactly at midnight, you might try something like: (zcat /var/log/maillog.0.gz; cat /var/log/maillog) \ |pflogsumm.pl -d yesterday See Also: 5. Processing Multiple Log Files 17. How Do I Get Pflogsumm To Email Reports To Me Daily/Weekly/etc.? 5. Processing Multiple Log Files When processing multiple log files (say: an entire weeks worth of logs that are rotated daily), it is important that Pflogsumm be fed them in chronological order. For performance and memory conservation reasons, Pflogsumm relies on log messages "arriving" in the order in which they were created. If you do something like this: pflogsumm /var/log/maillog* you might not get what you expect! Instead, try something like: pflogsumm `ls -rt /var/log/maillog*` A more complex example, where compressed logs are involved: (zcat `ls -rt /var/log/maillog.*.gz`; cat /var/log/maillog) \ |pflogsumm.pl Obviously, this depends on the file modification times for your logs being reflective of their chronological order. If that can't be trusted, you're gonna have to get ugly. Like in enumerating each file, or as in: (for each in 3 2 1 0; do zcat "/var/log/maillog.$each.gz" done cat /var/log/maillog) |pflogsumm.pl or (somewhat more efficiently--by running zcat only once): (zcat `for ea in 3 2 1 0; do echo "/var/log/maillog.$ea.gz"; done`; cat /var/log/maillog) |pflogsumm.pl [Note: I didn't actually run these. So you would be well-advised to double-check them.] See Also: 4. Built-In Support for Compressed Logs 17. How Do I Get Pflogsumm To Email Reports To Me Daily/Weekly/etc.? 6. Time-Based Reporting and Statistics There has been a small assortment of requests for different time statistics reporting. And adding this would be relatively straight- forward. (Just have to reach a consensus on exactly *what* should be reported, and how. This could easily get out of hand!) There's only one *small* problem. Ironically, it's time. I've experimented with Pflogsumm grokking the log timestamps. As a matter-of-fact: the enhancement added in the 19990110-05 version required that I do some of this. My first pass was to use the Perl timelocal() function to convert those sub-strings to an integer for subsequent comparison operations. Imagine my surprise when performance of the resulting code was a factor of five (5) times slower than that of its predecessor. After a "remove the statements until it got fast again" exercise, I found that the culprit was timelocal(). As of version 19990321-01, Pflogsumm does by-domain stats reporting of average and maximum delivery time by host/domain. And an even earlier version added by-hour and by-day message count reporting. Anything much beyond these is going to get "expensive." If/when any additional time-based stats reporting is added: I think they are definitely going to be optional. One way you can make up for Pflogsumm's deficiency in this respect is to use good ol' Unix tools like "grep" to pre-process your log files before feeding them to Pflogsumm. E.g.: grep "Feb 9" /var/log/maillog |pflogsumm what_ever_args Note that single-digit days-of-the-month have an additional leading space in the logfiles, where the digit for two-digit dates would be. 7. By-domain Listings I figured on the desire for this one from the start. There are many possibilities: 1) A single report, split by domain 2) An option to limit reporting to a particular domain This issue is kind of tricky. The popularity of Unix amongst SysAdmins is testimony to the beauty of being able to wire- together small, simple tools so that one can generate output to ones taste. Anything I do is likely to make some Admins happy and others wishing I'd done it "the other way". One thought that occurred is to perhaps provide a couple of options that would allow one to limit a particular report to sender=regular_expression and/or recipient=regular_expression The problem with this solution is that an Admin desiring to emit custom reports for multiple domains would have to re-process the same log multiple times--once for each desired domain. So I'm still thinking about this one. 8. Reject, Deferred and Bounced Detail Info I've actually only received one query about this so far, but there are bound to be more. So... The "detailed" information in the "Reject", "Deferred" and "Bounced" reports is a compromise. Just take a stroll through your postfix logs some day and observe the variation in how the "reason" for a particular reject, defer, or bounce is reported. Without putting a lot of static comparisons for each-and-every case into the analyzer, I have absolutely no hope is doing this very well. Emitting the entire "reason" is not good, either. The entire "reason" string can be very long. Depending on what somebody is using to display Pflogsumm's output, the lines may well wrap-- producing output that is no more readable than just grepping the logs. And anything more I do to this end may soon be rendered moot. After Wietse gets most of the more important functional stuff out of the way, Postfix logging is going to be completely re-written. (Oh boy, won't that be fun!) I'm hoping I'll be able to get some input into the process so the formatting is more amenable to automated processing. Wietse has indicated that such would be the case. Also, please note my primary objective behind Pflogsumm (besides the entertainment value): "just enough detail to give the administrator a ``heads up'' for potential trouble spots." It's not *supposed* to do away with manual inspection entirely. For those that really want all that extra detail in the log summary reports, specify the "--verbose-msg-detail" switch. See Also: 25. Sending Logfile Samples 9. "Orphaned" (no size) Messages The Problem: Message size is reported only by the queue manager. The message may be delivered long-enough after the (last) qmgr log entry that the information is not in the log(s) processed by a particular run of pflogsumm.pl. The Result: "Orphaned" messages. These are reported by Pflogsumm as "Messages with no size data." This, of course, throws off "Recipients by message size" and the total for "bytes delivered." ("bytes in messages" in earlier versions.) The Solution: "Memorize" message sizes by queue i.d. Easy in theory. Difficult in practice. At least at the moment. You see, if Pflogsumm's going to "memorize" message sizes, it has to have some definitive way to know when to delete a no- longer-needed reference. Otherwise the memory file will just grow forever. As with the "Reject, Deferred and Bounced Detail Info" issue above, I'm hoping the get some input into future changes in logging issues. In any event: maybe whatever comes out of the logging redesign will provide a solution. As of Pflogsumm version 1.0.12, the "Messages with no size data" report can be turned off. 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. Are you using a real old version of VMailer? As of pflogsumm.pl version 19990220-06, versions of VMailer prior to 19981023 are no longer supported. Sorry. Pflogsumm-19990121-01.pl will be made permanently available from now on for those with out-of-date versions of VMailer prior to 19981023. Are you processing your log files in chronological order? See item "5: "Processing Multiple Log Files". Pflogsumm.pl is being developed by me on my rather small-scale server at home. There are only two users on the system. And I do no mail-forwarding. So the log samples I have to work with are commensurately limited. If there's something that Pflogsumm is not doing, or not doing right, let me know what it is, what you think it ought to do, and send me a representative sample of *real* log entries with which to work. See Also: 5. Processing Multiple Log Files 12. Pflogsumm just doesn't work or doesn't report anything 15. Pflogsumm's numbers don't add up 19. New Red Hat install - Pflogsumm no longer works! 21. Pflogsumm doesn't understand my log file format 25. Sending Logfile Samples 11. Pflogsumm is generating lots of "uninitialized value" warnings Are you using a version of Perl lower than 5.004_04? Perhaps with a "beta" version of pflogsumm.pl? If so, try turning off the "-w" switch. Pflogsumm as of 19990413-02beta appeared to work correctly with Perl 5.003 in spite of the warnings. (Those warnings didn't appear with Perl 5.004.) I don't guarantee that I'll remember to test future versions of pflogsumm.pl against 5.003, but I'll try to :-). You really should consider upgrading your Perl to 5.004 or later. 12. Pflogsumm just doesn't work or doesn't report anything Did you *download* Pflogsumm as opposed to grabbing it by "copy-and-paste" from a browser? Copy-and-paste can result in lines being unintentionally wrapped and hard-tabs being converted to spaces. This will break Pflogsumm. Also, I've received a couple of reports by people downloading Pflogsumm with Lynx that the download has long lines wrapped. Naturally, this breaks Pflogsumm. See Also: 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 19. New Red Hat install - Pflogsumm no longer works! 21. Pflogsumm doesn't understand my log file format 13. Postfix Rejects Pflogsumm Reports Because Of Body Checks You configure Postfix to do body checks, Postfix does its thing, Pflogsumm reports it and Postfix catches the the same string in the Pflogsumm report. There are several solutions to this. Wolfgang Zeikat contributed this: #!/usr/bin/perl use MIME::Lite; ### Create a new message: $msg = MIME::Lite->new( From => 'your@send.er', To => 'your@recipie.nt', # Cc => 'some@other.com, some@more.com', Subject => 'pflogsumm', Date => `date`, Type => 'text/plain', Encoding => 'base64', Path =>'/tmp/pflogg', ); $msg->send; Where "/tmp/pflogg" is the output of Pflogsumm. This puts Pflogsumm's output in a base64 MIME attachment. In a follow-up to a thread in the postfix-users mailing list, Ralf Hildebrandt noted: "mpack does the same thing." The canonical FTP site for mpack is ftp.andrew.cmu.edu:pub/mpack/ The solution I came up with is to modify the body_checks statements to ignore the strings when they're in a Pflogsumm report, as follows: Bounce anything with 6 or more "$"s in a row... /\${6,}/ REJECT Which, of course, catches the line in the Pflogsumm report too. So... /^(?!\s+[0-9]+\s+).*?\${6,}/ REJECT which reads "anything with 6 or more '$'s in a row that is not a line beginning with one or more whitespace characters, followed by one or more digits, followed by one or more whitespace characters." (This is using PCRE's, btw.) Note that my solution will be more computationally expensive, by a *long* way, than encoding Pflogsumm's output into a format that body_checks won't catch. Robert L Mathews suggested the following solution /^ {6,11}[[:digit:]]{1,6}[ km] / OK Placed at the beginning of a body_checks file, this will "pre-approve" lines in Pflogsumm's output that might otherwise get caught. That's a POSIX regexp version. A PCRE version of the same thing would be: /^ {6,11}\d{1,6}[ km] / OK 14. Pflogsumm Reports Double Traffic When Anti-Virus Scanner Used Sadly, there's absolutely nothing I can do about this :-(. The problem arises because of the way in which anti-virus scanning is handled by Postfix. Basically, Postfix "delivers" each email to the anti-virus scanner and the anti-virus scanner re-sends it through Postfix. So each email really is received twice and sent/delivered twice. And yes, I tried. I really, really tried. If I recall correctly, I spent some two days mucking-about with this problem. Actually thought I had it once or twice. But the results inevitably failed regression testing. At the end of this, and with some more careful thought, I realized it just wasn't possible. If you think you can prove me wrong, please do so. I'd be quite pleased to be proven wrong on this one. johnfawcett at tiscali-dot-it believes he's done it. You may find prefiltering your log with his "prepflog" does it for you. You can find it at . Note: Because of the way John's pre-processing script works, which, given my own experiments, is probably the way it *has* to work to work correctly, integrating his code into Pflogsumm would be difficult, at best, if it's even possible within Pflogsumm's current structure. 15. Pflogsumm's numbers don't add up Pflogsumm reports more "delivered" than "received" Naturally. A single email message can have multiple recipients. Pflogsumm reports more "rejected" than "received" Why doesn't delivered + deferred + bounce + rejected = received? Some rejects (header and body checks, for example) happen in "cleanup," after alias lists are expanded. Thus a single received message will be rejected multiple times: once for each recipient. The "size=" fields, multiplied by their "nrcpt=" fields, when added-up yields a total higher than Pflogsumm's "bytes delivered" total. Pflogsumm doesn't count something delivered until it actually *is* delivered. Nrcpt only suggests the number of intended recipients, not how many are actually deliverable. Only if there were no bounces, rejects, defers or other undeliverables for everything that was received would a calculation such as that above yield the proper value. Pflogsumm's "% rejected" doesn't add up The "percent rejected" and "percent discarded" figures are only approximations. They are calculated as follows (example is for "percent rejected"): percent rejected = (rejected / (delivered + rejected + discarded)) * 100 Given the issues discussed above, this is really the best that can be hoped-for, IMO. I consistently see more "delivered" than "received." How is that possible? Any message that's got multiple recipients in the "To:," "Cc:," and "Bcc:" fields will result in a single "received" with multiple "delivered"s, as well as, possibly, multiple "rejects" (or reject warnings, discards or holds), depending on where in Postfix' processing the rule was that resulted in the reject, etc. See Also: 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 16. Hourly stats for reports run without "-d" option are halved Scenario: On day #1 of a fresh logfile, you run Pflogsumm with "-d today" and the next day you run it with no "-d" option. The "Per-Hour Traffic" statistics are approximately halved. How can this be? Note that when you run Pflogsumm on a logfile that contains multi-day logfile entries, Pflogsumm automatically changes the per-hour stats to daily traffic averages. If there's even *one* logfile entry from another day, all of the per-hour stats will be divided by two. Unless you rotate logfiles *precisely* at midnight--and it's unlikely you can guarantee that happening--there's no way to prevent this. 17. How Do I Get Pflogsumm To Email Reports To Me Daily/Weekly/etc.? Excuse me? You're running a mailserver and you don't know how to use cron to run things on a scheduled basis and pipe the output to something that'll email it to you? Oh. My. Lord. *sigh* Here's my crontab entries: 10 0 * * * /usr/local/sbin/pflogsumm -d yesterday /var/log/syslog \ 2>&1 |/usr/bin/mailx -s "`uname -n` daily mail stats" postmaster 10 4 * * 0 /usr/local/sbin/pflogsumm /var/log/syslog.0 \ 2>&1 |/usr/bin/mailx -s "`uname -n` weekly mail stats" postmaster (Those are actually each a single line. I line-broke them [and signified that with the "\"s] for readability.) The first generates stats for the previous day and is run *after* midnight. The second is run against the previous week's entire log. (I rotate my logs weekly.) If you rotate your logs on a different schedule, want monthly reports, etc., I leave it as an exercise to you, the reader, to figure out how to concatenate several logs to stdout and feed that to Pflogsumm. See Also: 4. Built-In Support for Compressed Logs 5. Processing Multiple Log Files The Unix manual pages for "cron," "crontab," "cat," "zcat," "gzip," "gunzip," "mail," "mailx," etc. 18. How Can I View Pflogsumm's Reports In My Web Browser? Just direct Pflogsumm's output to a file, call it "something.txt" or whatever, and look at it with your browser :). If you want to get fancy, create a post-processing shell script that'll create a date-tagged file, like "mailstats-20030216.txt". It's easy. See Also: Pflogsumm Through A Browser, on Pflogsumm's home page. 19. New Red Hat install - Pflogsumm no longer works! From some email exchanges with a couple of people that reported this... "It appears the Pflogsumm is broken with RedHat9. I can take the same log file and run it under Solaris9/RedHat 7.3 (perl 5.8 on both) without a problem, but it breaks on RH9." "Oops. Sorry about the false alarm. This is an issue with some of the other Perl scripts that are out there due to Red Hat 8/9 using LANG=en_US.UTF-8 Changing the locale to "POSIX" fixes this... LANG=C Note that Pflogsumm works fine when run through cron.daily, as cron has different environment settings." "Ah, the good old RH8/9 UTF-8 strikes again. I should have known. Setting LANG to either en_US or C fixes the problem." What the above means is that you have to change the "LANG" environment variable from "en_US.UTF-8" to "en_US" or "C". E.g.: LANG="en_US" export LANG in your shell. Or you could add these commands to your login profile. (I.e.: $HOME/.bash_profile, if you're using bash.) Or set the system-wide default in /etc/sysconfig/i18n. My RH boxes have LANG set to "en_US" there, and everything seems to work fine. (If you set it in your profile or the system-wide default, you'll need a fresh login for it to take effect, obviously.) See Also: 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 12. Pflogsumm just doesn't work or doesn't report anything 21. Pflogsumm doesn't understand my log file format 20. How can I best format my custom reject messages for display in Pflogsumm's output? Reject reason strings found in the mail log will be truncated at the first comma (","), colon (":") or semi-colon (";"). If you want a "clause" in your reject message to appear in Pflogsumm's output, without having to specify --verbose-msg-detail, use a punctuation mark other than one of those three, such as a dash ("-"). 21. Pflogsumm doesn't understand my log file format I've received several requests to modify Pflogsumm's log file format regular expression matching to accommodate "non-standard" log file formats. I'm not inclined to honour such requests. The regexp that identifies Postfix' log file entries is nearly incomprehensible as it is. If your log file format has extra fields (e.g.: FreeBSD syslogd with "-v -v" specified), or, as in one case (metalog), is lacking fields, and you insist on doing things that way, I recommend you code-up a little pre-filter to mung the format into a standard one. See Also: 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 12. Pflogsumm just doesn't work or doesn't report anything 19. New Red Hat install - Pflogsumm no longer works! 22. Why Isn't There Any Mention Of "Monkey Butler" In The FAQ? A friend of mine asked me if I'd put the phrase "monkey butler" in the FAQ. The answer is no. Pflogsumm is used by some rather large corporations. There are credibility issues. Sorry. :) 23. Translating Pflogsumm (Support for Internationalization) Unfortunately, Pflogsumm doesn't currently have i18n support. It wasn't until at least Perl 5.6 that i18n was included as part of the base distribution. Since, last time I looked, 5.005* was still the most widely-used version of Perl (that's what I'm still running everywhere, too), I can't put i18n in without chancing breaking things right-and-left for the majority of my customers. Even with Perl 5.6 and above, it was mentioned in postfix-users, by Liviu Daia, that "Perl 5.6+ has locales. Locales can give you localized dates, charsets, standard error messages etc., but it won't automatically switch languages of the strings defined in your program. For that, you still need gettext or something equivalent." So I'm not clear on the future of i18n support in Pflogsumm. But I'm keeping an eye on things. Proper i18n support has long been one of the top things on my own wish list! Prospective translators are urged to translate *only* the stable/production versions. Beta and Alpha versions can sometimes change rapidly. If you do translate Pflogsumm, let me know and I'll put a link to it on Pflogsumm's main web page. 24. Pflogsumm may sometimes calculate "yesterday" incorrectly As Wieland Chmielewski aptly noted: Subroutine get_datestr incorrectly assumes that each day of the year comprises 24 hours. In those countries which participate in Daylight Saving Time policy there is one day with 23 hours and one day with 25 hours. So, chances are (for 1 hour within those days) that get_datestr actually returns either "the day before yesterday" or "today" instead of "yesterday" as requested. Right you are, Wieland, and thanks for the catch. Problem is, of course, there's really no clean, easy, certain fix. The work-around is to stay well clear of DST never-never land with your cron jobs. 25. Sending Logfile Samples Here's the deal with whatever you may send me in the way of log samples: . Obfuscate them if you want. But take care not alter them in such a manner that they're not accurate wrt the "realism" of the data, make sure the field formatting is not altered, and that the order of the log entries is not altered. . The world is an unsafe place for your data, no matter where it might reside. But I'll do my level best to ensure that your data does not fall into the hands of others. . If you want, I'll PGP-encrypt the data when it's not in use. . You can PGP-encrypt it when you send it to me if you're concerned. My PGP public key can be found on my Web site and at the PGP public key servers. . If you want, I'll delete the sample data when the work is done. But I would *like* to keep it around for future regression-testing. It's your call. Let me know. 26. From Where Can I Obtain Pflogsumm? http://jimsun.LinxNet.com/postfix_contrib.html Created: 15 Feb., 1999 / Last updated: 22 March, 2010 pflogsumm-1.1.5/pflogsumm.10000640000175000017500000004726411713601765016156 0ustar jseymourjseymour.\" Automatically generated by Pod::Man 2.1801 (Pod::Simple 3.13) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is turned on, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .ie \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . nr % 0 . rr F .\} .el \{\ . de IX .. .\} .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "PFLOGSUMM 1" .TH PFLOGSUMM 1 "2012-02-05" "1.1.5" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" pflogsumm.pl \- Produce Postfix MTA logfile summary .PP Copyright (C) 1998\-2010 by James S. Seymour, Release 1.1.5 .SH "SYNOPSIS" .IX Header "SYNOPSIS" .Vb 8 \& pflogsumm.pl \-[eq] [\-d ] [\-\-detail ] \& [\-\-bounce\-detail ] [\-\-deferral\-detail ] \& [\-h ] [\-i|\-\-ignore\-case] [\-\-iso\-date\-time] [\-\-mailq] \& [\-m|\-\-uucp\-mung] [\-\-no\-no\-msg\-size] [\-\-problems\-first] \& [\-\-rej\-add\-from] [\-\-reject\-detail ] [\-\-smtp\-detail ] \& [\-\-smtpd\-stats] [\-\-smtpd\-warning\-detail ] \& [\-\-syslog\-name=string] [\-u ] [\-\-verbose\-msg\-detail] \& [\-\-verp\-mung[=]] [\-\-zero\-fill] [file1 [filen]] \& \& pflogsumm.pl \-[help|version] \& \& If no file(s) specified, reads from stdin. Output is to stdout. .Ve .SH "DESCRIPTION" .IX Header "DESCRIPTION" .Vb 4 \& Pflogsumm is a log analyzer/summarizer for the Postfix MTA. It is \& designed to provide an over\-view of Postfix activity, with just enough \& detail to give the administrator a "heads up" for potential trouble \& spots. \& \& Pflogsumm generates summaries and, in some cases, detailed reports of \& mail server traffic volumes, rejected and bounced email, and server \& warnings, errors and panics. .Ve .SH "OPTIONS" .IX Header "OPTIONS" .Vb 1 \& \-\-bounce\-detail \& \& Limit detailed bounce reports to the top . 0 \& to suppress entirely. \& \& \-d today generate report for just today \& \-d yesterday generate report for just "yesterday" \& \& \-\-deferral\-detail \& \& Limit detailed deferral reports to the top . 0 \& to suppress entirely. \& \& \-\-detail \& \& Sets all \-\-*\-detail, \-h and \-u to . Is \& over\-ridden by individual settings. \-\-detail 0 \& suppresses *all* detail. \& \& \-e extended (extreme? excessive?) detail \& \& Emit detailed reports. At present, this includes \& only a per\-message report, sorted by sender domain, \& then user\-in\-domain, then by queue i.d. \& \& WARNING: the data built to generate this report can \& quickly consume very large amounts of memory if a \& lot of log entries are processed! \& \& \-h top to display in host/domain reports. \& \& 0 = none. \& \& See also: "\-u" and "\-\-*\-detail" options for further \& report\-limiting options. \& \& \-\-help Emit short usage message and bail out. \& \& (By happy coincidence, "\-h" alone does much the same, \& being as it requires a numeric argument :\-). Yeah, I \& know: lame.) \& \& \-i \& \-\-ignore\-case Handle complete email address in a case\-insensitive \& manner. \& \& Normally pflogsumm lower\-cases only the host and \& domain parts, leaving the user part alone. This \& option causes the entire email address to be lower\- \& cased. \& \& \-\-iso\-date\-time \& \& For summaries that contain date or time information, \& use ISO 8601 standard formats (CCYY\-MM\-DD and HH:MM), \& rather than "Mon DD CCYY" and "HHMM". \& \& \-m modify (mung?) UUCP\-style bang\-paths \& \-\-uucp\-mung \& \& This is for use when you have a mix of Internet\-style \& domain addresses and UUCP\-style bang\-paths in the log. \& Upstream UUCP feeds sometimes mung Internet domain \& style address into bang\-paths. This option can \& sometimes undo the "damage". For example: \& "somehost.dom!username@foo" (where "foo" is the next \& host upstream and "somehost.dom" was whence the email \& originated) will get converted to \& "foo!username@somehost.dom". This also affects the \& extended detail report (\-e), to help ensure that by\- \& domain\-by\-name sorting is more accurate. \& \& \-\-mailq Run "mailq" command at end of report. \& \& Merely a convenience feature. (Assumes that "mailq" \& is in $PATH. See "$mailqCmd" variable to path thisi \& if desired.) \& \& \-\-no_bounce_detail \& \-\-no_deferral_detail \& \-\-no_reject_detail \& \& These switches are deprecated in favour of \& \-\-bounce\-detail, \-\-deferral\-detail and \& \-\-reject\-detail, respectively. \& \& Suppresses the printing of the following detailed \& reports, respectively: \& \& message bounce detail (by relay) \& message deferral detail \& message reject detail \& \& See also: "\-u" and "\-h" for further report\-limiting \& options. \& \& \-\-no\-no\-msg\-size \& \& Do not emit report on "Messages with no size data". \& \& Message size is reported only by the queue manager. \& The message may be delivered long\-enough after the \& (last) qmgr log entry that the information is not in \& the log(s) processed by a particular run of \& pflogsumm.pl. This throws off "Recipients by message \& size" and the total for "bytes delivered." These are \& normally reported by pflogsumm as "Messages with no \& size data." \& \& \-\-no\-smtpd\-warnings \& \& This switch is deprecated in favour of \& smtpd\-warning\-detail \& \& On a busy mail server, say at an ISP, SMTPD warnings \& can result in a rather sizeable report. This option \& turns reporting them off. \& \& \-\-problems\-first \& \& Emit "problems" reports (bounces, defers, warnings, \& etc.) before "normal" stats. \& \& \-\-rej\-add\-from \& For those reject reports that list IP addresses or \& host/domain names: append the email from address to \& each listing. (Does not apply to "Improper use of \& SMTP command pipelining" report.) \& \& \-q quiet \- don\*(Aqt print headings for empty reports \& \& note: headings for warning, fatal, and "master" \& messages will always be printed. \& \& \-\-reject\-detail \& \& Limit detailed smtpd reject, warn, hold and discard \& reports to the top . 0 to suppress entirely. \& \& \-\-smtp\-detail \& \& Limit detailed smtp delivery reports to the top . \& 0 to suppress entirely. \& \& \-\-smtpd\-stats \& \& Generate smtpd connection statistics. \& \& The "per\-day" report is not generated for single\-day \& reports. For multiple\-day reports: "per\-hour" numbers \& are daily averages (reflected in the report heading). \& \& \-\-smtpd\-warning\-detail \& \& Limit detailed smtpd warnings reports to the top . \& 0 to suppress entirely. \& \& \-\-syslog\-name=name \& \& Set syslog\-name to look for for Postfix log entries. \& \& By default, pflogsumm looks for entries in logfiles \& with a syslog name of "postfix," the default. \& If you\*(Aqve set a non\-default "syslog_name" parameter \& in your Postfix configuration, use this option to \& tell pflogsumm what that is. \& \& See the discussion about the use of this option under \& "NOTES," below. \& \& \-u top to display in user reports. 0 == none. \& \& See also: "\-h" and "\-\-*\-detail" options for further \& report\-limiting options. \& \& \-\-verbose\-msg\-detail \& \& For the message deferral, bounce and reject summaries: \& display the full "reason", rather than a truncated one. \& \& Note: this can result in quite long lines in the report. \& \& \-\-verp\-mung do "VERP" generated address (?) munging. Convert \& \-\-verp\-mung=2 sender addresses of the form \& "list\-return\-NN\-someuser=some.dom@host.sender.dom" \& to \& "list\-return\-ID\-someuser=some.dom@host.sender.dom" \& \& In other words: replace the numeric value with "ID". \& \& By specifying the optional "=2" (second form), the \& munging is more "aggressive", converting the address \& to something like: \& \& "list\-return@host.sender.dom" \& \& Actually: specifying anything less than 2 does the \& "simple" munging and anything greater than 1 results \& in the more "aggressive" hack being applied. \& \& See "NOTES" regarding this option. \& \& \-\-version Print program name and version and bail out. \& \& \-\-zero\-fill "Zero\-fill" certain arrays so reports come out with \& data in columns that that might otherwise be blank. .Ve .SH "RETURN VALUE" .IX Header "RETURN VALUE" .Vb 1 \& Pflogsumm doesn\*(Aqt return anything of interest to the shell. .Ve .SH "ERRORS" .IX Header "ERRORS" .Vb 1 \& Error messages are emitted to stderr. .Ve .SH "EXAMPLES" .IX Header "EXAMPLES" .Vb 1 \& Produce a report of previous day\*(Aqs activities: \& \& pflogsumm.pl \-d yesterday /var/log/maillog \& \& A report of prior week\*(Aqs activities (after logs rotated): \& \& pflogsumm.pl /var/log/maillog.0 \& \& What\*(Aqs happened so far today: \& \& pflogsumm.pl \-d today /var/log/maillog \& \& Crontab entry to generate a report of the previous day\*(Aqs activity \& at 10 minutes after midnight. \& \& 10 0 * * * /usr/local/sbin/pflogsumm \-d yesterday /var/log/maillog \& 2>&1 |/usr/bin/mailx \-s "\`uname \-n\` daily mail stats" postmaster \& \& Crontab entry to generate a report for the prior week\*(Aqs activity. \& (This example assumes one rotates ones mail logs weekly, some time \& before 4:10 a.m. on Sunday.) \& \& 10 4 * * 0 /usr/local/sbin/pflogsumm /var/log/maillog.0 \& 2>&1 |/usr/bin/mailx \-s "\`uname \-n\` weekly mail stats" postmaster \& \& The two crontab examples, above, must actually be a single line \& each. They\*(Aqre broken\-up into two\-or\-more lines due to page \& formatting issues. .Ve .SH "SEE ALSO" .IX Header "SEE ALSO" .Vb 1 \& The pflogsumm FAQ: pflogsumm\-faq.txt. .Ve .SH "NOTES" .IX Header "NOTES" .Vb 3 \& Pflogsumm makes no attempt to catch/parse non\-Postfix log \& entries. Unless it has "postfix/" in the log entry, it will be \& ignored. \& \& It\*(Aqs important that the logs are presented to pflogsumm in \& chronological order so that message sizes are available when \& needed. \& \& For display purposes: integer values are munged into "kilo" and \& "mega" notation as they exceed certain values. I chose the \& admittedly arbitrary boundaries of 512k and 512m as the points at \& which to do this\-\-my thinking being 512x was the largest number \& (of digits) that most folks can comfortably grok at\-a\-glance. \& These are "computer" "k" and "m", not 1000 and 1,000,000. You \& can easily change all of this with some constants near the \& beginning of the program. \& \& "Items\-per\-day" reports are not generated for single\-day \& reports. For multiple\-day reports: "Items\-per\-hour" numbers are \& daily averages (reflected in the report headings). \& \& Message rejects, reject warnings, holds and discards are all \& reported under the "rejects" column for the Per\-Hour and Per\-Day \& traffic summaries. \& \& Verp munging may not always result in correct address and \& address\-count reduction. \& \& Verp munging is always in a state of experimentation. The use \& of this option may result in inaccurate statistics with regards \& to the "senders" count. \& \& UUCP\-style bang\-path handling needs more work. Particularly if \& Postfix is not being run with "swap_bangpath = yes" and/or *is* being \& run with "append_dot_mydomain = yes", the detailed by\-message report \& may not be sorted correctly by\-domain\-by\-user. (Also depends on \& upstream MTA, I suspect.) \& \& The "percent rejected" and "percent discarded" figures are only \& approximations. They are calculated as follows (example is for \& "percent rejected"): \& \& percent rejected = \& \& (rejected / (delivered + rejected + discarded)) * 100 \& \& There are some issues with the use of \-\-syslog\-name. The problem is \& that, even with Postfix\*(Aq $syslog_name set, it will sometimes still \& log things with "postfix" as the syslog_name. This is noted in \& /etc/postfix/sample\-misc.cf: \& \& # Beware: a non\-default syslog_name setting takes effect only \& # after process initialization. Some initialization errors will be \& # logged with the default name, especially errors while parsing \& # the command line and errors while accessing the Postfix main.cf \& # configuration file. \& \& As a consequence, pflogsumm must always look for "postfix," in logs, \& as well as whatever is supplied for syslog_name. \& \& Where this becomes an issue is where people are running two or more \& instances of Postfix, logging to the same file. In such a case: \& \& . Neither instance may use the default "postfix" syslog name \& and... \& \& . Log entries that fall victim to what\*(Aqs described in \& sample\-misc.cf will be reported under "postfix", so that if \& you\*(Aqre running pflogsumm twice, once for each syslog_name, such \& log entries will show up in each report. \& \& The Pflogsumm Home Page is at: \& \& http://jimsun.LinxNet.com/postfix_contrib.html .Ve .SH "REQUIREMENTS" .IX Header "REQUIREMENTS" .Vb 3 \& For certain options (e.g.: \-\-smtpd\-stats), Pflogsumm requires the \& Date::Calc module, which can be obtained from CPAN at \& http://www.perl.com. \& \& Pflogsumm is currently written and tested under Perl 5.8.3. \& As of version 19990413\-02, pflogsumm worked with Perl 5.003, but \& future compatibility is not guaranteed. .Ve .SH "LICENSE" .IX Header "LICENSE" .Vb 4 \& This program is free software; you can redistribute it and/or \& modify it under the terms of the GNU General Public License \& as published by the Free Software Foundation; either version 2 \& of the License, or (at your option) any later version. \& \& This program is distributed in the hope that it will be useful, \& but WITHOUT ANY WARRANTY; without even the implied warranty of \& MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the \& GNU General Public License for more details. \& \& You may have received a copy of the GNU General Public License \& along with this program; if not, write to the Free Software \& Foundation, Inc., 59 Temple Place \- Suite 330, Boston, MA 02111\-1307, \& USA. \& \& An on\-line copy of the GNU General Public License can be found \& http://www.fsf.org/copyleft/gpl.html. .Ve