CUFlow-1.7/0002755011555000000140000000000010361017147012641 5ustar selskysrc00000000000000CUFlow-1.7/COPYING0000644011555000000140000004310610361017143013672 0ustar selskysrc00000000000000 GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc. 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Library General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) 19yy This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) 19yy name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Library General Public License instead of this License. CUFlow-1.7/CUFlow.cf0000644011555000000140000000332207753743440014325 0ustar selskysrc00000000000000# These are the subnets in our network # These are used only to determine whether a packet is inbound our # outbound Subnet 10.0.0.0/16 # These are networks we are particularly interested in, and want to # get separate rrd's for their aggregate traffic Network 10.0.1.0/24 routers # Where to put the rrd's # Make sure this is the same as $rrddir in CUGrapher.pl OutputDir /cflow/reports/rrds # Track multicast traffic Multicast # Keep top N lists # Show the top ten talkers, storing reports in /cflow/flows/reports # and keeping the current report in /etc/httpd/data/reports/topten.html Scoreboard 10 /cflow/reports/scoreboard /var/www/html/topten.html # Same, but build an over-time average top N list AggregateScore 10 /cflow/reports/scoreboard/agg.dat /var/www/html/overall.html # Our two netflow exporters. Produce service and protocol reports for the # total, and each of these. Router 10.0.1.1 router1 Router 10.0.1.2 router2 # Services we are interested in Service 20-21/tcp ftp Service 22/tcp ssh Service 23/tcp telnet Service 25/tcp smtp Service 53/udp,53/tcp dns Service 80/tcp http Service 110/tcp pop3 Service 119/tcp nntp Service 143/tcp imap Service 412/tcp,412/udp dc Service 443/tcp https Service 1214/tcp kazaa Service 4661-4662/tcp,4665/udp edonkey Service 5190/tcp aim Service 6346-6347/tcp gnutella Service 6665-6669/tcp irc Service 54320/tcp bo2k Service 7070/tcp,554/tcp,6970-7170/udp real # protocols we are interested in Protocol 1 icmp Protocol 4 ipinip Protocol 6 tcp Protocol 17 udp Protocol 47 gre Protocol 50 esp Protocol 51 ah Protocol 57 skip Protocol 88 eigrp Protocol 169 Protocol 255 # ToS bit percentages to graph TOS 0 normal TOS 1-255 other # Interested in traffic to/from AS 1 ASNumber 1 Genuity CUFlow-1.7/CUFlow.pm0000644011555000000140000015243410361016510014336 0ustar selskysrc00000000000000# CUFlow.pm - A (hopefully) slightly faster implementation of some of the # functionality of SubnetIO.pm, in a slightly more configurable fashion # Owes *VERY* heavily to Dave Plonka's SubnetIO.pm and CampusIO.pm # Thanks, Dave :) # To Add: # File creation routines subroutinized? # ICMP type handling as a Service, ie 6/icmp Echo # Make Networks record services? How, while still being fast? # Traffic with the same source and destination port can mess up Services # Also traffic whose source port is in one service and whose destination is # another. Hmmm... # (c) 2002 - 2005 The Trustees of Columbia University in the City of New York # License restrictions apply, see COPYING for details. use strict; package CUFlow; require 5; require Exporter; @CUFlow::ISA=qw(FlowScan Exporter); # convert the RCS revision to a reasonable Exporter VERSION: '$Revision: 1.66 $ ' =~ m/(\d+)\.(\d+)/ && (( $CUFlow::VERSION ) = sprintf("%d.%03d", $1, $2)); =head1 NAME CUFlow - flowscan module that is a little more configurable than SubnetIO.pm in return for sacrificing some modularity. =head1 SYNOPSIS $ flowscan CUFlow or in F: ReportClasses CUFlow =head1 DESCRIPTION CUFlow.pm creates rrds matching the configuration given in CUFlow.cf. It (by default) creates a 'total.rrd' file, representing the total in and out-bound traffic it receives. It also creates 2 rrd files for every B directive in CUFlow.cf, service_servicename_src.rrd and service_servicename_dst.rrd. It makes some assumptions about the nature of the flows exported to it: basically that they are either inbound to your network or outbount from it. It is designed to be run on a border router, and is not written to handle traffic exported to it about flows that source and end in your network, or outside it. It should be used to monitor the traffic being sent out by networks contained in B statements. =head1 CONFIGURATION CUFlow's configuration file is F. This configuration file is located in the directory in which the F script resides. In this file, blank lines, and any data after a pound sign (#) are ignored. Directives that can be put in this file include: =over 4 =item B By default, CUFlow does not care from which router the flow records it is processing are from. Unless you specify a Router statement, it just aggregates all the traffic it gets and produces rrd files based on the total. But, if you put # Separate out traffic from foorouter # Router Router 127.0.0.5 foorouter In addition to generating the totals rrd files it normally does in OutputDir, it will create a directory whose name is the IP address specified (or the alias, if one is provided), and create all the same service_*, protocol_*, and total.rrd files in it, except only for traffic passed from the router whose address is . Note that it does not make any sense to have Router statements in your config unless you have more than one router feeding flow records to flowscan (with one router, the results in the per-router directory will be identical to the total records in OutputDir) =item B If you are using sampled netflow (mandatory on Juniper) the router will export only 1/n samples. Specify the sample rate in the configfile by using # Sample rate pr. exporter in case we're using sampled netflow # SampleRate SampleRate 127.0.0.5 96 This will effectively multiply all data from that router by 96. =item B Each B entry in the file is an IP/length pair that represents a local subnet. E.g.: # Subnet for main campus Subnet 128.59.0.0/16 Add as many of these as is necessary. CUFlow does not generate additional reports per subnet, as does CampusIO, it simply treats any packet destined to an address *not* in any of its Subnet statements as an outbound packet. The Subnet statements are solely to determine if a given IP address is "in" your network or not. For subnet-specific reporting, see the Network item below. =item B Each B statement in the cf file is used to generate an rrd file describing the bytes, packets, and flows in and out of a group of IP addresses within your larger Subnet blocks. E.g.: # Watson Hall traffic Network 128.59.39.0/24,128.59.31.0/24 watson It consists of a comma separated list of 1 or more CIDR blocks, followed by a label to apply to traffic into/out of those blocks. It creates rrd files named 'network_label.rrd'. Note that these are total traffic seen only, unfortunately, and not per-exporter as Service and Protocol are. Note also that a Network must be subset of your defined Subnet's. =item B Each B entry in the file is a port/protocol name that we are interested in, followed by a label. E.g.: # Usenet news Service nntp/tcp news In this case, we are interested in traffic to or from port 119 on TCP, and wish to refer to such traffic as 'news'. The rrd files that will be created to track this traffic will be named 'service_news_src.rrd' (tracking traffic whose source port is 119) and 'service_news_dst.rrd' (for traffic with dst port 119). Each B entry will produce these 2 service files. The port and protocol can either be symbolic (nntp, tcp), or absolute numeric (119, 6). If a name is symbolic, we either getservbyname or getprotobyname as appropriate. B tags may also define a range or group of services that should be aggregated together. E.g: # RealServer traffic Service 7070/tcp,554/tcp,6970-7170/udp realmedia This means that we will produce a 'service_realmedia_dst.rrd' and 'service_realmedia_src.rrd' files, which will contain traffic data for the sum of the port/protocol pairs given above. Do not put spaces in the comma-separated list. The label that follows the set of port/protocol pairs must be unique, as it is used to create the rrd file for the matching data. CUFlow will not start if there is a duplicate label name. =item B Each B entry in the configuration file specifies a foreign Autonomous System number we are interested in. Specify these in the config file as: # Track our traffic to as 23517 ASNumber 23517 FooNET where the first argument is the AS number (as reported in the netflow records), and the second is a label describing that AS. (Do not use spaces or characters that have meaning to the filesystem/shell in these labels!) An rrd file is created for every exporting Router (given by a B entry). You may specify multiple AS numbers to graph together by providing a comma-separated list as the first argument, as in the B tag above. Also as in B tags, do not put spaces in the comma-separated list, and use unique labels for each ASNumber statement. =item B Add B to your CUFlow.cf file to enable our cheap multicast hack. E.g. : # Log multicast traffic Multicast Unfortunately, in cflow records, multicast traffic always has a nexthop address of 0.0.0.0 and an output interface of 0, meaning by default CUFlow drops it (but counts for purposes of total.rrd). If you enable this option, CUFlow will create protocol_multicast.rrd in OutputDir (and exporter-specific rrd's for any Router statements you have) =item B Each B entry means you are interested in gathering summary statistics for the protocol named in the entry. E.g.: # TCP Protocol 6 tcp Each protocol entry creates an rrd file named protocol_.rrd in B The protocol may be specified either numerically (6), or symbolically (tcp). It may be followed by an optional alias name. If symbolic, it will be resolved via getprotobyname. The rrd file will be named according to the alias, or if one is not present, the name/number supplied. =item B Each B entry means you are interested in gathering summary statistics for traffic whose TOS flag is contained in the range of the entry. E.g.: # Normal TOS 0 normal Each TOS entry creates an rrd file named tos_.rrd in B. The TOS value must be specified numerically. The rrd file will be named according to the alias. Similar to Service tags, you may define ranges or groups of TOS values to record together. E.g.: # first 8 values TOS 0-7 normal This will graph data about all flows with the matching TOS data. TOS values are between 0 and 255 inclusive. =item B This is the directory where the output rrd files will be written. E.g.: # Output to rrds OutputDir rrds =item B The Scoreboard directive is used to keep a running total of the top consumers of resources. It produces an html reports showing the top N (where N is specified in the directive) source addresses that sent the most (bytes, packets, flows) out, and the top N destination addresses that received the most (bytes, packets, flows) from the outside. Its syntax is # Scoreboard Scoreboard 10 /html/reports /html/current.html The above indicates that each table should show the top 10 of its category, to keep past reports in the /html/reports directory, and the latest report should be pointed to by current.html. Within RootDir, we create a directory per day, and within that, a directory per hour of the day. In each of these directories, we write the scoreboard reports. Scoreboarding measures all traffic we get flows for, it is unaffected by any Router statements in your config. =item B The AggregateScore directive indicates that CUFlow should keep running totals for the various Scoreboard categories, and generate an overall report based on them, updating it every time it creates a new Scoreboard file. E.g.: # AggregateScore AggregateScore 10 /html/reports/totals.dat /html/topten.html If you configure this option, you must also turn on Scoreboard. /html/reports/totals.dat is a data file containing an easily machine-readable form of the last several ScoreBoard reports. It then takes each entries average values for every time it has appeared in a ScoreBoard. Then it prints the top NumberToPrint of those. Every 100 samples, it drops all entries that have only appeared once, and divides all the others by 2 (including the number of times they have appeared). So, if a given host were always in the regular ScoreBoard, its appearance count would slowly grow from 50 to 100, then get cut in half, and repeat. This is usefull for trend analysis, as it enables you to see which hosts are *always* using bandwidth, as opposed to outliers and occasional users. AggregateScoreboarding measures all traffic we get flows for, it is unaffected by any Router statements in your config. =item B B is similar to B. It produces a similar graphing service, but shows the top N (where N is specified in the directive) AS'es sending or receiving the most (bytes, packets, flows). Its syntax is # ASScoreboard ASScoreboard 10 /html/reports/AS /html/currentAS.html This example indicates we should keep reports on the top 10 AS'es, keep past reports in the /html/reports/AS directory, and keep a symlink called currentAS.html that points to the latest report. Within RootDir, we create a directory per day, and within that, a directory per hour of the day. In each of these directories, we write the scoreboard reports. If you also have Scoreboard configured, you should use a separate RootDir for the ASScoreboard reports. Like Scoreboard, ASScoreboard measures all traffic we get flows for, it is unaffected by any Router statements in your config. =back =cut use Cflow qw(:flowvars 1.015); # for use in wanted sub use RRDs; # To actually produce results use Socket; # We need inet_aton use Net::Patricia; # Fast IP/mask lookups use POSIX; # We need floor() use FindBin; # To find our executable my(%ROUTERS); # A hash mapping exporter IP's to the name # we want them to be called, e.g. # $ROUTER{"127.0.0.1"} = 'localhost'; my($SUBNETS); # A trie of the subnets that are 'inside' my(%SERVICES); # A hashtable containing services we are # interested in. E.g.: # $SERVICES{'www'} = { 80 => { 6 } } # means that we are interested in www/tcp # and that it has label 'www' my(%PROTOCOLS); # A hashtable containing protocols we are # interested in. E.g.: # $PROTOCOLS{6} = 'tcp'; # means we are interested in protocol 6, and # wish to call the rrd ourput file tcp.rrd my($OUTDIR) = '.'; # The directory we will stow rrd files in my($scorekeep) = 0; # The top N addresses to report on. By default # don't keep scoreboards. my($scoredir) = undef; # The directory to put the tree of reports in my($scorepage) = undef; # The link to the current page my($aggscorekeep) = 0; # Do not create an overall rankings file $CUFlow::NUMKEEP = 50; # How many aggregates to keep $CUFlow::multicast = 0; # Do multicast? Default no. # Multicast address spec's, taken from CampusIO $CUFlow::MCAST_NET = unpack('N', inet_aton('224.0.0.0')); $CUFlow::MCAST_MASK = unpack('N', inet_aton('240.0.0.0')); $CUFlow::SUBNETS = new Net::Patricia || die "Could not create a trie ($!)\n"; &parseConfig("${FindBin::Bin}/CUFlow.cf"); # Read our config file sub parseConfig { my $file = shift; my($ip,$mask,$srv,$proto,$label,$tmp,$txt); my($num,$dir,$current,$start,$end,$hr,$i); open(FH,$file) || die "Could not open $file ($!)\n"; while() { s/\#.*$//; # Strip out everything after a # next if /^\s*$/; # Skip blank lines if (/^\s*Subnet (\d+\.\d+\.\d+\.\d+\/\d+)\s*$/) { # Stick this entry into our trie $CUFlow::SUBNETS->add_string($1); } elsif (/^\s*Router\s+(\d+\.\d+\.\d+\.\d+)\s*(\S+)?\s*$/) { if (defined $2) { $CUFlow::ROUTERS{$1} = $2; } else { $CUFlow::ROUTERS{$1} = $1; } } elsif (/^\s*SampleRate\s+(\d+\.\d+\.\d+\.\d+)\s+(\d+)\s*$/) { $CUFlow::SAMPLERATE{$1} = $2; } elsif (/^\s*Network\s+(\S+)\s+(\S+)\s*$/) { $ip = $1; $label = $2; $CUFlow::NETWORKS{$label} = new Net::Patricia || die "Could not create a trie ($!)\n"; foreach $current (split(/,/,$ip)) { # Parse each item $CUFlow::NETWORKS{$label}->add_string($current); } } elsif (/^\s*Multicast\s*$/) { $CUFlow::multicast = 1; } elsif (/^\s*Service\s+(\S+)\s+(\S+)\s*$/) { $txt = $1; $label = $2; die "Service $label redefined at $.\n" if (defined $CUFlow::SERVICES{$label}); $hr = { }; # New hashref # A Service is one or more port/proto ranges, separated by ,'s foreach $current (split(/,/,$txt)) { # Parse each item if ($current =~ /(\S+)\s*\/\s*(\S+)/) { $srv = $1; $proto = $2; if ($proto !~ /^\d+$/) { # Not an integer! Try getprotobyname $tmp = getprotobyname($proto) || die "Unknown protocol $proto on line $.\n"; $proto = $tmp; } if ($srv =~ /(\d+)-?(\d+)?/) { # Numeric or range $start = $1; $end = (defined($2)) ? $2 :$start; die "Bad range $start - $end on line $.\n" if ($end < $start); for($i=$start;$i<=$end;$i++) { $hr->{$proto}{$i} = 1; # Save all these ports } } else { # Symbolic or bad? if ($srv !~ /^\d+$/) { # Not an integer? # Try getservbyname $tmp = getservbyname($srv, getprotobynumber($proto)) || die "Unknown service $srv on line $.\n"; $srv = $tmp; } $hr->{$proto}{$srv} = 1; } } else { die "Bad Service item $current on line $.\n"; } } $CUFlow::SERVICES{$label} = $hr; } elsif (/^\s*ASNumber\s+(\S+)\s+(\S+)\s*$/) { $txt = $1; $label = $2; die "ASNumber $label redefined at $." if (defined $CUFlow::ASNUMBERS{$label}); $hr = { }; # An ASNumber statement is 1 or more AS numbers, separated by ,'s foreach $current (split(/,/,$txt)) { die "Bad AS number $current on line $.\n" if ($current !~ /^\d+$/); $hr->{$current} = 1; } $CUFlow::ASNUMBERS{$label} = $hr; } elsif (/^\s*TOS\s+(\S+)\s+(\S+)\s*$/) { $txt = $1; $label = $2; $hr = { }; # New hashref # a TOS value can be one or more ranges of ints, separated by ,'s foreach $current (split(/,/,$txt)) { # parse each item if ($current =~ /(\d+)-?(\d+)?/) { # A range $start = $1; $end = (defined($2)) ? $2 :$start; die "Bad range $start - $end on line $.\n" if ($end < $start); die "Bad TOS value $start on line $.\n" if (($start < 0) || ($start > 255)); die "Bad TOS value $end on line $.\n" if (($end < 0) || ($end > 255)); for($i=$start;$i<=$end;$i++) { $hr->{$i} = 1; # Save all these ports } } else { die "Bad TOS item $current on line $.\n"; } } $CUFlow::TOS{$label} = $hr; } elsif (/^\s*Scoreboard\s+(\d+)\s+(\S+)\s+(\S+)\s*$/) { $num = $1; $dir = $2; $current = $3; eval "use HTML::Table"; die "$@" if $@; $CUFlow::scorekeep = $num; $CUFlow::scoredir = $dir; $CUFlow::scorepage = $current; } elsif (/^\s*AggregateScore\s+(\d+)\s+(\S+)\s+(\S+)\s*$/) { $num = $1; $dir = $2; $current = $3; $CUFlow::aggscorekeep = $num; $CUFlow::aggscorefile = $dir; $CUFlow::aggscoreout = $current; } elsif (/^\s*ASScoreboard\s+(\d+)\s+(\S+)\s+(\S+)\s*$/) { $num = $1; $dir = $2; $current = $3; eval "use HTML::Table"; die "$@" if $@; $CUFlow::asScorekeep = $num; $CUFlow::asScoredir = $dir; $CUFlow::asScorepage = $current; } elsif (/^\s*Protocol\s+(\S+)\s*(\S+)?\s*$/) { $proto = $1; $label = $2; if ($proto !~ /\d+/) { # non-numeric protocol name # Try resolving $tmp = getprotobyname($proto) || die "Unknown protocol $proto on line $.\n"; if (defined($label)) { $CUFlow::PROTOCOLS{$tmp} = $label; } else { $CUFlow::PROTOCOLS{$tmp} = $proto; } } else { if (defined($label)) { $CUFlow::PROTOCOLS{$proto} = $label; } else { $CUFlow::PROTOCOLS{$proto} = $proto; } } } elsif (/^\s*OutputDir\s+(\S+)\s*$/) { $CUFlow::OUTDIR = $1; } else { die "Invalid line $. in $file\n\t$_\n"; } } close(FH); } sub new { my $self = {}; my $class = shift; return bless _init($self), $class } sub _init { my $self = shift; $self->{iniplog} = new Net::Patricia || die "Could not create a trie ($!)\n"; $self->{outiplog} = new Net::Patricia || die "Could not create a trie ($!)\n"; return $self } # This is called once per flow record, more than 800k times per file. It # needs to be as fast and as short as possible. sub wanted { my $self = shift; my $which = ''; my $hr; my $samplerate = 1; my $asn; my $lbytes = $bytes; my $lpkts = $pkts; # If this router is set up as a sampled source, normalize its data if (defined($CUFlow::SAMPLERATE{$exporterip})) { $samplerate = $CUFlow::SAMPLERATE{$exporterip}; $lbytes *= $samplerate; $lpkts *= $samplerate; } # Multicast handling here, it seems to have nexthop and # outputIf of zero if (($dstaddr & $CUFlow::MCAST_MASK) == $CUFlow::MCAST_NET) { # it is a multicast packet! # First, are we inbound or outbound? if ($CUFlow::SUBNETS->match_integer($srcaddr)) { $which = 'out'; $asn = $dst_as } else { $which = 'in'; $asn = $src_as } $hr = $self->{exporters}{$exporterip}; # Save stats for every protocol, sort out which to log in sub report() $hr->{'multicast'}{$which}{'flows'} += $samplerate; $hr->{'multicast'}{$which}{'bytes'} += $lbytes; $hr->{'multicast'}{$which}{'pkts'} += $lpkts; # Also update the totals... $hr->{'total'}{$which}{'flows'} += $samplerate; $hr->{'total'}{$which}{'bytes'} += $lbytes; $hr->{'total'}{$which}{'pkts'} += $lpkts; # Now update TOS counters $hr->{'tos'}{$which}{$tos}{'flows'} += $samplerate; $hr->{'tos'}{$which}{$tos}{'bytes'} += $lbytes; $hr->{'tos'}{$which}{$tos}{'pkts'} += $lpkts; # AS statistics $hr->{'as'}{$which}{$asn}{'flows'} += $samplerate; $hr->{'as'}{$which}{$asn}{'bytes'} += $lbytes; $hr->{'as'}{$which}{$asn}{'pkts'} += $lpkts; $self->{exporters}{$exporterip} = $hr; return 1; } # Had to rmeove this because too many cisco's don't fill out these # fields #return 0 if ($nexthop == 0); # Non-routable traffic is dumped. #return 0 if ($output_if == 0); # Dump traffic routed to nowhere # First, are we inbound or outbound? if ($CUFlow::SUBNETS->match_integer($dstaddr)) { # If the destination is in inside, this is an inbound flow # unless the source is also inside, in which case, return silently return 0 if ($CUFlow::SUBNETS->match_integer($srcaddr)); $which = 'in'; $asn = $src_as; # Inbound, which AS is it coming from? # Save stats for scoreboarding if (!($hr = $self->{iniplog}->match_integer($dstaddr))) { $hr = $self->{iniplog}->add_string($dstip, { addr => $dstip, bytes => 0, pkts => 0, flows => 0}); } } elsif ($CUFlow::SUBNETS->match_integer($srcaddr)) { # The source for this flow is in SUBNETS; it is outbound # unless the destination is also inside, in which case, return silently return 0 if ($CUFlow::SUBNETS->match_integer($dstaddr)); $which = 'out'; $asn = $dst_as; # outbound, which AS is it going to? # Save stats for scoreboarding if (!($hr = $self->{outiplog}->match_integer($srcaddr))) { $hr = $self->{outiplog}->add_string($srcip, { addr => $srcip, bytes => 0, pkts => 0, flows => 0}); } } else { # Neither source nor destination is inside, return silently return 0; } $hr->{bytes} += $lbytes; $hr->{pkts} += $lpkts; $hr->{flows} += $samplerate; $hr = $self->{exporters}{$exporterip}; # First update the total counters $hr->{'total'}{$which}{'flows'} += $samplerate; $hr->{'total'}{$which}{'bytes'} += $lbytes; $hr->{'total'}{$which}{'pkts'} += $lpkts; # Now update TOS counters $hr->{'tos'}{$which}{$tos}{'flows'} += $samplerate; $hr->{'tos'}{$which}{$tos}{'bytes'} += $lbytes; $hr->{'tos'}{$which}{$tos}{'pkts'} += $lpkts; # Save stats for every protocol, sort out which to log in sub report() $hr->{$protocol}{'total'}{$which}{'flows'} += $samplerate; $hr->{$protocol}{'total'}{$which}{'bytes'} += $lbytes; $hr->{$protocol}{'total'}{$which}{'pkts'} += $lpkts; # Next update service counters in the same fashion $hr->{$protocol}{'src'}{$srcport}{$which}{'flows'} += $samplerate; $hr->{$protocol}{'src'}{$srcport}{$which}{'bytes'} += $lbytes; $hr->{$protocol}{'src'}{$srcport}{$which}{'pkts'} += $lpkts; $hr->{$protocol}{'dst'}{$dstport}{$which}{'flows'} += $samplerate; $hr->{$protocol}{'dst'}{$dstport}{$which}{'bytes'} += $lbytes; $hr->{$protocol}{'dst'}{$dstport}{$which}{'pkts'} += $lpkts; # Save Data on a per-AS basis $hr->{'as'}{$which}{$asn}{'flows'} += $samplerate; $hr->{'as'}{$which}{$asn}{'bytes'} += $lbytes; $hr->{'as'}{$which}{$asn}{'pkts'} += $lpkts; $self->{exporters}{$exporterip} = $hr; return 1; } sub perfile { # Only do this, so we get the filetime from our super-class my $self = shift; $CUFlow::totals = (); # Clear this out $self->SUPER::perfile(@_); } sub report { my $self = shift; my($file) = $CUFlow::OUTDIR . "/total.rrd"; my($routerfile); my(@values) = (); my(@array); my($hr,$count, $i, $tmp); # Zeroth, this takes a lot of time. So let's fork and return if (fork()) { # We are the parent, wait wait; # This wait will finish quit, as our immediate # child just forks and dies. It's kid does the # work. Saves us fiddling with signals. return; # Then go back!!! } else { # We are the child, fork again exit if (fork()); } # Make sure directories exist foreach my $k (keys %CUFlow::ROUTERS) { if (! -d "$CUFlow::OUTDIR/" . $CUFlow::ROUTERS{$k} ) { mkdir("$CUFlow::OUTDIR/" . $CUFlow::ROUTERS{$k},0755); } } # Compute total values for my $dir ('in','out') { for my $type ('pkts','bytes','flows') { $i = 0; for my $exporter (keys %{$self->{exporters}}) { $i += $self->{exporters}{$exporter}{'total'}{$dir}{$type}; } $CUFlow::totals{$dir}{$type} = $i; } } # First, always generate a totals report # createGeneralRRD we get from our parent, FlowScan # Create a new rrd if one doesn't exist $self->createGeneralRRD($file, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $file; foreach my $i ('bytes','pkts','flows') { foreach my $j ('in','out') { push(@values, $CUFlow::totals{$j}{$i}); } } $self->updateRRD($file, @values); # Now do totals per-exporter foreach my $exporter (keys %CUFlow::ROUTERS) { @values = (); $routerfile = $CUFlow::OUTDIR . "/" . $CUFlow::ROUTERS{$exporter} . "/total.rrd"; $self->createGeneralRRD($routerfile, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $routerfile; foreach my $i ('bytes','pkts','flows') { foreach my $j ('in','out') { $hr = $self->{exporters}{$exporter}; push(@values, 0 + $hr->{'total'}{$j}{$i}); } } $self->updateRRD($routerfile, @values); } # Second, Multicast? # createGeneralRRD we get from our parent, FlowScan # Create a new rrd if one doesn't exist if ($CUFlow::multicast == 1) { $file = $CUFlow::OUTDIR . "/protocol_multicast.rrd"; $self->createGeneralRRD($file, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $file; @values = (); foreach my $i ('bytes','pkts','flows') { foreach my $j ('in','out') { $count = 0; foreach my $k (keys %{$self->{exporters}}) { $count += $self->{exporters}{$k}{'multicast'}{$j}{$i}; } push(@values,$count); } } $self->updateRRD($file, @values); # Now do totals per-exporter foreach my $exporter (keys %CUFlow::ROUTERS) { @values = (); $routerfile = $CUFlow::OUTDIR . "/" . $CUFlow::ROUTERS{$exporter} . "/protocol_multicast.rrd"; $self->createGeneralRRD($routerfile, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $routerfile; foreach my $i ('bytes','pkts','flows') { foreach my $j ('in','out') { $hr = $self->{exporters}{$exporter}; push(@values, 0 + $hr->{'multicast'}{$j}{$i}); } } $self->updateRRD($routerfile, @values); } } # Now do an AS report for each AS we are interested in for each exporter foreach my $label (keys %CUFlow::ASNUMBERS) { @values = (); # Initialize values to insert $hr = $CUFlow::ASNUMBERS{$label}; # First, total values $file = $CUFlow::OUTDIR . "/as_" . $label . ".rrd"; $self->createGeneralRRD($file, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $file; foreach my $type ('bytes','pkts','flows') { foreach my $dir ('in','out') { # Sum all the AS values here $tmp = 0; foreach my $exporter (keys %{$self->{exporters}}) { foreach my $current (keys %$hr) { $tmp += $self->{exporters}{$exporter}{as}{$dir}{$current}{$type}; } } push(@values,$tmp); } } $self->updateRRD($file, @values); # Next, per-exporter totals foreach my $exporter (keys %CUFlow::ROUTERS) { @values = (); $routerfile = $CUFlow::OUTDIR . "/" . $CUFlow::ROUTERS{$exporter} . "/as_" . $label . ".rrd"; $self->createGeneralRRD($routerfile, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $routerfile; foreach my $type ('bytes','pkts','flows') { foreach my $dir ('in','out') { # Sum counter for all AS'es in this label $tmp = 0; foreach my $current (keys %$hr) { $tmp += $self->{exporters}{$exporter}{as}{$dir}{$current}{$type}; } push(@values,$tmp); } } $self->updateRRD($routerfile, @values); } } # Next, generate TOS reports. Each label gets one rrd. foreach my $tos (keys %CUFlow::TOS) { @values = (); $file = $CUFlow::OUTDIR . "/tos_" . $tos . ".rrd"; $hr = $CUFlow::TOS{$tos}; $self->createGeneralRRD($file, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $file; foreach my $type ('bytes','pkts','flows') { foreach my $dir ('in','out') { # Sum counter for all TOS values in this label $tmp = 0; foreach my $exporter (keys %{$self->{exporters}}) { foreach my $current (keys %$hr) { $tmp += $self->{exporters}{$exporter}{tos}{$dir}{$current}{$type}; } } push(@values,$tmp); } } $self->updateRRD($file, @values); # Now do the same thing for each exporter. foreach my $exporter (keys %CUFlow::ROUTERS) { @values = (); $routerfile = $CUFlow::OUTDIR . "/" . $CUFlow::ROUTERS{$exporter} . "/tos_" . $tos . ".rrd"; $self->createGeneralRRD($routerfile, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $routerfile; foreach my $type ('bytes','pkts','flows') { foreach my $dir ('in','out') { # Sum counter for all services in this label $tmp = 0; foreach my $current (keys %$hr) { $tmp += $self->{exporters}{$exporter}{tos}{$dir}{$current}{$type}; } push(@values,$tmp); } } $self->updateRRD($routerfile, @values); } } # Next, see if we need to generate any per-protocol reports # Each protocol gets one rrd. foreach my $proto (keys %CUFlow::PROTOCOLS) { @values = (); $file = $CUFlow::OUTDIR . "/protocol_" . $CUFlow::PROTOCOLS{$proto} . ".rrd"; $self->createGeneralRRD($file, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $file; foreach my $i ('bytes','pkts','flows') { foreach my $j ('in','out') { $count = 0; foreach my $k (keys %{$self->{exporters}}) { $hr = $self->{exporters}{$k}; $count += $hr->{$proto}{'total'}{$j}{$i}; } push(@values, $count); } } $self->updateRRD($file, @values); # Now do totals per-exporter foreach my $exporter (keys %CUFlow::ROUTERS) { @values = (); $routerfile = $CUFlow::OUTDIR . "/" . $CUFlow::ROUTERS{$exporter} . "/protocol_" . $CUFlow::PROTOCOLS{$proto} . ".rrd"; $self->createGeneralRRD($routerfile, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $routerfile; foreach my $i ('bytes','pkts','flows') { foreach my $j ('in','out') { $hr = $self->{exporters}{$exporter}; push(@values, 0 + $hr->{$proto}{'total'}{$j}{$i}); } } $self->updateRRD($routerfile, @values); } } # Next, see which services require rrd files. Each one gets 2 files, # one for src port traffic, one for dst port traffic foreach my $label (keys %CUFlow::SERVICES) { $hr = $CUFlow::SERVICES{$label}; foreach my $direction ('src','dst') { @values = (); $file = $CUFlow::OUTDIR . "/service_$label" . "_$direction.rrd"; # createGeneralRRD we get from our parent, FlowScan # Create a new rrd if one doesn't exist $self->createGeneralRRD($file, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $file; foreach my $type ('bytes','pkts','flows') { foreach my $dir ('in','out') { # Sum counter for all services in this label $tmp = 0; foreach my $exporter (keys %{$self->{exporters}}) { foreach my $proto (keys %$hr) { foreach my $service (keys %{ $hr->{$proto} }) { # This is the worst thing ever. $tmp += $self->{exporters}{$exporter}{$proto}{$direction}{$service}{$dir}{$type}; } } } push(@values,$tmp); } } $self->updateRRD($file, @values); # Now do the same thing for each exporter. foreach my $exporter (keys %CUFlow::ROUTERS) { @values = (); $routerfile = $CUFlow::OUTDIR . "/" . $CUFlow::ROUTERS{$exporter} . "/service_$label" . "_$direction.rrd"; $self->createGeneralRRD($routerfile, qw( ABSOLUTE in_bytes ABSOLUTE out_bytes ABSOLUTE in_pkts ABSOLUTE out_pkts ABSOLUTE in_flows ABSOLUTE out_flows ) ) unless -f $routerfile; foreach my $type ('bytes','pkts','flows') { foreach my $dir ('in','out') { # Sum counter for all services in this label $tmp = 0; foreach my $proto (keys %$hr) { foreach my $service (keys %{ $hr->{$proto} }) { # This is the worst thing ever also. $tmp += $self->{exporters}{$exporter}{$proto}{$direction}{$service}{$dir}{$type}; } } push(@values,$tmp); } } $self->updateRRD($routerfile, @values); } } } # Should we do scoreboarding? if ($CUFlow::scorekeep > 0) { # Yes. $self->scoreboard(); } # Should we do AS scoreboarding? if ($CUFlow::asScorekeep > 0) { # Yes. $self->asScoreboard(); } # Per-network reporting? # # You need to iterate over all the IP's for each Network group because # You may have some IP's that match more than 1 network statement foreach my $label (keys %CUFlow::NETWORKS) { @values = (); $tmp = $CUFlow::NETWORKS{$label}; $file = $CUFlow::OUTDIR . "/network_" . "$label.rrd"; $self->createGeneralRRD($file, qw( ABSOLUTE in_bytes ABSOLUTE in_pkts ABSOLUTE in_flows ABSOLUTE out_bytes ABSOLUTE out_pkts ABSOLUTE out_flows ) ) unless -f $file; foreach my $dir ('in','out') { @array = (); $self->{$dir . "iplog"}->climb(sub {push(@array, $_[0]) if $tmp->match_string( $_[0]->{addr}) } ); foreach my $type ('bytes','pkts','flows') { $i = 0; foreach my $ip (@array) { $i += $ip->{$type}; } push(@values,$i); } } $self->updateRRD($file, @values); } exit 0; # We forked, no returning. } # Lifted totally and shamelessly from CampusIO.pm # I think perhaps this goes into FlowScan.pm, but... sub updateRRD { my $self = shift; my $file = shift; my @values = @_; RRDs::update($file, $self->{filetime} . ':' . join(':', @values)); my $err=RRDs::error; warn "ERROR updating $file: $err\n" if ($err); } # Function to read in the current aggregate data # Returns a hash of ip to (count of times in top ten, ttl bytes in, # ttl pkts in, ttl flows in, ttl bytes out, # ttl pkts out, ttl flows out sub readAggFile { my($ip,$cnt,$bin,$pin,$fin,$bout,$pout,$fout); my(%ret) = (); if (-f $CUFlow::aggscorefile) { # Exists, try reading it in open(AGG,$CUFlow::aggscorefile) || die "Cannot open $CUFlow::aggscorefile ($!)\n"; $ret{'numresults'} = ; chomp($ret{'numresults'}); while() { if ( ($ip,$cnt,$bin,$pin,$fin,$bout,$pout,$fout) = (/(\d+\.\d+\.\d+\.\d+) (\d+) (\d+) (\d+) (\d+) (\d+) (\d+) (\d+)/)) { # Skip any data that has rolled over if (($cnt < 0) || ($bin < 0) || ($bout < 0) || ($pin < 0) || ($pout < 0) || ($fin < 0) || ($fout < 0)) { print STDERR "Rollover for $ip\n"; next; # Skip it } $ret{$ip} = { 'count' => $cnt, 'bytesin' => $bin, 'bytesout' => $bout, 'pktsin' => $pin, 'pktsout' => $pout, 'flowsin' => $fin, 'flowsout' => $fout }; } } close AGG; } return %ret; } # Function to write the aggregate data out to a file sub writeAggFile (\%) { my %data = %{(shift)}; open(OUT,">$CUFlow::aggscorefile") || die "Cannot open $CUFlow::aggscorefile for write ($!)\n"; print OUT $data{'numresults'} . "\n"; foreach my $ip (keys %data) { next if ($ip =~ /numresults/); printf OUT "%s %d %d %d %d %d %d %d\n", $ip, $data{$ip}->{'count'}, $data{$ip}->{'bytesin'}, $data{$ip}->{'pktsin'}, $data{$ip}->{'flowsin'}, $data{$ip}->{'bytesout'}, $data{$ip}->{'pktsout'}, $data{$ip}->{'flowsout'}; } close OUT; } # Function to print the pretty table of over-all winners sub writeAggScoreboard (\%) { my %data = %{(shift)}; my($dir,$key, $i); my(@sorted); my(%dnscache); my($tmp) = $data{'numresults'}; delete $data{'numresults'}; open(OUT,">$CUFlow::aggscoreout") || die "Cannot open $CUFlow::aggscoreout for write ($!)\n"; print OUT "\n\n\n
\n"; print OUT "

Average rankings for the last $tmp topN reports\n
\n"; print OUT "

\n"; # Now, print out our 6 topN tables my %columns = ('bytes' => 3, 'pkts' => 5, 'flows' => 7); foreach my $dir ('in','out') { foreach my $key ('bytes','pkts','flows') { @sorted = sort { ($data{$b}->{"$key$dir"} / $data{$b}->{'count'}) <=> ($data{$a}->{"$key$dir"} / $data{$a}->{'count'}) } (keys %data); my $table = new 'HTML::Table'; die unless ref($table); $table->setBorder(1); $table->setCellSpacing(0); $table->setCellPadding(3); $table->setCaption("Top $CUFlow::aggscorekeep by " . "$key $dir
\n" . "built on aggregated topN " . "5 minute average samples to date", 'TOP'); my $row = 1; $table->addRow('rank', "$dir Address", 'bits/sec in', 'bits/sec out', 'pkts/sec in', 'pkts/sec out', 'flows/sec in', 'flows/sec out'); $table->setRowBGColor($row, '#FFFFCC'); # pale yellow # Highlight the current column (out is 1 off from in) $table->setCellBGColor($row, $columns{$key} + ('out' eq $dir), '#90ee90'); # light green $row++; for($i=0;$i < @sorted; $i++) { last unless $i < $CUFlow::aggscorekeep; my $ip = $sorted[$i]; if (!(defined($dnscache{$ip}))) { # No name here? if ($dnscache{$ip} = gethostbyaddr(pack("C4", split(/\./, $ip)), AF_INET)) { $dnscache{$ip} .= "
$ip (" . $data{$ip}->{'count'} . " samples)"; } else { $dnscache{$ip} = $ip . " (" . $data{$ip}->{'count'} . " samples)"; } } my $div = 300 * $data{$ip}->{'count'}; $table->addRow( sprintf("#%d",$i+1), $dnscache{$ip}, # IP Name/Address # Bits/sec in scale("%.1f", ($data{$ip}->{'bytesin'}*8) / $div), # Bits/sec out scale("%.1f", ($data{$ip}->{'bytesout'}*8) / $div), # Pkts/sec in scale("%.1f", ($data{$ip}->{'pktsin'}/$div)), # Pkts/sec out scale("%.1f", ($data{$ip}->{'pktsout'}/$div)), # Flows/sec in scale("%.1f", ($data{$ip}->{'flowsin'}/$div)), # Flows/sec out scale("%.1f", ($data{$ip}->{'flowsout'}/$div))); $table->setRowAlign($row, 'RIGHT'); $table->setCellBGColor($row, $columns{$key} + ('out' eq $dir), '#add8e6'); # light blue $row++; } print OUT "

\n$table

\n\n"; } } close OUT; $data{'numresults'} = $tmp; } # Handle writing our HTML scoreboard reports sub scoreboard { my $self = shift; my($i,$file,$ip,$hr); my (@values, @sorted); my(%dnscache) = (); my(%aggdata, %newaggdata); # First, should we read in the aggregate data? if ($CUFlow::aggscorekeep > 0) { %aggdata = &readAggFile(); } # Next, open the file, making any necessary directories my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($self->{filetime}); $mon++; $year += 1900; $file=sprintf("%s/%4.4d-%2.2d-%2.2d",$CUFlow::scoredir,$year,$mon,$mday); if (! -d $file) { mkdir($file,0755) || die "Cannot mkdir $file ($!)\n"; } $file = sprintf("%s/%2.2d",$file,$hour); if (! -d $file) { mkdir($file,0755) || die "Cannot mkdir $file ($!)\n"; } $file = sprintf("%s/%2.2d:%2.2d:%2.2d.html",$file,$hour,$min,$sec); open(HTML,">$file") || die "Could not write to $file ($!)\n"; # Now, print out our header stuff into the file print HTML "\n\n
\n\n"; # Now, print out our 6 topN tables my %columns = ('bytes' => 3, 'pkts' => 5, 'flows' => 7); foreach my $dir ('in','out') { @values = (); # fill values with hashrefs $self->{$dir . "iplog"}->climb(sub {push(@values, $_[0]) }); foreach my $key ('bytes','pkts','flows') { @sorted = sort { $b->{$key} <=> $a->{$key} } @values; # This part lifted totally from CampusIO.pm. Thanks, dave! my $table = new 'HTML::Table'; die unless ref($table); $table->setBorder(1); $table->setCellSpacing(0); $table->setCellPadding(3); $table->setCaption("Top $CUFlow::scorekeep by " . "$key $dir
\n" . "for five minute flow sample ending " . scalar(localtime($self->{filetime})), 'TOP'); my $row = 1; $table->addRow('rank', "$dir Address", 'bits/sec in', 'bits/sec out', 'pkts/sec in', 'pkts/sec out', 'flows/sec in', 'flows/sec out'); $table->setRowBGColor($row, '#FFFFCC'); # pale yellow # Highlight the current column (out is 1 off from in) $table->setCellBGColor($row, $columns{$key} + ('out' eq $dir), '#90ee90'); # light green $row++; # Get the in and out hr's for ease of use my ($in, $out); for($i=0;$i < @sorted; $i++) { last unless $i < $CUFlow::scorekeep; $ip = $sorted[$i]->{addr}; $in = $self->{"iniplog"}->match_string($ip); $out = $self->{"outiplog"}->match_string($ip); if (!(defined($newaggdata{$ip}))) { # Add this to aggdata 1x $newaggdata{$ip} = { 'bytesin' => $in->{bytes}, 'bytesout' => $out->{bytes}, 'pktsin' => $in->{pkts}, 'pktsout' => $out->{pkts}, 'flowsin' => $in->{flows}, 'flowsout' => $out->{flows} }; } if (!(defined($dnscache{$ip}))) { # No name here? if ($dnscache{$ip} = gethostbyaddr(pack("C4", split(/\./, $ip)), AF_INET)) { $dnscache{$ip} .= "
$ip"; } else { $dnscache{$ip} = $ip; } } $table->addRow( sprintf("#%d",$i+1), $dnscache{$ip}, # IP Name/Address # Bits/sec in scale("%.1f", ($in->{bytes}*8)/300) . sprintf(" (%.1f%%)", percent($in->{bytes}, $CUFlow::totals{in}{bytes})), # Bits/sec out scale("%.1f", ($out->{bytes}*8)/300) . sprintf(" (%.1f%%)", percent($out->{bytes}, $CUFlow::totals{out}{bytes})), # Pkts/sec in scale("%.1f", ($in->{pkts}/300)) . sprintf(" (%.1f%%)", percent($in->{pkts}, $CUFlow::totals{in}{pkts})), # Pkts/sec out scale("%.1f", ($out->{pkts}/300)) . sprintf(" (%.1f%%)", percent($out->{pkts}, $CUFlow::totals{out}{pkts})), # Flows/sec in scale("%.1f", ($in->{flows}/300)) . sprintf(" (%.1f%%)", percent($in->{flows}, $CUFlow::totals{in}{flows})), # Flows/sec out scale("%.1f", ($out->{flows}/300)) . sprintf(" (%.1f%%)", percent($out->{flows}, $CUFlow::totals{out}{flows}))); $table->setRowAlign($row, 'RIGHT'); $table->setCellBGColor($row, $columns{$key} + ('out' eq $dir), '#add8e6'); # light blue $row++; } print HTML "

\n$table

\n\n"; } } # Print footers print HTML "\n
\n\n\n"; # Close the file, and make $scorepage point at this page close HTML; unlink $CUFlow::scorepage || die "Could not remove $CUFlow::scorepage ($!)\n"; symlink $file, $CUFlow::scorepage || die "Could not create symlink to $CUFlow::scorepage ($!)\n"; if ($CUFlow::aggscorekeep > 0) { # Merge newaggdata and aggdata foreach $ip (keys %newaggdata) { $aggdata{$ip}->{'count'}++; $aggdata{$ip}->{'bytesin'} += $newaggdata{$ip}->{'bytesin'}; $aggdata{$ip}->{'bytesout'} += $newaggdata{$ip}->{'bytesout'}; $aggdata{$ip}->{'pktsin'} += $newaggdata{$ip}->{'pktsin'}; $aggdata{$ip}->{'pktsout'} += $newaggdata{$ip}->{'pktsout'}; $aggdata{$ip}->{'flowsin'} += $newaggdata{$ip}->{'flowsin'}; $aggdata{$ip}->{'flowsout'} += $newaggdata{$ip}->{'flowsout'}; } # Increment counter $aggdata{'numresults'}++; if ($aggdata{'numresults'} > $CUFlow::NUMKEEP) { # Prune this shit $aggdata{'numresults'} >>= 1; foreach $ip (keys %aggdata) { next if ($ip =~ /numresults/); # Skip this, not a ref if ($aggdata{$ip}->{'count'} == 1) { # Delete singletons delete $aggdata{$ip}; } else { $aggdata{$ip}->{'count'} >>= 1; # Divide by 2 $aggdata{$ip}->{'bytesin'} >>= 1; $aggdata{$ip}->{'bytesout'} >>= 1; $aggdata{$ip}->{'pktsin'} >>= 1; $aggdata{$ip}->{'pktsout'} >>= 1; $aggdata{$ip}->{'flowsin'} >>= 1; $aggdata{$ip}->{'flowsout'} >>= 1; } } } # Write the aggregate table &writeAggScoreboard(\%aggdata); # Save the aggregation data &writeAggFile(\%aggdata); } return; } # write out a scoreboard file for the topN AS'es. sub asScoreboard { my $self = shift; my $file; my @values; # Open the file, making any necessary directories my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($self->{filetime}); $mon++; $year += 1900; $file=sprintf("%s/%4.4d-%2.2d-%2.2d",$CUFlow::asScoredir,$year,$mon,$mday); if (! -d $file) { mkdir($file,0755) || die "Cannot mkdir $file ($!)\n"; } $file = sprintf("%s/%2.2d",$file,$hour); if (! -d $file) { mkdir($file,0755) || die "Cannot mkdir $file ($!)\n"; } $file = sprintf("%s/%2.2d:%2.2d:%2.2d.html",$file,$hour,$min,$sec); open(HTML,">$file") || die "Could not write to $file ($!)\n"; # Now, print out our header stuff into the file print HTML "\n\n
\n\n"; # Now, print out our 6 topN tables my %columns = ('bytes' => 3, 'pkts' => 5, 'flows' => 7); # Find the top AS'es # First, build totals over all exporters my $totals; foreach my $router (keys %{$self->{'exporters'}}) { foreach my $dir ('in','out') { foreach my $as (keys %{$self->{'exporters'}{$router}{'as'}{$dir}}) { foreach my $key ('bytes','pkts','flows') { $totals->{$as}{$dir}{$key} += $self->{'exporters'}{$router}{'as'}{$dir}{$as}{$key}; } } } } foreach my $dir ('in','out') { foreach my $key ('bytes','pkts','flows') { my(@sorted) = sort { $totals->{$b}{$dir}{$key} <=> $totals->{$a}{$dir}{$key} } (keys %{$totals}); # This part lifted totally from CampusIO.pm. Thanks, dave! my $table = new 'HTML::Table'; die unless ref($table); $table->setBorder(1); $table->setCellSpacing(0); $table->setCellPadding(3); $table->setCaption("Top $CUFlow::asScorekeep by " . "$key $dir
\n" . "for five minute flow sample ending " . scalar(localtime($self->{filetime})), 'TOP'); my $row = 1; $table->addRow('rank', "$dir AS", 'bits/sec in', 'bits/sec out', 'pkts/sec in', 'pkts/sec out', 'flows/sec in', 'flows/sec out'); $table->setRowBGColor($row, '#FFFFCC'); # pale yellow # Highlight the current column (out is 1 off from in) $table->setCellBGColor($row, $columns{$key} + ('out' eq $dir), '#90ee90'); # light green $row++; for(my $i=0;$i < @sorted; $i++) { last unless $i < $CUFlow::asScorekeep; my($in, $out); $in = $totals->{ $sorted[$i] }{'in'}; $out = $totals->{ $sorted[$i] }{'out'}; $table->addRow( sprintf("#%d",$i+1), $sorted[$i], # IP Name/Address # Bits/sec in scale("%.1f", ($in->{bytes}*8)/300) . sprintf(" (%.1f%%)", percent($in->{bytes}, $CUFlow::totals{in}{bytes})), # Bits/sec out scale("%.1f", ($out->{bytes}*8)/300) . sprintf(" (%.1f%%)", percent($out->{bytes}, $CUFlow::totals{out}{bytes})), # Pkts/sec in scale("%.1f", ($in->{pkts}/300)) . sprintf(" (%.1f%%)", percent($in->{pkts}, $CUFlow::totals{in}{pkts})), # Pkts/sec out scale("%.1f", ($out->{pkts}/300)) . sprintf(" (%.1f%%)", percent($out->{pkts}, $CUFlow::totals{out}{pkts})), # Flows/sec in scale("%.1f", ($in->{flows}/300)) . sprintf(" (%.1f%%)", percent($in->{flows}, $CUFlow::totals{in}{flows})), # Flows/sec out scale("%.1f", ($out->{flows}/300)) . sprintf(" (%.1f%%)", percent($out->{flows}, $CUFlow::totals{out}{flows}))); $table->setRowAlign($row, 'RIGHT'); $table->setCellBGColor($row, $columns{$key} + ('out' eq $dir), '#add8e6'); # light blue $row++; } print HTML "

\n$table

\n\n"; } } # Print footers print HTML "\n
\n\n\n"; # Close the file, and make $asScorepage point at this page close(HTML); unlink $CUFlow::asScorepage || die "Could not remove $CUFlow::asScorepage ($!)\n"; symlink $file, $CUFlow::asScorepage || die "Could not create symlink to $CUFlow::asScorepage ($!)\n"; } # Simple percentifier, usage percent(1,10) returns 10 # Also stolen from CampusIO.pm sub percent($$) { my $num = shift; my $denom = shift; return(0) if (0 == $denom); return 100*($num/$denom) } # Print a large number in sensible units. # Arg1 = sprintf format string # Arg2 = value to put in it. # Also stolen from CampusIO.pm, where Dave Plonka says... # This is based somewhat on Tobi Oetiker's code in rrd_graph.c: sub scale($$) { my $fmt = shift; my $value = shift; my @symbols = ("a", # 10e-18 Ato "f", # 10e-15 Femto "p", # 10e-12 Pico "n", # 10e-9 Nano "u", # 10e-6 Micro "m", # 10e-3 Milli " ", # Base "k", # 10e3 Kilo "M", # 10e6 Mega "G", # 10e9 Giga "T", # 10e12 Terra "P", # 10e15 Peta "E");# 10e18 Exa my $symbcenter = 6; my $digits = (0 == $value)? 0 : floor(log($value)/log(1000)); return sprintf(${fmt} . " %s", $value/pow(1000, $digits), $symbols[$symbcenter+$digits]) } sub DESTROY { my $self = shift; $self->SUPER::DESTROY } =head1 BUGS Some. Majorly, flows that have the same source and destination port will end up being counted twice (in either the inbound or outbound direction) for the purposes of measuring the percentages in CUGrapher. The total traffic is correct, but the assumptions CUGrapher makes lead to a negative percentage of "other traffic". This may just be a cosmetic bug with CUGrapher. If an inside host transfers 5megs to an outside host from port 80 to port 80, we need to update both counters. The bug is in assuming that the sum of all the services totals will be less than the total traffic, which may not be the case if some traffic belongs to more than 1 service. More stuff for 2.0... scoreboard() assumes our flowfile is 300 seconds worth of data. It should figure out over how many seconds the records run, and divide by that instead. report and its subroutines need to do locking =head1 AUTHOR Johan Andersen =head1 CONTRIBUTORS Matt Selsky - CUGrapher and co-developer Terje Krogdahl - Sampled netflow support and AS graphing Joerg Borchain - CUGrapher menu on displayed graphs page =head1 REPORT PROBLEMS Please contact to get help with CUFlow. =cut 1 CUFlow-1.7/CUGrapher.pl0000755011555000000140000011324410361016516015023 0ustar selskysrc00000000000000#! /usr/bin/perl -w # CUGrapher.pl # $Revision: 1.53 $ # Author: Matt Selsky # Contact for help: # (c) 2002 - 2005 The Trustees of Columbia University in the City of New York # License restrictions apply, see COPYING for details. use strict; use CGI::Pretty qw(-nosticky :standard); use RRDs; use Digest::MD5 qw(md5_hex); ### Local settings ### # directory with rrd files my $rrddir = "/cflow/reports/rrds"; # default number of hours to go back my $hours = 48; # duration of graph, starting from $hours ago my $duration; # organization name my $organization = "Estimated Columbia University Campus"; # default graph width my $width = 640; # default graph height my $height = 320; # default image type (png/gif) my $imageType = 'png'; ### End local settings ### # auto-flush STDOUT $| = 1; # report -> proper name my %reportName = ( 'bits' => 'bits', 'pkts' => 'packets', 'flows' => 'flows' ); unless( param() ) { &showMenu(); } if (param('showmenu')) { Delete('showmenu'); &showMenu(self_url()); } # protocol/service -> filename my %filename; # lists of networks/protocols/services/routers my (%network, %as, %protocol, %service, %tos, @router); # should we show totals also? my %total; # hash for colors/CDEFs my (%color, %cdef); # are we in debug mode? my $debug; &getRouter(); &getProtocols(); &getServices(); &getTOS(); &getNetworks(); &getASs(); &getImageType(); &setColors(); &getHours(); &getDuration(); &getWidth(); &getHeight(); &getTotal(); &getDebug(); &doReport(); ################################################################ # Image generation and display sub generateImage { my @args = @_; my ($err1, $err2); unless( $debug ) { print header( -type => "image/${imageType}", -expires => 'now' ); RRDs::graph( '-', @args); $err1 = RRDs::error; if( $err1 ) { # log & attempt to create image w/ error message in it warn $err1, "\n"; RRDs::graph( '-', "--width=".$width, "COMMENT:graph failed ($err1)\\n"); $err2 = RRDs::error; if( $err2 ) { # log to stderr since we are totally broken warn "graph failed ($err1) and warning failed ($err2)\n"; } } } else { print header, '
', join(" \\\n", @args), "\n"; exit;
    }
}

sub showMenu {
    my ($imgurl) = @_;
    my $q = new CGI::Pretty(""); # Avoid inheriting the parameter list
    

    print $q->header, $q->start_html( -title => 'Generate FlowScan graphs on the fly',
				      -bgcolor => 'ffffff' );

    if ($imgurl) {
	if ($imgurl =~ /debug=1/) {
	    (my $cleanimgurl = $imgurl) =~ s/debug=1//;
	    print $q->center( $q->a( { href => $imgurl},
				     $q->img({ -src => $cleanimgurl, -align => 'center', 
					       -alt => 'The Graph you requested'}) ) );

	} else {
	    print $q->center( $q->img({ -src => $imgurl, -align => 'center', 
					-alt => 'The Graph you requested'}));
	}
    }
    
    print $q->start_form( -action => $q->url(), -method => 'get' ); # Just the url, without query string

    print $q->start_table( { -align => 'center',
			     -cellspacing => '10' } );

    print $q->start_Tr( { -align => 'center',
			  -valign => 'top' } );

    print $q->td( { -rowspan => '2' },
		  "Report: ",
		  $q->popup_menu( -name => 'report',
				  -values => [sort keys %reportName],
				  -default => '' ) );
    
    my %hours = ( 24 => '24 hours',
		  36 => '36 hours',
		  48 => '48 hours',
		  168 => '1 week',
		  720 => '1 month' );

    print $q->td( { -align => 'right' },
		  "Time period: ",
		  $q->popup_menu( -name => 'hours',
				  -values => [sort {$a <=> $b} keys %hours],
				  -default => $hours,
				  -labels => \%hours ) );
    
    print $q->td( { -rowspan => '2' },
		  "Image type: ",
		  $q->popup_menu( -name => 'imageType',
				  -values => ['png', 'gif'],
				  -default => 'png' ) );
    
    print $q->td( { -rowspan => '2' },
		  "Width:",
		  $q->textfield( -name => "width",
				 -default => $width,
				 -size => 7 ) );
    
    print $q->td( { -rowspan => '2' },
		  "Height:",
		  $q->textfield( -name => "height",
				 -default => $height,
				 -size => 7 ) );

    print $q->end_Tr();

    print $q->start_Tr( { -align => 'center' } );

    print $q->td( { -align => 'right' },
		  "Duration: ",
		  $q->popup_menu( -name => 'duration',
				  -values => ['', sort {$a <=> $b} keys %hours],
				  -labels => \%hours ) );

    print $q->end_Tr();

    print $q->end_table();

    print $q->start_table( { align => 'center',
			     -border => '1' } );
    my @headerTableRow;
    push @headerTableRow, $q->td( i('Router') );
    push @headerTableRow, $q->td( i('Protocol') ), $q->td( i('All Protos') );
    push @headerTableRow, $q->td( i('Service') ), $q->td( i('All Svcs') );
    if (scalar &getTOSList()) {
	push @headerTableRow, $q->td( i('TOS') ), $q->td( i('All TOS') );
    }
    if (scalar &getASList()) {
	push @headerTableRow, $q->td( i('AS') ), $q->td( i('All ASes') );
    }
    push @headerTableRow, $q->td( i('Network') );
    push @headerTableRow, $q->td( i('Total') );

    print $q->Tr( { -align => 'center' }, @headerTableRow );

    foreach my $router ( 'all', sort &getRouterList() ) {
	print $q->start_Tr;

	print $q->td( { -align => 'center' }, $q->b($router),
		      $q->hidden( -name => 'router', -default => $router ) );

	print $q->td( $q->scrolling_list( -name => "${router}_protocol",
					  -values => [sort &getProtocolList()],
					  -size => 5,
					  -multiple => 'true' ) );

	print $q->td( $q->checkbox( -name => "${router}_all_protocols",
				    -value => '1',
				    -label => 'Yes' ) );
	
	print $q->td( $q->scrolling_list( -name => "${router}_service",
					  -values => [sort &getServiceList()],
					  -size => 5,
					  -multiple => 'true' ) );

	print $q->td( $q->checkbox( -name => "${router}_all_services",
				    -value => '1',
				    -label => 'Yes' ) );

	if (scalar &getTOSList()) {
	    print $q->td( $q->scrolling_list( -name => "${router}_tos",
					      -values => [sort &getTOSList()],
					      -size => 5,
					      -multiple => 'true' ) );

	    print $q->td( $q->checkbox( -name => "${router}_all_tos",
					-value => '1',
					-label => 'Yes' ) );
	}

	if (scalar &getASList()) {
	    print $q->td( $q->scrolling_list( -name => "${router}_as",
					      -values => [sort &getASList()],
					      -size => 5,
					      -multiple => 'true' ) );

	    print $q->td( $q->checkbox( -name => "${router}_all_as",
					-value => '1',
					-label => 'Yes' ) );
	}

	if ($router eq 'all') {
	    print $q->td( $q->scrolling_list( -name => "${router}_network",
					      -values => [sort &getNetworkList()],
					      -size => 5,
					      -multiple => 'true' ) );
	} else {
	    print $q->td( ' ' );
	}
	
	print $q->td( $q->checkbox( -name => "${router}_total",
				    -value => '1',
				    -label => 'Yes') );
	
	print $q->end_Tr;
    }
    print $q->end_table();
    
    print $q->br;

    print $q->hidden('showmenu','1');

    print $q->center( $q->submit( -name => '',
				  -value => 'Generate graph' ),
		      $q->checkbox( -name => 'legend',
				    -checked => 'ON',
				    -value => '1',
				    -label => 'Legend?' ) );

    print $q->end_form;

    print $q->end_html;    
    exit;
}

sub browserDie {
    print header;
    print start_html(-title => 'Error Occurred',
		     -bgcolor => 'ffffff');
    print '
', "\n";
    print @_;
    print "\n", '
', "\n";
    exit;
}

## Parse param()

sub getImageType {
    if( param('imageType') ) {
	if( param('imageType') eq 'png' || param('imageType') eq 'gif' ) {
	    $imageType = param('imageType');
	} else { &browserDie('Invalid imageType parameter') }
    }
}

sub getRouter {
    if( !param('router') ) {
	push @router, 'all';
    }
    # XXX how much is tainting a problem? .. attacks, etc
    else {
	foreach ( param('router') ) {
	    s/\.\./_/g;
	    if( $_ eq 'all' ) { push @router, 'all' }
	    elsif( -d $rrddir.'/'.$_ ) { push @router, $_ }
	    else { &browserDie('Invalid router parameter') }
	}
    }
}

sub getHours {
    if( param('hours') ) {
	if( param('hours') =~ /^\d+$/ ) { $hours = param('hours') }
	else { &browserDie( "Invalid hours parameter" ) }
    }
}

sub getDuration {
    if( param('duration') ) {
	if( param('duration') =~ /^\d+$/ ) { $duration = param('duration') }
	else { &browserDie( "Invalid duration parameter" ) }
    } else { $duration = $hours; }
}

sub getWidth {
    if( param('width') ) {
	if( param('width') =~ /^\d+$/ ) { $width = param('width') }
	else { &browserDie( "Invalid width parameter" ) }
    }
}

sub getHeight {
    if( param('height') ) {
	if( param('height') =~ /^\d+$/ ) { $height = param('height') }
	else { &browserDie( "Invalid height parameter" ) }
    }
}

sub getTotal {
    foreach my $r (@router) {
	if( param("${r}_all") ) {
	    $total{$r} = 1;
	}
	elsif( param("${r}_total") ) {
	    $total{$r} = param("${r}_total");
	}
    }
}

sub getDebug {
    if( param('debug') && param('debug') eq '1' ) {
	$debug = 1;
    } else { $debug = 0 }
}

sub doReport {
    defined param('report') or &browserDie( "You must specify report type" );

    return &generateImage( &io_report( param('report'), param('legend') ) );
}

# Generate list of protocols and resolve filenames
sub getProtocols {
    foreach my $r (@router) {
	if( param("${r}_all_protocols") ) {
	    push @{$protocol{$r}}, &getProtocolList();
	}
	elsif( param("${r}_protocol") ) {
	    push @{$protocol{$r}}, param("${r}_protocol");
	}
	foreach my $p ( @{$protocol{$r}} ) {
	    my $file;
	    if( $r eq 'all' ) { $file = "${rrddir}/protocol_${p}.rrd"}
	    else { $file =  "${rrddir}/${r}/protocol_${p}.rrd"}
	    -f $file or &browserDie("cannot find $file");
	    $filename{$r}{$p} = $file;
	}
	if( $r eq 'all' ) { $filename{$r}{'total'} = "${rrddir}/total.rrd" }
	else { $filename{$r}{'total'} = "${rrddir}/${r}/total.rrd" }
    }
}

# Generate list of services and resolve filenames
sub getServices {
    foreach my $r (@router) {
	if( param("${r}_all_services") ) {
	    push @{$service{$r}}, &getServiceList();
	}
	elsif( param("${r}_service") ) {
	    push @{$service{$r}}, param("${r}_service");
	}
	foreach my $s ( @{$service{$r}} ) {
	    my ($file_base, $file_src, $file_dst);
	    if( $r eq 'all' ) {
		$file_base = "${rrddir}/service_${s}";
		$file_src = "${rrddir}/service_${s}_src.rrd";
		$file_dst = "${rrddir}/service_${s}_dst.rrd";
	    } else {
		$file_base = "${rrddir}/${r}/service_${s}";
		$file_src = "${rrddir}/${r}/service_${s}_src.rrd";
		$file_dst = "${rrddir}/${r}/service_${s}_dst.rrd";
	    }
	    -f $file_src or &browserDie("cannot find $file_src");
	    -f $file_dst or &browserDie("cannot find $file_dst");
	    $filename{$r}{$s} = $file_base;
	}
    }
}

# Generate list of TOS and resolve filenames
sub getTOS {
    foreach my $r (@router) {
	if( param("${r}_all_tos") ) {
	    push @{$tos{$r}}, &getTOSList();
	}
	elsif( param("${r}_tos") ) {
	    push @{$tos{$r}}, param("${r}_tos");
	}
	foreach my $t ( @{$tos{$r}} ) {
	    my $file;
	    if( $r eq 'all' ) { $file = "${rrddir}/tos_${t}.rrd"}
	    else { $file =  "${rrddir}/${r}/tos_${t}.rrd"}
	    -f $file or &browserDie("cannot find $file");
	    $filename{$r}{$t} = $file;
	}
	if( $r eq 'all' ) { $filename{$r}{'total'} = "${rrddir}/total.rrd" }
	else { $filename{$r}{'total'} = "${rrddir}/${r}/total.rrd" }
    }
}

# Generate list of networks and resolve filenames
sub getNetworks {
    # Networks are only in the all category, for total traffic

    if( param("all_network") ) {
	push @{$network{'all'}}, param("all_network");
    }
    
    foreach my $n ( param("all_network") ) {
	my $file = "${rrddir}/network_${n}.rrd";

	-f $file or &browserDie("cannot find $file");

	$filename{'all'}{$n} = $file;
    }
}

# Generate a list of ASes and resolve filenames
sub getASs {
    foreach my $r (@router) {
	if( param("${r}_all_as") ) {
	    push @{$as{$r}}, &getASList();
	}
	elsif( param("${r}_as") ) {
	    push @{$as{$r}}, param("${r}_as");
	}
	foreach my $t ( @{$as{$r}} ) {
	    my $file;
	    if( $r eq 'all' ) { $file = "${rrddir}/as_${t}.rrd"}
	    else { $file =  "${rrddir}/${r}/as_${t}.rrd"}
	    -f $file or &browserDie("cannot find $file");
	    $filename{$r}{$t} = $file;
	}
	if( $r eq 'all' ) { $filename{$r}{'total'} = "${rrddir}/total.rrd" }
	else { $filename{$r}{'total'} = "${rrddir}/${r}/total.rrd" }
    }
}

## Assign each protocol/service a color

sub setColors {
    # "nice" colors. taken from Packeteer PacketShaper's web interface
    # (via Dave Plonka) and other places
    my @safe_colors = ( 0x746FAE, # lavender
			0x993366, # maroon
  			0xB8860B, # dark goldenrod
  			0xCCFFFF, # lt. cyan
  			0x660066, # purple
  			0xFF6666, # orange
  			0x0066CC, # med. blue
  			0xCCCCFF, # pale lavender
  			0x000066, # dk. blue
  			0x0000FF, # blue
  			0xFFFF00 # yellow
  			);

    foreach my $r (@router) {
	foreach my $n (@{$network{$r}}) {
	    $color{$r}{$n} = &iterateColor(\@safe_colors);
	}

	foreach my $a (@{$as{$r}}) {
	    $color{$r}{$a} = &iterateColor(\@safe_colors);
	}

	foreach my $p (@{$protocol{$r}}) {
	    $color{$r}{$p} = &iterateColor(\@safe_colors);
	}
	
	foreach my $s (@{$service{$r}}) {
	    $color{$r}{$s}{'src'} = &iterateColor(\@safe_colors);
	    $color{$r}{$s}{'dst'} = &iterateColor(\@safe_colors);
	}

	foreach my $t (@{$tos{$r}}) {
	    $color{$r}{$t} = &iterateColor(\@safe_colors);
	}

	$color{$r}{'total'} = &iterateColor(\@safe_colors);
    }
}

# use a color and move it to the back
sub iterateColor {
    my $color = shift @{$_[0]};
    push @{$_[0]}, $color;

    return sprintf('#%06x', $color);
}

# Generate list of available protocols
sub getProtocolList {
    opendir( DIR, $rrddir ) or &browserDie("open $rrddir failed ($!)");
    @_ = grep { /^protocol_.*\.rrd$/ } readdir( DIR );
    closedir DIR;
    
    foreach (@_) {
	s/^protocol_(.*)\.rrd$/$1/;
    }

    return @_;
}

# Generate list of available services
sub getServiceList {
    opendir( DIR, $rrddir ) or &browserDie("open $rrddir failed ($!)");
    @_ = grep { /^service_.*_src\.rrd$/ } readdir( DIR );
    closedir DIR;

    foreach (@_) {
	s/^service_(.*)_src\.rrd$/$1/;
    }

    return @_;
}

# Generate list of available TOS
sub getTOSList {
    opendir( DIR, $rrddir ) or &browserDie("open $rrddir failed ($!)");
    @_ = grep { /^tos_.*\.rrd$/ } readdir( DIR );
    closedir DIR;

    foreach (@_) {
	s/^tos_(.*)\.rrd$/$1/;
    }

    return @_;
}

# Generate list of available ASes
sub getASList {
    opendir( DIR, $rrddir ) or &browserDie("open $rrddir failed ($!)");
    @_ = grep { /^as_.*\.rrd$/ } readdir( DIR );
    closedir DIR;

    foreach (@_) {
	s/^as_(.*)\.rrd$/$1/;
    }

    return @_;
}

# Generate list of available networks
sub getNetworkList {
    opendir( DIR, $rrddir ) or &browserDie("open $rrddir failed ($!)");
    @_ = grep { /^network_.*\.rrd$/ } readdir( DIR );
    closedir DIR;

    foreach (@_) {
	s/^network_(.*)\.rrd$/$1/;
    }

    return @_;
}

# Generate list of available routers
sub getRouterList {
    opendir( DIR, $rrddir ) or &browserDie("open $rrddir failed ($!)");
    while( $_ = readdir( DIR ) ) {
	if( !/^\.\.?/ && -d $rrddir.'/'.$_ ) {
	    push @_, $_;
	}
    }
    closedir DIR;

    return @_;
}

# rrdtool has annoying format rules so just use MD5 like cricket does
sub cleanDEF {
    my $def = shift;

    return $def if $debug;

    unless( exists $cdef{$def} ) {
	$cdef{$def} = substr( md5_hex($def), 0, 29);
    }
    return $cdef{$def};
}

# make service labels a consistent length
sub cleanServiceLabel {
    my $labelLength = 15;
    my $s = shift;
    my $txt = shift;
    return uc($s) . ' ' x ($labelLength - length $s) . $txt;
}

# make protocol labels a consistent length
sub cleanProtocolLabel {
    my $labelLength = 47;
    my $p = shift;
    return uc($p) . ' ' x ($labelLength - length $p);
}

# make other percentage labels a consistent length
sub cleanOtherLabel {
    my $labelLength = 51;
    my $label = shift;
    my $format = shift;
    return $label . ' ' x ($labelLength - length $label) . $format;
}

sub io_report {
    my $reportType = shift;
    my $legend = shift;
    my @args;
    my @str;

    unless( exists $reportName{$reportType} ) {
	&browserDie('invalid report parameter');
    }

    push @args, ('--interlaced',
		 '--imgformat='.uc($imageType),
		 '--vertical-label='.$reportName{$reportType}.' per second',
		 "--title=${organization} Well Known Protocols/Services, ".
		 "\u${reportName{$reportType}}, +out/-in",
		 "--start=".(time - $hours*60*60),
		 "--end=".(time - $hours*60*60 + $duration*60*60),
		 "--width=${width}",
		 "--height=${height}",
		 '--alt-autoscale');

    push @args, '--no-legend' unless($legend);

    # CDEF for total
    foreach my $r (@router) {
	if( $reportType eq 'bits' ) {
	    push @args, ('DEF:'.&cleanDEF("${r}_total_out_bytes").'='.$filename{$r}{'total'}.':out_bytes:AVERAGE',
			 'DEF:'.&cleanDEF("${r}_total_in_bytes").'='.$filename{$r}{'total'}.':in_bytes:AVERAGE',
			 'CDEF:'.&cleanDEF("${r}_total_out_bits").'='.&cleanDEF("${r}_total_out_bytes").',8,*',
			 'CDEF:'.&cleanDEF("${r}_total_in_bits").'='.&cleanDEF("${r}_total_in_bytes").',8,*',
			 'CDEF:'.&cleanDEF("${r}_total_in_bits_neg").'='.&cleanDEF("${r}_total_in_bits").',-1,*');
	} else {
	    push @args, ('DEF:'.&cleanDEF("${r}_total_out_${reportType}").'='.$filename{$r}{'total'}.":out_${reportType}:AVERAGE",
			 'DEF:'.&cleanDEF("${r}_total_in_${reportType}").'='.$filename{$r}{'total'}.":in_${reportType}:AVERAGE",
			 'CDEF:'.&cleanDEF("${r}_total_in_${reportType}_neg").'='.&cleanDEF("${r}_total_in_${reportType}").',-1,*');
	}
    }
	
    # CDEFs for each service
    foreach my $r (@router) {
	foreach my $s (@{$service{$r}}) {
	    if( $reportType eq 'bits' ) {
		push @args, ('DEF:'.&cleanDEF("${r}_${s}_src_out_bytes").'='.$filename{$r}{$s}.'_src.rrd:out_bytes:AVERAGE',
			     'DEF:'.&cleanDEF("${r}_${s}_src_in_bytes").'='.$filename{$r}{$s}.'_src.rrd:in_bytes:AVERAGE',
			     'CDEF:'.&cleanDEF("${r}_${s}_src_out_bits").'='.&cleanDEF("${r}_${s}_src_out_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${s}_src_in_bits").'='.&cleanDEF("${r}_${s}_src_in_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${s}_src_in_bits_neg").'='.&cleanDEF("${r}_${s}_src_in_bytes").',8,*,-1,*',
			     'DEF:'.&cleanDEF("${r}_${s}_dst_out_bytes").'='.$filename{$r}{$s}.'_dst.rrd:out_bytes:AVERAGE',
			     'DEF:'.&cleanDEF("${r}_${s}_dst_in_bytes").'='.$filename{$r}{$s}.'_dst.rrd:in_bytes:AVERAGE',
			     'CDEF:'.&cleanDEF("${r}_${s}_dst_out_bits").'='.&cleanDEF("${r}_${s}_dst_out_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${s}_dst_in_bits").'='.&cleanDEF("${r}_${s}_dst_in_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${s}_dst_in_bits_neg").'='.&cleanDEF("${r}_${s}_dst_in_bytes").',8,*,-1,*');
	    } else {
		push @args, ('DEF:'.&cleanDEF("${r}_${s}_src_out_${reportType}").'='.$filename{$r}{$s}."_src.rrd:out_${reportType}:AVERAGE",
			     'DEF:'.&cleanDEF("${r}_${s}_src_in_${reportType}").'='.$filename{$r}{$s}."_src.rrd:in_${reportType}:AVERAGE",
			     'CDEF:'.&cleanDEF("${r}_${s}_src_in_${reportType}_neg").'='.&cleanDEF("${r}_${s}_src_in_${reportType}").',-1,*',
			     'DEF:'.&cleanDEF("${r}_${s}_dst_out_${reportType}").'='.$filename{$r}{$s}."_dst.rrd:out_${reportType}:AVERAGE",
			     'DEF:'.&cleanDEF("${r}_${s}_dst_in_${reportType}").'='.$filename{$r}{$s}."_dst.rrd:in_${reportType}:AVERAGE",
			     'CDEF:'.&cleanDEF("${r}_${s}_dst_in_${reportType}_neg").'='.&cleanDEF("${r}_${s}_dst_in_${reportType}").',-1,*');
	    }
	}
    }

    # CDEFs for service by number, percentage
    foreach my $r (@router) {
	if( scalar @{$service{$r}} ) {
	    foreach my $s ( @{$service{$r}} ) {
		push @args, 'CDEF:'.&cleanDEF("${r}_${s}_in_nr").'='.&cleanDEF("${r}_${s}_src_in_${reportType}").','.&cleanDEF("${r}_${s}_dst_in_${reportType}").',+';
		push @args, 'CDEF:'.&cleanDEF("${r}_${s}_in_pct").'='.&cleanDEF("${r}_${s}_src_in_${reportType}").','.&cleanDEF("${r}_${s}_dst_in_${reportType}").',+,'.&cleanDEF("${r}_total_in_${reportType}").',/,100,*';
	    }
	    
	    $str[0] = 'CDEF:'.&cleanDEF("${r}_other_service_in_nr").'='.&cleanDEF("${r}_total_in_${reportType}");
	    $str[1] = 'CDEF:'.&cleanDEF("${r}_other_service_in_pct").'=100';
	    foreach my $s ( @{$service{$r}} ) {
		$str[0] .= ','.&cleanDEF("${r}_${s}_in_nr").',-';
		$str[1] .= ','.&cleanDEF("${r}_${s}_in_pct").',-';
	    }
	    push @args, @str;
	    
	    foreach my $s ( @{$service{$r}} ) {
		push @args, 'CDEF:'.&cleanDEF("${r}_${s}_out_nr").'='.&cleanDEF("${r}_${s}_src_out_${reportType}").','.&cleanDEF("${r}_${s}_dst_out_${reportType}").',+';
		push @args, 'CDEF:'.&cleanDEF("${r}_${s}_out_pct").'='.&cleanDEF("${r}_${s}_src_out_${reportType}").','.&cleanDEF("${r}_${s}_dst_out_${reportType}").',+,'.&cleanDEF("${r}_total_out_${reportType}").',/,100,*';
	    }
	    
	    $str[0] = 'CDEF:'.&cleanDEF("${r}_other_service_out_nr").'='.&cleanDEF("${r}_total_out_${reportType}");
	    $str[1] = 'CDEF:'.&cleanDEF("${r}_other_service_out_pct").'=100';
	    foreach my $s ( @{$service{$r}} ) {
		$str[0] .= ','.&cleanDEF("${r}_${s}_out_pct").',-';
		$str[1] .= ','.&cleanDEF("${r}_${s}_out_pct").',-';
	    }
	    push @args, @str;
	}
    }
	
    # CDEFs for each protocol
    foreach my $r (@router) {
	foreach my $p ( @{$protocol{$r}} ) {
	    if( $reportType eq 'bits' ) {
		push @args, ('DEF:'.&cleanDEF("${r}_${p}_out_bytes").'='.$filename{$r}{$p}.':out_bytes:AVERAGE',
			     'DEF:'.&cleanDEF("${r}_${p}_in_bytes").'='.$filename{$r}{$p}.':in_bytes:AVERAGE',
			     'CDEF:'.&cleanDEF("${r}_${p}_out_bits").'='.&cleanDEF("${r}_${p}_out_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${p}_in_bits").'='.&cleanDEF("${r}_${p}_in_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${p}_in_bits_neg").'='.&cleanDEF("${r}_${p}_in_bytes").',8,*,-1,*');
	    } else {
		push @args, ('DEF:'.&cleanDEF("${r}_${p}_out_${reportType}").'='.$filename{$r}{$p}.":out_${reportType}:AVERAGE",
			     'DEF:'.&cleanDEF("${r}_${p}_in_${reportType}").'='.$filename{$r}{$p}.":in_${reportType}:AVERAGE",
			     'CDEF:'.&cleanDEF("${r}_${p}_in_${reportType}_neg").'='.&cleanDEF("${r}_${p}_in_${reportType}").',-1,*');
	    }
	}
    }

    # CDEFs for protocol by percentage
    foreach my $r (@router) {
	if( scalar @{$protocol{$r}} ) { 
	    foreach my $p ( @{$protocol{$r}} ) {
		push @args, 'CDEF:'.&cleanDEF("${r}_${p}_in_pct").'='.&cleanDEF("${r}_${p}_in_${reportType}").','.&cleanDEF("${r}_total_in_${reportType}").',/,100,*';
	    }
	    
	    $str[0] = 'CDEF:'.&cleanDEF("${r}_other_protocol_in_nr").'='.&cleanDEF("${r}_total_in_${reportType}");
	    $str[1] = 'CDEF:'.&cleanDEF("${r}_other_protocol_in_pct").'=100';
	    foreach my $p ( @{$protocol{$r}} ) {
		$str[0] .= ','.&cleanDEF("${r}_${p}_in_".$reportType).',-';
		$str[1] .= ','.&cleanDEF("${r}_${p}_in_pct").',-';
	    }
	    push @args, @str;
	    
	    foreach my $p ( @{$protocol{$r}} ) {
		push @args, 'CDEF:'.&cleanDEF("${r}_${p}_out_pct").'='.&cleanDEF("${r}_${p}_out_${reportType}").','.&cleanDEF("${r}_total_out_${reportType}").',/,100,*';
	    }
	    
	    $str[0] = 'CDEF:'.&cleanDEF("${r}_other_protocol_out_nr").'='.&cleanDEF("${r}_total_out_${reportType}");
	    $str[1] = 'CDEF:'.&cleanDEF("${r}_other_protocol_out_pct").'=100';
	    foreach my $p ( @{$protocol{$r}} ) {
		$str[0] .= ','.&cleanDEF("${r}_${p}_out_".$reportType).',-';
		$str[1] .= ','.&cleanDEF("${r}_${p}_out_pct").',-';
	    }
	    push @args, @str;
	}
    }

    # CDEFs for each AS
    foreach my $r (@router) {
	foreach my $p ( @{$as{$r}} ) {
	    if( $reportType eq 'bits' ) {
		push @args, ('DEF:'.&cleanDEF("${r}_${p}_out_bytes").'='.$filename{$r}{$p}.':out_bytes:AVERAGE',
			     'DEF:'.&cleanDEF("${r}_${p}_in_bytes").'='.$filename{$r}{$p}.':in_bytes:AVERAGE',
			     'CDEF:'.&cleanDEF("${r}_${p}_out_bits").'='.&cleanDEF("${r}_${p}_out_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${p}_in_bits").'='.&cleanDEF("${r}_${p}_in_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${p}_in_bits_neg").'='.&cleanDEF("${r}_${p}_in_bytes").',8,*,-1,*');
	    } else {
		push @args, ('DEF:'.&cleanDEF("${r}_${p}_out_${reportType}").'='.$filename{$r}{$p}.":out_${reportType}:AVERAGE",
			     'DEF:'.&cleanDEF("${r}_${p}_in_${reportType}").'='.$filename{$r}{$p}.":in_${reportType}:AVERAGE",
			     'CDEF:'.&cleanDEF("${r}_${p}_in_${reportType}_neg").'='.&cleanDEF("${r}_${p}_in_${reportType}").',-1,*');
	    }
	}
    }

    # CDEFs for AS by percentage
    foreach my $r (@router) {
	if( scalar @{$as{$r}} ) { 
	    foreach my $p ( @{$as{$r}} ) {
		push @args, 'CDEF:'.&cleanDEF("${r}_${p}_in_pct").'='.&cleanDEF("${r}_${p}_in_${reportType}").','.&cleanDEF("${r}_total_in_${reportType}").',/,100,*';
	    }
	    
	    $str[0] = 'CDEF:'.&cleanDEF("${r}_other_as_in_nr").'='.&cleanDEF("${r}_total_in_${reportType}");
	    $str[1] = 'CDEF:'.&cleanDEF("${r}_other_as_in_pct").'=100';
	    foreach my $p ( @{$as{$r}} ) {
		$str[0] .= ','.&cleanDEF("${r}_${p}_in_".$reportType).',-';
		$str[1] .= ','.&cleanDEF("${r}_${p}_in_pct").',-';
	    }
	    push @args, @str;
	    
	    foreach my $p ( @{$as{$r}} ) {
		push @args, 'CDEF:'.&cleanDEF("${r}_${p}_out_pct").'='.&cleanDEF("${r}_${p}_out_${reportType}").','.&cleanDEF("${r}_total_out_${reportType}").',/,100,*';
	    }
	    
	    $str[0] = 'CDEF:'.&cleanDEF("${r}_other_as_out_nr").'='.&cleanDEF("${r}_total_out_${reportType}");
	    $str[1] = 'CDEF:'.&cleanDEF("${r}_other_as_out_pct").'=100';
	    foreach my $p ( @{$as{$r}} ) {
		$str[0] .= ','.&cleanDEF("${r}_${p}_out_".$reportType).',-';
		$str[1] .= ','.&cleanDEF("${r}_${p}_out_pct").',-';
	    }
	    push @args, @str;
	}
    }

    # CDEFs for each TOS
    foreach my $r (@router) {
	foreach my $t ( @{$tos{$r}} ) {
	    if( $reportType eq 'bits' ) {
		push @args, ('DEF:'.&cleanDEF("${r}_${t}_out_bytes").'='.$filename{$r}{$t}.':out_bytes:AVERAGE',
			     'DEF:'.&cleanDEF("${r}_${t}_in_bytes").'='.$filename{$r}{$t}.':in_bytes:AVERAGE',
			     'CDEF:'.&cleanDEF("${r}_${t}_out_bits").'='.&cleanDEF("${r}_${t}_out_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${t}_in_bits").'='.&cleanDEF("${r}_${t}_in_bytes").',8,*',
			     'CDEF:'.&cleanDEF("${r}_${t}_in_bits_neg").'='.&cleanDEF("${r}_${t}_in_bytes").',8,*,-1,*');
	    } else {
		push @args, ('DEF:'.&cleanDEF("${r}_${t}_out_${reportType}").'='.$filename{$r}{$t}.":out_${reportType}:AVERAGE",
			     'DEF:'.&cleanDEF("${r}_${t}_in_${reportType}").'='.$filename{$r}{$t}.":in_${reportType}:AVERAGE",
			     'CDEF:'.&cleanDEF("${r}_${t}_in_${reportType}_neg").'='.&cleanDEF("${r}_${t}_in_${reportType}").',-1,*');
	    }
	}
    }

    # CDEFs for TOS by percentage
    foreach my $r (@router) {
	if( scalar @{$tos{$r}} ) { 
	    foreach my $t ( @{$tos{$r}} ) {
		push @args, 'CDEF:'.&cleanDEF("${r}_${t}_in_pct").'='.&cleanDEF("${r}_${t}_in_${reportType}").','.&cleanDEF("${r}_total_in_${reportType}").',/,100,*';
	    }
	    
	    $str[0] = 'CDEF:'.&cleanDEF("${r}_other_tos_in_nr").'='.&cleanDEF("${r}_total_in_${reportType}");
	    $str[1] = 'CDEF:'.&cleanDEF("${r}_other_tos_in_pct").'=100';
	    foreach my $t ( @{$tos{$r}} ) {
		$str[0] .= ','.&cleanDEF("${r}_${t}_in_".$reportType).',-';
		$str[1] .= ','.&cleanDEF("${r}_${t}_in_pct").',-';
	    }
	    push @args, @str;
	    
	    foreach my $t ( @{$tos{$r}} ) {
		push @args, 'CDEF:'.&cleanDEF("${r}_${t}_out_pct").'='.&cleanDEF("${r}_${t}_out_${reportType}").','.&cleanDEF("${r}_total_out_${reportType}").',/,100,*';
	    }
	    
	    $str[0] = 'CDEF:'.&cleanDEF("${r}_other_tos_out_nr").'='.&cleanDEF("${r}_total_out_${reportType}");
	    $str[1] = 'CDEF:'.&cleanDEF("${r}_other_tos_out_pct").'=100';
	    foreach my $t ( @{$tos{$r}} ) {
		$str[0] .= ','.&cleanDEF("${r}_${t}_out_".$reportType).',-';
		$str[1] .= ','.&cleanDEF("${r}_${t}_out_pct").',-';
	    }
	    push @args, @str;
	}
    }

    # CDEFs for each network
    foreach my $n ( @{$network{'all'}} ) {
	if( $reportType eq 'bits' ) {
	    push @args, ('DEF:'.&cleanDEF("all_${n}_out_bytes").'='.$filename{'all'}{$n}.':out_bytes:AVERAGE',
			 'DEF:'.&cleanDEF("all_${n}_in_bytes").'='.$filename{'all'}{$n}.':in_bytes:AVERAGE',
			 'CDEF:'.&cleanDEF("all_${n}_out_bits").'='.&cleanDEF("all_${n}_out_bytes").',8,*',
			 'CDEF:'.&cleanDEF("all_${n}_in_bits").'='.&cleanDEF("all_${n}_in_bytes").',8,*',
			 'CDEF:'.&cleanDEF("all_${n}_in_bits_neg").'='.&cleanDEF("all_${n}_in_bytes").',8,*,-1,*');
	} else {
	    push @args, ('DEF:'.&cleanDEF("all_${n}_out_${reportType}").'='.$filename{'all'}{$n}.":out_${reportType}:AVERAGE",
			 'DEF:'.&cleanDEF("all_${n}_in_${reportType}").'='.$filename{'all'}{$n}.":in_${reportType}:AVERAGE",
			 'CDEF:'.&cleanDEF("all_${n}_in_${reportType}_neg").'='.&cleanDEF("all_${n}_in_${reportType}").',-1,*');
	}
    }

    # CDEFs for network by percentage
    if( scalar @{$network{'all'}} ) { 
	foreach my $n ( @{$network{'all'}} ) {
	    push @args, 'CDEF:'.&cleanDEF("all_${n}_in_pct").'='.&cleanDEF("all_${n}_in_${reportType}").','.&cleanDEF("all_total_in_${reportType}").',/,100,*';
	}
	    
	$str[0] = 'CDEF:'.&cleanDEF("all_other_network_in_nr").'='.&cleanDEF("all_total_in_${reportType}");
	$str[1] = 'CDEF:'.&cleanDEF("all_other_network_in_pct").'=100';
	foreach my $n ( @{$network{'all'}} ) {
	    $str[0] .= ','.&cleanDEF("all_${n}_in_".$reportType).',-';
	    $str[1] .= ','.&cleanDEF("all_${n}_in_pct").',-';
	}
	push @args, @str;
	    
	foreach my $n ( @{$network{'all'}} ) {
	    push @args, 'CDEF:'.&cleanDEF("all_${n}_out_pct").'='.&cleanDEF("all_${n}_out_${reportType}").','.&cleanDEF("all_total_out_${reportType}").',/,100,*';
	}
	    
	$str[0] = 'CDEF:'.&cleanDEF("all_other_network_out_nr").'='.&cleanDEF("all_total_out_${reportType}");
	$str[1] = 'CDEF:'.&cleanDEF("all_other_network_out_pct").'=100';
	foreach my $n ( @{$network{'all'}} ) {
	    $str[0] .= ','.&cleanDEF("all_${n}_out_".$reportType).',-';
	    $str[1] .= ','.&cleanDEF("all_${n}_out_pct").',-';
	}
	push @args, @str;
    }
	
    # Graph commands
    my $count;
	
    foreach my $r (@router) {
	$count = 0;

	# router name
	if( scalar @{$service{$r}} || scalar @{$protocol{$r}} || scalar @{$tos{$r}}
	    || exists $total{$r} ) {
	    if( $RRDs::VERSION >= 1.2 ) {
		push @args, 'COMMENT: Router\: '.$r.'\n';
	    } else {
		push @args, 'COMMENT: Router: '.$r.'\n';
	    }
	}

	# service outbound, numbers & percentages
	foreach my $s ( @{$service{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${s}_src_out_".$reportType).$color{$r}{$s}{'src'}.':'.&cleanServiceLabel($s, ' src  +');
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${s}_src_out_".$reportType).$color{$r}{$s}{'src'}.':'.&cleanServiceLabel($s, ' src  +');
	    }
	    push @args, 'STACK:'.&cleanDEF("${r}_${s}_dst_out_".$reportType).$color{$r}{$s}{'dst'}.':'.&cleanServiceLabel($s, ' dst  ');
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${s}_out_nr").':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${s}_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${s}_in_nr").':AVERAGE:%.1lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${s}_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}
	
	# protocol outbound, numbers & percentages
	foreach my $p ( @{$protocol{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${p}_out_".$reportType).$color{$r}{$p}.':'.&cleanProtocolLabel($p);
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${p}_out_".$reportType).$color{$r}{$p}.':'.&cleanProtocolLabel($p);
	    }
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${p}_out_".$reportType).':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${p}_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${p}_in_".$reportType).':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${p}_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}

	# tos outbound, numbers & percentages
	foreach my $t ( @{$tos{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${t}_out_".$reportType).$color{$r}{$t}.':'.&cleanProtocolLabel($t);
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${t}_out_".$reportType).$color{$r}{$t}.':'.&cleanProtocolLabel($t);
	    }
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${t}_out_".$reportType).':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${t}_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${t}_in_".$reportType).':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${t}_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}
	
	# network outbound, numbers & percentages
	foreach my $n ( @{$network{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${n}_out_".$reportType).$color{$r}{$n}.':'.&cleanProtocolLabel($n);
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${n}_out_".$reportType).$color{$r}{$n}.':'.&cleanProtocolLabel($n);
	    }
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${n}_out_".$reportType).':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${n}_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${n}_in_".$reportType).':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${n}_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}

	# AS outbound, numbers & percentages
	foreach my $n ( @{$as{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${n}_out_".$reportType).$color{$r}{$n}.':'.&cleanProtocolLabel($n);
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${n}_out_".$reportType).$color{$r}{$n}.':'.&cleanProtocolLabel($n);
	    }
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${n}_out_".$reportType).':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${n}_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${n}_in_".$reportType).':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_${n}_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}

	# service other, numbers & percentages
	if( scalar @{$service{$r}} ) {
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_service_out_nr").':AVERAGE:'.&cleanOtherLabel('Other services','%.2lf%S');
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_service_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_service_in_nr").':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_service_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}
	# protocol other, numbers & percentages
	if( scalar @{$protocol{$r}} ) {
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_protocol_out_pct").':AVERAGE:'.&cleanOtherLabel('Other protocols','%.2lf%S');
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_protocol_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_protocol_in_pct").':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_protocol_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}
	# tos other, numbers & percentages
	if( scalar @{$tos{$r}} ) {
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_tos_out_pct").':AVERAGE:'.&cleanOtherLabel('Other TOS','%.2lf%S');
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_tos_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_tos_in_pct").':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_tos_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}
	# network other, numbers & percentages
	if( scalar @{$network{$r}} ) {
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_network_out_pct").':AVERAGE:'.&cleanOtherLabel('Other networks','%.2lf%S');
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_network_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_network_in_pct").':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_network_in_pct").':AVERAGE:%.1lf%% In\n';
	}
	# AS other, numbers & percentages
	if( scalar @{$as{$r}} ) {
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_as_out_pct").':AVERAGE:'.&cleanOtherLabel('Other ASes','%.2lf%S');
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_as_out_pct").':AVERAGE:(%.1lf%%) Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_as_in_pct").':AVERAGE:%.2lf%S';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_other_as_in_pct").':AVERAGE:(%.1lf%%) In\n';
	}
	# total outbound, numbers
	if( exists $total{$r} ) {
	    push @args, 'LINE1:'.&cleanDEF("${r}_total_out_".$reportType).$color{$r}{'total'}.':'.&cleanProtocolLabel('TOTAL');
	    push @args, 'GPRINT:'.&cleanDEF("${r}_total_out_".$reportType).':AVERAGE:%.2lf%S Out ';
	    push @args, 'GPRINT:'.&cleanDEF("${r}_total_in_".$reportType).':AVERAGE:%.2lf%S In\n';
	}
	
	$count = 0;

	# service inbound
	foreach my $s ( @{$service{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${s}_src_in_".$reportType.'_neg').$color{$r}{$s}{'src'};
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${s}_src_in_".$reportType.'_neg').$color{$r}{$s}{'src'};
	    }
	    push @args, 'STACK:'.&cleanDEF("${r}_${s}_dst_in_".$reportType.'_neg').$color{$r}{$s}{'dst'};
	}

	# protocol inbound
	foreach my $p ( @{$protocol{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${p}_in_".$reportType.'_neg').$color{$r}{$p};
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${p}_in_".$reportType.'_neg').$color{$r}{$p};
	    }
	}

	# TOS inbound
	foreach my $t ( @{$tos{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${t}_in_".$reportType.'_neg').$color{$r}{$t};
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${t}_in_".$reportType.'_neg').$color{$r}{$t};
	    }
	}

	# network inbound
	foreach my $n ( @{$network{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${n}_in_".$reportType.'_neg').$color{$r}{$n};
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${n}_in_".$reportType.'_neg').$color{$r}{$n};
	    }
	}

	# AS inbound
	foreach my $n ( @{$as{$r}} ) {
	    $count++;
	    if( $count == 1 ) {
		push @args, 'AREA:'.&cleanDEF("${r}_${n}_in_".$reportType.'_neg').$color{$r}{$n};
	    } else {
		push @args, 'STACK:'.&cleanDEF("${r}_${n}_in_".$reportType.'_neg').$color{$r}{$n};
	    }
	}

	# total inbound
	if( exists $total{$r} ) {
	    push @args, 'LINE1:'.&cleanDEF("${r}_total_in_".$reportType.'_neg').$color{$r}{'total'};
	}

	# blank line after router
	if( scalar @{$service{$r}} || scalar @{$protocol{$r}} || scalar @{$tos{$r}} ||
	    exists $total{$r} ) {
	    push @args, 'COMMENT:\n';
	}
    }

    push @args, 'HRULE:0#000000';
	
    return @args;
}
CUFlow-1.7/README.txt0000644011555000000140000000522010361016551014332 0ustar  selskysrc00000000000000CUFlow - a module for FlowScan and an accompanying graphing cgi
by Johan Andersen 
and Matt Selsky 
January 2006


SYNOPSIS

To execute CUFlow properly, type:

	flowscan CUFlow


OVERVIEW:

CUFlow.pm is a FlowScan (http://net.doit.wisc.edu/~plonka/FlowScan/)
module designed to combine the features of CampusIO and SubNetIO as we
needed them and also hopefully process data more quickly.  CUFlow
allows you to differentiate traffic by protocol, service, TOS, router,
and network and then generate TopN reports over 5 minutes periods and
over an extended period of time.  CUGrapher is the companion graphing
tool and is designed as a CGI which generates images on the fly based
on user input with data supplied by CUFlow.

INSTALLATION AND EXECUTION:

Follow the instructions for installing FlowScan
(http://net.doit.wisc.edu/~plonka/FlowScan/INSTALL.html) If you are
only using CUFlow and not the other modules distributed with FlowScan
then you probably don't need to install ksh and the Boulder and
ConfigReader Perl modules.

CUFlow.pm and CUFlow.cf should be placed in the $flowscan_home/bin
directory along with the other modules.  To use CUFlow add it to the
Report section of flowscan.cf and fire up flowscan.  For more detailed
information on the format of CUFlow.cf see the perldoc documentation.

CUGrapher.pl should be placed in your cgi-bin and the Local Settings
section should be checked to make sure the values make sense for your
site.  You will most certainly need to change $rrddir and
$organization.  The other settings should be reasonable defaults and
should be self-explanatory.

KNOWN ISSUES:

If you are not using a Cisco Catalyst 6509, you will want to uncomment
lines 495 and 496.  These lines deal with the fact that the 6509
doesn't fill out the NextHop and OutputInterface fields in the NetFlow
records sent from the PFC.

MAILING LISTS:

There is a mailing list having to do with CUFlow:

    * cuflow-users: a general mailing list for CUFlow users.

The list's archives are available at:

https://www1.columbia.edu/sec/bboard/mj/cuflow-users/

This list is hosted by the Academic Information Systems department
at Columbia University. To subscribe to the list, send email to:

majordomo@columbia.edu

containing:

subscribe cuflow-users

You should receive an automatic response that will request that you
verify your request to become a member of the list, to which you must
reply with the authentication information there-in. Then, in response
to your reply, you should receive a welcome message. If you have any
questions about the administrative policies of this list's manager,
please contact:

owner-cuflow-users@columbia.edu