pax_global_header00006660000000000000000000000064140450103070014504gustar00rootroot0000000000000052 comment=3e8706952f7caaf8ff479b4cd49d1488bff824ce nfdump-1.6.23/000077500000000000000000000000001404501030700130665ustar00rootroot00000000000000nfdump-1.6.23/.gitignore000066400000000000000000000017541404501030700150650ustar00rootroot00000000000000# Object files *.o *.ko *.obj *.elf # Precompiled Headers *.gch *.pch # Libraries *.lib *.a *.la *.lo # Shared objects (inc. Windows DLLs) *.dll *.so *.so.* *.dylib # Executables *.exe *.out *.app *.i*86 *.x86_64 *.hex # Debug files *.dSYM/ # Backup files *~ # Automake generated files /INSTALL /Makefile.in /aclocal.m4 /autom4te.cache/ /bin/Makefile.in /compile /config.guess /config.h.in /config.sub /configure /depcomp /install-sh /ltmain.sh /man/Makefile.in /missing /test-driver /ylwrap /doc/Makefile.in # Configure generated files /Makefile /bin/.deps/ /bin/Makefile /config.h /config.log /config.status /extra/nftrack/.deps/ /libtool /man/Makefile /stamp-h1 /doc/Doxyfile /doc/Makefile # Binaries generated by make /bin/.libs/ /bin/grammar.c /bin/grammar.h /bin/nfanon /bin/nfcapd /bin/nfdump /bin/nfexpire /bin/nfpcapd /bin/nfprofile /bin/nfreplay /bin/nftrack /bin/scanner.c /bin/sfcapd /extra/nftrack/.dirstamp # Generated docs /doc/doxygen-build.stamp /doc/html/ /doc/man/ /doc/xml/ nfdump-1.6.23/AUTHORS000077500000000000000000000000461404501030700141410ustar00rootroot00000000000000Peter Haag peter@people.ops-trust.net nfdump-1.6.23/BSD-license.txt000066400000000000000000000031201404501030700156530ustar00rootroot00000000000000 The nfdump project is distributed under the BSD license: Copyright (c) 2016, Peter Haag All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. nfdump-1.6.23/COPYING000077700000000000000000000000001404501030700167052BSD-license.txtustar00rootroot00000000000000nfdump-1.6.23/ChangeLog000077500000000000000000000711031404501030700146450ustar00rootroot000000000000002021-05-05 - Release 1.6.23 2021-04-28 - Fix potential FreeNode without valid Node in nfpcapd. 2021-04-21 - Add all non TCP/UDP IP protocols as streams in nfpcapd 2021-04-20 - Add mpls unwrap in nfpcapd. Skip MPLS labels - Add ESP to processed protocols in nfpcapd. - Some Code cleanup 2021-04-10 - Change spin lock to native C11 lock - Cleanup code for issue #283 2021-04-09 - Fix minor nfpcapd issues - Add mpls unwrap in sflow code - adds mpls labels if available 2021-03-12 - Update rbtree. - Fix potential deadlock in nfpcapd if it terminates. 2021-03-06 - Add packet capture buffer size to nfpcapd 2021-02-20 - Fix sflow code extended field parsing. #262 and #273 - Fix endless loop of nfexpire, if it does not find files 2021-01-30 - Fix processing deoding error for yaf exporter - Zero out tcp flags for non TCP records 2021-01-16 - Add reverse element enterprise ID 29305 for counter values - Add biFlow direction element 239 - Add flow end reason element 136 - Make -Tall the default for nfcapd to collect extensions 2020-12-22 - Code cleanup and boundary checks in option template processing 2020-12-19 - Implement element 160 (SystemInitTime) in option template 2020-12-19 - Add Element 160 (SystemInitTime) in flow record used by Huawei 2020-12-05 - Fix path handling for -l - Fix print plain numbers #263 2020-11-21 - Release 1.6.22 2020-10-18 - Fix nfreplay v5 time shift bug 2020-10-17 - add support for >=, <= comparators. #256. Thanks to piorek94 - Fix yacc/bison warnings. Cleanup unused tokens - Fix syntax error 'flags AS' as AS is a reserved word. #255 - Add element 139 for ICMP type/code in IPv6. #250 - Fix IPv4/IPv6 statistics representation #252 2020-09-12 - Cleanup nip/xip filter syntax. Add filter syntax 'nip in [ ]'. Request #246 2020-09-03 - Add nfversion to nfpcapd 2020-08-31 - Add collected netflow/sflow version in nfdump record. Request #242 - Fix GuessDir bug - issue #215 2020-08-02 - Re-address issue #231 - remove strict rule rfc 7011 2020-08-02 - Release 1.6.21 - Address issue #159. Implement rfc 7011 and include sender UDP port into unique template identification - Address issue #236 Add token 'dir' equivalent to 'flowdir' in filter syntax 2020-07-25 - Add optional print direction ascending or descending to output of statistics -s and oredered printing -O. Request #235 - Fix issue #234 - Fix #230 - Avoid use_syslog name clash on certain OS 2020-06-20 - Honor -n flag when printing sorted flow cache - Fix uninitialized variable printPlain - Fix bug #223 limit matchig flows -c - Restore old behaviour unlimiting output flows unless in -s stat - Fix ft2nfdump nexthop fields - Fix ft2nfdump extension map size - internal: put output parameters in a single struct - Fix GuessDir bug - issue #215 - Compact Changelog - Fix GuessDir bug - issue #215 in flow exporter 2020-03-29 - Release 1.6.20 - More cleanup on plain number printing - Fix plain numbers bug #213 - Fix profiler filer bug 2020-02-22 - Release 1.6.19 - Add Source security group tag (SGT 34000) issue #192 - Modify heuristic for bidir flows #59 - Add Push for new sgt tag. - Cleanup unused nffile records. Add nffileV2.h - Fix various compiler warnings and automake issues - Remove external global vars - Set verbose logging off by default in non daemon mode - Fix make check - Remove old legacy records - Major code rearrangement in order to prepare for futur versions of nfdump Move extension definitions into extension files nfx.h Move exporter definition into extension.h nffile.h should only contain nffile data block handlich definitions - Fix compile error for FreeBSD #203 - Cleanup header code - add new filter.h - free nfdump.h from dust - More code cleanup - delete nf_common - add output_fmt - Fix bug for IPv6 network cidr representation in raw and cvs output mode. - Remove unused old code. Fix for #197 - Fix bidir export issues. Cleanup code. Fix for #195 - Enable small time intervals < 60s. - #185 - Cleanup old code - Fix minor nfpcapd issues - Fix gcc compile issue - Fix corrupt file handling #193 - More code cleanup. Move code to dedicated files. - Restructure code for output into separate files. - Cleanup old code. - Fix bug #189 - valid json output - Fix issue #190. Add compile time option for JunOS. - Cleanup configure.ac - Fix various c11 compile issues - Cleanup old compat15 comments - Add vlan tags dot1qVlanId, 243, 254 - #182 - Fix sflow issue with Arista switches - Fix ft2nfdump next hop and ip router fields - Cleanup and fix IPv6 network display in war records. - Fix compile issues - Fix output buffer size for lzo1x_decompress_safe() - Fix VerifyExtensionMap #179 - Fix compile errors 2019-08-05 - Release 1.6.18 - Fix nfdump.1 man page. #175 - Fix off by 1 array. #173 - Fix use after free in ModifyCompressFile - Add bound checks in AddExporterStat #174 - Add bound checks in AddSamplerInfo #176 - Add bound checks in AddExporterInfo - Fix checks in InsertExtensionMap #177 - Remove COMPAT15 code - should no longer be needed. - Merge pull request #167 - Cleanup old code - Replace depricated pcap_lookupdev call in nfpcapd 2019-07-31 - Add early record size sanity check also for nfprofile, nfanon and nfreplay 2019-07-26 - nfpcapd cleanup, add some more monitoring - Fix hbo_exporter.c:249_1 segfault - Fix hbo_nffile_inline.c:85_1 segfault - Fix hbo_nfx.c:216_3 segfault - Update minilzo to v2.10 - Change to safe lzo decompress function 2019-07-25 - Rework nfpcapd and add it officially to the nfdump collection. - Add nfpcapd man page - Fix potential unsigned integer underflow #171 2019-07-16 - Add latency extension if dumping flowcache 2019-07-15 - Fix typos - Fix exporter struct inconsistancies. Coredump on ARM otherwise. 2019-07-02 - Add ipfix element #150, #151 unix time start/end - Fix display bug raw record 2019-06-01 - Add ipfix dyn element handling. - Add empty m4 directory - keep autoconf happy 2019-06-01 - Fix issue #162 - ipfix mpls sequece. - Fix issue #156 - print flowtable index error 2019-03-17 - Fix spec file - Remove non thread safe logging in nfpcapd 2018-11-24 - Fix protocol tag for protocol 87 - TCF - #130 - Add TCP flags ECN,CVR - #132 - Fix some error messages to be printed to the correct stream #135 - Add missing -M command line help to nfcapd - Remove padding byte warning in log #141 - Fix bug to accept -y compression flag in nfcapd. - #145 2018-06-24 - Fix bookkeeper type - use key_t - Add multiple packet repeaters to nfcapd/sfcapd. Up to 8 repeaters (-R) can be defined. - Ignore OSX .DS_Store files in -R file list - Add CISCO ASA elements initiatorPackets (298) responderPackets (299) - Merge #120 pull request for -z parameter to nfreplay - Update man page nfreplay 2018-05-06 - New bookkeeper hash broke NfSen. Fixed. ported back to release 1.6.17 2018-04-20 - Release 1.6.17 2018-04-20 - Fix bug in sorting when guessing flow direction. Issue #92 - Update nfdump.1 man page for xsrcport & xdstport aggregations. Request #109 - Fix minor bugs - Fix definition for InfluxDB in configure.ac Issue #98 2018-04-01 - Add program exit in nfx.c after panic with correupt data file - Add missing size check when reading nfdump 1.5.x common record blocks - Add missing option -M in man page. Issue #103 - Add Fix processing of influx URL in nfprofile 2018-02-11 - Add missing json output format in nfdump help text - Add missing -v option in nfreplay help text 2018-01-06 - Merge pull request #51 Influxdb from Luca. Thx for the patch 2018-01-01 - IPFIX time stamps - Fix elements #21,#22 offset calculation, but timestamps not yet evaluated. (#160) - IPFIX add fwd status tag #89 compatible to v9 (1byte) 2017-12-31 - IPFIX sampling - sampling algorithm no longer required for tag #34 - IPFIX sampling add tags #305 and #304 - set them identical to #34, #35 2017-12-30 - Add new output format json. Print each record as individual json object 2017-12-28 - Add sampling elements ID 302,304,305. put them identical to ID 48,49,50 - Add option to label filter terms. syntax: () %labelname. - Add %lbl option to print flow label in output - Update nfdump(1) man page for flowlabels 2017-12-27 - Add ipfix delta timestamp elements 158/159. - Update sflow code to commit 7322984 of https://github.com/sflow/sflowtool - Cleanup sflow code - uncomment unnecessary code - Fix header includes" - Fix 64bit fts compat issue in fts_compat.c - Add more detailed autogen.sh - softlink bootstrap 2017-12-22 - Fix potential memory leaks in nfpcapd 2017-12-21 - Fix wrong offset calculation if unknown options are found - Add x-late src/dst ip aggregation, if compiled with NSEL support 2017-12-17 - Add ipfix sampling. Process option template/record with sampling elements 34 and 35 - Report updates on existing samplers in v9 only if values change. issue 84 2017-11-05 v1.6.16 2017-12-10 - Add lz4 compression - Remove old xstat legancy code, not needed - Remove automake files from git 2017-12-03 - Fix old 1.6.15 tags - Fix minor issues and compiler warnings 2017-10-22 - Add support for CISCO IOS 8 bytes timestamps ID 21/22 - Fix issue #72 - multiple stat output - Change -B behaviour as proposed in issue #59. Should not impact with previous use, but is more flexible - Add bzip compress switch in usage output of nfpcapd - Fix compile issues on some platforms - nfpcapd improvements - still beta software. - Minor bug fixes 2016-11-25 - Add latency extension to nfpcapd - Smaller bug fixes to nfpcapd 2016-07-23 - Replace unreliable _ftok with more reliable string hash 2016-07-20 - Aggregate using in+out bytes for bidirectional flows 2016-06-05 v.1.6.15 - Fix Security issue http://www.security-assessment.com/files/documents/advisory/Nfdump%20nfcapd%201.6.14%20-%20Multiple%20Vulnerabilities.pdf - Fix obyte, opps and obps output records - Fix wrong bps type case in cvs output. Fix opbs ipbs typos 2016-01-10 v.1.6.14 - Fix CentOS compile issues with flow-tools converter - Fix FreeBSD,OpenBSD build problems - Fix timestamp overflow in sflow.c 2015-12-23 - Fix IP Fragmentation in sflow collector - Create libnfdump for dynamic linking 2015-10-02 - Fix compile errors on other platforms - Add -R to ModifyCompression - Add std sampler ID 4 Bytes and allow random sampler (tag 50) - Add BZ2 compression along existing LZ0 - Add direct write to flowtools converter ft2nfdump - Fix zero alignment bug, if only half of an extension is sent - Fix nfanon time window bug in subsequent files in -R list - Fix CommonRecordV0Type conversion bug - Fix nfexport bug, if only one single map exists 2014-11-16 v.1.6.13 - Fix v1 extension size bug - Add htonll check for autoconf - Fix AddExtensionMap compare bug - Fix ipfix templare withdraw problems - free all maps correctly - Add minilzo 2.08 - fixes CVE-2014-4607 - Cleanup some stat code. more needs to be done .. - Cleanup man pages for -O -n - Remove SunPro test in configure - no longer supported anyway - Cleanup NAT/NSEL filter differences 2014-06-15 v1.6.12p1 - Add pblock compare functions - Update extended filter: Allow modification left/right values 2014-02-16 v1.6.12 - Add NAT pool port allocation - Modify/fix NAT vrf tags. Add egress vrf ID - Modify common record due to exporter exhaustion. new common record type 10 adds 4 extra bytes. Reads v1 common record transparently - Fix sflow potential crash 2013-11-13 v1.6.11 - Add ASA/NSEL 9.x protcol changes - Make it llvm compilable 2013-08-12 v1.6.10p1 - Fix -t +/- n timeslot option - Fix bug in nfanon - stat record update. - Fix bug in netflow v5 mudule: extension map size wrong. - Fix bug nfexport: In some cases could result in wrong flow counter. - Fix nftrack - could coredump in some cases. 2013-05-16 v1.6.10 - Fix SPARC compile/optimise bug - Add output packet/bytes counter to global stat - importatnt for NSEL flows ASA > 8.5 - Add NSEL filter options xnet - Modify extension descriptor code for nfdump1.7. Still use 1.6 extension map layout for compatibility - Add prototype for nfpcapd - pcap -> nfdump collector. Converts traffoc directly to nfdump files. - Fix bug in ipfix module: uninitialised variable - Cleanup syslog/LogError calls - Fix minor non critical bugs and compile issues 2013-03-02 v1.6.9 - Fix some bugs in beta 1.6.9 NSEL code - Fix bug statistics update with aggreagted flow records - Fix sflow bug sfcapd stores wrong (ghost) dump by past samples in same sflow datagram 2012-12-31 - Add time received in csv output - ICMP should handled better now - somewhat - Implement ASA NSEL records - Add definitions in nffile and nx for ASA NSEL extensions 2012-11-09 v1.6.8p1 - Add dynamic source directory tree for multiple exporters - Fix exporter bug: 'too many exporters' with large time windows - Fix uninitialised exporter sysid in default sampler record - v9 - Fix v9/ipfix cache initialisation with no templates > 1 in same packet 2012-10-26 v1.6.8 - Add ip list option for 'next ip' in filter syntax - Accept v9 sampler_id in 2bytes - Fix IPFIX mac address bug - did not get collected - Add IPFIX packet/octet TotalCount fields 85/86 - Add received timestamp to sflow collector - Fix long flow duration calculation - 32bit overflow - Fix v9 sampling ID: allow 2 byte ID - Add IPFIX options as rfc5101 section-6.2 - Add exporter records for sflow collector - Fix bug for MAC address printing %idmc and %odmc. - Add received time stamp extension - Add recursive format parser. Allows to extend predefined formats. - Change flow record sorting to heapsort. remove limit 1000 - Merge -m option to -O tstart. -m now depricated. - Add -O tend. Print order according to tend of flows ascending - Apply -O print order for printing flow cache. Applies to -A 2012-07-31 v1.6.7-tc-1 - Special version for TC - Print exporter and sampling records with nfdump -E - Added exporter and sampling records to file. 2012-07-30 v1.6.7 - Prepare for file catalog in current file format. - Fix bug in ReadBlock when reading flow from stdin pipe - Add new more flexible translation engine for v9 - Add nprobe client/server delay fields - Prepare for NSEL merging - Fix memory corruption with double -A flags - Fix bug in nfreader with compat15 mode files 2012-03-12 v1.6.6 - Minor IPFIX bug. - IPFIX implement template withdraw - For IPFIX, check packet sequence per template and observation domain - Fix time window, when no flows collected or no flows matched while processing - Fixed typos - Fix seg fault bug - test for EMPTY_LIST was missing at several places. 2012-02-19 v1.6.6b1 - Fix bps/pps. make it uint64_t, as bps/pps > 4Gb/s overflows. - In record raw print mode: decode ICMP instead of src/dst ports - sflow use announced exporter IP instead of sending IP for router ID - sflow: Ignore extra fill bytes. Do not complain. - sflow: fix packet length issue. - Add IPFIX protokoll support 2011-12-31 v1.6.5 - Fix 64bit bug when using byte/packet limits - for v5 and sampling use 64bit counters to prevent overflow for large sampled flows. - Fixed Ident printig bug 2011-07-11 v1.6.4 - some code restructuring - prepare for IPFIX module - Add netflow v1 module. Some routers still use that - Add %sn, %dn output tags for src/dst networks - Fix buffer length check in v5. - Fix export bug: include last flow cache bucket, when exporting - number in all filter expressions accept hex values - fix an sflow colletor bug. Missing extension maps in rotated files - implement extended statistics. Currently ports and bpp distribution vectors can be collected automatically be nfcapd. Still experimental 2011-02-26 v1.6.3p1 - Fix timebug fix :(, make it a compile time option - fix v7 sequence errors 2011-02-15 - Zero out unused fields after aggregation 2011-02-05 - Fix SysUptime 32bit overflow in v5 header - Add fix for strange first/last swap reported by some users. 2011-01-09 v1.6.3 - Fix extension size bug - Move IP anonymisation to separate binary nfanon - Fix initialise bug of -o fmt: and not available fields 2010-09-09 v1.6.2 - released - fixes some sflow bugs in sfcapd 2010-04-28 v1.6.1p0 - Update flow tools converter to build with Google-Code version 0.68.5 - Fix sflow bugs 2010-03-05 v1.6.1 - Fix bug in man page for -t - Test sampler infos before using them ( nfcapd startup ) - Add sampling tags #34, #35 used by JunOS - nfexpire: Fix empty .nfsat, when setting limits on an empty directory - Fix coredump for -B -m (-w) combination - Optimise some extension map code 2009-12-28 stable v1.6 - Few bug fixes in release candidates rc1, rc2 and rc3 2009-11-16 snapshot-1.6b-20091116 - Update sflow collector with new tags - Add router IP extension - Add router ID (engine type/ID) extension 2009-09-30 snapshot-1.6b-20090930 - snapshot bugfix release 2009-11-0801 snapshot-1.6b-20090806 - Add srcmask and dstmask aggregation - Add csv output mode. -o csv - Fix some bugs of previous beta - Add bidirectional aggregation of flows ( -b, -B ) - Add possibility to save aggregated flows into file ( -w ) Note: This results in a behaviour change for -w in combination with aggragation ) - Extend -N ( do not scale numbers ) to all text output not just summary - Make extension handling more robust for some moody IOSes. - Remove header lines of -s stat, when using -q ( quiet ) Note: This results in a behaviour change for -N - Remove -S option from nfdump ( legacy 1.4 compatibility ) - Make use of log (syslog) functions for nfprofile. - Move log functions to util.c 2009-06-19 snapshot-1.6b-20090717 - Flow-tools converter updated - supports more common elements. - Sflow collector updated. Supports more common elements. - Add sampling to nfdump. Sampling is automatically recognised in v5 undocumented header fields and in v9 option templates. see nfcapd.1(1) - Add @include option for filter to include more filter files. - Add flexible aggregation comparable to Flexible Netflow (FNF) - All new tags can be selected in -o fmt:... see nfdump(1) - topN stat for all new tags is implemented - Integrate developer code to read from pcap files into stable - Update filter syntax for new tags - Added more v9 tags for netflow v9. The detailed tags are listed in nfcapd(1) Adding new tags also extended the binary file format with data block format 2, which is extension based. File format for version <= 1.5.* ( Data block format 1 ) is read transparently. Data block 2 are skipped by nfdump 1.5.7. 32bit but AS and interface numbers are supported. - Add flexible storage option for nfcapd. To save disk space, the data extensions to be stored in the data file are user selectable. - Added option for multiple netflow stream to same port. -n Example: -n router1,192.168.100.1,/var/nfdump/router1 So multiple -n options may be given at the command line Old style syntax still works for compatibility, ( -I .. -l ... ) but then only one source is supported. - Move to automake for building nfdump - Switch scaling factor ( k, M, G ) from 1024 to 1000. - Make nfdump fully 64bit compliant. ( 8bit data alignments and access ) 2009-04-17 stable 1.5.8 - Fix daylight summer time bug, when guessing sub dirs. file access ( -M, -r ) - Bug fixes for 64bits CPUs 2008-02-22 stable-1,5.7 - Add icmp type/code decoding - Add proper icmp v9 decoding - Fix memory leaks in -e auto expire mode in nfcapd. - Fix somee potential dead locks with file locking, when expiring - Fix multicast bug in nfreplay - Add hostname lookup for IP addresses in filter. 2007-10-15 stable-1.5.6 - Fix odd CISCO behaviour for ICMP type/code in src port. - Add fast LZO1X-1 compression option (-z) for output file. - Add lists for port in syntax -> port in [ 135 137 445] - Add lists for AS syntax -> as in [ 1024 1025 ] - Bug fix in filter for syntax 'src as and dst as' 2007-08-24 stable-1.5.5 - Fix nfprofile bug, nfprofile crashes when last opts line is not valid for some reason. - Fix potential hand for nfexpire, on empty flow directories. 2007-08-08 snapshot-20070808 - Idents may contain '-' in name. - Fixed install bugs in Makefile.in and configure.in - Installs now cleanly on Solaris - Handle 4byte interface numbers in v9. Quick fix: 4bytes reduced to 2bytes. - Fix aggregation bug in statistics. - ftok(3) C library call replaced by more reliable own implementation. Did result in error messages like "Another collector is already running" - Fix minor bugs iin file range selction -R. - Add recursive behaviour for -R - New option -i can canche Ident descriptor in data files. 2007-03-12 snapshot-20070312 - Bug fix release of 20070306 2007-03-06 snapshot-20070306 - Fix bug in flist.c. Resulted in a coredump when using sub dirs and -R . ( all files ) - Fix minor bug in nfcapd.c. - Extend nfprofile for alerting system of nfsen - special version of profiles - Extend nfprofile for shadow profiles. 2007-08-10 snapshot-20070110 - Fix some compiler warnings, when compiled on a 64bit LINUX - Fixes an sflow bug: IP address was printed in wrong direction. ( lower bits first ) - Add new IP addr taging option -T for easy parsing for nfsen lookups - Add new IP list for massive address filtering: syntax: ip in [ 12345 23456 3456 ....] - Change nfprofile for channel based profiling. This breaks with old nfprofile functionality. - Remove space from ICMP type/code when followed by an IP address 2006-07-21 snapshot-20060809 - Make nfexpire ready for profile expiration - Fix bug in nfrpofile. sub dir hierarchy not handled correctly. 2006-07-21 snapshot-20060721 - Add -N option for plain number output in summary line 2006-07-21 snapshot-20060721 - Do recursive file selection when a directory is given by -R 2006-06-14 snapshot-20060621 - Add srcas/dstas/proto aggregation. Note: This changes the default aggregation behaviour, but gives more flexibility - Add tos to element statistics list 2006-06-14 snapshot-20060614 - Add additional stat line at the end of output - Add new binary nfexpire. Manages data expiry on time and/or size based limits Includes new bookkeeping records in nfcapd. See nfexpire(1) - Add ICMP type/code decoding in flow listing instead of dst port - Add packet repeater in nfcapd/sfcapd. In addition, incoming UDP packets can be directly forwarded to another IP address/Port. See new option -R - Add sub directory hierarchies: Files can be stored into various sub dir levels based on different time formats. see new option -S - Some minor bug fixes. - Code cleanup in nfcapd. better daemonize code and communication with launcher. 2006-04-xx v.1.5.1 Fix bug in nfdump.c: Writing anonymized flows to file did not work corretly stdin input format now compatible with file format, therefore 'nfdump < file' works again as it did in nfdump 1.4. Fix bug in nfcapd.c: Error handling not correct when receiving a non recognized netflow packet. Resulted in an endless loop 2006-03-27 snapshot 1.5-20060327 Make all element statistics -s transport layer protocol independant by default. Add :p to stat name ( e.g. srcip:p ) to enable transport layer dependant statistics on request. 2006-03-20 snapshot 1.5-20060320 Fix bug in filter engine: 'not flags xyz' produces wrong results when more than a single flag is specified. Minor man page fixes. 2006-03-06 v1.5 Fix bug nfcapd. Laucher signaled too early. File not yet properly closed. 2006-02-14 v1.5-beta-5 Add srcas, dstas, input and output interfaces in aggregated output. Fix IPv6 bug in filter: accept 1234:: address. rename nfcapd.curent tmp file to nfcapd.curren.. Poorly configured nfcapd processes may mess up themselves otherwise. 2006-02-02 v1.5-beta-4 Fix netflow v5 dPkts <-> dOctets collector bug. Update pipe format to include more information Allow AS number 0 in filter syntax. Add some more boundary checking - netflow exporters aren't bug free either - sigh .. 2006-01-11 v1.5-beta-3 Fix isnumber incompatibility in grammar.y Add 'if' statistics 2006-01-10 v1.5-beta-2 nf_common.c Fix bug in format parser. Extended 'proto ' syntax to support all protocols Change time format in summary line to ISO format 2005-12-20 v1.5-beta-1 *.* A lot of internal changes, not mentioned here. :( nfdump Add subnet aggregation for option -A A new syntax e.g. srcip4/24, dstip6/64 is supported for subnet wise aggregation. example: traffic of a whole subnet -A srcip4/24 -s srcip/bytes nfdump Add more stat element option. -s now supports: srcip, dstip, ip, srcport, dstport, port, srcas, dstas, as, inif, outif, proto nfdump Add -z. Suppress writing flows to data files. Only stat information is written. nfprofile Used only be nfsen for upcoming shadow profiles. If you don't understand this simply ignore it. nfdump Add -q option to suppress header as well as stat information at the bottom nfprofile for easier post processing with external programms. nf_common.c Output format processsing rewritting for more flexibility. Besides standard nfdump.c output formats line, long extended etc., user defined output formats are now possible and can even be compiled into nfdump for easy access. See -o fmt: and nfdump.c around line 100. *.* Integrate netflow v9 into nfdump. Only a subset of v9 is stored into the data files, basically everything needed for nfdump to work as it did before. This also includes IPv6 support for any nfdump options. CryptoPAN extended to work with IPv6. IPv6 condensed output format for better readability. Output formats available in long and condensed mode: e.g. line -> line6 extended -> extended6 *.* Replace binary data file format. Old format not flexible enough for upcoming netflow v9/sflow data. *.stat files are gone. The same information is now available under nfdump -I New format about 5% larger in size, but faster for reading and writing. speed gain eaten up by more complex processing - sigh .. compat14 mode enables transparent reading of old style format. nffile.[ch] now handles all data file stuff. nfreplay Multicast enabled: Add -j . Joins the specified multicast group ( v4 or v6 ) sending flows to this group. nfreplay IPv6 enabled: Add option -4 and -6 to force a specific protocol, otherwise protocol is automatically selected according the hostname to send flows to. Add -K key, to send data anonymized, using CryptoPAn nfcapd Multicast enabled: Add -j . Joins the specified multicast group ( v4 or v6 ) for listening. nfcapd IPv6 enabled: Add option -4 and -6 for IPv4 and IPv6. By default, listen on IPv4. Option -b to bind for a specific host/IP address automatically selects appropriate protocol. nfnet.c All functions to setup network sockets for listening/sending are put into this file. 2005-08-22 v1.4 - nfreplay: Bug fix sending flows. - nfdump: Add CryptoPAn code to anonymize IP addresses. New option -K - nfdump: Change time format in output to ISO 8601 compatible: e.g. 1981-04-05 14:30:30.100 - nfdump: Add scaling factor k,m,g to number in filter syntax: e.g. bytes > 1m - nfdump: Create new output format extended with additional fields pps, bps and bpp - nfdump: Rename output format extended to raw - nfdump: More than one single flow element statistic ( -s ) is now possible - nfdump: Add user defined sort order in flow element statistic - nfdump: Flow element statistic can be ordered by more than one order in the same run - nfdump: Add pps, bps and bpp fields in flow element statistics - nfdump: Add more symbolic protocols ESP, AH, GRP and RVSP to filter syntax - nfdump: Add duration, pps, bps and bpp to filter syntax - nfdump: Make nfdump miliseconds aware. Older versions skipped msecs. Binary nfdump file format changed due to this. output formats changed, due to this. - nfdump: Add interface in/out if syntax to filter - nfcapd: Add flow_sequence check. Reports missing flows now. - nfcapd: Report statistics to syslog LOG_INFO when data file is rotated. - ft2nfdump: Add ft2nfdump to read netflow data from flow-tools 2005-04-21 v1.3 - Add option -A for more flexible aggregation. - Correct spelling errors :( 2005-03-04 v1.2.1 Bug fix release - nfcapd: launcher subprocess may hang on Linux 2.6.x kernels. Cleaned up interrupt handling. - nfcapd: fix include order of socket.h and types.h in order to compile cleanly under FreeBSD 4.x - nfcapd: clean up syslog logging. - nfdump: Multiple sources ( -M ) and sort flows ( -m ) with -c did not list the correct flows. - nfprofile: Profiling with multiple sources may produce incorrect profiles. 2004-12-20 v1.2 - nfcapd handles transparent v5 and v7 flows. v7 gets converted into v5 - nfcapd can execute any command at the end of interval. New option -x - nfdump Extended filter syntax for flags, to, bytes and packets - Rearrange output formats in nfdump: new switch -o, remove switch -E output formats: 'line', 'long', 'extended' and 'pipe' - More flexible statistic handling in nfdump: cleanup ugly -s -s -s syntax. Replaced by -s option. New statistics for Port and AS. 2004-09-20 v 1.1 First public Version. nfdump-1.6.23/CreateSubHierarchy.pl000077500000000000000000000100271404501030700171420ustar00rootroot00000000000000#!/usr/bin/perl # # # Sample script to clean old data. # Run this script each hour to cleanup old files to make room for # new data. When max_size_spool is reached the oldest files are # deleted down to high_water. # # Copyright (c) 2004, SWITCH - Teleinformatikdienste fuer Lehre und Forschung # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: # # * Redistributions of source code must retain the above copyright notice, # this list of conditions and the following disclaimer. # * Redistributions in binary form must reproduce the above copyright notice, # this list of conditions and the following disclaimer in the documentation # and/or other materials provided with the distribution. # * Neither the name of SWITCH nor the names of its contributors may be # used to endorse or promote products derived from this software without # specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. # # $Author: peter $ # # $Id: CreateSubHierarchy.pl 77 2006-06-14 14:52:25Z peter $ # # $LastChangedRevision: 77 $ # use strict; use warnings; use POSIX qw(strftime); use Time::Local; use Getopt::Std; our( $opt_l, # Data directory $opt_S, # Sub hierarchy format. Correspondes to -S to nfcapd. See nfcapd(1) ); getopts('l:S:'); my $subdir_format; my @subdir_formats = ( "", "%Y/%m/%d", "%Y/%m/%d/%H", "%Y/%W/%u", "%Y/%W/%u/%H", "%Y/%j", "%Y/%j/%H", "%F", "%F/%H" ); sub usage { print "$0 [options]\n", " -l datadir Data directory\n", " -S Sub hierarchy format. Correspondes to -S to nfcapd. See nfcapd(1)\n", "\n"; exit(0); } sub ISO2UNIX { my $isotime = shift; $isotime =~ s/\-//g; # allow '-' to structur time string # 2004 02 13 12 45 / my $sec = 0; my ( $year, $mon, $mday, $hour, $min ) = $isotime =~ /(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})/; $mon--; # round down to nearest 5 min slot my $diff = $min % 5; if ( $diff ) { $min -= $diff; } my $unixtime = Time::Local::timelocal($sec,$min,$hour,$mday,$mon,$year); return $unixtime; } # End of ISO2UNIX if ( !defined $opt_l || !defined $opt_S ) { usage(); } my $data_dir = $opt_l; if ( !defined $subdir_formats[$opt_S] ) { die "Unknown format number $opt_S"; } $subdir_format = $subdir_formats[$opt_S]; opendir DIR, "$data_dir" || die "Can't open current directory: $!\n"; $| = 1; print "Reorganizing data files ... "; while ( my $entry = readdir DIR ) { next if $entry =~ /^\./; next unless -f "$data_dir/$entry"; next unless $entry =~ /nfcapd\.(\d{12})$/; my $date = $1; my $unix_time = ISO2UNIX($date); my $sub_path = strftime $subdir_format, localtime($unix_time); if ( !-d "$data_dir/$sub_path" ) { print "Need to create '$data_dir/$sub_path'\n"; my @dirlist = split '\/', $sub_path; my $all_dirs = undef; foreach my $dir ( @dirlist ) { $all_dirs = defined $all_dirs ? "$all_dirs/$dir" : $dir; if ( !-d "$data_dir/$all_dirs" ) { mkdir "$data_dir/$all_dirs" || die "Can't create subdir '$data_dir/$all_dirs'\n"; } } } rename "$data_dir/$entry", "$data_dir/$sub_path/$entry" || die "Can't move file: $!\n"; } print "done.\n"; nfdump-1.6.23/LICENSE000066400000000000000000000031201404501030700140670ustar00rootroot00000000000000 The nfdump project is distributed under the BSD license: Copyright (c) 2016, Peter Haag All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. nfdump-1.6.23/Makefile.am000077500000000000000000000002061404501030700151230ustar00rootroot00000000000000 SUBDIRS = . bin man doc EXTRA_DIST = CreateSubHierarchy.pl LICENSE BSD-license.txt extra/PortTracker.pm extra/nfdump.spec bootstrap nfdump-1.6.23/NEWS000077700000000000000000000000001404501030700153322ChangeLogustar00rootroot00000000000000nfdump-1.6.23/README000077700000000000000000000000001404501030700152202README.mdustar00rootroot00000000000000nfdump-1.6.23/README.md000077500000000000000000000372341404501030700143610ustar00rootroot00000000000000# nfdump Stable Release v1.6.23 See the Changelog file for all changes in release 1.6.23 nfdump is a toolset in order to collect and process netflow and sflow data, sent from netflow/sflow compatible devices. The toolset supports netflow __v1__, __v5/v7__,__v9__,__IPFIX__ and __SFLOW__. nfdump supports IPv4 as well as IPv6. __Note:__ nfdump 1.6.18 and newer versions __not longer__ support nfdump-1.5.x files. If you have nfdump-1.5.x please convert them before upgrading. nfdump is used as backend toolset for __NfSen__. --- ## NSEL/ASA, NEL/NAT support __NSEL__ (Network Event Security Logging) as well as NEL (NAT Event Logging) are technologies invented by __CISCO__ and also use the netflow v9 protocol. However, NSEL and NEL are not flows as commonly known but rather *__Events__!* exported from specific devices such as CISCO ASA. nfdump supports Event looging as part of netflow v9. __Note:__ The older nfdump-1.5.8-2-NSEL is __not compatible__ with nfdump > 1.6.9 which supports NSEL/NEL. __Junos NAT Event Logging__ is mostly compatible with CISCO's NAT Event Logging - mostly - it needs another data interpretation. See __--enable-jnat__ below --- ## IPFIX nfdump contains an IPFIX module for decoding IPFIX flow data. It does not support the full IPFIX definition. * Supports basically same feature set of elements as netflow_v9 module * Only UDP traffic is accepted no TCP/SCTP * If you would like to see more IPFIX support, please contact me. --- ## Overview ### Building and config options The toolset is build upon the autotools framework. Run `./autogen.sh` first. Afterwards `./configure` `make` and `make install` should do the trick. The following config options are available: * __--enable-nsel__ Compile nfdump, to read and process NSEL/NEL event data; default is __NO__ * __--enable-jnat__ compile nfdump, to read and process JunOS NAT event logging __NO__ * __--enable-ftconv__ Build the flow-tools to nfdump converter; default is __NO__ * __--enable-sflow__ Build sflow collector sfcpad; default is __NO__ * __--enable-nfprofile__ Build nfprofile used by NfSen; default is __NO__ * __--enable-nftrack__ Build nftrack used by PortTracker; default is __NO__ This code no longer reads nfdump-1.5.x data files. If needed use nfdump up to v1.6.17 Development and beta options * __--enable-devel__ Insert lots of debug and development code into nfdump for testing and debugging; default is __NO__ * __--enable-readpcap__ Add code to nfcapd to read flow data also from pcap files; default is __NO__ * __--enable-nfpcapd__ Build nfpcapd collector to create netflow data from interface traffic or precollected pcap traffic, similar to softflowd; default is __NO__ ### The tools __nfcapd__ - netflow collector daemon. Collects the netflow data, sent from exporters and stores the flow records into files. Automatically rotates files every n minutes. ( typically every 5 min ) The netflow versions mentioned above are read transparently Multiple netflow streams can be collected by a single or collector. nfcapd can listen on IPv6 or IPv4. Furthermore multicast is supported. __nfdump__ - process collected netflow records. Nfdump reads the netflow data from one or many files stored by nfcapd. It's filter syntax is similar to tcpdump ( pcap like ) but adapted for netflow. If you like tcpdump you will like nfdump. nfdump displays netflow data and/or creates top N statistics of flows, bytes, packets. nfdump has a powerful and flexible flow aggregation including bi-directional flows. The output format is user selectable and also includes a simple csv format for post processing. __nfanon__ - anonymize netflow records IP addresses in flow records are anonimized using the CryptoPAn method. __nfexpire__ - expire old netflow data Manages data expiration. Sets appropriate limits. Used by NfSen. __nfreplay__ - netflow replay Reads the netflow data from the files stored by nfcapd and sends it over the network to another host. #### Optional binaries: __nfpcapd__ - pcap to netflow collector daemon nfpcapd listens on a network interface, or reads precollected pcap traffic and stores flow records into nfcapd comaptible files. It is nfcapd's companion to convert traffic directly into nfdump records. __sfcapd__ - sflow collector daemon scfapd collects sflow data and stores it into nfcapd comaptible files. "sfcapd includes sFlow(TM) code, freely available from https://github.com/sflow/sflowtool. __nfprofile__ - netflow profiler. Required by NfSen Reads the netflow data from the files stored by nfcapd. Filters the netflow data according to the specified filter sets ( profiles ) and stores the filtered data into files for later use. __nftrack__ - Port tracking decoder for NfSen plugin PortTracker. __ft2nfdump__ - flow-tools flow converter ft2nfdump converts flow-tools data into nfdump format. __nfreader__ - Framework for programmers nfreader is a framework to read nfdump files for any other purpose. Own C code can be added to process flows. nfreader is not installed __parse_csv.pl__ - Simple reader, written in Perl. parse_csv.pl reads nfdump csv output and print the flows to stdout. This program is intended to be a framework for post processing flows for any other purpose. #### Notes for sflow users: sfcapd and nfcapd can be used concurrently to collect netflow and sflow data at the same time. Generic command line options apply to both collectors likewise. sfcapd's sflow decoding module is based on InMon's sflowtool code and supports similar fields as nfcapd does for netflow v9, which is a subset of all available sflow fields in an sflow record. More fields may be integrated in future versions of sfcapd. --- ### Compression Binary data files can optionally be compressed using either the fast LZO1X-1 compression, LZ4 or the efficient but slow bzip2 method. If you compress automatically flows while they are collected, LZO1X-1 or LZ4 methods are recommended. bzip2 uses about 30 times more CPU than LZO1X-1. Use bzip2 to archive netflow data, which may reduce the disk usage again by a factor of 2. The compression of flow files can be changed any time with nfdump -J For more details on each methde, see: LZO1X-1: http://www.oberhumer.com/opensource/lzo LZ4: https://github.com/lz4/lz4 bzip2: http://www.bzip.org You can check the compression speed for your system by running ./nftest . --- ## General Operation of nfdump The goal of the design is to able to analyze netflow data from the past as well as to track interesting traffic patterns continuously. The amount of time back in the past is limited only by the disk storage available for all the netflow data. The tools are optimized for speed for efficient filtering. The filter rules should look familiar to the syntax of tcpdump ( pcap compatible ). All data is stored to disk, before it gets analyzed. This separates the process of storing and analyzing the data. The data is organized in a time-based fashion. Every n minutes - typically 5 min - nfcapd rotates and renames the output file with the timestamp nfcapd.YYYYMMddhhmm of the interval e.g. nfcapd.200907110845 contains data from July 11th 2009 08:45 onward. Based on a 5min time interval, this results in 288 files per day. Analyzing the data can be done for a single file, or by concatenating several files for a single output. The output is either ASCII text or binary data, when saved into a file, ready to be processed again with the same tools. You may have several netflow sources - let's say 'router1' 'router2' and so on. The data is organized as follows: /flow_base_dir/router1 /flow_base_dir/router2 which means router1 and router2 are subdirs of the flow_base_dir. Although several flow sources can be sent to a single collector, It's recommended to have multiple collector on busy networks for each source. Example: Start two collectors on different ports: nfcapd -w -D -S 2 -B 1024000 -l /flow_base_dir/router1 -p 23456 nfcapd -w -D -S 2 -B 1024000 -l /flow_base_dir/router2 -p 23457 nfcapd can handle multiple flow sources. All sources can go into a single file or can be split: All into the same file: nfcapd -w -D -S 2 -l /flow_base_dir/routers -p 23456 Collected on one port and split per source: nfcapd -w -D -S 2 -n router1,172.16.17.18,/flow_base_dir/router1 \-n router2,172.16.17.20,/flow_base_dir/router2 -p 23456 See nfcapd(1) for a detailed explanation of all options. Security: none of the tools requires root privileges, unless you have a port < 1024. However, there is no access control mechanism in nfcapd. It is assumed, that host level security is in place to filter the proper IP addresses. See the manual pages or use the -h switch for details on using each of the programs. For any questions send email to peter@people.ops-trust.net Configure your router to export netflow. See the relevant documentation for your model. A generic Cisco sample configuration enabling NetFlow on an interface: ip address 192.168.92.162 255.255.255.224 interface fastethernet 0/0 ip route-cache flow To tell the router where to send the NetFlow data, enter the following global configuration command: ip flow-export 192.168.92.218 9995 ip flow-export version 5 ip flow-cache timeout active 5 This breaks up long-lived flows into 5-minute segments. You can choose any number of minutes between 1 and 60; Netflow v9 full export example of a cisco 7200 with sampling enabled: interface Ethernet1/0 ip address 192.168.92.162 255.255.255.224 duplex half flow-sampler my-map ! ! flow-sampler-map my-map mode random one-out-of 5 ! ip flow-cache timeout inactive 60 ip flow-cache timeout active 1 ip flow-capture fragment-offset ip flow-capture packet-length ip flow-capture ttl ip flow-capture vlan-id ip flow-capture icmp ip flow-capture ip-id ip flow-capture mac-addresses ip flow-export version 9 ip flow-export template options export-stats ip flow-export template options sampler ip flow-export template options timeout-rate 1 ip flow-export template timeout-rate 1 ip flow-export destination 192.168.92.218 9995 See the relevant documentation for a full description of netflow commands Note: Netflow version v5 and v7 have 32 bit counter values. The number of packets or bytes may overflow this value, within the flow-cache timeout on very busy routers. To prevent overflow, you may consider to reduce the flow-cache timeout to lower values. All nfdump tools use 64 bit counters internally, which means, all aggregated values are correctly reported. The binary format of the data files is netflow version independent. For speed reasons the binary format is machine architecture dependent, and as such can not be exchanged between little and big endian systems. Internally nfdump does all processing IP protocol independent, which means everything works for IPv4 as well as IPv6 addresses. See the nfdump(1) man page for details. netflow version 9: nfcapd supports a large range of netflow v9 tags. Version 1.6 nfdump supports the following fields. This list can be found in netflow_v9.h --- ### Flowset record types Tag | ID ----|--- NF9_IN_BYTES | 1 IN_PACKETS | 2 NF9_FLOWS_AGGR | 3 NF9_IN_PROTOCOL | 4 NF9_SRC_TOS | 5 NF9_TCP_FLAGS | 6 NF9_L4_SRC_PORT | 7 NF9_IPV4_SRC_ADDR | 8 NF9_SRC_MASK | 9 NF9_INPUT_SNMP | 10 NF9_L4_DST_PORT | 11 NF9_IPV4_DST_ADDR | 12 NF9_DST_MASK | 13 NF9_OUTPUT_SNMP | 14 NF9_V4_NEXT_HOP | 15 NF9_SRC_AS | 16 NF9_DST_AS | 17 NF9_BGP_V4_NEXT_HOP | 18 NF9_LAST_SWITCHED | 21 NF9_FIRST_SWITCHED | 22 NF9_OUT_BYTES | 23 NF9_OUT_PKTS | 24 NF9_IPV6_SRC_ADDR | 27 NF9_IPV6_DST_ADDR | 28 NF9_IPV6_SRC_MASK | 29 NF9_IPV6_DST_MASK | 30 NF9_IPV6_FLOW_LABEL | 31 NF9_ICMP_TYPE | 32 NF9_SAMPLING_INTERVAL | 34 NF9_SAMPLING_ALGORITHM | 35 NF9_ENGINE_TYPE | 38 NF9_ENGINE_ID | 39 NF9_FLOW_SAMPLER_ID | 48 FLOW_SAMPLER_MODE | 49 NF9_FLOW_SAMPLER_RANDOM_INTERVAL | 50 NF9_MIN_TTL | 52 NF9_MAX_TTL | 53 NF9_IPV4_IDENT | 54 NF9_DST_TOS | 55 NF9_IN_SRC_MAC | 56 NF9_OUT_DST_MAC | 57 NF9_SRC_VLAN | 58 NF9_DST_VLAN | 59 NF9_DIRECTION | 61 NF9_V6_NEXT_HOP | 62 NF9_BPG_V6_NEXT_HOP | 63 // NF9_V6_OPTION_HEADERS | 64 NF9_MPLS_LABEL_1 | 70 NF9_MPLS_LABEL_2 | 71 NF9_MPLS_LABEL_3 | 72 NF9_MPLS_LABEL_4 | 73 NF9_MPLS_LABEL_5 | 74 NF9_MPLS_LABEL_6 | 75 NF9_MPLS_LABEL_7 | 76 NF9_MPLS_LABEL_8 | 77 NF9_MPLS_LABEL_9 | 78 NF9_MPLS_LABEL_10 | 79 NF9_IN_DST_MAC | 80 NF9_OUT_SRC_MAC | 81 NF9_FORWARDING_STATUS | 89 NF9_BGP_ADJ_NEXT_AS | 128 NF9_BGP_ADJ_PREV_AS | 129 ### CISCO ASA NSEL extension - Network Security Event Logging__ Tag | ID ----|--- NF_F_FLOW_BYTES | 85 NF_F_CONN_ID | 148 NF_F_FLOW_CREATE_TIME_MSEC | 152 NF_F_ICMP_TYPE | 176 NF_F_ICMP_CODE | 177 NF_F_ICMP_TYPE_IPV6 | 178 NF_F_ICMP_CODE_IPV6 | 179 NF_F_FWD_FLOW_DELTA_BYTES | 231 NF_F_REV_FLOW_DELTA_BYTES | 232 NF_F_FW_EVENT84 | 233 NF_F_EVENT_TIME_MSEC | 323 NF_F_INGRESS_ACL_ID | 33000 NF_F_EGRESS_ACL_ID | 33001 NF_F_FW_EXT_EVENT | 33002 NF_F_USERNAME | 40000 NF_F_XLATE_SRC_ADDR_IPV4 | 40001 NF_F_XLATE_DST_ADDR_IPV4 | 40002 NF_F_XLATE_SRC_PORT | 40003 NF_F_XLATE_DST_PORT | 40004 NF_F_FW_EVENT | 40005 ### Cisco ASR 1000 series NEL extension - Nat Event Logging__ Tag | ID ----|--- NF_N_NAT_EVENT | 230 NF_N_INGRESS_VRFID | 234 NF_N_EGRESS_VRFID | 235 NF_N_NAT_INSIDE_GLOBAL_IPV4 | 225 NF_N_NAT_OUTSIDE_GLOBAL_IPV4 | 226 NF_N_POST_NAPT_SRC_PORT | 227 NF_N_POST_NAPT_DST_PORT | 228 ### latency extensions for nfpcapd and nprobe__ Tag | ID ----|--- NF9_NPROBE_CLIENT_NW_DELAY_SEC | 57554 NF9_NPROBE_CLIENT_NW_DELAY_USEC | 57555 NF9_NPROBE_SERVER_NW_DELAY_SEC | 57556 NF9_NPROBE_SERVER_NW_DELAY_USEC | 57557 NF9_NPROBE_APPL_LATENCY_SEC | 57558 NF9_NPROBE_APPL_LATENCY_USEC | 57559 32 and 64 bit counters are supported for any counters. However, internally nfdump stores packets and bytes counters always as 64bit counters. 16 and 32 bit AS numbers are supported. Extensions: nfcapd supports a large number of v9 tags. In order to optimise disk space and performance, v9 tags are grouped into a number of extensions which may or may not be stored into the data file. Therefore the v9 templates configured on the exporter may be tuned with the collector. Only the tags common to both are stored into the data files. Extensions can be switch on/off by using the -T option. If you want to collect all data, use __-Tall__ ### Sampling By default, the sampling rate is set to 1 (unsampled) or to any given value specified by the -s cmd line option. If sampling information is found in the netflow stream, it overwrites the default value. Sampling is automatically recognised when announced in v9 option templates (tags #48, #49, #50 ), (tag #34, #35) or in the unofficial v5 header hack. Note: Not all platforms (or IOS versions) support exporting sampling information in netflow data, even if sampling is configured. The number of bytes/packets in each netflow record is automatically multiplied by the sampling rate. The total number of flows is not changed as this is not accurate enough. (Small flows versus large flows) ### InfluxDB You can send nfprofile stats data to an influxdb database. The data are the same of rrd files. For enable this option you need libcurl dev package installed, use --enable-influxdb for configure the project and the nfprofile command should be invoked with option: -i . Example: -i http://localhost:8086/write?db=mydb&u=user&p=pass The parameters for auth (&u=user&p=pass) are optional. Then you get the stats data on influxdb mydb in the measurement nfsen_stats. For put the stats of live profile you need to apply a patch to nfsen (in extra/nfsen) and add in nfsen.conf the option: $influxdb_url="http://mydbhost.local:8086/write?db=nfsen"; as example I added a preconfigured grafana dashboard in extra/grafana/Nfsen_Stats.json . --- For more information, see the GitHub Wiki nfdump-1.6.23/autogen.sh000077500000000000000000000041451404501030700150730ustar00rootroot00000000000000#!/bin/sh set -e case "$(uname)" in Darwin) LIBTOOLIZE=${LIBTOOLIZE:-glibtoolize} ;; *) LIBTOOLIZE=${LIBTOOLIZE:-libtoolize} ;; esac AUTORECONF=${AUTORECONF:-autoreconf} ACLOCAL=${ACLOCAL:-aclocal} AUTOCONF=${AUTOCONF:-autoconf} AUTOHEADER=${AUTOHEADER:-autoheader} AUTOMAKE=${AUTOMAKE:-automake} # Check we have all tools installed check_command() { command -v "${1}" > /dev/null 2>&1 || { >&2 echo "autogen.sh: could not find \`$1'. \`$1' is required to run autogen.sh." exit 1 } } check_command "$LIBTOOLIZE" check_command "$AUTORECONF" check_command "$ACLOCAL" check_command "$AUTOCONF" check_command "$AUTOHEADER" check_command "$AUTOMAKE" # Absence of pkg-config or misconfiguration can make some odd error # messages, we check if it is installed correctly. See: # https://blogs.oracle.com/mandy/entry/autoconf_weirdness # # We cannot just check for pkg-config command, we need to check for # PKG_* macros. The pkg-config command can be defined in ./configure, # we cannot tell anything when not present. check_pkg_config() { grep -q '^AC_DEFUN.*PKG_CHECK_MODULES' aclocal.m4 || { cat <&2 autogen.sh: could not find PKG_CHECK_MODULES macro. Either pkg-config is not installed on your system or \`pkg.m4' is missing or not found by aclocal. If \`pkg.m4' is installed at an unusual location, re-run \`autogen.sh' by setting \`ACLOCAL_FLAGS': ACLOCAL_FLAGS="-I /share/aclocal" ./autogen.sh EOF exit 1 } } echo "autogen.sh: reconfigure with autoreconf" ${AUTORECONF} -vif -I m4 || { echo "autogen.sh: autoreconf has failed ($?), let's do it manually" [ -f ./configure.ac ] || [ -f ./configure.in ] || continue echo "autogen.sh: configure `basename $PWD`" ${ACLOCAL} -I m4 ${ACLOCAL_FLAGS} check_pkg_config ${LIBTOOLIZE} --automake --copy --force ${ACLOCAL} -I m4 ${ACLOCAL_FLAGS} ${AUTOCONF} --force ${AUTOHEADER} ${AUTOMAKE} --add-missing --copy --force-missing } echo "autogen.sh: for the next step, run './configure' [or './configure --help' to check available options]" exit 0 nfdump-1.6.23/bin/000077500000000000000000000000001404501030700136365ustar00rootroot00000000000000nfdump-1.6.23/bin/AddExtension.txt000077500000000000000000000005421404501030700167700ustar00rootroot00000000000000Add extensions: =============== 1. Add appropriate definitions for new extension in nffile.h 2. Update extension_descriptor array in nfx.c to reflect the new extension. Increase user index ( 3rd value ) 3. Add filter in grammar.y 4. test filter with nftest. 5. nffile_inline.c 6. nf_common.c define printing functions 7. nfsta.c add stat definitions nfdump-1.6.23/bin/Makefile.am000077500000000000000000000103401404501030700156730ustar00rootroot00000000000000 bin_PROGRAMS = nfcapd nfdump nfreplay nfexpire nfanon check_PROGRAMS = nftest nfgen nfreader EXTRA_DIST = applybits_inline.c nffile_inline.c collector_inline.c inline.c nfdump_inline.c heapsort_inline.c test.sh nfdump.test.out nfdump.test.diff check_PROGRAMMS = test.sh TESTS = nftest test.sh if SFLOW bin_PROGRAMS += sfcapd endif if NFPROFILE bin_PROGRAMS += nfprofile endif if NFTRACK bin_PROGRAMS += nftrack endif if FT2NFDUMP bin_PROGRAMS += ft2nfdump endif BUILT_SOURCES = grammar.h AM_YFLAGS = -d AM_CPPFLAGS = $(DEPS_CFLAGS) LDADD = $(DEPS_LIBS) #Add extra debug info for gdb AM_CFLAGS = -ggdb # libnfdump sources output = output_util.h output_util.c output_raw.h output_raw.c output += output_json.c output_json.h output_csv.c output_csv.h output_pipe.c output_pipe.h output += output_fmt.c output_fmt.h util = util.c util.h filelzo = minilzo.c minilzo.h lzoconf.h lzodefs.h lz4.c lz4.h nffile = nffile.c nffile.h nfx.c nfx.h nflist = flist.c flist.h fts_compat.c fts_compat.h filter = grammar.y scanner.l nftree.c nftree.h ipconv.c ipconv.h rbtree.h exporter = exporter.c exporter.h nfprof = nfprof.c nfprof.h nfnet = nfnet.c nfnet.h anon = panonymizer.c panonymizer.h rijndael.c rijndael.h collector = collector.c collector.h nfv1 = netflow_v1.c netflow_v1.h nfv9 = netflow_v9.c netflow_v9.h # pcaproc = pcaproc.c pcaproc.h flowtree.c flowtree.h ipfrag.c ipfrag.h malloc_hook.c pcaproc = pcaproc.c pcaproc.h flowtree.c flowtree.h ipfrag.c ipfrag.h content = content_dns.c content_dns.h netflow_pcap = netflow_pcap.c netflow_pcap.h ipfix = ipfix.c ipfix.h nfv5v7 = netflow_v5_v7.c netflow_v5_v7.h nfstatfile = nfstatfile.c nfstatfile.h nflowcache = nflowcache.c nflowcache.h bookkeeper = bookkeeper.c bookkeeper.h expire= expire.c expire.h launch = launch.c launch.h lib_LTLIBRARIES = libnfdump.la libnfdump_la_SOURCES = $(output) $(util) $(filelzo) $(nffile) $(nflist) $(filter) $(exporter) libnfdump_la_LDFLAGS = -release 1.6.23 nfdump_SOURCES = nfdump.c nfdump.h nfstat.c nfstat.h nfexport.c nfexport.h \ $(nflowcache) $(nfprof) nfdump_LDADD = libnfdump.la nfreplay_SOURCES = nfreplay.c $(nfprof) \ $(nfnet) $(collector) $(nfv1) $(nfv9) $(nfv5v7) $(ipfix) nfreplay_LDADD = libnfdump.la nfprofile_SOURCES = nfprofile.c profile.c profile.h $(nfstatfile) nfprofile_LDADD = libnfdump.la -lrrd nftrack_SOURCES = ../extra/nftrack/nftrack.c \ ../extra/nftrack/nftrack_rrd.c ../extra/nftrack/nftrack_rrd.h \ ../extra/nftrack/nftrack_stat.c ../extra/nftrack/nftrack_stat.h nftrack_CFLAGS = -I ../extra/nftrack nftrack_LDADD = libnfdump.la -lrrd nfcapd_SOURCES = nfcapd.c \ $(nfstatfile) $(launch) \ $(nfnet) $(collector) $(nfv1) $(nfv5v7) $(nfv9) $(ipfix) $(bookkeeper) $(expire) nfcapd_LDADD = libnfdump.la nfpcapd_SOURCES = nfpcapd.c \ $(pcaproc) $(netflow_pcap) \ $(nfstatfile) $(launch) \ $(nfnet) $(collector) $(bookkeeper) $(expire) $(content) nfpcapd_LDADD = libnfdump.la if READPCAP nfcapd_CFLAGS = -DPCAP nfcapd_SOURCES += pcap_reader.c pcap_reader.h nfcapd_LDADD += -lpcap endif if BUILDNFPCAPD bin_PROGRAMS += nfpcapd nfpcapd_CFLAGS = -D_BSD_SOURCE -D_DEFAULT_SOURCE nfpcapd_LDADD += -lpcap nfpcapd_LDFLAGS = -pthread endif sfcapd_SOURCES = sfcapd.c sflow_nfdump.c sflow_nfdump.h sflow.h sflow_v2v4.h sflow_process.c sflow_process.h\ $(nfstatfile) $(launch) \ $(nfnet) $(collector) $(bookkeeper) $(expire) sfcapd_LDADD = libnfdump.la if READPCAP sfcapd_CFLAGS = -DPCAP sfcapd_SOURCES += pcap_reader.c pcap_reader.h sfcapd_LDADD += -lpcap endif nfreader_SOURCES = nfreader.c nfreader_LDADD = libnfdump.la nfanon_SOURCES = nfanon.c $(anon) nfanon_LDADD = libnfdump.la nfgen_SOURCES = nfgen.c nfgen_LDADD = libnfdump.la nfexpire_SOURCES = nfexpire.c \ $(bookkeeper) $(expire) $(nfstatfile) nfexpire_LDADD = libnfdump.la @FTS_OBJ@ nftest_SOURCES = nftest.c nftest_LDADD = libnfdump.la nftest_DEPENDENCIES = nfgen if FT2NFDUMP ft2nfdump_SOURCES = ft2nfdump.c ft2nfdump_CFLAGS = @FT_INCLUDES@ ft2nfdump_LDADD = libnfdump.la -lft -lz ft2nfdump_LDADD += @FT_LDFLAGS@ endif check_DIST = inline.c collector_inline.c nffile_inline.c nfdump_inline.c heapsort_inline.c applybits_inline.c check_DIST += test.sh nfdump.test.out parse_csv.pl AddExtension.txt nfdump.test.diff CLEANFILES = lex.yy.c grammar.c grammar.h scanner.c scanner.h $(check_PROGRAMS) *.gch nfdump-1.6.23/bin/anon.flows000066400000000000000000000117701404501030700156530ustar00rootroot00000000000000 ¥N`‹E l“æEåKŠE ŒòÏ´3¤¢å½&Äå'ˆúð@†üð@ Ø(|  ¬ ˆúð@’úð@ê¬Ù¤HØÊ/ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ nx’úð@¦úð@ ê¬Ø¤HØÊ/ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ ÒÜœúð@ºúð@ ê¬Ø¤HØef €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ 6@¦úð@Îúð@ ê¬Ú¤HØef €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ š¤°úð@âúð@èê¬Û¤HØéê €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ þºúð@öúð@3Ð M ê¬Ð¤HØ'' €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ blÄúð@ ûð@¸= ê¬Ñ¤HØ¡†¢† €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ ÆÐÎúð@ûð@ " ê¬Ò¤HØABBB €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ *4Øúð@2ûð@  ½ê¬Ó¤HØ–˜é €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ Ž˜âúð@Fûð@ ˆê¬Ô¤HØô–˜ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ ìúð@Zûð@ ˆbê¬Õ¤HØô–˜ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ nxöúð@nûð@ ÿp-ê¬Ö¤H؈áõ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ ÒÜûð@‚ûð@ ?XÍê¬×¤H؈Êš; €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ 6@ ûð@–ûð@bê¬Î¤HØRÃPà €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ¬ š¤ûð@ªûð@('¨a¦¨]¬µ¨]¬!¡ ¡ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ Ä þûð@¾ûð@ÿÿÿö+ÍÔ]ÿÿÿö¡½*íÍ«ï¡ ü: €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ Ä bl(ûð@Òûð@(yÌÌÿ»ªÌýþ æ×ùÃ$îÿ ߘ?­þ?Eüýt'Àáä €X0’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ È ÆÐ2ûð@æûð@(yÌÌÿ»ªÌýþ æ×ùÃ$îÿ ߘ?­þ?Eüý š €X0’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ È *4<ûð@úûð@(yÌÌÿ»ªÌýþ æ×ùÃ$îÿ ߘ?­þ?EüýÀáä €X0’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ Ì Ž˜Fûð@üð@(yÌÌÿ»ªÌýþ æ×ùÃ$îÿ ߘ?­þ?Eüý €X0’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ° Pûð@"üð@(yÌê¬Ï¤HØ š €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ° nxZûð@6üð@(yÌòê¬Í¤HØÀáä €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€ ´ ÒÜdûð@Jüð@(yÌ’ê¬Ì¤HØ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€(€  ¸ 6@nûð@^üð@(yÌ’ê¬Ì¤HØ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€(€  ¸ š¤xûð@rüð@(yÌ’ê¬Ì¤HØ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€(„  ¼ þ‚ûð@†üð@(yÌ’ê¬Ì¤HØ €’ž‰¬òˆ¬R]Ô¢ ªxV4ª»ÌÝîÿÿ»ÌÝîªxV4ª ?@~`½€ü ;Àzà¹ù 8Apø‚~ªªì²€nfdump-1.6.23/bin/applybits_inline.c000077500000000000000000000067111404501030700173570ustar00rootroot00000000000000/* * Copyright (c) 2017, Peter Haag * Copyright (c) 2014, Peter Haag * Copyright (c) 2009, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * * */ static inline void ApplyNetMaskBits(master_record_t *flow_record, int apply_netbits); static inline void ApplyNetMaskBits(master_record_t *flow_record, int apply_netbits) { if ( (flow_record->flags & FLAG_IPV6_ADDR ) != 0 ) { // IPv6 if ( apply_netbits & 1 ) { uint64_t mask; uint32_t mask_bits = flow_record->src_mask; if ( mask_bits > 64 ) { mask = 0xffffffffffffffffLL << ( 128 - mask_bits ); flow_record->V6.srcaddr[1] &= mask; } else { mask = 0xffffffffffffffffLL << ( 64 - mask_bits ); flow_record->V6.srcaddr[0] &= mask; flow_record->V6.srcaddr[1] = 0; } } if ( apply_netbits & 2 ) { uint64_t mask; uint32_t mask_bits = flow_record->dst_mask; if ( mask_bits > 64 ) { mask = 0xffffffffffffffffLL << ( 128 - mask_bits ); flow_record->V6.dstaddr[1] &= mask; } else { mask = 0xffffffffffffffffLL << ( 64 - mask_bits ); flow_record->V6.dstaddr[0] &= mask; flow_record->V6.dstaddr[1] = 0; } } } else { // IPv4 if ( apply_netbits & 1 ) { uint32_t srcmask = 0xffffffff << ( 32 - flow_record->src_mask ); flow_record->V4.srcaddr &= srcmask; } if ( apply_netbits & 2 ) { uint32_t dstmask = 0xffffffff << ( 32 - flow_record->dst_mask ); flow_record->V4.dstaddr &= dstmask; } } } // End of ApplyNetMaskBits static inline void ApplyAggrMask(master_record_t *record, master_record_t *mask) { uint64_t *r = (uint64_t *)record; uint64_t *m = (uint64_t *)mask; int i, max_offset; max_offset = offsetof(master_record_t, map_ref) >> 3; for (i=2; i #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #ifndef HAVE_SEMUN union semun { int val; // value for SETVAL struct semid_ds *buf; // buffer for IPC_STAT & IPC_SET u_short *array; // array for GETALL & SETALL }; #endif #include "util.h" #include "bookkeeper.h" static bookkeeper_list_t *bookkeeper_list = NULL; /* function prototypes */ static key_t hash(char *str, int flag); static void sem_lock(int sem_set_id); static void sem_unlock(int sem_set_id); static inline bookkeeper_list_t *Get_bookkeeper_list_entry(bookkeeper_t *bookkeeper); /* Create shared memory object and set its size */ /* hash: compute hash value of string */ #define MULTIPLIER 37 static key_t hash(char *str, int flag) { uint32_t h; unsigned char *p; char cleanPath[MAXPATHLEN]; if ( realpath(str, cleanPath) == NULL ) { return -1; } h = 0; for (p = (unsigned char*)cleanPath; *p != '\0'; p++) h = MULTIPLIER * h + *p; if ( flag ) { h = MULTIPLIER * h + 'R'; } // LogError("Bookeeper hash for path: '%s' -> '%s': %u flag: %i", str, cleanPath, h, flag); return (key_t)h; // or, h % ARRAY_SIZE; } // End of hash // locks the semaphore, for exclusive access to the bookkeeping record static void sem_lock(int sem_set_id) { struct sembuf sem_op; /* wait on the semaphore, unless it's value is non-negative. */ sem_op.sem_num = 0; sem_op.sem_op = -1; sem_op.sem_flg = 0; if ( semop(sem_set_id, &sem_op, 1) == 0 ) return; LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); } // End of sem_lock // sem_unlock. un-locks the semaphore. static void sem_unlock(int sem_set_id) { struct sembuf sem_op; /* signal the semaphore - increase its value by one. */ sem_op.sem_num = 0; sem_op.sem_op = 1; sem_op.sem_flg = 0; if ( semop(sem_set_id, &sem_op, 1) == 0 ) return; LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); } // End of sem_unlock static inline bookkeeper_list_t *Get_bookkeeper_list_entry(bookkeeper_t *bookkeeper) { bookkeeper_list_t *bookkeeper_list_entry; if ( bookkeeper == NULL ) return NULL; bookkeeper_list_entry = bookkeeper_list; while ( bookkeeper_list_entry != NULL && bookkeeper_list_entry->bookkeeper != bookkeeper ) bookkeeper_list_entry = bookkeeper_list_entry->next; return bookkeeper_list_entry; } // End of Get_bookkeeper_list_entry int InitBookkeeper(bookkeeper_t **bookkeeper, char *path, pid_t nfcapd_pid, pid_t launcher_pid) { int sem_key, shm_key, shm_id, sem_id; union semun sem_val; bookkeeper_list_t **bookkeeper_list_entry; *bookkeeper = NULL; shm_key = hash(path, 0); if ( shm_key == - 1 ) return ERR_PATHACCESS; // check if the shared memory is already allocated shm_id = shmget(shm_key, sizeof(bookkeeper_t), 0600); if ( shm_id >= 0 ) { // the segment already exists. Either a running process is active // or an unclean shutdown happened // map the segement and check the record *bookkeeper = (bookkeeper_t *)shmat(shm_id, NULL, 0); if ( *bookkeeper == (bookkeeper_t *)-1 ) { LogError("shmat() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } if ( (*bookkeeper)->nfcapd_pid <= 0 ) { // rubbish or invalid pid of nfcapd process. // Assume unclean shutdown or something else. We clean up and take this record. memset((void *)(*bookkeeper), 0, sizeof(bookkeeper_t)); } else { // check if the process created this record is still alive int ret = kill((*bookkeeper)->nfcapd_pid, 0); if ( ret == - 1 ) { switch (errno) { case ESRCH: // process does not exist, we can clean up this record and use it memset((void *)(*bookkeeper), 0, sizeof(bookkeeper_t)); break; case EPERM: // A process exists, but we are not allowed to signal this process LogError("Another collector with pid %i but different user ID is already running, and configured for '%s'", (*bookkeeper)->nfcapd_pid, path); return ERR_EXISTS; break; default: // This should never happen, but catch it anyway LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } } else { // process exists; LogError("Another collector with pid %i is already running, and configured for '%s'", (*bookkeeper)->nfcapd_pid, path); return ERR_EXISTS; } // if we pass this point, we have recycled an existing record } } else { // no valid shared segment was found switch (errno) { case ENOENT: // this is ok - no shared segemtn exists, we can create a new one below break; case EACCES: // there is such a segment, but we are not allowed to get it // Assume it's another nfcapd LogError("Access denied to collector bookkeeping record."); return ERR_EXISTS; break; default: // This should never happen, but catch it anyway LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } // we now create a new segement, this hould not fail now shm_id = shmget(shm_key, sizeof(bookkeeper_t), IPC_CREAT | 0600); if ( shm_id == - 1 ) { // but did anyway - give up LogError("shmget() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } *bookkeeper = (bookkeeper_t *)shmat(shm_id, NULL, 0); if ( (*bookkeeper) == (bookkeeper_t *)-1 ) { LogError("shmget() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } memset((void *)(*bookkeeper), 0, sizeof(bookkeeper_t)); } // at this point we now have a valid record and can proceed (*bookkeeper)->nfcapd_pid = nfcapd_pid; (*bookkeeper)->launcher_pid = launcher_pid; (*bookkeeper)->sequence++; // create semaphore sem_key = hash(path, 1); // this should never fail, as we aleady got a key for the shared memory if ( sem_key == - 1 ) { // .. but catch it anyway .. and release shared memory. something is fishy struct shmid_ds buf; shmdt((void *)(*bookkeeper)); shmctl(shm_id, IPC_RMID, &buf); return ERR_FAILED; } // get the semaphore sem_id = semget(sem_key, 1, IPC_CREAT | 0600); if ( sem_id == - 1 ) { struct shmid_ds buf; // this should not have failed LogError("semget() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); // release shared memory shmdt((void *)(*bookkeeper)); shmctl(shm_id, IPC_RMID, &buf); return ERR_FAILED; } // initialize the semaphore sem_val.val = 1; if ( semctl(sem_id, 0, SETVAL, sem_val) == -1) { struct shmid_ds buf; // this should not have failed LogError("semctl() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); // release shared memory shmdt((void *)(*bookkeeper)); shmctl(shm_id, IPC_RMID, &buf); return ERR_FAILED; } bookkeeper_list_entry = &bookkeeper_list; while ( *bookkeeper_list_entry != NULL ) bookkeeper_list_entry = &((*bookkeeper_list_entry)->next); (*bookkeeper_list_entry) = (bookkeeper_list_t *)malloc(sizeof(bookkeeper_list_t)); if ( !*bookkeeper_list_entry ) { struct shmid_ds buf; LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); shmdt((void *)(*bookkeeper)); shmctl(shm_id, IPC_RMID, &buf); semctl( sem_id, 0, IPC_RMID ); return ERR_FAILED; } memset((void *)*bookkeeper_list_entry, 0, sizeof(bookkeeper_list_t)); (*bookkeeper_list_entry)->shm_id = shm_id; (*bookkeeper_list_entry)->sem_id = sem_id; (*bookkeeper_list_entry)->bookkeeper = *bookkeeper; (*bookkeeper_list_entry)->next = NULL; // we are done return BOOKKEEPER_OK; } // End of InitBookkeeper int AccessBookkeeper(bookkeeper_t **bookkeeper, char *path) { bookkeeper_list_t **bookkeeper_list_entry; int sem_key, shm_key, shm_id, sem_id; *bookkeeper = NULL; shm_key = hash(path, 0); if ( shm_key == - 1 ) return ERR_PATHACCESS; // check if the shared memory is already allocated shm_id = shmget(shm_key, sizeof(bookkeeper_t), 0600); if ( shm_id < 0 ) { // the segment does not exists. Check why switch (errno) { case ENOENT: // no shared segemtn exists. return ERR_NOTEXISTS; break; case EACCES: // there is such a segment, but we are not allowed to get it // Assume it's another nfcapd LogError("Access denied to collector bookkeeping record."); return ERR_FAILED; break; default: // This should never happen, but catch it anyway LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } // not reached } // at this point we now have a valid record and can proceed // create semaphore sem_key = hash(path, 1); // this should never fail, as we aleady got a key for the shared memory if ( sem_key == - 1 ) { // .. but catch it anyway .. and release shared memory. something is fishy return ERR_FAILED; } // get the semaphore sem_id = semget(sem_key, 1, 0600); if ( sem_id == - 1 ) { // this should not have failed LogError("semget() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } // map the shared segment *bookkeeper = (bookkeeper_t *)shmat(shm_id, NULL, 0); if ( *bookkeeper == (bookkeeper_t *)-1 ) { LogError("shmat() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } bookkeeper_list_entry = &bookkeeper_list; while ( *bookkeeper_list_entry != NULL && (*bookkeeper_list_entry)->bookkeeper != NULL ) bookkeeper_list_entry = &((*bookkeeper_list_entry)->next); // allocate new slot, else use unused slot if ( *bookkeeper_list_entry == NULL ) { (*bookkeeper_list_entry) = (bookkeeper_list_t *)malloc(sizeof(bookkeeper_list_t)); if ( !*bookkeeper_list_entry ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return ERR_FAILED; } memset((void *)*bookkeeper_list_entry, 0, sizeof(bookkeeper_list_t)); } (*bookkeeper_list_entry)->shm_id = shm_id; (*bookkeeper_list_entry)->sem_id = sem_id; (*bookkeeper_list_entry)->bookkeeper = *bookkeeper; (*bookkeeper_list_entry)->next = NULL; return BOOKKEEPER_OK; } // End of AccessBookkeeper void ReleaseBookkeeper(bookkeeper_t *bookkeeper, int destroy) { bookkeeper_list_t *bookkeeper_list_entry; struct shmid_ds buf; if ( !bookkeeper ) return; bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper); if ( !bookkeeper_list_entry ) { // this should never happen LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list"); return; } // detach from my process addr space memory if ( shmdt((void *)bookkeeper) == -1 ) { // ups .. LogError("shmdt() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); } bookkeeper = NULL; if ( destroy == 0 ) { // Entry no longer valid bookkeeper_list_entry->bookkeeper = NULL; bookkeeper_list_entry->shm_id = 0; bookkeeper_list_entry->sem_id = 0; return; } // prevent other proceeses to access the share memory, while we are removing it // try to clean up. sem_lock(bookkeeper_list_entry->sem_id); if ( shmctl(bookkeeper_list_entry->shm_id, IPC_RMID, &buf) ) { // ups .. LogError("shmctl() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); } sem_unlock(bookkeeper_list_entry->sem_id); if ( semctl( bookkeeper_list_entry->sem_id, 0, IPC_RMID ) == -1 ) { // ups .. LogError("semctl() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); } // Entry no longer valid bookkeeper_list_entry->bookkeeper = NULL; bookkeeper_list_entry->shm_id = 0; bookkeeper_list_entry->sem_id = 0; } // End of ReleaseBookkeeper int LookBooks(bookkeeper_t *bookkeeper) { bookkeeper_list_t *bookkeeper_list_entry; if ( !bookkeeper ) return 0; bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper); if ( !bookkeeper_list_entry ) { // this should never happen LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list"); return 1; } sem_lock(bookkeeper_list_entry->sem_id); return 0; } // End of LookBooks int UnlookBooks(bookkeeper_t *bookkeeper) { bookkeeper_list_t *bookkeeper_list_entry; if ( !bookkeeper ) return 0; bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper); if ( !bookkeeper_list_entry ) { // this should never happen LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list"); return 1; } sem_unlock(bookkeeper_list_entry->sem_id); return 0; } // End of UnlookBooks void ClearBooks(bookkeeper_t *bookkeeper, bookkeeper_t *tmp_books) { bookkeeper_list_t *bookkeeper_list_entry; if ( !bookkeeper ) return; bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper); if ( !bookkeeper_list_entry ) { // this should never happen LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list"); return; } sem_lock(bookkeeper_list_entry->sem_id); // backup copy if ( tmp_books != NULL ) { memcpy((void *)tmp_books, (void *)bookkeeper, sizeof(bookkeeper_t)); } bookkeeper->first = 0; bookkeeper->last = 0; bookkeeper->numfiles = 0; bookkeeper->filesize = 0; bookkeeper->sequence++; sem_unlock(bookkeeper_list_entry->sem_id); } // End of ClearBooks uint64_t BookSequence(bookkeeper_t *bookkeeper) { bookkeeper_list_t *bookkeeper_list_entry; uint64_t seq; if ( !bookkeeper ) return 0; bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper); if ( !bookkeeper_list_entry ) { // this should never happen LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list"); return 0; } sem_lock(bookkeeper_list_entry->sem_id); seq = bookkeeper->sequence; sem_unlock(bookkeeper_list_entry->sem_id); return seq; } // End of BookSequence void UpdateBooks(bookkeeper_t *bookkeeper, time_t when, uint64_t size) { bookkeeper_list_t *bookkeeper_list_entry; if ( !bookkeeper ) return; bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper); if ( !bookkeeper_list_entry ) { // this should never happen LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list"); return; } sem_lock(bookkeeper_list_entry->sem_id); if ( bookkeeper->first == 0 ) bookkeeper->first = when; bookkeeper->last = when; bookkeeper->numfiles++; bookkeeper->filesize += size; bookkeeper->sequence++; sem_unlock(bookkeeper_list_entry->sem_id); } // End of UpdateBooks void UpdateBooksParam(bookkeeper_t *bookkeeper, time_t lifetime, uint64_t maxsize) { bookkeeper_list_t *bookkeeper_list_entry; if ( !bookkeeper ) return; bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper); if ( !bookkeeper_list_entry ) { // this should never happen LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list"); return; } sem_lock(bookkeeper_list_entry->sem_id); bookkeeper->max_lifetime = lifetime; bookkeeper->max_filesize = maxsize; bookkeeper->sequence++; sem_unlock(bookkeeper_list_entry->sem_id); } // End of UpdateBooksParam void PrintBooks(bookkeeper_t *bookkeeper) { bookkeeper_list_t *bookkeeper_list_entry; struct tm *ts; time_t t; char string[32]; if ( !bookkeeper ) { printf("No bookkeeper record available!\n"); return; } bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper); if ( !bookkeeper_list_entry ) { // this should never happen LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list"); return; } sem_lock(bookkeeper_list_entry->sem_id); printf("Collector process: %lu\n", (unsigned long)bookkeeper->nfcapd_pid); if ( bookkeeper->launcher_pid ) printf("Launcher process: %lu\n", (unsigned long)bookkeeper->launcher_pid); else printf("Launcher process: \n"); printf("Record sequence : %llu\n", (unsigned long long)bookkeeper->sequence); t = bookkeeper->first; ts = localtime(&t); strftime(string, 31, "%Y-%m-%d %H:%M:%S", ts); string[31] = '\0'; printf("First : %s\n", bookkeeper->first ? string : ""); t = bookkeeper->last; ts = localtime(&t); strftime(string, 31, "%Y-%m-%d %H:%M:%S", ts); string[31] = '\0'; printf("Last : %s\n", bookkeeper->last ? string : ""); printf("Number of files : %llu\n", (unsigned long long)bookkeeper->numfiles); printf("Total file size : %llu\n", (unsigned long long)bookkeeper->filesize); printf("Max file size : %llu\n", (unsigned long long)bookkeeper->max_filesize); printf("Max life time : %llu\n", (unsigned long long)bookkeeper->max_lifetime); sem_unlock(bookkeeper_list_entry->sem_id); } // End of PrintBooks nfdump-1.6.23/bin/bookkeeper.h000066400000000000000000000063431404501030700161430ustar00rootroot00000000000000/* * Copyright (c) 2017, Peter Haag * Copyright (c) 2014, Peter Haag * Copyright (c) 2009, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _BOOKKEEPER_H #define _BOOKKEEPER_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include #include enum { BOOKKEEPER_OK = 0, ERR_FAILED, ERR_NOTEXISTS, ERR_PATHACCESS, ERR_EXISTS }; #define DETACH_ONLY 0 #define DESTROY_BOOKKEEPER 1 typedef struct bookkeeper_s { // collector infos pid_t nfcapd_pid; pid_t launcher_pid; // track info uint64_t sequence; // file infos time_t first; time_t last; uint64_t numfiles; uint64_t filesize; uint64_t max_filesize; uint64_t max_lifetime; } bookkeeper_t; // All bookkeepers are put into a linked list, to have all the shm_id,sem_id typedef struct bookkeeper_list_s { struct bookkeeper_list_s *next; bookkeeper_t *bookkeeper; // shared parameters int sem_id; int shm_id; } bookkeeper_list_t; /* function prototypes */ int InitBookkeeper(bookkeeper_t **bookkeeper, char *path, pid_t nfcapd_pid, pid_t launcher_pid); int AccessBookkeeper(bookkeeper_t **bookkeeper, char *path); void ReleaseBookkeeper(bookkeeper_t *bookkeeper, int destroy); void ClearBooks(bookkeeper_t *bookkeeper, bookkeeper_t *tmp_books); int LookBooks(bookkeeper_t *bookkeeper); int UnlookBooks(bookkeeper_t *bookkeeper); uint64_t BookSequence(bookkeeper_t *bookkeeper); void UpdateBooks(bookkeeper_t *bookkeeper, time_t when, uint64_t size); void UpdateBooksParam(bookkeeper_t *bookkeeper, time_t lifetime, uint64_t maxsize); void PrintBooks(bookkeeper_t *bookkeeper); #endif //_BOOKKEEPER_H nfdump-1.6.23/bin/collector.c000077500000000000000000000455401404501030700160030ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifdef HAVE_STDINT_H #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "bookkeeper.h" #include "collector.h" #include "nfx.h" #include "nffile_inline.c" /* local variables */ static uint32_t exporter_sysid = 0; static char *DynamicSourcesDir = NULL; /* local prototypes */ static uint32_t AssignExporterID(void); /* local functions */ static uint32_t AssignExporterID(void) { if ( exporter_sysid >= 0xFFFF ) { LogError("Too many exporters (id > 65535). Flow records collected but without reference to exporter"); return 0; } return ++exporter_sysid; } // End of AssignExporterID /* global functions */ int SetDynamicSourcesDir(FlowSource_t **FlowSource, char *dir) { if ( *FlowSource ) return 0; DynamicSourcesDir = dir; return 1; } // End of SetDynamicSourcesDir int AddFlowSource(FlowSource_t **FlowSource, char *ident) { FlowSource_t **source; struct stat fstat; char *p, *q, s[MAXPATHLEN]; int has_any_source = 0; int ok; if ( DynamicSourcesDir ) return 0; source = FlowSource; while ( *source ) { has_any_source |= (*source)->any_source; source = &((*source)->next); } if ( has_any_source ) { fprintf(stderr, "Ambiguous idents not allowed\n"); return 0; } *source = (FlowSource_t *)calloc(1, sizeof(FlowSource_t)); if ( !*source ) { LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return 0; } (*source)->next = NULL; (*source)->bookkeeper = NULL; (*source)->any_source = 0; (*source)->exporter_data = NULL; (*FlowSource)->exporter_count = 0; // separate IP address from ident if ( ( p = strchr(ident, ',')) == NULL ) { fprintf(stderr, "Syntax error for netflow source definition. Expect -n ident,IP,path\n"); return 0; } *p++ = '\0'; // separate path from IP if ( ( q = strchr(p, ',')) == NULL ) { fprintf(stderr, "Syntax error for netflow source definition. Expect -n ident,IP,path\n"); return 0; } *q++ = '\0'; if ( strchr(p, ':') != NULL ) { uint64_t _ip[2]; ok = inet_pton(PF_INET6, p, _ip); (*source)->sa_family = PF_INET6; (*source)->ip.V6[0] = ntohll(_ip[0]); (*source)->ip.V6[1] = ntohll(_ip[1]); } else { uint32_t _ip; ok = inet_pton(PF_INET, p, &_ip); (*source)->sa_family = PF_INET; (*source)->ip.V6[0] = 0; (*source)->ip.V6[1] = 0; (*source)->ip.V4 = ntohl(_ip); } switch (ok) { case 0: fprintf(stderr, "Unparsable IP address: %s\n", p); return 0; case 1: // success break; case -1: fprintf(stderr, "Error while parsing IP address: %s\n", strerror(errno)); return 0; break; } // fill in ident if ( strlen(ident) >= IDENTLEN ) { fprintf(stderr, "Source identifier too long: %s\n", ident); return 0; } if ( strchr(ident, ' ') ) { fprintf(stderr,"Illegal characters in ident %s\n", ident); exit(255); } strncpy((*source)->Ident, ident, IDENTLEN-1 ); (*source)->Ident[IDENTLEN-1] = '\0'; if ( strlen(q) >= MAXPATHLEN ) { fprintf(stderr,"Path too long: %s\n", q); exit(255); } char *path = realpath(q, NULL); if ( !path ) { fprintf(stderr, "realpath() error %s: %s\n", q, strerror(errno)); return 0; } // check for existing path if ( stat(path, &fstat) ) { fprintf(stderr, "stat() error %s: %s\n", path, strerror(errno)); return 0; } if ( !(fstat.st_mode & S_IFDIR) ) { fprintf(stderr, "Not a directory: %s\n", path); return 0; } // remember path (*source)->datadir = path; // cache current collector file if ( snprintf(s, MAXPATHLEN-1, "%s/%s.%lu", (*source)->datadir , NF_DUMPFILE, (unsigned long)getpid() ) >= (MAXPATHLEN-1)) { fprintf(stderr, "Path too long: %s\n", q); return 0; } (*source)->current = strdup(s); if ( !(*source)->current ) { fprintf(stderr, "strdup() error: %s\n", strerror(errno)); return 0; } return 1; } // End of AddFlowSource int AddFlowSourceFromFile(FlowSource_t **FlowSource, char *path) { struct stat fstat; char entry[MAXPATHLEN]; FILE *inputfile = NULL; int ret = 0; if ( strlen(path) >= MAXPATHLEN ) { fprintf(stderr, "Path too long: %s\n", path); return 1; } if ( stat(path, &fstat) ) { fprintf(stderr, "stat() error %s: %s\n", path, strerror(errno)); return 2; } if ( !(fstat.st_mode & S_IFREG) ) { fprintf(stderr, "Not a file: %s\n", path); return 3; } inputfile = fopen(path, "r"); if ( NULL == inputfile ) { fprintf(stderr, "Cannot open file %s: %s\n", path, strerror(errno)); return 4; } for( ; fgets(entry,sizeof(entry),inputfile) != NULL ; ) { int len = strlen(entry); if ( entry[len-2] == '\n' || entry[len-2] == '\r' ) entry[len-2] = 0; if ( entry[len-1] == '\n' || entry[len-1] == '\r' ) entry[len-1] = 0; if ( !AddFlowSource(FlowSource, entry) ) { fprintf(stderr, "Could not add flow source %s\n", entry); ret = 5; } } if ( fclose(inputfile) ) { fprintf(stderr, "Cannot close file %s: %s\n", path, strerror(errno)); return 6; } return ret; } // End of AddFlowSourceFromFile int AddDefaultFlowSource(FlowSource_t **FlowSource, char *ident, char *path) { struct stat fstat; char s[MAXPATHLEN]; if ( DynamicSourcesDir ) return 0; *FlowSource = (FlowSource_t *)calloc(1,sizeof(FlowSource_t)); if ( !*FlowSource ) { fprintf(stderr, "calloc() allocation error: %s\n", strerror(errno)); return 0; } (*FlowSource)->next = NULL; (*FlowSource)->bookkeeper = NULL; (*FlowSource)->any_source = 1; (*FlowSource)->exporter_data = NULL; (*FlowSource)->exporter_count = 0; // fill in ident if ( strlen(ident) >= IDENTLEN ) { fprintf(stderr, "Source identifier too long: %s\n", ident); return 0; } if ( strchr(ident, ' ') ) { fprintf(stderr,"Illegal characters in ident %s\n", ident); return 0; } strncpy((*FlowSource)->Ident, ident, IDENTLEN-1 ); (*FlowSource)->Ident[IDENTLEN-1] = '\0'; if ( strlen(path) >= MAXPATHLEN ) { fprintf(stderr,"Path too long: %s\n",path); return 0; } // check for existing path if ( stat(path, &fstat) ) { fprintf(stderr, "stat() error %s: %s\n", path, strerror(errno)); return 0; } if ( !(fstat.st_mode & S_IFDIR) ) { fprintf(stderr, "No such directory: %s\n", path); return 0; } // remember path (*FlowSource)->datadir = strdup(path); if ( !(*FlowSource)->datadir ) { fprintf(stderr, "strdup() error: %s\n", strerror(errno)); return 0; } // cache current collector file if ( snprintf(s, MAXPATHLEN-1, "%s/%s.%lu", (*FlowSource)->datadir , NF_DUMPFILE, (unsigned long)getpid() ) >= (MAXPATHLEN-1)) { fprintf(stderr, "Path too long: %s\n", path); return 0; } (*FlowSource)->current = strdup(s); if ( !(*FlowSource)->current ) { fprintf(stderr, "strdup() error: %s\n", strerror(errno)); return 0; } return 1; } // End of AddDefaultFlowSource FlowSource_t *AddDynamicSource(FlowSource_t **FlowSource, struct sockaddr_storage *ss) { FlowSource_t **source; void *ptr; char *s, ident[100], path[MAXPATHLEN]; int err; union { struct sockaddr_storage *ss; struct sockaddr *sa; struct sockaddr_in *sa_in; struct sockaddr_in6 *sa_in6; } u; u.ss = ss; if ( !DynamicSourcesDir ) return NULL; source = FlowSource; while ( *source ) { source = &((*source)->next); } *source = (FlowSource_t *)calloc(1, sizeof(FlowSource_t)); if ( !*source ) { LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return NULL; } (*source)->next = NULL; (*source)->bookkeeper = NULL; (*source)->any_source = 0; (*source)->exporter_data = NULL; (*FlowSource)->exporter_count = 0; switch (ss->ss_family) { case PF_INET: { #ifdef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN if (ss->ss_len != sizeof(struct sockaddr_in) ) { // malformed struct LogError("Malformed IPv4 socket struct in '%s', line '%d'", __FILE__, __LINE__ ); free(*source); *source = NULL; return NULL; } #endif (*source)->sa_family = PF_INET; (*source)->ip.V6[0] = 0; (*source)->ip.V6[1] = 0; (*source)->ip.V4 = ntohl(u.sa_in->sin_addr.s_addr); ptr = &u.sa_in->sin_addr; } break; case PF_INET6: { uint64_t *ip_ptr = (uint64_t *)u.sa_in6->sin6_addr.s6_addr; #ifdef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN if (ss->ss_len != sizeof(struct sockaddr_in6) ) { // malformed struct LogError("Malformed IPv6 socket struct in '%s', line '%d'", __FILE__, __LINE__ ); free(*source); *source = NULL; return NULL; } #endif // ptr = &((struct sockaddr_in6 *)sa)->sin6_addr; (*source)->sa_family = PF_INET6; (*source)->ip.V6[0] = ntohll(ip_ptr[0]); (*source)->ip.V6[1] = ntohll(ip_ptr[1]); ptr = &u.sa_in6->sin6_addr; } break; default: // keep compiler happy (*source)->ip.V6[0] = 0; (*source)->ip.V6[1] = 0; ptr = NULL; LogError("Unknown sa fanily: %d in '%s', line '%d'", ss->ss_family, __FILE__, __LINE__ ); free(*source); *source = NULL; return NULL; } if ( !ptr ) { free(*source); *source = NULL; return NULL; } inet_ntop(ss->ss_family, ptr, ident, sizeof(ident)); ident[99] = '\0'; dbg_printf("Dynamic Flow Source IP: %s\n", ident); s = ident; while ( *s != '\0' ) { if ( *s == '.' || *s == ':' ) *s = '-'; s++; } dbg_printf("Dynamic Flow Source ident: %s\n", ident); strncpy((*source)->Ident, ident, IDENTLEN-1 ); (*source)->Ident[IDENTLEN-1] = '\0'; snprintf(path, MAXPATHLEN-1, "%s/%s", DynamicSourcesDir, ident); path[MAXPATHLEN-1] = '\0'; err = mkdir(path, 0755); if ( err != 0 && errno != EEXIST ) { LogError("mkdir() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); free(*source); *source = NULL; return NULL; } (*source)->datadir = strdup(path); if ( snprintf(path, MAXPATHLEN-1, "%s/%s.%lu", (*source)->datadir, NF_DUMPFILE, (unsigned long)getpid() ) >= (MAXPATHLEN-1)) { fprintf(stderr, "Path too long: %s\n", path); free(*source); *source = NULL; return NULL; } (*source)->current = strdup(path); LogInfo("Dynamically add source ident: %s in directory: %s", ident, path); return *source; } // End of AddDynamicSource int InitExtensionMapList(FlowSource_t *fs) { fs->extension_map_list.maps = (extension_map_t **)calloc(BLOCK_SIZE, sizeof(extension_map_t *)); if ( !fs->extension_map_list.maps ) { LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return 0; } fs->extension_map_list.max_maps = BLOCK_SIZE; fs->extension_map_list.next_free = 0; fs->extension_map_list.num_maps = 0; return 1; } // End of InitExtensionMapList int ReInitExtensionMapList(FlowSource_t *fs) { dbg_printf("Flush all extension maps!\n"); free(fs->extension_map_list.maps); fs->extension_map_list.maps = NULL; return InitExtensionMapList(fs); } // End of ReInitExtensionMapList int RemoveExtensionMap(FlowSource_t *fs, extension_map_t *map) { int slot = map->map_id; dbg_printf("Remove extension map ID: %d\n", slot); if ( slot >= fs->extension_map_list.max_maps ) { // XXX hmm .. is simply return correct /// LogError("*** software error *** Try to remove extension map %d, while only %d slots are available\n", slot, fs->extension_map_list.max_maps); return 0; } fs->extension_map_list.maps[slot] = NULL; return 1; } // End of RemoveExtensionMap int AddExtensionMap(FlowSource_t *fs, extension_map_t *map) { int next_slot = fs->extension_map_list.next_free; dbg_printf("Add extension map\n"); // is it a new map, we have not yet in the list if ( map->map_id == INIT_ID ) { if ( next_slot >= fs->extension_map_list.max_maps ) { // extend map list dbg_printf("List full - extend extension list to %d slots\n", fs->extension_map_list.max_maps + BLOCK_SIZE); extension_map_t **p = realloc((void *)fs->extension_map_list.maps, (fs->extension_map_list.max_maps + BLOCK_SIZE ) * sizeof(extension_map_t *)); if ( !p ) { LogError("realloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return 0; } fs->extension_map_list.maps = p; fs->extension_map_list.max_maps += BLOCK_SIZE; } dbg_printf("Add map to slot %d\n", next_slot); fs->extension_map_list.maps[next_slot] = map; map->map_id = next_slot; fs->extension_map_list.num_maps++; if ( (next_slot + 1) == fs->extension_map_list.num_maps ) { // sequencially filled slots // next free is next slot fs->extension_map_list.next_free++; dbg_printf("Next slot is sequential: %d\n", fs->extension_map_list.next_free); } else { // fill gap in list - search for next free int i; dbg_printf("Search next slot ... \n"); for ( i = (next_slot + 1); i < fs->extension_map_list.max_maps; i++ ) { if ( fs->extension_map_list.maps[i] == NULL ) { // empty slot found dbg_printf("Empty slot found at %d\n", i); break; } } // assign next_free - if none found up to max, the list will get extended // in the next round dbg_printf("Set next free to %d\n", i); fs->extension_map_list.next_free = i; } } AppendToBuffer(fs->nffile, (void *)map, map->size); return 1; } // End of AddExtensionMap int FlushInfoExporter(FlowSource_t *fs, exporter_info_record_t *exporter) { exporter->sysid = AssignExporterID(); fs->exporter_count++; AppendToBuffer(fs->nffile, (void *)exporter, exporter->header.size); #ifdef DEVEL { #define IP_STRING_LEN 40 char ipstr[IP_STRING_LEN]; printf("Flush Exporter: "); if ( exporter->sa_family == AF_INET ) { uint32_t _ip = htonl(exporter->ip.V4); inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr)); printf("SysID: %u, IP: %16s, version: %u, ID: %2u\n", exporter->sysid, ipstr, exporter->version, exporter->id); } else if ( exporter->sa_family == AF_INET6 ) { uint64_t _ip[2]; _ip[0] = htonll(exporter->ip.V6[0]); _ip[1] = htonll(exporter->ip.V6[1]); inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr)); printf("SysID: %u, IP: %40s, version: %u, ID: %2u\n", exporter->sysid, ipstr, exporter->version, exporter->id); } else { strncpy(ipstr, "", IP_STRING_LEN); printf("**** Exporter IP version unknown ****\n"); } } #endif return 1; } // End of FlushInfoExporter int FlushInfoSampler(FlowSource_t *fs, sampler_info_record_t *sampler) { AppendToBuffer(fs->nffile, (void *)sampler, sampler->header.size); #ifdef DEVEL { printf("Flush Sampler: "); if ( sampler->id < 0 ) { printf("Exporter SysID: %u, Generic Sampler: mode: %u, interval: %u\n", sampler->exporter_sysid, sampler->mode, sampler->interval); } else { printf("Exporter SysID: %u, Sampler: id: %i, mode: %u, interval: %u\n", sampler->exporter_sysid, sampler->id, sampler->mode, sampler->interval); } } #endif return 1; } // End of FlushInfoSampler void FlushStdRecords(FlowSource_t *fs) { exporter_t *e = fs->exporter_data; int i; while ( e ) { sampler_t *sampler = e->sampler; AppendToBuffer(fs->nffile, (void *)&(e->info), e->info.header.size); while ( sampler ) { AppendToBuffer(fs->nffile, (void *)&(sampler->info), sampler->info.header.size); sampler = sampler->next; } e = e->next; } for ( i=0; iextension_map_list.next_free; i++ ) { extension_map_t *map = fs->extension_map_list.maps[i]; if ( map ) AppendToBuffer(fs->nffile, (void *)map, map->size); } } // End of FlushStdRecords void FlushExporterStats(FlowSource_t *fs) { exporter_t *e = fs->exporter_data; exporter_stats_record_t *exporter_stats; uint32_t i, size; // idle collector .. if ( !fs->exporter_count ) return; size = sizeof(exporter_stats_record_t) + (fs->exporter_count -1) * sizeof(struct exporter_stat_s); exporter_stats = ( exporter_stats_record_t *)malloc(size); if ( !exporter_stats ) { LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return; } exporter_stats->header.type = ExporterStatRecordType; exporter_stats->header.size = size; exporter_stats->stat_count = fs->exporter_count; #ifdef DEVEL printf("Flush Exporter Stats: %u exporters, size: %u\n", fs->exporter_count, size); #endif i = 0; while ( e ) { // prevent memory corruption - just in case .. should not happen anyway // continue loop for error reporting after while if ( i >= fs->exporter_count ) { i++; e = e->next; continue; } exporter_stats->stat[i].sysid = e->info.sysid; exporter_stats->stat[i].sequence_failure = e->sequence_failure; exporter_stats->stat[i].packets = e->packets; exporter_stats->stat[i].flows = e->flows; #ifdef DEVEL printf("Stat: SysID: %u, version: %u, ID: %2u, Packets: %llu, Flows: %llu, Sequence Failures: %u, Padding Errors: %u\n", e->info.sysid, e->info.version, e->info.id, e->packets, e->flows, e->sequence_failure, e->padding_errors); #endif // reset counters e->sequence_failure = 0; e->padding_errors = 0; e->packets = 0; e->flows = 0; i++; e = e->next; } AppendToBuffer(fs->nffile, (void *)exporter_stats, size); free(exporter_stats); if ( i != fs->exporter_count ) { LogError("ERROR: exporter stats: Expected %u records, but found %u in %s line %d: %s\n", fs->exporter_count, i, __FILE__, __LINE__, strerror(errno) ); } } // End of FlushExporterStats nfdump-1.6.23/bin/collector.h000077500000000000000000000103531404501030700160020ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _COLLECTOR_H #define _COLLECTOR_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include #include #include "exporter.h" #include "bookkeeper.h" #include "nffile.h" #define FNAME_SIZE 256 /* common minimum netflow header for all versions */ typedef struct common_flow_header { uint16_t version; uint16_t count; } common_flow_header_t; typedef struct FlowSource_s { // link struct FlowSource_s *next; // exporter identifiers char Ident[IDENTLEN]; ip_addr_t ip; uint32_t sa_family; in_port_t port; int any_source; bookkeeper_t *bookkeeper; // all about data storage char *datadir; // where to store data for this source char *current; // current file name - typically nfcad.current.pid nffile_t *nffile; // the writing file handle // statistical data per source uint32_t bad_packets; uint64_t first_seen; // in msec uint64_t last_seen; // in msec // Any exporter specific data exporter_t *exporter_data; uint32_t exporter_count; struct timeval received; // extension map list struct { #define BLOCK_SIZE 16 int next_free; int max_maps; int num_maps; extension_map_t **maps; } extension_map_list; } FlowSource_t; /* input buffer size, to read data from the network */ #define NETWORK_INPUT_BUFF_SIZE 65535 // Maximum UDP message size // prototypes int AddFlowSource(FlowSource_t **FlowSource, char *ident); int AddFlowSourceFromFile(FlowSource_t **FlowSource, char *path); int AddDefaultFlowSource(FlowSource_t **FlowSource, char *ident, char *path); int SetDynamicSourcesDir(FlowSource_t **FlowSource, char *dir); FlowSource_t *AddDynamicSource(FlowSource_t **FlowSource, struct sockaddr_storage *ss); int InitExtensionMapList(FlowSource_t *fs); int ReInitExtensionMapList(FlowSource_t *fs); int RemoveExtensionMap(FlowSource_t *fs, extension_map_t *map); int AddExtensionMap(FlowSource_t *fs, extension_map_t *map); void FlushStdRecords(FlowSource_t *fs); void FlushExporterStats(FlowSource_t *fs); int FlushInfoExporter(FlowSource_t *fs, exporter_info_record_t *exporter); int FlushInfoSampler(FlowSource_t *fs, sampler_info_record_t *sampler); /* Default time window in seconds to rotate files */ #define TIME_WINDOW 300 /* overdue time: * if nfcapd does not get any data, wake up the receive system call * at least after OVERDUE_TIME seconds after the time window */ #define OVERDUE_TIME 10 // time nfcapd will wait for launcher to terminate #define LAUNCHER_TIMEOUT 60 #define SYSLOG_FACILITY "daemon" #endif //_COLLECTOR_H nfdump-1.6.23/bin/collector_inline.c000077500000000000000000000075361404501030700173440ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ static inline FlowSource_t *GetFlowSource(struct sockaddr_storage *ss) { FlowSource_t *fs; void *ptr; ip_addr_t ip; in_port_t port; char as[100]; union { struct sockaddr_storage *ss; struct sockaddr *sa; struct sockaddr_in *sa_in; struct sockaddr_in6 *sa_in6; } u; u.ss = ss; switch (ss->ss_family) { case PF_INET: { #ifdef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN if (ss->ss_len != sizeof(struct sockaddr_in) ) { // malformed struct LogError("Malformed IPv4 socket struct in '%s', line '%d'", __FILE__, __LINE__ ); return NULL; } #endif ip.V6[0] = 0; ip.V6[1] = 0; ip.V4 = ntohl(u.sa_in->sin_addr.s_addr); port = u.sa_in->sin_port; ptr = &u.sa_in->sin_addr; } break; case PF_INET6: { uint64_t *ip_ptr = (uint64_t *)u.sa_in6->sin6_addr.s6_addr; #ifdef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN if (ss->ss_len != sizeof(struct sockaddr_in6) ) { // malformed struct LogError("Malformed IPv6 socket struct in '%s', line '%d'", __FILE__, __LINE__ ); return NULL; } #endif // ptr = &((struct sockaddr_in6 *)sa)->sin6_addr; ip.V6[0] = ntohll(ip_ptr[0]); ip.V6[1] = ntohll(ip_ptr[1]); port = u.sa_in6->sin6_port; ptr = &u.sa_in6->sin6_addr; } break; default: // keep compiler happy ip.V6[0] = 0; ip.V6[1] = 0; port = 0; ptr = NULL; LogError("Unknown sa fanily: %d in '%s', line '%d'", ss->ss_family, __FILE__, __LINE__ ); return NULL; } #ifdef DEVEL inet_ntop(ss->ss_family, ptr, as, sizeof(as)); as[99] = '\0'; printf("Flow Source IP: %s\n", as); #endif fs = FlowSource; while ( fs ) { if ( ip.V6[0] == fs->ip.V6[0] && ip.V6[1] == fs->ip.V6[1] ) { fs->port = port; return fs; } // if we match any source, store the current IP address - works as faster cache next time // and identifies the current source by IP if ( fs->any_source ) { fs->ip = ip; fs->port = port; fs->sa_family = ss->ss_family; return fs; } fs = fs->next; } if ( ptr ) { inet_ntop (ss->ss_family, ptr, as, 100); } else strncpy(as, "", 99); as[99] = '\0'; LogError("Unknown flow source: '%s'" , as); return NULL; } // End of GetFlowSource nfdump-1.6.23/bin/content_dns.c000066400000000000000000000174071404501030700163310ustar00rootroot00000000000000/* * Copyright (c) 2013-2020, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #ifdef HAVE_FEATURES_H #include #endif #include #include #include #include #include #include #include #include #ifdef HAVE_ARPA_NAMESER_H #include #endif #ifdef HAVE_ARPA_NAMESER_COMPAT_H #include #endif #include #include #ifdef HAVE_STDINT_H #include #endif #include #include #include #include #include #include #include #include #include #include #include #include "util.h" #include "nfdump.h" #include "nffile.h" #include "bookkeeper.h" #include "collector.h" #include "flowtree.h" #include "content_dns.h" #include "inline.c" /* * Structure for query header. The order of the fields is machine- and * compiler-dependent, depending on the byte/bit order and the layout * of bit fields. We use bit fields only in int variables, as this * is all ANSI requires. This requires a somewhat confusing rearrangement. */ typedef struct dns_header_s { unsigned id :16; /* query identification number */ #ifdef WORDS_BIGENDIAN /* fields in third byte */ unsigned qr: 1; /* response flag */ unsigned opcode: 4; /* purpose of message */ unsigned aa: 1; /* authoritive answer */ unsigned tc: 1; /* truncated message */ unsigned rd: 1; /* recursion desired */ /* fields in fourth byte */ unsigned ra: 1; /* recursion available */ unsigned unused :3; /* unused bits (MBZ as of 4.9.3a3) */ unsigned rcode :4; /* response code */ #else /* fields in third byte */ unsigned rd :1; /* recursion desired */ unsigned tc :1; /* truncated message */ unsigned aa :1; /* authoritive answer */ unsigned opcode :4; /* purpose of message */ unsigned qr :1; /* response flag */ /* fields in fourth byte */ unsigned rcode :4; /* response code */ unsigned unused :3; /* unused bits (MBZ as of 4.9.3a3) */ unsigned ra :1; /* recursion available */ #endif /* remaining bytes */ unsigned qdcount :16; /* number of question entries */ unsigned ancount :16; /* number of answer entries */ unsigned nscount :16; /* number of authority entries */ unsigned arcount :16; /* number of resource entries */ } dns_header_t; #define DNS_QUERY_TYPE_A (1) #define DNS_QUERY_TYPE_AAAA (2) #define DNS_QUERY_TYPE_SRV (3) typedef struct dns_host_st { struct dns_host_st *next; unsigned int type; unsigned int class; unsigned int ttl; void *rr; } dns_host_t; typedef struct dns_srv_st { unsigned int priority; unsigned int weight; unsigned int port; unsigned int rweight; char name[256]; } dns_srv_t; static void *_a_rr(void **p) { struct in_addr in; in.s_addr = ntohl(Get_val32(*p)); *p += 4; return strdup(inet_ntoa(in)); } static void *_aaaa_rr(void **p) { char addr[INET6_ADDRSTRLEN]; uint64_t sa6[2]; memcpy((void *)sa6, *p, 16); inet_ntop(AF_INET6, (struct sockaddr_storage *)&sa6, addr, sizeof(addr)); return strdup(addr); } void content_decode_dns(struct FlowNode *node, uint8_t *payload, uint32_t payload_size) { dns_header_t *dns_header = (dns_header_t *)payload; uint32_t qdcount, ancount; void *p, *eod; #define DN_LENGTH 256 char dn[DN_LENGTH]; int i; if ( payload_size < sizeof(dns_header_t) ) { LogError("Unable to decode short DNS packet"); return; } // no of query packets qdcount = ntohs(dns_header->qdcount); dbg_printf("DNS Queries: %u\n", qdcount); // no of answer packets ancount = ntohs(dns_header->ancount); dbg_printf("DNS Answers: %u\n", ancount); // end of dns packet eod = (void *)(payload + payload_size); // reord pointer p = (void *)(payload + sizeof(dns_header_t)); for (i=0; i\n"); p = (void *)(p + len); continue; } } } // End of content_decode_dns nfdump-1.6.23/bin/content_dns.h000066400000000000000000000034101404501030700163230ustar00rootroot00000000000000/* * Copyright (c) 2014, Peter Haag * Copyright (c) 2013, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * * $Author$ * * $Id$ * * $LastChangedRevision$ * */ void content_decode_dns(struct FlowNode *node, uint8_t *payload, uint32_t payload_size); nfdump-1.6.23/bin/expire.c000066400000000000000000000546251404501030700153120ustar00rootroot00000000000000/* * Copyright (c) 2009-2019, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_FTS_H # include #else # include "fts_compat.h" #define fts_children fts_children_compat #define fts_close fts_close_compat #define fts_open fts_open_compat #define fts_read fts_read_compat #define fts_set fts_set_compat #endif #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "bookkeeper.h" #include "nfstatfile.h" #include "expire.h" static uint32_t timeout = 0; static void PrepareDirLists(channel_t *channel); static int compare(const FTSENT **f1, const FTSENT **f2); static void IntHandler(int signal) { switch (signal) { case SIGALRM: timeout = 1; break; case SIGHUP: case SIGINT: case SIGTERM: timeout = 1; break; break; case SIGCHLD: default: // ignore everything we don't know break; } } /* End of IntHandler */ static void SetupSignalHandler(void) { struct sigaction act; memset((void *)&act,0,sizeof(struct sigaction)); act.sa_handler = IntHandler; sigemptyset(&act.sa_mask); act.sa_flags = 0; sigaction(SIGTERM, &act, NULL); sigaction(SIGINT, &act, NULL); sigaction(SIGHUP, &act, NULL); sigaction(SIGALRM, &act, NULL); sigaction(SIGCHLD, &act, NULL); } // End of SetupSignalHandler uint64_t ParseSizeDef(char *s, uint64_t *value) { char *p; uint64_t fac; int dot; dot = 0; *value = 0; p = s; while ( *p ) { if ( *p < '0' || *p > '9' ) { if ( *p == '.' ) { if ( dot ) { break; } else { dot = 1; } } else break; } p++; } if ( p == s ) { LogError("Missing number in '%s'", s); return 0; } fac = 0; switch (*p) { case '\0': case 'b': case 'B': fac = 1; break; case 'k': case 'K': fac = 1024; break; case 'm': case 'M': fac = 1024 * 1024; break; case 'g': case 'G': fac = 1024 * 1024 * 1024; break; case 't': case 'T': fac = 1024LL * 1024LL * 1024LL * 1024LL; break; default: LogError("Garbage character(s) '%s' in '%s'", p, s); return 0; } if ( *p ) { char *r = p++; // skip optional 'B' for Bytes in KB etc. if ( fac != 1 && (*p == 'B' || *p == 'b') ) p++; if ( *p ) { // extra garbage after factor LogError("Garbage character(s) '%s''in '%s'", p, s); return 0; } *r = '\0'; } // *value = strtoll(s, NULL, 10) * fac; *value = (uint64_t)(atof(s) * (double)fac); return 1; } // End of ParseSizeDef /* * Accepted time scales * * w week * d day * H hour * M minute */ uint64_t ParseTimeDef(char *s, uint64_t *value) { char *p; uint64_t weeks, days, hours, minutes; *value = 0; weeks=days=hours=minutes=0; p = s; while ( *p ) { char *q = p; while ( *p ) { if ( *p < '0' || *p > '9' ) { break; } p++; } if ( p == q ) { LogError("Missing number before '%s'", q); return 0; } switch (*p) { case 'w': *p++ = '\0'; if ( weeks ) { LogError("Ambiguous weeks %sw", q); return 0; } weeks = strtoll(q, NULL, 10); break; case 'd': *p++ = '\0'; if ( days ) { LogError("Ambiguous days %sD", q); return 0; } days = strtoll(q, NULL, 10); break; case '\0': // without any units, assume hours case 'H': if ( *p ) *p++ = '\0'; if ( hours ) { LogError("Ambiguous hours %sH", q); return 0; } hours = strtoll(q, NULL, 10); break; case 'M': *p++ = '\0'; if ( minutes ) { LogError("Ambiguous minutes %sM", q); return 0; } minutes = strtoll(q, NULL, 10); break; default: LogError("Unknown time unit '%s'", q); return 0; } } *value = minutes*60 + hours*3600 + days*24*3600 + weeks*7*24*3600; return 1; } // End of ParseTimeDef static int compare(const FTSENT **f1, const FTSENT **f2) { return strcmp( (*f1)->fts_name, (*f2)->fts_name); } // End of compare void RescanDir(char *dir, dirstat_t *dirstat) { FTS *fts; FTSENT *ftsent; char *const path[] = { dir, NULL }; char first_timestring[16], last_timestring[16]; dirstat->filesize = dirstat->numfiles = 0; dirstat->first = 0; dirstat->last = 0; strncpy(first_timestring, "99999999999999", 15); strncpy(last_timestring, "00000000000000", 15); fts = fts_open(path, FTS_LOGICAL, compare); if ( !fts ) { LogError( "fts_open() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return; } while ( (ftsent = fts_read(fts)) != NULL) { if ( ftsent->fts_info == FTS_F && ((ftsent->fts_namelen == 19) || (ftsent->fts_namelen == 21)) ) { // nfcapd.200604301200 strlen = 19 // nfcapd.20190430120010 strlen = 21 if ( strncmp(ftsent->fts_name, "nfcapd.", 7) == 0 ) { char *s, *p = &(ftsent->fts_name[7]); // make sure, we have only digits s = p; while ( *s ) { if ( *s < '0' || *s > '9' ) break; s++; } // otherwise skip if ( *s ) continue; if ( strcmp(p, first_timestring) < 0 ) { first_timestring[0] = '\0'; strncat(first_timestring, p, 15); } if ( strcmp(p, last_timestring) > 0 ) { last_timestring[0] = '\0'; strncat(last_timestring, p, 15); } dirstat->filesize += 512 * ftsent->fts_statp->st_blocks; dirstat->numfiles++; } } else { switch (ftsent->fts_info) { case FTS_D: // skip all '.' entries as well as hidden directories if ( ftsent->fts_level > 0 && ftsent->fts_name[0] == '.' ) fts_set(fts, ftsent, FTS_SKIP); // any valid dirctory need to start with a digit ( %Y -> year ) if ( ftsent->fts_level > 0 && !isdigit(ftsent->fts_name[0]) ) fts_set(fts, ftsent, FTS_SKIP); break; case FTS_DP: break; } } } fts_close(fts); // no files means do rebuild next time, otherwise the stat record may not be accurate if ( dirstat->numfiles == 0 ) { dirstat->first = dirstat->last = time(NULL); dirstat->status = FORCE_REBUILD; } else { dirstat->first = ISO2UNIX(first_timestring); dirstat->last = ISO2UNIX(last_timestring); dirstat->status = STATFILE_OK; } dbg_printf("Rescan dir - first: %s, last: %s\n", first_timestring, last_timestring); } // End of RescanDir void ExpireDir(char *dir, dirstat_t *dirstat, uint64_t maxsize, uint64_t maxlife, uint32_t runtime ) { FTS *fts; FTSENT *ftsent; uint64_t sizelimit, num_expired; int done, size_done, lifetime_done, dir_files; char *const path[] = { dir, NULL }; char *expire_timelimit = NULL; time_t now = time(NULL); dir_files = 0; if ( dirstat->low_water == 0 ) dirstat->low_water = 95; if ( runtime ) { SetupSignalHandler(); alarm(runtime); } if ( maxlife ) { // build an appropriate string for comparing time_t t_expire = now - maxlife; time_t t_watermark = now - (time_t)((maxlife * dirstat->low_water)/100); dbg_printf("Expire files before %s", ctime(&t_expire)); expire_timelimit = strdup(UNIX2ISO(t_watermark)); dbg_printf("down to %s", ctime(&t_watermark)); dbg_printf("Diff: %li\n", t_watermark - t_expire ); if ( dirstat->last < t_expire && (isatty(STDIN_FILENO) ) ) { // this means all files will get expired - are you sure ? char *s, s1[32], s2[32]; time_t t; struct tm *when; t = t_expire; when = localtime(&t); strftime(s1, 31, "%Y-%m-%d %H:%M:%S", when); s1[31] = '\0'; t = dirstat->last; when = localtime(&t); strftime(s2, 31, "%Y-%m-%d %H:%M:%S", when); s2[31] = '\0'; printf("Your max lifetime of %s will expire all file before %s\n", ScaleTime(maxlife), s1); printf("Your latest files are dated %s. This means all files will be deleted!\n", s2); printf("Are you sure? yes/[no] "); s = fgets(s1, 31, stdin); s1[31] = '\0'; if ( s && strncasecmp(s1, "yes\n", 31) == 0 ) { printf("Ok - you've beeen warned!\n"); } else { printf("Expire canceled!\n"); return; } } } done = 0; size_done = maxsize == 0 || dirstat->filesize < maxsize; lifetime_done = maxlife == 0 || ( now - dirstat->first ) < maxlife; sizelimit = (dirstat->low_water * maxsize)/100; num_expired = 0; fts = fts_open(path, FTS_LOGICAL, compare); while ( !done && ((ftsent = fts_read(fts)) != NULL) ) { if ( ftsent->fts_info == FTS_F ) { dir_files++; // count files in directories if ( (ftsent->fts_namelen == 19 || ftsent->fts_namelen == 21) && strncmp(ftsent->fts_name, "nfcapd.", 7) == 0 ) { // nfcapd.200604301200 strlen = 19 // nfcapd.20190430120010 strlen = 21 char *s, *p = &(ftsent->fts_name[7]); // process only nfcapd. files // make sure it's really an nfcapd. file and we have // only digits in the rest of the file name s = p; while ( *s ) { if ( *s < '0' || *s > '9' ) break; s++; } // otherwise skip if ( *s ) continue; // expire size-wise if needed if ( !size_done ) { if ( dirstat->filesize > sizelimit ) { if ( unlink(ftsent->fts_path) == 0 ) { dirstat->filesize -= 512 * ftsent->fts_statp->st_blocks; num_expired++; dir_files--; } else { LogError( "unlink() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); } continue; // next file if file was unlinked } else { dirstat->first = ISO2UNIX(p); // time of first file not expired size_done = 1; } } // expire time-wise if needed // this part of the code is executed only when size-wise is fullfilled if ( !lifetime_done ) { if ( expire_timelimit && strcmp(p, expire_timelimit) < 0 ) { if ( unlink(ftsent->fts_path) == 0 ) { dirstat->filesize -= 512 * ftsent->fts_statp->st_blocks; num_expired++; dir_files--; } else { LogError( "unlink() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); } lifetime_done = 0; } else { dirstat->first = ISO2UNIX(p); // time of first file not expired lifetime_done = 1; } } done = (size_done && lifetime_done) || timeout; } } else { switch (ftsent->fts_info) { case FTS_D: // set pre-order flag dir_files = 0; // skip all '.' entries as well as hidden directories if ( ftsent->fts_level > 0 && ftsent->fts_name[0] == '.' ) fts_set(fts, ftsent, FTS_SKIP); // any valid directory needs to start with a digit ( %Y -> year ) if ( ftsent->fts_level > 0 && !isdigit(ftsent->fts_name[0]) ) fts_set(fts, ftsent, FTS_SKIP); break; case FTS_DP: // do not delete base data directory ( level == 0 ) if ( dir_files == 0 && ftsent->fts_level > 0 ) { // directory is empty and can be deleted dbg_printf("Will remove directory %s\n", ftsent->fts_path); if ( rmdir(ftsent->fts_path) != 0 ) { LogError( "rmdir() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); } } break; } } } fts_close(fts); if ( !done ) { // all files expired and limits not reached // this may be possible, when files get time-wise expired and // the time limit is shorter than the latest file dirstat->first = dirstat->last; } if ( runtime ) alarm(0); if ( timeout ) { LogError( "Maximum execution time reached! Interrupt expire.\n"); } if ( num_expired > dirstat->numfiles ) { LogError( "Error updating stat record: Number of files inconsistent!\n"); LogError( "Will automatically rebuild this directory next time\n"); dirstat->numfiles = 0; dirstat->status = FORCE_REBUILD; } else { dirstat->numfiles -= num_expired; } if ( dirstat->numfiles == 0 ) { dirstat->first = dirstat->last = time(NULL); dirstat->status = FORCE_REBUILD; } free(expire_timelimit); } // End of ExpireDir static void PrepareDirLists(channel_t *channel) { channel_t *current_channel = channel; while ( current_channel ) { char *const path[] = { current_channel->datadir, NULL }; current_channel->fts = fts_open(path, FTS_LOGICAL|FTS_NOCHDIR, compare); if ( !current_channel->fts ) { LogError( "fts_open() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); continue; } // get first entry current_channel->ftsent = fts_read(current_channel->fts); if ( current_channel->ftsent ) // use fts_number as the number of files already seen in this directory. current_channel->ftsent->fts_number = 0; while ( current_channel->ftsent ) { if ( current_channel->ftsent->fts_info == FTS_ERR || current_channel->ftsent->fts_info == FTS_NS) { LogError( "fts_read() %s error in %s line %d: %s\n", current_channel->ftsent->fts_path, __FILE__, __LINE__, strerror(current_channel->ftsent->fts_errno) ); continue; } if ( current_channel->ftsent->fts_info != FTS_F ) { current_channel->ftsent = fts_read(current_channel->fts); continue; } // it's now FTS_F current_channel->ftsent->fts_number++; // if ftsent points to first valid file, break if ( (current_channel->ftsent->fts_namelen == 19 || current_channel->ftsent->fts_namelen == 21) && strncmp(current_channel->ftsent->fts_name, "nfcapd.", 7) == 0 ) break; // otherwise loop current_channel->ftsent = fts_read(current_channel->fts); } current_channel = current_channel->next; } } // End of PrepareDirLists void ExpireProfile(channel_t *channel, dirstat_t *current_stat, uint64_t maxsize, uint64_t maxlife, uint32_t runtime ) { int size_done, lifetime_done, done; char *expire_timelimit = ""; time_t now = time(NULL); uint64_t sizelimit, num_expired; if ( !channel ) return; done = 0; SetupSignalHandler(); if ( maxlife ) { // time_t t_expire = now - maxlife; // build an appropriate string for comparing time_t t_watermark = now - (time_t)((maxlife * current_stat->low_water)/100); expire_timelimit = strdup(UNIX2ISO(t_watermark)); dbg_printf("down to %s", ctime(&t_watermark)); } size_done = maxsize == 0 || current_stat->filesize < maxsize; sizelimit = (current_stat->low_water * maxsize)/100; lifetime_done = maxlife == 0 || ( now - current_stat->first ) < maxlife; num_expired = 0; PrepareDirLists(channel); if ( runtime ) alarm(runtime); while ( !done ) { char *p; int file_removed; // search for the channel with oldest file. If all channel have same age, // get the last in the list channel_t *expire_channel = channel; channel_t *compare_channel = expire_channel->next; while ( compare_channel ) { if ( expire_channel->ftsent == NULL ) { expire_channel = compare_channel; } if ( compare_channel->ftsent == NULL ) { compare_channel = compare_channel->next; continue; } // at this point expire_channel and current_channel fts entries are valid if ( strcmp(expire_channel->ftsent->fts_name, compare_channel->ftsent->fts_name) >= 0 ) { expire_channel = compare_channel; } compare_channel = compare_channel->next; } if ( !expire_channel->ftsent ) { // no more entries in any channel - we are done done = 1; continue; } // flag is file got removed file_removed = 0; // expire_channel now points to the channel with oldest file // do expire p = &(expire_channel->ftsent->fts_name[7]); dbg_printf("File: %s\n", expire_channel->ftsent->fts_path); if ( !size_done ) { // expire size-wise if needed dbg_printf(" Size expire %llu %llu\n", current_stat->filesize, sizelimit); if ( current_stat->filesize > sizelimit ) { // need to delete this file if ( unlink(expire_channel->ftsent->fts_path) == 0 ) { // Update profile stat current_stat->filesize -= 512 * expire_channel->ftsent->fts_statp->st_blocks; current_stat->numfiles--; // Update channel stat expire_channel->dirstat->filesize -= 512 * expire_channel->ftsent->fts_statp->st_blocks; expire_channel->dirstat->numfiles--; // decrement number of files seen in this directory expire_channel->ftsent->fts_number--; num_expired++; } else { LogError( "unlink() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); } file_removed = 1; } else { // we are done size-wise // time of first file not expired = start time of channel/profile expire_channel->dirstat->first = current_stat->first = ISO2UNIX(p); size_done = 1; } } else if ( !lifetime_done ) { dbg_printf(" Time expire \n"); // expire time-wise if needed // this part of the code is executed only when size-wise is already fullfilled if ( strcmp(p, expire_timelimit) < 0 ) { // need to delete this file if ( unlink(expire_channel->ftsent->fts_path) == 0 ) { // Update profile stat current_stat->filesize -= 512 * expire_channel->ftsent->fts_statp->st_blocks; current_stat->numfiles--; // Update channel stat expire_channel->dirstat->filesize -= 512 * expire_channel->ftsent->fts_statp->st_blocks; expire_channel->dirstat->numfiles--; // decrement number of files seen in this directory expire_channel->ftsent->fts_number--; num_expired++; } else { LogError( "unlink() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); } file_removed = 1; } else { // we are done time-wise // time of first file not expired = start time of channel/profile expire_channel->dirstat->first = current_stat->first = ISO2UNIX(p); lifetime_done = 1; } } else // all done done = 1; if ( timeout ) done = 1; // advance fts entry in expire channel to next file, if file was removed if ( file_removed ) { expire_channel->ftsent = fts_read(expire_channel->fts); while ( expire_channel->ftsent ) { if ( expire_channel->ftsent->fts_info == FTS_F ) { // entry is a file expire_channel->ftsent->fts_number++; if ( (expire_channel->ftsent->fts_namelen == 19 || expire_channel->ftsent->fts_namelen == 21) && strncmp(expire_channel->ftsent->fts_name, "nfcapd.", 7) == 0 ) { // if ftsent points to next valid file char *p = &(expire_channel->ftsent->fts_name[7]); // next file is first (oldest) for channel and for profile - update first mark expire_channel->dirstat->first = current_stat->first = ISO2UNIX(p); break; } } else { switch (expire_channel->ftsent->fts_info) { case FTS_D: // entry is a directory // set number of files seen in this directory = 0 expire_channel->ftsent->fts_number = 0; // skip all '.' entries as well as hidden directories if ( expire_channel->ftsent->fts_level > 0 && expire_channel->ftsent->fts_name[0] == '.' ) fts_set(expire_channel->fts, expire_channel->ftsent, FTS_SKIP); // any valid directory needs to start with a digit ( %Y -> year ) if ( expire_channel->ftsent->fts_level > 0 && !isdigit(expire_channel->ftsent->fts_name[0]) ) fts_set(expire_channel->fts, expire_channel->ftsent, FTS_SKIP); break; case FTS_DP: // do not delete base data directory ( level == 0 ) if ( expire_channel->ftsent->fts_number == 0 && expire_channel->ftsent->fts_level > 0 ) { // directory is empty and can be deleted dbg_printf("Will remove directory %s\n", expire_channel->ftsent->fts_path); if ( rmdir(expire_channel->ftsent->fts_path) != 0 ) { LogError( "rmdir() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); } } break; } } // otherwise loop expire_channel->ftsent = fts_read(expire_channel->fts); } // end advance fts entry file_removed = 0; } if ( expire_channel->ftsent == NULL ) { // this channel has no more files now expire_channel->dirstat->first = expire_channel->dirstat->last; if ( expire_channel->dirstat->numfiles ) { // if channel is empty, no files must be reported, but rebuild is done anyway LogError( "Inconsitency detected in channel %s. Will rebuild automatically.\n", expire_channel->datadir); LogError( "No more files found, but %llu expected.\n", expire_channel->dirstat->numfiles); } expire_channel->dirstat->numfiles = 0; expire_channel->dirstat->status = FORCE_REBUILD; } } // while ( !done ) if ( runtime ) alarm(0); if ( timeout ) { LogError( "Maximum execution time reached! Interrupt expire.\n"); } } // End of ExpireProfile void UpdateBookStat(dirstat_t *dirstat, bookkeeper_t *books) { if ( books->numfiles ) { /* prevent some faults and dublicates: * book records can never be timewise smaller than directory records => fishy! * in case book records == directory records, the user stopped and restarted nfcapd * this is not necessarily wrong, but results in overwriting an existing file * which results in wrong stats => rescan needed */ if ( books->last <= dirstat->last || books->first <= dirstat->first) { dirstat->status = FORCE_REBUILD; return; } dirstat->last = books->last; dirstat->numfiles += books->numfiles; dirstat->filesize += books->filesize; } } // End of UpdateBookStat nfdump-1.6.23/bin/expire.h000066400000000000000000000052621404501030700153100ustar00rootroot00000000000000/* * Copyright (c) 2017, Peter Haag * Copyright (c) 2014, Peter Haag * Copyright (c) 2009, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _EXPIRE_H #define _EXPIRE_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #ifdef HAVE_FTS_H # include #else # include "fts_compat.h" #endif #include "nfstatfile.h" #include "bookkeeper.h" typedef struct channel_s { struct channel_s *next; char *datadir; dirstat_t *dirstat; bookkeeper_t *books; int books_stat; int do_rescan; int status; FTS *fts; FTSENT *ftsent; } channel_t; enum { OK = 0, NOFILES }; uint64_t ParseSizeDef(char *s, uint64_t *value); uint64_t ParseTimeDef(char *s, uint64_t *value); void RescanDir(char *dir, dirstat_t *dirstat); void ExpireDir(char *dir, dirstat_t *dirstat, uint64_t maxsize, uint64_t maxlife, uint32_t runtime ); void ExpireProfile(channel_t *channel, dirstat_t *current_stat, uint64_t maxsize, uint64_t maxlife, uint32_t runtime ); void UpdateBookStat(dirstat_t *dirstat, bookkeeper_t *books); #endif //_EXPIRE_H nfdump-1.6.23/bin/exporter.c000077500000000000000000000334461404501030700156670ustar00rootroot00000000000000/* * Copyright (c) 2012-2020, Peter Haag * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifdef HAVE_CONFIG_H #include "config.h" #endif #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "nfnet.h" #include "bookkeeper.h" #include "collector.h" #include "exporter.h" #include "netflow_v1.h" #include "netflow_v5_v7.h" #include "netflow_v9.h" #include "ipfix.h" /* global */ exporter_t **exporter_list; /* local variables */ #define MAX_EXPORTERS 65536 static exporter_t *exporter_root; #include "nffile_inline.c" /* local prototypes */ /* functions */ int InitExporterList(void) { exporter_list = calloc(MAX_EXPORTERS, sizeof(exporter_t *)); if ( !exporter_list ) { LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return 0; } exporter_root = NULL; return 1; } // End of InitExporterList int AddExporterInfo(exporter_info_record_t *exporter_record) { uint32_t id; int i; char *p1, *p2; if ( exporter_record->header.size != sizeof(exporter_info_record_t) ) { LogError("Corrupt exporter record in %s line %d\n", __FILE__, __LINE__); return 0; } // sanity check id = exporter_record->sysid; if ( id >= MAX_EXPORTERS ) { LogError("Corrupt exporter record in %s line %d\n", __FILE__, __LINE__); return 0; } if ( exporter_list[id] != NULL ) { // slot already taken - check if exporters are identical exporter_record->sysid = exporter_list[id]->info.sysid; if ( memcmp((void *)exporter_record, (void *)&(exporter_list[id]->info), sizeof(exporter_info_record_t)) == 0 ) { dbg_printf("Found identical exporter record at SysID: %i, Slot: %u\n", exporter_record->sysid, id); // we are done return 2; } else { // exporters not identical - move current slot int i; // search first emty slot at the top of the list for ( i = id+1; i < MAX_EXPORTERS && exporter_list[i] != NULL; i++ ) {;} if ( i >= MAX_EXPORTERS ) { // all slots taken LogError("Too many exporters (>256)\n"); return 0; } dbg_printf("Move existing exporter from slot %u, to %i\n", id, i); // else - move slot exporter_list[i] = exporter_list[id]; exporter_list[id] = NULL; exporter_record->sysid = i; } } // slot[id] is now available exporter_list[id] = (exporter_t *)calloc(1, sizeof(exporter_t)); if ( !exporter_list[id] ) { LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return 0; } // SPARC gcc fails here, if we use directly a pointer to the struct. // SPARC barfs and core dumps otherwise // memcpy((void *)&(exporter_list[id]->info), (void *)exporter_record, sizeof(exporter_info_record_t)); p1 = (char *)&(exporter_list[id]->info); p2 = (char *)exporter_record; for ( i=0; isysid); #ifdef DEVEL { #define IP_STRING_LEN 40 char ipstr[IP_STRING_LEN]; if ( exporter_record->sa_family == AF_INET ) { uint32_t _ip = htonl(exporter_record->ip.V4); inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr)); printf("SysID: %u, IP: %16s, version: %u, ID: %2u, Slot: %u\n", exporter_record->sysid, ipstr, exporter_record->version, exporter_record->id, id); } else if ( exporter_record->sa_family == AF_INET6 ) { uint64_t _ip[2]; _ip[0] = htonll(exporter_record->ip.V6[0]); _ip[1] = htonll(exporter_record->ip.V6[1]); inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr)); printf("SysID: %u, IP: %40s, version: %u, ID: %2u, Slot: %u\n", exporter_record->sysid, ipstr, exporter_record->version, exporter_record->id, id); } else { strncpy(ipstr, "", IP_STRING_LEN); printf("**** Exporter IP version unknown ****\n"); } } printf("\n"); #endif if ( !exporter_root ) { exporter_root = exporter_list[id]; } return 1; } // End of AddExporterInfo int AddSamplerInfo(sampler_info_record_t *sampler_record) { uint32_t id; sampler_t **sampler; if ( sampler_record->header.size != sizeof(sampler_info_record_t) ) { LogError("Corrupt sampler record in %s line %d\n", __FILE__, __LINE__); return 0; } id = sampler_record->exporter_sysid; if ( id >= MAX_EXPORTERS) { LogError("Corrupt sampler record in %s line %d\n", __FILE__, __LINE__); return 0; } if ( !exporter_list[id] ) { LogError("Exporter SysID: %u not found! - Skip sampler record", id); return 0; } sampler = &exporter_list[id]->sampler; while ( *sampler ) { if ( memcmp((void *)&(*sampler)->info, (void *)sampler_record, sizeof(sampler_info_record_t)) == 0 ) { // Found identical sampler already registered dbg_printf("Identical sampler already registered: Exporter SysID: %u, Sampler: id: %i, mode: %u, interval: %u\n", sampler_record->exporter_sysid, sampler_record->id, sampler_record->mode, sampler_record->interval); return 2; } sampler = &((*sampler)->next); } *sampler = (sampler_t *)malloc(sizeof(sampler_t)); if ( !*sampler ) { LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return 0; } (*sampler)->next = NULL; sampler_record->exporter_sysid = exporter_list[id]->info.sysid; memcpy((void *)&(*sampler)->info, (void *)sampler_record, sizeof(sampler_info_record_t)); dbg_printf("Insert sampler record for exporter at slot %i:\n", id); #ifdef DEVEL { if ( sampler_record->id < 0 ) { printf("Exporter SysID: %u, Generic Sampler: mode: %u, interval: %u\n", sampler_record->exporter_sysid, sampler_record->mode, sampler_record->interval); } else { printf("Exporter SysID: %u, Sampler: id: %i, mode: %u, interval: %u\n", sampler_record->exporter_sysid, sampler_record->id, sampler_record->mode, sampler_record->interval); } } #endif return 1; } // End of AddSamplerInfo int AddExporterStat(exporter_stats_record_t *stat_record) { int i, use_copy; exporter_stats_record_t *rec; size_t required; if ( stat_record->header.size < sizeof(exporter_stats_record_t) ) { LogError("Corrupt exporter record in %s line %d\n", __FILE__, __LINE__); return 0; } required = sizeof(exporter_stats_record_t) + (stat_record->stat_count-1) * sizeof(struct exporter_stat_s); if ((stat_record->stat_count == 0) || (stat_record->header.size != required)) { LogError("Corrupt exporter record in %s line %d\n", __FILE__, __LINE__); return 0; } // 64bit counters can be potentially unaligned if ( ((ptrdiff_t)stat_record & 0x7) != 0 ) { rec = malloc(stat_record->header.size); if ( !rec ) { LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); exit(255); } memcpy(rec, stat_record, stat_record->header.size); use_copy = 1; } else { rec = stat_record; use_copy = 0; } for (i=0; istat_count; i++ ) { uint32_t id = rec->stat[i].sysid; if ( id >= MAX_EXPORTERS ) { LogError("Corrupt exporter record in %s line %d\n", __FILE__, __LINE__); return 0; } if ( !exporter_list[id] ) { LogError("Exporter SysID: %u not found! - Skip stat record record.\n"); continue; } exporter_list[id]->sequence_failure += rec->stat[i].sequence_failure; exporter_list[id]->packets += rec->stat[i].packets; exporter_list[id]->flows += rec->stat[i].flows; dbg_printf("Update exporter stat for SysID: %i: Sequence failures: %u, packets: %llu, flows: %llu\n", id, exporter_list[id]->sequence_failure, exporter_list[id]->packets, exporter_list[id]->flows); } if ( use_copy ) free(rec); return 1; } // End of AddExporterStat void ExportExporterList( nffile_t *nffile ) { int i; // sysid 0 unused -> no exporter available i = 1; while ( i < MAX_EXPORTERS && exporter_list[i] != NULL ) { exporter_info_record_t *exporter; sampler_t *sampler; exporter = &exporter_list[i]->info; AppendToBuffer(nffile, (void *)exporter, exporter->header.size); sampler = exporter_list[i]->sampler; while ( sampler ) { AppendToBuffer(nffile, (void *)&(sampler->info), sampler->info.header.size); sampler = sampler->next; } i++; } } // End of ExportExporterList void PrintExporters(char *filename) { int i, done, found = 0; nffile_t *nffile; record_header_t *record; uint32_t skipped_blocks; uint64_t total_bytes; printf("Exporters:\n"); nffile = OpenFile(filename, NULL); if ( !nffile ) { return; } total_bytes = 0; skipped_blocks = 0; done = 0; while ( !done ) { int i, ret; // get next data block from file ret = ReadBlock(nffile); switch (ret) { case NF_CORRUPT: case NF_ERROR: if ( ret == NF_CORRUPT ) LogError("Corrupt data file '%s': '%s'\n",filename); else LogError("Read error in file '%s': %s\n",filename, strerror(errno) ); done = 1; continue; break; // fall through - get next file in chain case NF_EOF: done = 1; continue; break; default: // successfully read block total_bytes += ret; } if ( nffile->block_header->id != DATA_BLOCK_TYPE_2 ) { skipped_blocks++; continue; } // block type = 2 record = (record_header_t *)nffile->buff_ptr; for ( i=0; i < nffile->block_header->NumRecords; i++ ) { switch ( record->type ) { case LegacyRecordType1: case LegacyRecordType2: LogError("Legacy record type: %i no longer supported\n", record->type); break; case ExporterInfoRecordType: found = 1; if ( !AddExporterInfo((exporter_info_record_t *)record) ) { LogError("Failed to add Exporter Record\n"); } break; case ExporterStatRecordType: AddExporterStat((exporter_stats_record_t *)record); break; case SamplerInfoRecordype: if ( !AddSamplerInfo((sampler_info_record_t *)record) ) { LogError("Failed to add Sampler Record\n"); } break; } // Advance pointer by number of bytes for netflow record record = (record_header_t *)((pointer_addr_t)record + record->size); } } CloseFile(nffile); DisposeFile(nffile); if ( !found ) { printf("No Exporter records found\n"); } i = 1; while ( i < MAX_EXPORTERS && exporter_list[i] != NULL ) { #define IP_STRING_LEN 40 char ipstr[IP_STRING_LEN]; exporter_info_record_t *exporter; sampler_t *sampler; printf("\n"); exporter = &exporter_list[i]->info; if ( exporter->sa_family == AF_INET ) { uint32_t _ip = htonl(exporter->ip.V4); inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr)); if ( exporter_list[i]->flows ) printf("SysID: %u, IP: %16s, version: %u, ID: %2u, Sequence failures: %u, packets: %llu, flows: %llu\n", exporter->sysid, ipstr, exporter->version, exporter->id, exporter_list[i]->sequence_failure, (long long unsigned)exporter_list[i]->packets, (long long unsigned)exporter_list[i]->flows); else printf("SysID: %u, IP: %16s, version: %u, ID: %2u\n", exporter->sysid, ipstr, exporter->version, exporter->id); } else if ( exporter->sa_family == AF_INET6 ) { uint64_t _ip[2]; _ip[0] = htonll(exporter->ip.V6[0]); _ip[1] = htonll(exporter->ip.V6[1]); inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr)); if ( exporter_list[i]->flows ) printf("SysID: %u, IP: %40s, version: %u, ID: %2u, Sequence failures: %u, packets: %llu, flows: %llu\n ", exporter->sysid, ipstr, exporter->version, exporter->id, exporter_list[i]->sequence_failure, (long long unsigned)exporter_list[i]->packets, (long long unsigned)exporter_list[i]->flows); else printf("SysID: %u, IP: %40s, version: %u, ID: %2u\n ", exporter->sysid, ipstr, exporter->version, exporter->id); } else { strncpy(ipstr, "", IP_STRING_LEN); printf("**** Exporter IP version unknown ****\n"); } sampler = exporter_list[i]->sampler; while ( sampler ) { if ( sampler->info.id < 0 ) { printf(" Sampler for Exporter SysID: %u, Generic Sampler: mode: %u, interval: %u\n", sampler->info.exporter_sysid, sampler->info.mode, sampler->info.interval); } else { printf(" Sampler for Exporter SysID: %u, Sampler: id: %i, mode: %u, interval: %u\n", sampler->info.exporter_sysid, sampler->info.id, sampler->info.mode, sampler->info.interval); } sampler = sampler->next; } i++; } } // End of PrintExporters nfdump-1.6.23/bin/exporter.h000077500000000000000000000201271404501030700156640ustar00rootroot00000000000000/* * Copyright (c) 2012-2020, Peter Haag * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _EXPORTER_H #define _EXPORTER_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include "nffile.h" typedef struct optionTag_s { uint16_t offset; uint16_t length; } optionTag_t; /* * * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | - | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 0 | record type == 7 | size | version | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 1 | | * +----+--------------+--------------+--------------+---------- ip ------------+--------------+--------------+--------------+ * | 2 | | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 3 | sa_family | sysid | id | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ */ typedef struct exporter_info_record_s { record_header_t header; // exporter version uint32_t version; #define SFLOW_VERSION 9999 // IP address ip_addr_t ip; uint16_t sa_family; // internal assigned ID uint16_t sysid; // exporter ID/Domain ID/Observation Domain ID assigned by the device uint32_t id; } exporter_info_record_t; /* * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | - | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 0 | record type == 8 | size | stat_count | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 1 | sysid[0] | sequence_failure[0] | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 2 | packets[0] | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 3 | flows[0] | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * ... more stat records [x], one for each exporter * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ */ typedef struct exporter_stats_record_s { record_header_t header; uint32_t stat_count; // number of stat records struct exporter_stat_s { uint32_t sysid; // identifies the exporter uint32_t sequence_failure; // number of sequence failues uint64_t packets; // number of packets sent by this exporter uint64_t flows; // number of flow records sent by this exporter } stat[1]; } exporter_stats_record_t; /* * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | - | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 0 | record type == 9 | size | id | * +----+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+ * | 1 | interval | mode | exporter_sysid | * +----+--------------+--------------+--------------+-----------------------------+--------------+--------------+--------------+ */ typedef struct sampler_info_record_s { record_header_t header; // sampler data int32_t id; // id assigned by the exporting device uint32_t interval; // sampling interval uint16_t mode; // sampling mode uint16_t exporter_sysid; // internal reference to exporter } sampler_info_record_t; typedef struct sampler_s { struct sampler_s *next; sampler_info_record_t info; // sampler record nffile } sampler_t; typedef struct exporter_s { // linked chain struct exporter_s *next; // exporter information exporter_info_record_t info; // exporter record nffile uint64_t packets; // number of packets sent by this exporter uint64_t flows; // number of flow records sent by this exporter uint32_t sequence_failure; // number of sequence failues uint32_t padding_errors; // number of sequence failues sampler_t *sampler; // list of samplers associated with this exporter } exporter_t; typedef struct samplerOption_s { struct samplerOption_s *next; uint32_t tableID; // table id #define STDSAMPLING34 1 #define STDSAMPLING35 2 #define STDMASK 0x3 #define STDFLAGS 0x3 #define SAMPLER302 4 #define SAMPLER304 8 #define SAMPLER305 16 #define SAMPLERMASK 0x1C #define SAMPLERFLAGS 0x1C uint32_t flags; // info about this map // sampling offset/length values optionTag_t id; optionTag_t mode; optionTag_t interval; } samplerOption_t; int InitExporterList(void); int AddExporterInfo(exporter_info_record_t *exporter_record); int AddSamplerInfo(sampler_info_record_t *sampler_record); int AddExporterStat(exporter_stats_record_t *stat_record); void ExportExporterList( nffile_t *nffile ); void PrintExporters(char *filename); #endif //_EXPORTER_H nfdump-1.6.23/bin/filter.h000066400000000000000000000111141404501030700152720ustar00rootroot00000000000000/* * Copyright (c) 2020, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _FILTER_H #define _FILTER_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include "rbtree.h" #define NSEL_EVENT_IGNORE 0LL #define NSEL_EVENT_CREATE 1LL #define NSEL_EVENT_DELETE 2LL #define NSEL_EVENT_DENIED 3LL #define NSEL_EVENT_ALERT 4LL #define NSEL_EVENT_UPDATE 5LL #define NEL_EVENT_INVALID 0LL #define NEL_EVENT_ADD 1LL #define NEL_EVENT_DELETE 2LL /* * Definitions */ enum { CMP_EQ = 0, CMP_GT, CMP_LT, CMP_GE, CMP_LE, CMP_IDENT, CMP_FLAGS, CMP_IPLIST, CMP_ULLIST }; /* * filter functions: * For some filter functions, netflow records need to be processed first in order to filter them * This involves all data not directly available in the netflow record, such as packets per second etc. * Filter speed is a bit slower due to extra netflow processsing * The sequence of the enum values must correspond with the entries in the flow_procs array */ enum { FUNC_NONE = 0, /* no function - just plain filtering - just to be complete here */ FUNC_PPS, /* function code for pps ( packet per second ) filter function */ FUNC_BPS, /* function code for bps ( bits per second ) filter function */ FUNC_BPP, /* function code for bpp ( bytes per packet ) filter function */ FUNC_DURATION, /* function code for duration ( in miliseconds ) filter function */ FUNC_MPLS_EOS, /* function code for matching End of MPLS Stack label */ FUNC_MPLS_ANY, /* function code for matching any MPLS label */ FUNC_PBLOCK /* function code for matching ports against pblock start */ }; typedef struct FilterParam { uint16_t comp; uint16_t direction; uint32_t data; uint32_t inout; uint32_t acl; uint32_t self; } FilterParam_t; /* Definition of the IP list node */ struct IPListNode { RB_ENTRY(IPListNode) entry; uint64_t ip[2]; uint64_t mask[2]; }; /* Definition of the port/AS list node */ struct ULongListNode { RB_ENTRY(ULongListNode) entry; uint64_t value; }; /* IP tree type */ typedef RB_HEAD(IPtree, IPListNode) IPlist_t; /* Port/AS tree type */ typedef RB_HEAD(ULongtree, ULongListNode) ULongtree_t; // Insert the RB prototypes here RB_PROTOTYPE(IPtree, IPListNode, entry, IPNodeCMP); RB_PROTOTYPE(ULongtree, ULongListNode, entry, ULNodeCMP); /* parser/scanner prototypes */ int yyparse(void); int yylex(void); void lex_cleanup(void); void lex_init(char *buf); int ScreenIdentString(char *string); /* * Returns next free slot in blocklist */ uint32_t NewBlock(uint32_t offset, uint64_t mask, uint64_t value, uint16_t comp, uint32_t function, void *data); /* * Connects the to blocks b1 and b2 ( AND ) and returns index of superblock */ uint32_t Connect_AND(uint32_t b1, uint32_t b2); /* * Connects the to blocks b1 and b2 ( OR ) and returns index of superblock */ uint32_t Connect_OR(uint32_t b1, uint32_t b2); /* * Inverts OnTrue and OnFalse */ uint32_t Invert(uint32_t a ); /* * Add label to filter index */ void AddLabel(uint32_t index, char *label); /* * Add Ident to Identlist */ uint32_t AddIdent(char *Ident); #endif //_FILTER_H nfdump-1.6.23/bin/flist.c000066400000000000000000001054251404501030700151320ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H # include #endif #ifdef HAVE_FTS_H # include #else # include "fts_compat.h" #define fts_children fts_children_compat #define fts_close fts_close_compat #define fts_open fts_open_compat #define fts_read fts_read_compat #define fts_set fts_set_compat #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "flist.h" /* * Select a single file * -------------------- * -r [/]path/to/single_file * Select a single file: absolute or relativ path to a single file. * Recursive: no * * Selecting a range of files * -------------------------- * -R [/]path/to/first_file * Select a range of files in directory specified by absolut or relative path [/]path/to/ * Files are selected in alphabetical order starting with 'first_file' to the end of * the directory. * * -R [/]path/to/first_file:last_file * Select a range of files in directory specified by absolut or relative path [/]path/to/ * Files are selected in alphabetical order starting with 'first_file' and ending with * 'last_file'. * * -R [/]path/to/directory * Select all files in alphabetical order in directory specified by absolut or relative * path [/]path/to/directory * * Selecting files over multiple sources * ------------------------------------- * -M /path/to/multiple/source1:source2[:..:sourceN] * It is assumed, that each source directory 'source1', 'source2' etc. exists in directory * /path/to/multiple. This will expand to multiple directories: * /path/to/multiple/source1 * /path/to/multiple/source2 * .. * /path/to/multiple/sourceN * Each of these directories contain the same files. * Used in combination with -r and -R to prepend file selections. * * Select a single file from multiple directories * ---------------------------------------------- * -M /path/to/source1:source2 -r single_file * Select the same file 'single_file' from each source directory: e.g. * /path/to/source1/single_file * /path/to/source2/single_file * * * Select a range of files from multiple directories * ------------------------------------------------- * -M /path/to/source1:source2[:..] -R first_file * For each expanded directory specified by -M /path/to/source1:source2 * select a range of files as described above. Would be identical to * -R /path/to/source1/first_file -R /path/to/source2/first_file * * -M /path/to/source1:source2[:..] -R first_file:last_file * For each expanded directory specified by -M /path/to/source1:source2 * select a range of files as described above. Would be identical to * -R /path/to/source1/first_file:last_file -R /path/to/source2/first_file:last_file [-R .. ] * * -M /path/to/source1:source2[:..] -R . * For each expanded directory specified by -M /path/to/source1:source2 * select all files of the directory as described above. Would be to * -R /path/to/source1 -R /path/to/source2 [-R ...] * * * Hierarchical file organinisation: * For performance reasons, files may be store in various sub directries instead of a * single directory. These sub directories are assumed to be created in alpabetical order. * For example daily sub directories: 2006/04/01 .. 2006/04/30 as created by nfcapd with * option -S %y/%m/%d * * Single file selection is identical to the flat file layout: * -r [/]path/to/sub1/sub2/sub3/single_file * * Selecting a range of files in a hierarchical file layout * -------------------------------------------------------- * -R [/]path/to/sub1/sub2/first_file * Select a range of files in directory specified by absolut or relative path * [/]path/to/sub1/sub2/. Files are selected in alphabetical order starting with * 'first_file' to the end of the directory. The hierarchy has no impact here. * * -R [/]path/to/first_sub1/first_sub2/first_file:last_sub1/last_sub2/last_file * Select a range of files over multiple sub directories starting at absolut or * relative path [/]path/to/first_sub1/first_sub2/first_file up to and including * [/]path/to/last_sub1/last_sub2/last_file. Files are selected in alphabetical * order by iterating over the required sub directory hierachy * Example: * -R /path/to/2006/03/31/nfcapd.200603312300:2006/04/01/nfcapd.200604010600 * * -R [/]path/to/directory * Select all files in alphabetical order in directory specified by absolut or relative * path [/]path/to/directory, identical to flat layout * * The same methode applies for selecting a range of files over multiple sub directories * and multiple sources. * * Example: * -M /path/to/source1:source2 -R 2006/03/31/nfcapd.200603312300:2006/04/01/nfcapd.200604010600 * */ /* * syntax for possible sub dir definitions: * * %Y is replaced by the year with century as a decimal number. * %y is replaced by the year without century as a decimal number (00-99). * %m is replaced by the month as a decimal number (01-12). * %d is replaced by the day of the month as a decimal number (01-31). * %j is replaced by the day of the year as a decimal number (001-366). * %H is replaced by the hour (24-hour clock) as a decimal number (00-23). * %M is replaced by the minute as a decimal number (00-59). * %s is replaced by the number of seconds since the Epoch, UTC * %U is replaced by the week number of the year (Sunday as the first day * of the week) as a decimal number (00-53). * %W is replaced by the week number of the year (Monday as the first day * of the week) as a decimal number (00-53). * %w is replaced by the weekday (Sunday as the first day of the week) as * a decimal number (0-6). * %u is replaced by the weekday (Monday as the first day of the week) as * a decimal number (1-7). * %F is equivalent to ``%Y-%m-%d''. */ // predefined and accpeted formats static const char *subdir_def[] = { "", // default index 0 - no subdir hierarchy "%Y/%m/%d", "%Y/%m/%d/%H", "%Y/%W/%u", "%Y/%W/%u/%H", "%Y/%j", "%Y/%j/%H", "%F", "%F/%H", NULL }; // all accpeted char in a string #define AcceptedFormatChar "YymdjHMsUWwuF" static mode_t mode, dir_mode; static const char *subdir_format; static struct entry_filter_s { char *first_entry; char *last_entry; int list_files; } *dir_entry_filter = NULL; #define NUM_PTR 16 // globals extern uint32_t twin_first, twin_last; static char *first_file, *last_file; static char *current_file = NULL; static stringlist_t source_dirs, file_list; /* Function prototypes */ static inline int CheckTimeWindow(uint32_t t_start, uint32_t t_end, stat_record_t *stat_record); static void GetFileList(char *path); static void CleanPath(char *entry); static void Getsource_dirs(char *dirs); static int mkpath(char *path, char *p, mode_t mode, mode_t dir_mode, char *error, size_t errlen); static char *GuessSubDir(char *channeldir, char *filename); static char *VerifyFileRange(char *path, char *last_file); /* Functions */ static int compare(const FTSENT **f1, const FTSENT **f2) { return strcmp( (*f1)->fts_name, (*f2)->fts_name); } // End of compare static void CleanPath(char *entry) { char *p, *q; size_t len; // wash out any '//' in entry while ( (p = strstr(entry, "//")) != NULL ) { p++; q = p+1; // q points to first char after '//' while ( *p ) *p++ = *q++; } // remove trailing '/' len = strlen(entry); if ( entry[len-1] == '/' ) entry[len-1] = '\0'; // wash out any '/./' in entry while ( (p = strstr(entry, "/./")) != NULL ) { p++; q = p+2; // q points to first char after '/./' while ( *p ) *p++ = *q++; } // remove leading './' in entry if ( strstr(entry, "./") == entry ) { p = entry; q = p + 2; while ( *p ) *p++ = *q++; } } // End of CleanPath static inline int CheckTimeWindow(uint32_t t_start, uint32_t t_end, stat_record_t *stat_record) { /* printf("t start %u %s", t_start, ctime(&t_start)); printf("t end %u %s", t_end, ctime(&t_end)); printf("f start %u %s", NetflowStat.first_seen, ctime(&NetflowStat.first_seen)); printf("f end %u %s", NetflowStat.last_seen, ctime(&NetflowStat.last_seen)); */ // if no time window is set, return true if ( t_start == 0 ) return 1; if ( stat_record->first_seen == 0 ) return 0; if ( t_start >= stat_record->first_seen && t_start <= stat_record->last_seen ) return 1; if ( t_end >= stat_record->first_seen && t_end <= stat_record->last_seen ) return 1; if ( t_start < stat_record->first_seen && t_end > stat_record->last_seen ) return 1; return 0; } // End of CheckTimeWindow // file filter for scandir function static int dirlevels(char *dir) { int num; if ( !dir ) return 0; num = 0; if ( dir[0] == '/' ) dir++; while ( *dir ) { if ( *dir == '/' ) num++; dir++; } return num; } // End of dirlevels static void CreateDirListFilter(char *first_path, char *last_path, int file_list_level) { int i; char *p, *q, *first_mark, *last_mark; // printf("First Dir: '%s', first_path: '%s', last_path '%s', first_file '%s', last_file '%s', list_level: %i\n", // source_dirs.list[0], first_path, last_path, first_file, last_file, file_list_level); if ( file_list_level == 0 ) return; if ( file_list_level < 0 ) { LogError("software error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); exit(250); } dir_entry_filter = (struct entry_filter_s *)malloc((file_list_level+1) * sizeof(struct entry_filter_s)); if ( !dir_entry_filter ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); exit(250); } // first default entry - the directory itself dir_entry_filter[0].first_entry = NULL; dir_entry_filter[0].last_entry = NULL; dir_entry_filter[0].list_files = 0; first_mark = first_path; last_mark = last_path; // intermediate directory level filters for ( i=1; i 0 ) LogError("WARNING: Entry '%s' > '%s'. Will not match anything!", dir_entry_filter[i].first_entry, dir_entry_filter[i].last_entry); // printf("%i first: '%s', last: '%s'\n", // i, dir_entry_filter[i].first_entry, dir_entry_filter[i].last_entry); } // the last level - files are listed here dir_entry_filter[file_list_level].first_entry = first_file; dir_entry_filter[file_list_level].last_entry = last_file; dir_entry_filter[file_list_level].list_files = 1; if ( dir_entry_filter[file_list_level].first_entry && dir_entry_filter[file_list_level].last_entry && strcmp(dir_entry_filter[file_list_level].first_entry, dir_entry_filter[file_list_level].last_entry) > 0 ) LogError("WARNING: File '%s' > '%s'. Will not match anything!", dir_entry_filter[file_list_level].first_entry, dir_entry_filter[file_list_level].last_entry); /* printf("%i first: '%s', last: '%s'\n", file_list_level, dir_entry_filter[file_list_level].first_entry, dir_entry_filter[file_list_level].last_entry); */ } // End of CreateDirListFilter static void GetFileList(char *path) { struct stat stat_buf; char *last_file_ptr, *first_path, *last_path; int levels_first_file, levels_last_file, file_list_level; int sub_index; FTS *fts; FTSENT *ftsent; CleanPath(path); // Check for last_file option last_file_ptr = strchr(path, ':'); first_path = last_path = NULL; levels_first_file = levels_last_file = 0; if ( last_file_ptr ) { // make sure we have only a single ':' in path if ( strrchr(path, ':') != last_file_ptr ) { LogError("Multiple file separators ':' in path not allowed!"); exit(250); } *last_file_ptr++ = '\0'; // last_file_ptr points to last_file if ( strlen(last_file_ptr) == 0 ) { LogError("Missing last file option after ':'!"); exit(250); } CleanPath(last_file_ptr); // make sure last_file option is not a full path if ( last_file_ptr[0] == '/') { LogError("Last file name in -R list must not start with '/'"); exit(250); } // how may sub dir levels has last_file option? levels_last_file = dirlevels(last_file_ptr); // if no subdirs are given for last_file, try to find out, if the last_file // exists in any possible subdirs if ( levels_last_file == 0 ) { char s[MAXPATHLEN]; char *r = VerifyFileRange(path, last_file_ptr); if ( r != last_file_ptr && r[0] != '\0' ) { snprintf(s, MAXPATHLEN-1, "%s/%s", r, last_file_ptr); s[MAXPATHLEN-1] = '\0'; last_file_ptr = strdup(s); levels_last_file = dirlevels(last_file_ptr); } } } levels_first_file = dirlevels(path); if ( source_dirs.num_strings == 0 ) { // No multiple sources option -M // path contains the path to a file/directory // stat this entry if ( stat(path, &stat_buf) ) { LogError("stat() error '%s': %s", path, strerror(errno)); exit(250); } if ( !S_ISDIR(stat_buf.st_mode) && !S_ISREG(stat_buf.st_mode) ) { LogError("Not a file or directory: '%s'", path); exit(250); } // Check, how many levels of directory in path levels_first_file = dirlevels(path); if ( last_file_ptr ) { // path is [/]path/to/any/dir|file:last_file_ptr // make sure first_file is a file if ( S_ISDIR(stat_buf.st_mode) ) { LogError("Not a file: '%s'", path); exit(250); } if ( levels_last_file ) { // we have levels_last_file number of sub dirs // sub dir levels of first_file mus have at least the same number of levels as last_file if ( levels_first_file < levels_last_file ) { LogError("Number of sub dirs for sub level hierarchy for file list -R do not match"); exit(250); } if ( levels_first_file == levels_last_file ) { char *p, *q; // path = [/]sub1[/..]/first_file:sub1[/...]/last_file if ( path[0] == '/' ) { // this is rather strange, but strctly spoken, valid anyway InsertString(&source_dirs, "/"); path++; } else { InsertString(&source_dirs, "."); } // path = sub_first[/..]/first_file:sub_last[/...]/last_file p = strrchr(path, '/'); q = strrchr(last_file_ptr, '/'); if ( !p || !q ) { // this should never happen LogError("software error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); exit(250); } *p++ = '\0'; *q++ = '\0'; first_file = strdup(p); last_file = strdup(q); file_list_level = levels_last_file + 1; first_path = path; last_path = last_file_ptr; } else { // path = [/]path/to/sub_first[/..]/first_file:sub_last[/...]/last_file int i; char *p, *r, *s; p = strrchr(path, '/'); // levels_first_file > levels_last_file // step back the number of sub dirs in first_file for ( i=0; i= path && *p != '/'); } *p++ = '\0'; InsertString(&source_dirs, path); r = strrchr(p, '/'); s = strrchr(last_file_ptr, '/'); if ( !r || !s ) { // this must never happen LogError("software error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); exit(250); } *r++ = '\0'; *s++ = '\0'; first_file = strdup(r); last_file = strdup(s); // files are listed at this sub dir level file_list_level = levels_last_file + 1; first_path = p; last_path = last_file_ptr; } } else { // we have no sub dir levels given // path is [/]path/to/any/file char *p = strrchr(path, '/'); if ( p ) { // path is [/]path/to/any/first_file:last_file *p++ = '\0'; // path is the direcory containing all the files InsertString(&source_dirs, path); first_file = strdup(p); } else { // path is first_file:last_file InsertString(&source_dirs, "."); first_file = strdup(path); } // set last_file filter last_file = strdup(last_file_ptr); // in any case we list the files of directory level 1 file_list_level = 1; } } else { // path is [/]path/to/any/dir|file if ( S_ISDIR(stat_buf.st_mode) ) { // path is [/]path/to/any/dir // list all files in this directory InsertString(&source_dirs, path); first_file = NULL; file_list_level = 0; } else { // path is [/]path/to/any/file char *p = strrchr(path, '/'); if ( p ) { // path is [/]path/to/any/file *p++ = '\0'; // path is the direcory containing all the files InsertString(&source_dirs, path); first_file = strdup(p); } else { // path is file InsertString(&source_dirs, "."); first_file = strdup(path); } // in any case we list the files of directory level 1 file_list_level = 1; } // in any case, no last_file filter last_file = NULL; } } else { char pathbuff[MAXPATHLEN]; // multiple sources option -M given if ( path[0] == '/') { LogError("File list -R must not start with '/' when combined with a source list -M"); exit(250); } // special case for all files in directory if ( strcmp(path, ".") == 0 ) { first_file = NULL; last_file = NULL; file_list_level = 0; } else { // pathbuff contains the path to a file/directory, compiled using the first entry // in the source_dirs snprintf(pathbuff, MAXPATHLEN-1, "%s/%s", source_dirs.list[0], path); pathbuff[MAXPATHLEN-1] = '\0'; // pathbuff must point to a file if ( stat(pathbuff, &stat_buf) ) { if ( errno == ENOENT ) { // file not found - try to guess a possible subdir char *sub_dir = GuessSubDir(source_dirs.list[0], path); if ( sub_dir ) { // subdir found snprintf(pathbuff, MAXPATHLEN-1, "%s/%s", sub_dir, path); pathbuff[MAXPATHLEN-1] = '\0'; // update path path = strdup(pathbuff); free(sub_dir); // need guessing subdir with last_file too if ( last_file_ptr ) { sub_dir = GuessSubDir(source_dirs.list[0], last_file_ptr); if ( sub_dir ) { // subdir found snprintf(pathbuff, MAXPATHLEN-1, "%s/%s", sub_dir, last_file_ptr); pathbuff[MAXPATHLEN-1] = '\0'; last_file_ptr = strdup(pathbuff); free(sub_dir); // update dir levels of extended file path levels_last_file = dirlevels(last_file_ptr); } else { LogError("'%s': %s", last_file_ptr, "File not found!"); exit(250); } } } else { // no file in any possible subdir found LogError("stat() error '%s': %s", pathbuff, "File not found!"); exit(250); } } else { // Any other stat error LogError("stat() error '%s': %s", pathbuff, strerror(errno)); exit(250); } } else if ( !S_ISREG(stat_buf.st_mode) ) { LogError("Not a file : '%s'", pathbuff); exit(250); } // Check, how many levels of directory in path levels_first_file = dirlevels(path); if ( last_file_ptr ) { // path is path/to/any/first_file:last_file_ptr char *p, *q; // the number of sub dirs must be eqal for first_file and last_file if ( levels_first_file != levels_last_file ) { LogError("Number of sub dirs must agree in '%s' and '%s'", path, last_file_ptr); exit(250); } p = strrchr(path, '/'); if ( p ) { // path is fist_sub/to/any/first_file // recursive all files in sub dirs file_list_level = dirlevels(path) + 1; *p++ = '\0'; first_file = strdup(p); first_path = path; } else { // path is first_file first_file = strdup(path); file_list_level = 1; } q = strrchr(last_file_ptr, '/'); if ( q ) { *q++ = '\0'; last_file = strdup(q); last_path = last_file_ptr; } else { last_file = strdup(last_file_ptr); } } else { // path is path/to/any/first_file char *p = strrchr(path, '/'); if ( p ) { // path is fist_sub/to/any/first_file // recursive all files in sub dirs file_list_level = dirlevels(path) + 1; *p++ = '\0'; first_file = strdup(p); first_path = path; } else { // path is first_file first_file = strdup(path); file_list_level = 1; } last_file = NULL; } } } CreateDirListFilter(first_path, last_path, file_list_level ); // last entry must be NULL InsertString(&source_dirs, NULL); fts = fts_open(source_dirs.list, FTS_LOGICAL, compare); sub_index = 0; while ( (ftsent = fts_read(fts)) != NULL) { int fts_level = ftsent->fts_level; char *fts_path; // printf("DBG: %u %i %s %s\n", ftsent->fts_info, ftsent->fts_level, ftsent->fts_path, ftsent->fts_name); if ( fts_level == 0 ) { sub_index = ftsent->fts_pathlen + 1; continue; } if ( dir_entry_filter && (fts_level > file_list_level) ) { LogError("ERROR: fts_level error at %s line %d", __FILE__, __LINE__); exit(250); } if ( ftsent->fts_pathlen < sub_index ) { LogError("ERROR: fts_pathlen error at %s line %d", __FILE__, __LINE__); exit(250); } fts_path = &ftsent->fts_path[sub_index]; /* if ( file_list_level ) printf("DGB: short fts: '%s', filer_first: '%s', filter_last: '%s'\n", fts_path, dir_entry_filter[fts_level].first_entry , dir_entry_filter[fts_level].last_entry); */ switch (ftsent->fts_info) { case FTS_D: // dir entry pre descend if ( file_list_level && file_list_level && ( ( dir_entry_filter[fts_level].first_entry && ( strcmp(fts_path, dir_entry_filter[fts_level].first_entry ) < 0 ) ) || ( dir_entry_filter[fts_level].last_entry && ( strcmp(fts_path, dir_entry_filter[fts_level].last_entry ) > 0 ) ) )) fts_set(fts, ftsent, FTS_SKIP ); break; case FTS_DP: break; case FTS_F: // file entry // printf("==> Check: %s\n", ftsent->fts_name); // skip stat file if ( strcmp(ftsent->fts_name, ".nfstat") == 0 || strncmp(ftsent->fts_name, NF_DUMPFILE , strlen(NF_DUMPFILE)) == 0) continue; if ( strstr(ftsent->fts_name, ".stat") != NULL ) continue; // skip OSX DS_Store files if ( strstr(ftsent->fts_name, ".DS_Store") != NULL ) continue; // skip pcap file if ( strstr(ftsent->fts_name, "pcap") != NULL ) continue; if ( file_list_level && ( ( fts_level != file_list_level ) || ( dir_entry_filter[fts_level].first_entry && ( strcmp(ftsent->fts_name, dir_entry_filter[fts_level].first_entry) < 0 ) ) || ( dir_entry_filter[fts_level].last_entry && ( strcmp(ftsent->fts_name, dir_entry_filter[fts_level].last_entry) > 0 ) ) ) ) continue; // printf("==> Listed: %s\n", ftsent->fts_path); InsertString(&file_list, ftsent->fts_path); break; } } fts_close(fts); } // End of GetFileList /* * Get the list of directories * dirs: user supplied parameter: /any/path/dir1:dir2:dir3:... * source_dirs must result in * /any/path/dir1 * /any/path/dir2 * /any/path/dir3 * /any/path is dir prefix, which may be NULL e.g. dir1:dir2:dir3:... * dir1, dir2 etc entrys */ void Getsource_dirs(char *dirs) { struct stat stat_buf; char *p, *q, *dirprefix; char path[MAXPATHLEN]; q = strchr(dirs, ':'); if ( q ) { // we have /path/to/firstdir:dir1:dir2:... *q = 0; p = strrchr(dirs, '/'); if ( p ) { *p++ = 0; // p points now to the first name in the dir list dirprefix = dirs; } else { // we have a source_dirs in current directory p = dirs; // p points now to the first name in the dir list dirprefix = "."; // current directory } *q = ':'; // restore ':' in source_dirs while ( p ) { // iterate over all elements in the dir list q = strchr(p, ':'); if ( q ) *q = 0; // p point to a dir name snprintf(path, 1023, "%s/%s", dirprefix, p); path[MAXPATHLEN-1] = 0; if ( stat(dirs, &stat_buf) ) { LogError("Can't stat '%s': %s", path, strerror(errno)); return; } if ( !S_ISDIR(stat_buf.st_mode) ) { LogError("Not a directory: '%s'", path); return; } // save path into source_dirs InsertString(&source_dirs, path); p = q ? q + 1 : NULL; } } else { // we have only one directory if ( stat(dirs, &stat_buf) ) { LogError("Can't stat '%s': %s", dirs, strerror(errno)); return; } if ( !S_ISDIR(stat_buf.st_mode) ) { LogError("Not a directory: '%s'", dirs); return; } // save the path into source_dirs InsertString(&source_dirs, dirs); } } // End of Getsource_dirs void SetupInputFileSequence(char *multiple_dirs, char *single_file, char *multiple_files) { twin_first = 0; twin_last = 0xffffffff; first_file = NULL; last_file = NULL; InitStringlist(&source_dirs, NUM_PTR); InitStringlist(&file_list, 64); if ( multiple_dirs ) Getsource_dirs(multiple_dirs); if ( multiple_files ) { // use multiple files GetFileList(multiple_files); // get time window spanning all the files if ( file_list.num_strings ) { stat_record_t stat_ptr; // read the stat record if ( !GetStatRecord(file_list.list[0], &stat_ptr) ) { exit(250); } twin_first = stat_ptr.first_seen; // read the stat record of last file if ( !GetStatRecord(file_list.list[file_list.num_strings-1], &stat_ptr) ) { exit(250); } twin_last = stat_ptr.last_seen; } } else if ( single_file ) { CleanPath(single_file); if ( source_dirs.num_strings == 0 ) { stat_record_t stat_ptr; InsertString(&file_list, single_file); if ( !GetStatRecord(single_file, &stat_ptr) ) { exit(250); } twin_first = stat_ptr.first_seen; twin_last = stat_ptr.last_seen; } else { int i; if ( single_file[0] == '/' ) { LogError("File -r must not start with '/', when combined with a source list -M"); exit(250); } for ( i=0; i"); #endif nffile = OpenFile(file_list.list[cnt], nffile); // Open the file if ( !nffile ) { return NULL; } current_file = file_list.list[cnt]; cnt++; // stdin if ( nffile->fd == STDIN_FILENO ) { current_file = NULL; return nffile; } if ( CheckTimeWindow(twin_start, twin_end, nffile->stat_record) ) { // printf("Return file: %s\n", string); return nffile; } CloseFile(nffile); } current_file = NULL; return EMPTY_LIST; } // End of GetNextFile int InitHierPath(int num) { int i; subdir_format = NULL; i=0; while ( subdir_def[i] != NULL ) { if ( i == num ) break; i++; } if ( subdir_def[i] == NULL ) { LogError("No such subdir level %i", num); return 0; } subdir_format = subdir_def[i]; /* * The default file mode is a=rwx (0777) with selected permissions * removed in accordance with the file mode creation mask. For * intermediate path name components, the mode is the default modified * by u+wx so that the subdirectories can always be created. */ // get umask mode = umask(0); umask(mode); mode = 0777 & ~mode; dir_mode = mode | S_IWUSR | S_IXUSR; return 1; } // End of InitHierPath static char *VerifyFileRange(char *path, char *last_file) { char *p, *q, *r; r = strdup(path); p = strrchr(r, '/'); while ( p ) { *p = '\0'; q = GuessSubDir(r, last_file); if ( q ) { free(r); return q; } p = strrchr(r, '/'); } free(r); return last_file; } // End of VerifyFileRange static char *GuessSubDir(char *channeldir, char *filename) { char s[MAXPATHLEN]; struct tm *t_tm; int i; size_t len = strlen(filename); if ( (len == 19 || len == 21) && (strncmp(filename, "nfcapd.", 7) == 0) ) { char *p = &filename[7]; time_t t = ISO2UNIX(p); t_tm = localtime(&t); } else return NULL; i = 0; // if the file exists, it must be in any of the possible subdirs // so try one after the next - one will match while ( subdir_def[i] ) { char const *sub_fmt = subdir_def[i]; char subpath[255]; struct stat stat_buf; strftime(subpath, 254, sub_fmt, t_tm); subpath[254] = '\0'; snprintf(s, MAXPATHLEN-1, "%s/%s/%s", channeldir, subpath, filename); if ( stat(s, &stat_buf) == 0 && S_ISREG(stat_buf.st_mode) ) { // found file in subdir return strdup(subpath); } i++; } return NULL; } // End of GuessSubDir char *GetSubDir(struct tm *now ) { static char subpath[255]; size_t sublen; sublen = strftime(subpath, 254, subdir_format, now); return sublen == 0 ? NULL : subpath; } // End of GetSubDir int SetupSubDir(char *dir, char *subdir, char *error, size_t errlen ) { char *p, path[MAXPATHLEN]; struct stat stat_buf; size_t sublen, pathlen; int err; error[0] = '\0'; path[0] = '\0'; strncat(path, dir, MAXPATHLEN-1); path[MAXPATHLEN-1] = '\0'; sublen = strlen(subdir); pathlen = strlen(path); // set p as reference between path and subdir if ( (sublen + pathlen + 2) >= (MAXPATHLEN-1) ) { // +2 : add 1 for '/' snprintf(error, errlen, "Path '%s': too long", path); return 0; } p = path + pathlen; // points to '\0' of path *p++ = '/'; *p = '\0'; strncat(path, subdir, MAXPATHLEN-pathlen-2); // +2: add 1 for '/' // our cwd is basedir ( -l ) so test if, dir exists if ( stat(path, &stat_buf) == 0 ) { if ( S_ISDIR(stat_buf.st_mode) ) { // sub directory already exists return 1; } else { // an entry with this name exists, but it's not a directory snprintf(error, errlen, "Path '%s': %s ", path, strerror(ENOTDIR)); return 0; } } // no such entry exists - try to create the directory, assuming path below exists err = mkdir(path, dir_mode); if ( err == 0 ) // success return 1; // else errno is set if ( errno == ENOENT ) { // we need to create intermediate directories as well err = mkpath(path, p, mode, dir_mode, error, errlen); if ( err == 0 ) // creation was successful return 1; } else { snprintf(error, errlen, "mkdir() error for '%s': %s\n", path, strerror(errno)); } // anything else failed and error string is set return 0; } // End of SetupSubDir /* * mkpath -- create directories. * path - path * p - separator path/subpath * mode - file mode of terminal directory * dir_mode - file mode of intermediate directories */ static int mkpath(char *path, char *p, mode_t mode, mode_t dir_mode, char *error, size_t errlen) { struct stat sb; char *slash; int done = 0; slash = p; while (!done) { slash += strspn(slash, "/"); slash += strcspn(slash, "/"); done = (*slash == '\0'); *slash = '\0'; if (stat(path, &sb)) { if (errno != ENOENT || (mkdir(path, done ? mode : dir_mode) && errno != EEXIST)) { snprintf(error, errlen, "mkdir() error for '%s': %s\n", path, strerror(errno)); return (-1); } } else if (!S_ISDIR(sb.st_mode)) { snprintf(error, errlen, "Path '%s': %s ", path, strerror(ENOTDIR)); return (-1); } *slash = '/'; } return (0); } // End of mkpath nfdump-1.6.23/bin/flist.h000066400000000000000000000041041404501030700151270ustar00rootroot00000000000000/* * Copyright (c) 2017, Peter Haag * Copyright (c) 2014, Peter Haag * Copyright (c) 2009, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _FLIST_H #define _FLIST_H 1 #include #include "nffile.h" #define EMPTY_LIST ((nffile_t *)-1) int InitHierPath(int num); char *GetSubDir(struct tm *now); int SetupSubDir(char *dir, char *subdir, char *error, size_t errlen ); nffile_t *GetNextFile(nffile_t *nffile, time_t twin_start, time_t twin_end); #endif //_FLIST_H nfdump-1.6.23/bin/flowtree.c000066400000000000000000000376171404501030700156470ustar00rootroot00000000000000/* * Copyright (c) 2011-2020, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifdef HAVE_CONFIG_H #include "config.h" #endif #include #include #include #include #include #include #include #include #include #include #include #include #include "util.h" #include "nfdump.h" #include "nffile.h" #include "bookkeeper.h" #include "collector.h" #include "netflow_pcap.h" #include "flowtree.h" /* Lock operation. */ #define spin_lock(lck) do { \ int zero = 0; \ while (!atomic_compare_exchange_weak(&(lck), &zero, 1)) \ zero = 0; \ } while (0) /* Unlock operation. */ #define spin_unlock(lck) atomic_store(&(lck), 0); #define GetTreeLock(a) spin_lock((a)->list_lock) #define ReleaseTreeLock(a) spin_unlock((a)->list_lock) static int FlowNodeCMP(struct FlowNode *e1, struct FlowNode *e2); // Insert the IP RB tree code here RB_GENERATE(FlowTree, FlowNode, entry, FlowNodeCMP); // Flow Cache to store all nodes static uint32_t FlowCacheSize = 512 * 1024; static uint32_t expireActiveTimeout = 300; static uint32_t expireInactiveTimeout = 60; static struct FlowNode *FlowElementCache; // free list static struct FlowNode *FlowNode_FreeList; static pthread_mutex_t m_FreeList = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t c_FreeList = PTHREAD_COND_INITIALIZER; static uint32_t EmptyFreeList; static uint32_t EmptyFreeListEvents = 0; static uint32_t Allocated; // Flow tree static FlowTree_t *FlowTree; static int NumFlows = 0; // Simple unprotected list typedef struct FlowNode_list_s { struct FlowNode *list; struct FlowNode *tail; uint32_t size; } Linked_list_t; /* Free list handling functions */ // Get next free node from free list struct FlowNode *New_Node(void) { struct FlowNode *node; pthread_mutex_lock(&m_FreeList); while ( FlowNode_FreeList == NULL ) { EmptyFreeList = 1; EmptyFreeListEvents++; pthread_cond_wait(&c_FreeList, &m_FreeList); } node = FlowNode_FreeList; if ( node == NULL ) { // should never happen , as we were waiting for a free node LogError("*** Software ERROR *** New_Node() unexpected error in %s line %d: %s: %u\n", __FILE__, __LINE__, "Node list exhausted", NumFlows); pthread_mutex_unlock(&m_FreeList); return NULL; } if ( node->memflag != NODE_FREE ) { LogError("*** Software ERROR *** New_Node() unexpected error in %s line %d: %s\n", __FILE__, __LINE__, "Tried to allocate a non free Node"); abort(); } FlowNode_FreeList = node->right; Allocated++; pthread_mutex_unlock(&m_FreeList); node->left = NULL; node->right = NULL; node->memflag = NODE_IN_USE; return node; } // End of New_Node // return node into free list void Free_Node(struct FlowNode *node) { if ( node->memflag == NODE_FREE ) { LogError("Free_Node() Fatal: Tried to free an already freed Node"); abort(); } if ( node->memflag != NODE_IN_USE ) { LogError("Free_Node() Fatal: Tried to free a Node not in use"); abort(); } dbg_assert(node->left == NULL); dbg_assert(node->right == NULL); memset((void *)node, 0, sizeof(struct FlowNode)); pthread_mutex_lock(&m_FreeList); node->right = FlowNode_FreeList; node->left = NULL; node->memflag = NODE_FREE; FlowNode_FreeList = node; Allocated--; if ( EmptyFreeList ) { EmptyFreeList = 0; pthread_cond_signal(&c_FreeList); } pthread_mutex_unlock(&m_FreeList); } // End of Free_Node /* safety check - this must never become 0 - otherwise the cache is too small */ void CacheCheck(FlowSource_t *fs, time_t when, int live) { uint32_t num; // live = 1; // if the cache is exhausted - force expire now if (FlowCacheSize == NumFlows || (live && FlowNode_FreeList == NULL) ) { LogInfo("Node cache exhausted! - Force expire"); Expire_FlowTree(fs, when); } // if still exhausted force flush of all flows if (FlowCacheSize == NumFlows || (live && FlowNode_FreeList == NULL)) { LogError("Node cache exhausted! - Force immediate flush - increase flow cache > %u", FlowCacheSize); num = Flush_FlowTree(fs); LogError("Expired flows: %u", num); } } // End of CacheCheck /* flow tree functions */ int Init_FlowTree(uint32_t CacheSize, int32_t expireActive, int32_t expireInactive) { int i; if ( expireActive ) { if ( expireActive < 0 || expireActive > 3600 ) { LogError("Active flow timeout %d out of range", expireActive); return 0; } expireActiveTimeout = expireActive; LogInfo("Set active flow expire timout to %us", expireActiveTimeout); } if ( expireInactive ) { if ( expireInactive < 0 || expireInactive > 3600 ) { LogError("Inactive flow timeout %d out of range", expireInactive); return 0; } expireInactiveTimeout = expireInactive; LogInfo("Set inactive flow expire timout to %us", expireInactiveTimeout); } FlowTree = malloc(sizeof(FlowTree_t)); if ( !FlowTree ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return 0; } RB_INIT(FlowTree); if ( CacheSize == 0 ) CacheSize = FlowCacheSize; else FlowCacheSize = CacheSize; FlowElementCache = calloc(CacheSize, sizeof(struct FlowNode)); if ( !FlowElementCache ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); free(FlowTree); FlowTree = NULL; return 0; } // init free list FlowNode_FreeList = FlowElementCache; FlowNode_FreeList->left = NULL; FlowNode_FreeList->right = &FlowElementCache[1]; FlowNode_FreeList->memflag = NODE_FREE; for (i=1; i < (CacheSize-1); i++ ) { FlowElementCache[i].memflag = NODE_FREE; FlowElementCache[i].left = &FlowElementCache[i-1]; FlowElementCache[i].right = &FlowElementCache[i+1]; } FlowElementCache[i].left = &FlowElementCache[i-1]; FlowElementCache[i].right = NULL; FlowElementCache[i].memflag = NODE_FREE; EmptyFreeList = 0; Allocated = 0; NumFlows = 0; return 1; } // End of Init_FlowTree void Dispose_FlowTree(void) { struct FlowNode *node, *nxt; // Dump all incomplete flows to the file nxt = NULL; for (node = RB_MIN(FlowTree, FlowTree); node != NULL; node = nxt) { nxt = RB_NEXT(FlowTree, FlowTree, node); Remove_Node(node); } free(FlowElementCache); FlowElementCache = NULL; FlowNode_FreeList = NULL; EmptyFreeList = 0; } // End of Dispose_FlowTree static int FlowNodeCMP(struct FlowNode *e1, struct FlowNode *e2) { uint64_t *a = e1->src_addr.v6; uint64_t *b = e2->src_addr.v6; int i; #define CMPLEN (offsetof(struct FlowNode, _ENDKEY_) - offsetof(struct FlowNode, src_addr)) i = memcmp((void *)a, (void *)b, CMPLEN ); return i; } // End of FlowNodeCMP struct FlowNode *Lookup_Node(struct FlowNode *node) { return RB_FIND(FlowTree, FlowTree, node); } // End of Lookup_FlowTree struct FlowNode *Insert_Node(struct FlowNode *node) { struct FlowNode *n; dbg_assert(node->left == NULL); dbg_assert(node->right == NULL); // return RB_INSERT(FlowTree, FlowTree, node); n = RB_INSERT(FlowTree, FlowTree, node); if ( n ) { // existing node return n; } else { NumFlows++; return NULL; } } // End of Insert_Node void Remove_Node(struct FlowNode *node) { struct FlowNode *rev_node; #ifdef DEVEL assert(node->memflag == NODE_IN_USE); if ( NumFlows == 0 ) { LogError("Remove_Node() Fatal Tried to remove a Node from empty tree"); return; } #endif rev_node = node->rev_node; if ( rev_node ) { // unlink rev node on both nodes dbg_assert(rev_node->rev_node == node); rev_node->rev_node = NULL; node->rev_node = NULL; } RB_REMOVE(FlowTree, FlowTree, node); Free_Node(node); NumFlows--; } // End of Remove_Node int Link_RevNode(struct FlowNode *node) { struct FlowNode lookup_node, *rev_node; dbg_printf("Link node: "); dbg_assert(node->rev_node == NULL); lookup_node.src_addr = node->dst_addr; lookup_node.dst_addr = node->src_addr; lookup_node.src_port = node->dst_port; lookup_node.dst_port = node->src_port; lookup_node.version = node->version; lookup_node.proto = node->proto; rev_node = Lookup_Node(&lookup_node); if ( rev_node ) { dbg_printf("Found revnode "); // rev node must not be linked already - otherwise there is an inconsistency dbg_assert(node->rev_node == NULL); if (node->rev_node == NULL ) { // link both nodes node->rev_node = rev_node; rev_node->rev_node = node; dbg_printf(" - linked\n"); } else { dbg_printf("Rev-node != NULL skip linking - inconsitency\n"); LogError("Rev-node != NULL skip linking - inconsitency\n"); } return 1; } else { dbg_printf("no revnode node\n"); return 0; } /* not reached */ } // End of Link_RevNode uint32_t Flush_FlowTree(FlowSource_t *fs) { struct FlowNode *node, *nxt; uint32_t n = NumFlows; // Dump all incomplete flows to the file nxt = NULL; for (node = RB_MIN(FlowTree, FlowTree); node != NULL; node = nxt) { StorePcapFlow(fs, node); nxt = RB_NEXT(FlowTree, FlowTree, node); #ifdef DEVEL if ( node->left || node->right ) { assert(node->proto == 17); node->left = node->right = NULL; } #endif Remove_Node(node); } #ifdef DEVEL if ( NumFlows != 0 ) LogError("### Flush_FlowTree() remaining flows: %u\n", NumFlows); #endif return n; } // End of Flush_FlowTree uint32_t Expire_FlowTree(FlowSource_t *fs, time_t when) { struct FlowNode *node, *nxt; if ( NumFlows == 0 ) return NumFlows; uint32_t expireCnt = 0; // Dump all incomplete flows to the file nxt = NULL; for (node = RB_MIN(FlowTree, FlowTree); node != NULL; node = nxt) { nxt = RB_NEXT(FlowTree, FlowTree, node); if ( when == 0 || (when - node->t_last.tv_sec) > expireInactiveTimeout || (when - node->t_first.tv_sec) > expireActiveTimeout) { StorePcapFlow(fs, node); Remove_Node(node); expireCnt++; } } if ( expireCnt ) LogInfo("Expired Nodes: %u, in use: %u, total flows: %u", expireCnt, Allocated, NumFlows); return NumFlows; } // End of Expire_FlowTree /* Node list functions */ NodeList_t *NewNodeList(void) { NodeList_t *NodeList; NodeList = (NodeList_t *)malloc(sizeof(NodeList_t)); if ( !NodeList ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return NULL; } NodeList->list = NULL; NodeList->last = NULL; NodeList->length = 0; NodeList->list_lock = 0; NodeList->waiting = 0; NodeList->waits = 0; pthread_mutex_init(&NodeList->m_list, NULL); pthread_cond_init(&NodeList->c_list, NULL); return NodeList; } // End of NewNodeList void DisposeNodeList(NodeList_t *NodeList) { if ( !NodeList ) return; if ( NodeList->length ) { LogError("Try to free non empty NodeList"); return; } free(NodeList); } // End of DisposeNodeList #ifdef DEVEL void ListCheck(NodeList_t *NodeList); void ListCheck(NodeList_t *NodeList) { uint32_t len = 0, mem = 0, proto; static uint32_t loops = 0; struct FlowNode *node, *n; // DumpList(NodeList); loops++; node = NodeList->list; while (node) { len++; if ( node == NodeList->last ) { mem = len; } if ( node->memflag != NODE_IN_USE ) { printf("mem flag error : len: %u, last: %u, Nodelist: %u, loops: %u\n", len, mem, NodeList->length, loops); } if ( node->right == NULL ) { proto = node->proto; n = node; } node=node->right; } if ( len != NodeList->length) { printf("Len miss match: len: %u, last: %u, proto: %u, Nodelist: %u, loops: %u, allocated: %u, node: %llx\n", len, mem, proto, NodeList->length, loops, Allocated, (long long unsigned)n); assert(len==NodeList->length); } else { printf("Len: %u ok last: %u, proto: %u in loop %u, allocated: %u\n", len, mem, proto, loops, Allocated); } } #endif void Push_Node(NodeList_t *NodeList, struct FlowNode *node) { GetTreeLock(NodeList); // pthread_mutex_lock(&NodeList->m_list); if ( NodeList->length == 0 ) { // empty list NodeList->list = node; node->left = NULL; node->right = NULL; } else { NodeList->last->right = node; node->left = NodeList->last; node->right = NULL; } NodeList->last = node; NodeList->length++; #ifdef DEVEL printf("pushed node 0x%llx proto: %u, length: %u first: %llx, last: %llx\n", (unsigned long long)node, node->proto, NodeList->length, (unsigned long long)NodeList->list, (unsigned long long)NodeList->last); ListCheck(NodeList); #endif if ( NodeList->waiting ) { pthread_cond_signal(&NodeList->c_list); } ReleaseTreeLock(NodeList); // pthread_mutex_unlock(&NodeList->m_list); // pthread_cond_signal(&NodeList->c_list); } // End of Push_Node struct FlowNode *Pop_Node(NodeList_t *NodeList, int *done) { struct FlowNode *node; GetTreeLock(NodeList); while ( NodeList->length == 0 && !*done ) { pthread_mutex_lock(&NodeList->m_list); NodeList->waiting = 1; NodeList->waits++; ReleaseTreeLock(NodeList); // sleep ad wait pthread_cond_wait(&NodeList->c_list, &NodeList->m_list); // wake up GetTreeLock(NodeList); NodeList->waiting = 0; pthread_mutex_unlock(&NodeList->m_list); } if ( NodeList->length == 0 && *done ) { ReleaseTreeLock(NodeList); dbg_printf("Pop_Node done\n"); return NULL; } if ( NodeList->list == NULL ) { // should never happen - list is supposed to have at least one item ReleaseTreeLock(NodeList); LogError("Unexpected empty FlowNode_ProcessList"); return NULL; } node = NodeList->list; NodeList->list = node->right; if ( NodeList->list ) NodeList->list->left = NULL; else NodeList->last = NULL; node->left = NULL; node->right = NULL; NodeList->length--; #ifdef DEVEL printf("popped node 0x%llx proto: %u, length: %u first: %llx, last: %llx\n", (unsigned long long)node, node->proto, NodeList->length, (unsigned long long)NodeList->list, (unsigned long long)NodeList->last); ListCheck(NodeList); #endif ReleaseTreeLock(NodeList); return node; } // End of Pop_Node #ifdef DEVEL void DumpList(NodeList_t *NodeList) { struct FlowNode *node; printf("FlowNode_ProcessList: 0x%llx, length: %u\n", (unsigned long long)NodeList->list, NodeList->length); node = NodeList->list; while ( node ) { printf("node: 0x%llx\n", (unsigned long long)node); printf(" ->left: 0x%llx\n", (unsigned long long)node->left); printf(" ->right: 0x%llx\n", (unsigned long long)node->right); node = node->right; } printf("tail: 0x%llx\n\n", (unsigned long long)NodeList->last); } // End of DumpList #endif void DumpNodeStat(NodeList_t *NodeList) { LogInfo("Nodes in use: %u, Flows: %u, Nodes list length: %u, Waiting for freelist: %u", Allocated, NumFlows, NodeList->length, EmptyFreeListEvents); EmptyFreeListEvents = 0; } // End of DumpNodeStat nfdump-1.6.23/bin/flowtree.h000066400000000000000000000104611404501030700156400ustar00rootroot00000000000000/* * Copyright (c) 2011-2019, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _FLOWTREE_H #define _FLOWTREE_H 1 #ifdef HAVE_CONFIG_H #include "config.h" #endif #include #ifdef HAVE_STDINT_H #include #endif #include #include #include #include #include "collector.h" #include "rbtree.h" #define v4 ip_union._v4 #define v6 ip_union._v6 struct FlowNode { // tree RB_ENTRY(FlowNode) entry; // linked list struct FlowNode *left; struct FlowNode *right; struct FlowNode *biflow; // flow key // IP addr ip_addr_t src_addr; ip_addr_t dst_addr; uint16_t src_port; uint16_t dst_port; uint8_t proto; uint8_t version; uint16_t _ENDKEY_; // End of flow key ip_addr_t tun_src_addr; ip_addr_t tun_dst_addr; uint8_t tun_proto; #define NODE_FREE 0xA5 #define NODE_IN_USE 0x5A uint16_t memflag; // internal houskeeping flag uint8_t flags; #define FIN_NODE 1 #define SIGNAL_NODE 255 uint8_t fin; // double use: 1: fin received - flow can be exported, if complete // 255: empty node - used to wake up flow thread priodically on quite lines // flow stat data struct timeval t_first; struct timeval t_last; uint32_t packets; // summed up number of packets uint32_t bytes; // summed up number of bytes struct FlowNode *rev_node; struct latency_s { uint64_t client; uint64_t server; uint64_t application; uint32_t flag; struct timeval t_request; } latency; }; typedef struct NodeList_s { struct FlowNode *list; struct FlowNode *last; atomic_int list_lock; pthread_mutex_t m_list; pthread_cond_t c_list; uint32_t length; uint32_t waiting; uint64_t waits; } NodeList_t; /* flow tree type */ typedef RB_HEAD(FlowTree, FlowNode) FlowTree_t; // Insert the RB prototypes here RB_PROTOTYPE(FlowTree, FlowNode, entry, FlowNodeCMP); int Init_FlowTree(uint32_t CacheSize, int32_t expireActive, int32_t expireInactive); void Dispose_FlowTree(void); uint32_t Flush_FlowTree(FlowSource_t *fs); uint32_t Expire_FlowTree(FlowSource_t *fs, time_t when); struct FlowNode *Lookup_Node(struct FlowNode *node); struct FlowNode *New_Node(void); void Free_Node(struct FlowNode *node); void CacheCheck(FlowSource_t *fs, time_t when, int live); int AddNodeData(struct FlowNode *node, uint32_t seq, void *payload, uint32_t size); struct FlowNode *Insert_Node(struct FlowNode *node); void Remove_Node(struct FlowNode *node); int Link_RevNode(struct FlowNode *node); // Node list functions NodeList_t *NewNodeList(void); void DisposeNodeList(NodeList_t *NodeList); void Push_Node(NodeList_t *NodeList, struct FlowNode *node); struct FlowNode *Pop_Node(NodeList_t *NodeList, int *done); void DumpList(NodeList_t *NodeList); // Stat functions void DumpNodeStat(NodeList_t *NodeList); #endif // _FLOWTREE_H nfdump-1.6.23/bin/ft2nfdump.c000066400000000000000000000264111404501030700157130ustar00rootroot00000000000000/* * All rights reserved. * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * Copyright (c) 2001 Mark Fullmer and The Ohio State University * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * * Flow-Tools related code taken from flow-tools-0.67 cretated by Mark Fullmer * */ #ifdef HAVE_CONFIG_H #include "config.h" #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "exporter.h" #include "ftlib.h" #include "output_raw.h" /* Global defines */ #define MAXRECORDS 30 /* Global consts */ extern extension_descriptor_t extension_descriptor[]; const char *nfdump_version = VERSION; typedef struct v5_block_s { uint32_t srcaddr; uint32_t dstaddr; uint32_t dPkts; uint32_t dOctets; uint8_t data[4]; // link to next record } v5_block_t; /* externals */ extern uint32_t Max_num_extensions; /* prototypes */ void usage(char *name); extension_info_t *GenExtensionMap(struct ftio *ftio); int flows2nfdump(struct ftio *ftio, char *wfile, int compress, extension_info_t *extension_info, int extended, uint32_t limitflows); #define NEED_PACKRECORD #include "nffile_inline.c" #undef NEED_PACKRECORD void usage(char *name) { printf("usage %s [options] \n" "-h\t\tthis text you see right here.\n" "-E\t\tDump records in ASCII extended format to stdout.\n" "-c\t\tLimit number of records to convert.\n" "-m\t\tPrint the extension map and exit.\n" "-V\t\tPrint version and exit.\n" "-r \tread flow-tools records from file\n" "-w \twrite nfdump records to file\n" "-j\t\tBZ2 compress flows in output file.\n" "-z\t\tLZO compress flows in output file.\n" "Convert flow-tools format to nfdump format:\n" "ft2nfdump -r -w [-z]\n" , name); } // End of usage extension_info_t *GenExtensionMap(struct ftio *ftio) { extension_info_t *extension_info; int i; if (ftio_check_xfield(ftio, FT_XFIELD_DPKTS | FT_XFIELD_DOCTETS | FT_XFIELD_FIRST | FT_XFIELD_LAST | FT_XFIELD_SRCADDR | FT_XFIELD_DSTADDR | FT_XFIELD_SRCPORT | FT_XFIELD_DSTPORT | FT_XFIELD_UNIX_SECS | FT_XFIELD_UNIX_NSECS | FT_XFIELD_SYSUPTIME | FT_XFIELD_TOS | FT_XFIELD_TCP_FLAGS | FT_XFIELD_PROT)) { fprintf(stderr,"Flow-tools record missing required fields."); return NULL; } InitExtensionMaps(NO_EXTENSION_LIST); extension_info = (extension_info_t *)malloc(sizeof(extension_info_t)); if ( !extension_info ) { fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return NULL; } memset((void *)extension_info, 0, sizeof(extension_info_t)); extension_info->map = (extension_map_t *)malloc(sizeof(extension_map_t) + Max_num_extensions * sizeof(uint16_t)); if ( !extension_info->map ) { fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return NULL; } i = 0; extension_info->map->type = ExtensionMapType; extension_info->map->map_id = 0; if ( !ftio_check_xfield(ftio, FT_XFIELD_INPUT | FT_XFIELD_OUTPUT )) { extension_info->map->ex_id[i++] = EX_IO_SNMP_2; } if (!ftio_check_xfield(ftio, FT_XFIELD_SRC_AS | FT_XFIELD_DST_AS)) { extension_info->map->ex_id[i++] = EX_AS_2; } if (!ftio_check_xfield(ftio, FT_XFIELD_SRC_MASK | FT_XFIELD_DST_MASK)) { extension_info->map->ex_id[i++] = EX_MULIPLE; } if (!ftio_check_xfield(ftio, FT_XFIELD_NEXTHOP )) { extension_info->map->ex_id[i++] = EX_NEXT_HOP_v4; } if (!ftio_check_xfield(ftio, FT_XFIELD_EXADDR )) { extension_info->map->ex_id[i++] = EX_ROUTER_IP_v4; } if (!ftio_check_xfield(ftio, FT_XFIELD_ENGINE_TYPE )) { extension_info->map->ex_id[i++] = EX_ROUTER_ID; } extension_info->map->ex_id[i] = 0; extension_info->map->size = sizeof(extension_map_t) + i * sizeof(uint16_t); // align 32bits if (( extension_info->map->size & 0x3 ) != 0 ) { extension_info->map->size += 4 - ( extension_info->map->size & 0x3 ); } extension_info->map->extension_size = 0; i=0; while (extension_info->map->ex_id[i]) { int id = extension_info->map->ex_id[i]; extension_info->map->extension_size += extension_descriptor[id].size; i++; } return extension_info; } // End of GenExtensionMap int flows2nfdump(struct ftio *ftio, char *wfile, int compress, extension_info_t *extension_info, int extended, uint32_t limitflows) { // required flow tools variables struct fttime ftt; struct fts3rec_offsets fo; struct ftver ftv; char *rec; // nfdump variables nffile_t *nffile; master_record_t record; char *s; uint32_t cnt; s = "flow-tools"; nffile = OpenNewFile( wfile , NULL, compress, 0, s); if ( !nffile ) { fprintf(stderr, "%s\n", s); return 1; } AppendToBuffer(nffile, (void *)extension_info->map, extension_info->map->size); ftio_get_ver(ftio, &ftv); fts3rec_compute_offsets(&fo, &ftv); memset((void *)&record, 0, sizeof(record)); record.map_ref = extension_info->map; record.type = CommonRecordType; record.exporter_sysid = 0; // only v4 addresses ClearFlag(record.flags, FLAG_IPV6_ADDR); cnt = 0; while ((rec = ftio_read(ftio))) { uint32_t when, unix_secs, unix_nsecs, sysUpTime; int i, id; unix_secs = *((uint32_t*)(rec+fo.unix_secs)); unix_nsecs = *((uint32_t*)(rec+fo.unix_nsecs)); sysUpTime = *((uint32_t*)(rec+fo.sysUpTime)); when = *((uint32_t*)(rec+fo.First)); ftt = ftltime(sysUpTime, unix_secs, unix_nsecs, when); record.first = ftt.secs; record.msec_first = ftt.msecs; when = *((uint32_t*)(rec+fo.Last)); ftt = ftltime(sysUpTime, unix_secs, unix_nsecs, when); record.last = ftt.secs; record.msec_last = ftt.msecs; record.V4.srcaddr = *((uint32_t*)(rec+fo.srcaddr)); record.V4.dstaddr = *((uint32_t*)(rec+fo.dstaddr)); record.srcport = *((uint16_t*)(rec+fo.srcport)); record.dstport = *((uint16_t*)(rec+fo.dstport)); record.prot = *((uint8_t*)(rec+fo.prot)); record.tcp_flags = *((uint8_t*)(rec+fo.tcp_flags)); record.tos = *((uint8_t*)(rec+fo.tos)); record.dOctets = *((uint32_t*)(rec+fo.dOctets)); record.dPkts = *((uint32_t*)(rec+fo.dPkts)); i = 0; while ( (id = extension_info->map->ex_id[i]) != 0 ) { switch (id) { case EX_IO_SNMP_2: record.input = *((uint16_t*)(rec+fo.input)); record.output = *((uint16_t*)(rec+fo.output)); break; case EX_AS_2: record.srcas = *((uint16_t*)(rec+fo.src_as)); record.dstas = *((uint16_t*)(rec+fo.dst_as)); break; case EX_MULIPLE: record.src_mask = *((uint8_t*)(rec+fo.src_mask)); record.dst_mask = *((uint8_t*)(rec+fo.dst_mask)); record.dir = 0; record.dst_tos = 0; break; case EX_NEXT_HOP_v4: record.ip_nexthop.V4 = *((uint32_t*)(rec+fo.nexthop)); break; case EX_ROUTER_IP_v4: record.ip_router.V4 = *((uint32_t*)(rec+fo.exaddr)); break; case EX_ROUTER_ID: record.engine_type = *((uint8_t*)(rec+fo.engine_type)); record.engine_id = *((uint8_t*)(rec+fo.engine_id)); break; // default: Other extensions can not be sent with v5 } i++; } PackRecord(&record, nffile); if ( extended ) { char *string; flow_record_to_raw(&record, &string, 0); fprintf(stderr, "%s\n", string); } cnt++; if ( cnt == limitflows ) break; } /* while */ // write the last records in buffer if ( nffile->block_header->NumRecords ) { if ( WriteBlock(nffile) <= 0 ) { fprintf(stderr, "Failed to write output buffer: '%s'" , strerror(errno)); } } free((void *)extension_info->map); free((void *)extension_info); DisposeFile(nffile); return 0; } // End of flows2nfdump int main(int argc, char **argv) { struct ftio ftio; extension_info_t *extension_info; struct stat statbuf; uint32_t limitflows; int i, extended, printmap, ret, fd, compress;; char *ftfile, *wfile; /* init fterr */ fterr_setid(argv[0]); extended = 0; printmap = 0; limitflows = 0; ftfile = NULL; wfile = "-"; compress = NOT_COMPRESSED; while ((i = getopt(argc, argv, "jzEVc:hmr:w:?")) != -1) switch (i) { case 'h': /* help */ case '?': usage(argv[0]); exit (0); break; case 'V': printf("%s: Version: %s\n",argv[0], nfdump_version); exit(0); break; case 'E': extended = 1; break; case 'c': limitflows = atoi(optarg); if ( !limitflows ) { fprintf(stderr, "Option -c needs a number > 0\n"); exit(255); } break; case 'm': printmap = 1; break; case 'j': compress = LZO_COMPRESSED; break; case 'z': compress = BZ2_COMPRESSED; break; case 'r': ftfile = optarg; if ( (stat(ftfile, &statbuf) < 0 ) || !(statbuf.st_mode & S_IFREG) ) { fprintf(stderr, "No such file: '%s'\n", ftfile); exit(255); } break; case 'w': wfile = optarg; break; default: usage(argv[0]); exit (1); break; } /* switch */ // End while if (argc - optind) fterr_errx(1, "Extra arguments starting with %s.", argv[optind]); if ( ftfile ) { fd = open(ftfile, O_RDONLY, 0); if ( fd < 0 ) { fprintf(stderr, "Can't open file '%s': %s.", ftfile, strerror(errno)); exit(255); } } else { fd = 0; } /* read from fd */ if (ftio_init(&ftio, fd, FT_IO_FLAG_READ) < 0) fterr_errx(1, "ftio_init(): failed"); extension_info = GenExtensionMap(&ftio); if ( !extension_info ) exit(255); if ( printmap ) { PrintExtensionMap(extension_info->map); exit(255); } ret = flows2nfdump(&ftio, wfile, compress, extension_info, extended, limitflows); return ret; } // End of main nfdump-1.6.23/bin/fts_compat.c000077500000000000000000000712251404501030700161530ustar00rootroot00000000000000/* $Id: fts_compat.c 16 2009-06-19 09:26:19Z haag $ */ /* TNFTPD ORIGINAL: libnetbsd/fts_open.c */ /* $TNFTPPD: fts_open.c,v 1.4 2003/12/17 01:42:45 lukem Exp $ */ /* from NetBSD: __fts13.c,v 1.36 2001/11/28 22:31:39 christos Exp */ /*- * Copyright (c) 1990, 1993, 1994 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include "config.h" #include #include #include #include #include #include #include #include #ifdef HAVE_ISO_LIMITS_ISO_H #include #endif #if HAVE_DIRENT_H # include # define NAMLEN(dirent) strlen((dirent)->d_name) #else # define dirent direct # define NAMLEN(dirent) (dirent)->d_namlen # if HAVE_SYS_NDIR_H # include # endif # if HAVE_SYS_DIR_H # include # endif # if HAVE_NDIR_H # include # endif #endif #include "fts_compat.h" #if ! defined(MIN) # define MIN(a, b) ((a) < (b) ? (a) : (b)) #endif #if ! defined(MAX) # define MAX(a, b) ((a) < (b) ? (b) : (a)) #endif #define STAT stat static FTSENT *fts_alloc(FTS *, const char *, size_t); static FTSENT *fts_build(FTS *, int); static void fts_lfree(FTSENT *); static void fts_load(FTS *, FTSENT *); static size_t fts_maxarglen(char * const *); static size_t fts_pow2(size_t); static int fts_palloc(FTS *, size_t); static void fts_padjust(FTS *, FTSENT *); static FTSENT *fts_sort(FTS *, FTSENT *, size_t); static u_short fts_stat(FTS *, FTSENT *, int); static int fts_safe_changedir(const FTS *, const FTSENT *, int, const char *); #define ISDOT(a) (a[0] == '.' && (!a[1] || (a[1] == '.' && !a[2]))) #define CLR(opt) (sp->fts_options &= ~(opt)) #define ISSET(opt) (sp->fts_options & (opt)) #define SET(opt) (sp->fts_options |= (opt)) #define CHDIR(sp, path) (!ISSET(FTS_NOCHDIR) && chdir(path)) #define FCHDIR(sp, fd) (!ISSET(FTS_NOCHDIR) && fchdir(fd)) /* fts_build flags */ #define BCHILD 1 /* fts_children */ #define BNAMES 2 /* fts_children, names only */ #define BREAD 3 /* fts_read */ #ifndef DTF_HIDEW #undef FTS_WHITEOUT #endif FTS * fts_open_compat(char * const *argv, int options, int (*compar)(const FTSENT **, const FTSENT **)) { FTS *sp; FTSENT *p, *root; size_t nitems; FTSENT *parent, *tmp = NULL; /* pacify gcc */ size_t len; /* Options check. */ if (options & ~FTS_OPTIONMASK) { errno = EINVAL; return (NULL); } /* Allocate/initialize the stream */ if ((sp = malloc((u_int)sizeof(FTS))) == NULL) return (NULL); memset(sp, 0, sizeof(FTS)); sp->fts_compar = compar; sp->fts_options = options; /* Logical walks turn on NOCHDIR; symbolic links are too hard. */ if (ISSET(FTS_LOGICAL)) SET(FTS_NOCHDIR); /* * Start out with 1K of path space, and enough, in any case, * to hold the user's paths. */ if (fts_palloc(sp, MAX(fts_maxarglen(argv), MAXPATHLEN))) goto mem1; /* Allocate/initialize root's parent. */ if ((parent = fts_alloc(sp, "", 0)) == NULL) goto mem2; parent->fts_level = FTS_ROOTPARENTLEVEL; /* Allocate/initialize root(s). */ for (root = NULL, nitems = 0; *argv; ++argv, ++nitems) { /* Don't allow zero-length paths. */ if ((len = strlen(*argv)) == 0) { errno = ENOENT; goto mem3; } if ((p = fts_alloc(sp, *argv, len)) == NULL) goto mem3; p->fts_level = FTS_ROOTLEVEL; p->fts_parent = parent; p->fts_accpath = p->fts_name; p->fts_info = fts_stat(sp, p, ISSET(FTS_COMFOLLOW)); /* Command-line "." and ".." are real directories. */ if (p->fts_info == FTS_DOT) p->fts_info = FTS_D; /* * If comparison routine supplied, traverse in sorted * order; otherwise traverse in the order specified. */ if (compar) { p->fts_link = root; root = p; } else { p->fts_link = NULL; if (root == NULL) tmp = root = p; else { tmp->fts_link = p; tmp = p; } } } if (compar && nitems > 1) root = fts_sort(sp, root, nitems); /* * Allocate a dummy pointer and make fts_read think that we've just * finished the node before the root(s); set p->fts_info to FTS_INIT * so that everything about the "current" node is ignored. */ if ((sp->fts_cur = fts_alloc(sp, "", 0)) == NULL) goto mem3; sp->fts_cur->fts_link = root; sp->fts_cur->fts_info = FTS_INIT; /* * If using chdir(2), grab a file descriptor pointing to dot to insure * that we can get back here; this could be avoided for some paths, * but almost certainly not worth the effort. Slashes, symbolic links, * and ".." are all fairly nasty problems. Note, if we can't get the * descriptor we run anyway, just more slowly. */ if (!ISSET(FTS_NOCHDIR) && (sp->fts_rfd = open(".", O_RDONLY, 0)) < 0) SET(FTS_NOCHDIR); return (sp); mem3: fts_lfree(root); free(parent); mem2: free(sp->fts_path); mem1: free(sp); return (NULL); } static void fts_load(FTS *sp, FTSENT *p) { size_t len; char *cp; /* * Load the stream structure for the next traversal. Since we don't * actually enter the directory until after the preorder visit, set * the fts_accpath field specially so the chdir gets done to the right * place and the user can access the first node. From fts_open it's * known that the path will fit. */ len = p->fts_pathlen = p->fts_namelen; memmove(sp->fts_path, p->fts_name, len + 1); if ((cp = strrchr(p->fts_name, '/')) && (cp != p->fts_name || cp[1])) { len = strlen(++cp); memmove(p->fts_name, cp, len + 1); p->fts_namelen = len; } p->fts_accpath = p->fts_path = sp->fts_path; sp->fts_dev = p->fts_dev; } int fts_close_compat(FTS *sp) { FTSENT *freep, *p; int saved_errno = 0; /* * This still works if we haven't read anything -- the dummy structure * points to the root list, so we step through to the end of the root * list which has a valid parent pointer. */ if (sp->fts_cur) { for (p = sp->fts_cur; p->fts_level >= FTS_ROOTLEVEL;) { freep = p; p = p->fts_link ? p->fts_link : p->fts_parent; free(freep); } free(p); } /* Free up child linked list, sort array, path buffer. */ if (sp->fts_child) fts_lfree(sp->fts_child); if (sp->fts_array) free(sp->fts_array); free(sp->fts_path); /* Return to original directory, save errno if necessary. */ if (!ISSET(FTS_NOCHDIR)) { if (fchdir(sp->fts_rfd)) saved_errno = errno; (void)close(sp->fts_rfd); } /* Free up the stream pointer. */ free(sp); /* ISSET() is illegal after this, since the macro touches sp */ /* Set errno and return. */ if (saved_errno) { errno = saved_errno; return (-1); } return (0); } /* * Special case a root of "/" so that slashes aren't appended which would * cause paths to be written as "//foo". */ #define NAPPEND(p) \ (p->fts_level == FTS_ROOTLEVEL && p->fts_pathlen == 1 && \ p->fts_path[0] == '/' ? 0 : p->fts_pathlen) FTSENT * fts_read_compat(FTS *sp) { FTSENT *p, *tmp; int instr; char *t; int saved_errno; /* If finished or unrecoverable error, return NULL. */ if (sp->fts_cur == NULL || ISSET(FTS_STOP)) return (NULL); /* Set current node pointer. */ p = sp->fts_cur; /* Save and zero out user instructions. */ instr = p->fts_instr; p->fts_instr = FTS_NOINSTR; /* Any type of file may be re-visited; re-stat and re-turn. */ if (instr == FTS_AGAIN) { p->fts_info = fts_stat(sp, p, 0); return (p); } /* * Following a symlink -- SLNONE test allows application to see * SLNONE and recover. If indirecting through a symlink, have * keep a pointer to current location. If unable to get that * pointer, follow fails. */ if (instr == FTS_FOLLOW && (p->fts_info == FTS_SL || p->fts_info == FTS_SLNONE)) { p->fts_info = fts_stat(sp, p, 1); if (p->fts_info == FTS_D && !ISSET(FTS_NOCHDIR)) { if ((p->fts_symfd = open(".", O_RDONLY, 0)) < 0) { p->fts_errno = errno; p->fts_info = FTS_ERR; } else p->fts_flags |= FTS_SYMFOLLOW; } return (p); } /* Directory in pre-order. */ if (p->fts_info == FTS_D) { /* If skipped or crossed mount point, do post-order visit. */ if (instr == FTS_SKIP || (ISSET(FTS_XDEV) && p->fts_dev != sp->fts_dev)) { if (p->fts_flags & FTS_SYMFOLLOW) (void)close(p->fts_symfd); if (sp->fts_child) { fts_lfree(sp->fts_child); sp->fts_child = NULL; } p->fts_info = FTS_DP; return (p); } /* Rebuild if only read the names and now traversing. */ if (sp->fts_child && ISSET(FTS_NAMEONLY)) { CLR(FTS_NAMEONLY); fts_lfree(sp->fts_child); sp->fts_child = NULL; } /* * Cd to the subdirectory. * * If have already read and now fail to chdir, whack the list * to make the names come out right, and set the parent errno * so the application will eventually get an error condition. * Set the FTS_DONTCHDIR flag so that when we logically change * directories back to the parent we don't do a chdir. * * If haven't read do so. If the read fails, fts_build sets * FTS_STOP or the fts_info field of the node. */ if (sp->fts_child) { if (fts_safe_changedir(sp, p, -1, p->fts_accpath)) { p->fts_errno = errno; p->fts_flags |= FTS_DONTCHDIR; for (p = sp->fts_child; p; p = p->fts_link) p->fts_accpath = p->fts_parent->fts_accpath; } } else if ((sp->fts_child = fts_build(sp, BREAD)) == NULL) { if (ISSET(FTS_STOP)) return (NULL); return (p); } p = sp->fts_child; sp->fts_child = NULL; goto name; } /* Move to the next node on this level. */ next: tmp = p; if ((p = p->fts_link) != NULL) { free(tmp); /* * If reached the top, return to the original directory, and * load the paths for the next root. */ if (p->fts_level == FTS_ROOTLEVEL) { if (FCHDIR(sp, sp->fts_rfd)) { SET(FTS_STOP); return (NULL); } fts_load(sp, p); return (sp->fts_cur = p); } /* * User may have called fts_set on the node. If skipped, * ignore. If followed, get a file descriptor so we can * get back if necessary. */ if (p->fts_instr == FTS_SKIP) goto next; if (p->fts_instr == FTS_FOLLOW) { p->fts_info = fts_stat(sp, p, 1); if (p->fts_info == FTS_D && !ISSET(FTS_NOCHDIR)) { if ((p->fts_symfd = open(".", O_RDONLY, 0)) < 0) { p->fts_errno = errno; p->fts_info = FTS_ERR; } else p->fts_flags |= FTS_SYMFOLLOW; } p->fts_instr = FTS_NOINSTR; } name: t = sp->fts_path + NAPPEND(p->fts_parent); *t++ = '/'; memmove(t, p->fts_name, (size_t)(p->fts_namelen + 1)); return (sp->fts_cur = p); } /* Move up to the parent node. */ p = tmp->fts_parent; free(tmp); if (p->fts_level == FTS_ROOTPARENTLEVEL) { /* * Done; free everything up and set errno to 0 so the user * can distinguish between error and EOF. */ free(p); errno = 0; return (sp->fts_cur = NULL); } /* Nul terminate the pathname. */ sp->fts_path[p->fts_pathlen] = '\0'; /* * Return to the parent directory. If at a root node or came through * a symlink, go back through the file descriptor. Otherwise, cd up * one directory. */ if (p->fts_level == FTS_ROOTLEVEL) { if (FCHDIR(sp, sp->fts_rfd)) { SET(FTS_STOP); return (NULL); } } else if (p->fts_flags & FTS_SYMFOLLOW) { if (FCHDIR(sp, p->fts_symfd)) { saved_errno = errno; (void)close(p->fts_symfd); errno = saved_errno; SET(FTS_STOP); return (NULL); } (void)close(p->fts_symfd); } else if (!(p->fts_flags & FTS_DONTCHDIR) && fts_safe_changedir(sp, p->fts_parent, -1, "..")) { SET(FTS_STOP); return (NULL); } p->fts_info = p->fts_errno ? FTS_ERR : FTS_DP; return (sp->fts_cur = p); } /* * Fts_set takes the stream as an argument although it's not used in this * implementation; it would be necessary if anyone wanted to add global * semantics to fts using fts_set. An error return is allowed for similar * reasons. */ /* ARGSUSED */ int fts_set_compat(FTS *sp, FTSENT *p, int instr) { if (instr && instr != FTS_AGAIN && instr != FTS_FOLLOW && instr != FTS_NOINSTR && instr != FTS_SKIP) { errno = EINVAL; return (1); } p->fts_instr = instr; return (0); } FTSENT * fts_children_compat(FTS *sp, int instr) { FTSENT *p; int fd; if (instr && instr != FTS_NAMEONLY) { errno = EINVAL; return (NULL); } /* Set current node pointer. */ p = sp->fts_cur; /* * Errno set to 0 so user can distinguish empty directory from * an error. */ errno = 0; /* Fatal errors stop here. */ if (ISSET(FTS_STOP)) return (NULL); /* Return logical hierarchy of user's arguments. */ if (p->fts_info == FTS_INIT) return (p->fts_link); /* * If not a directory being visited in pre-order, stop here. Could * allow FTS_DNR, assuming the user has fixed the problem, but the * same effect is available with FTS_AGAIN. */ if (p->fts_info != FTS_D /* && p->fts_info != FTS_DNR */) return (NULL); /* Free up any previous child list. */ if (sp->fts_child) fts_lfree(sp->fts_child); if (instr == FTS_NAMEONLY) { SET(FTS_NAMEONLY); instr = BNAMES; } else instr = BCHILD; /* * If using chdir on a relative path and called BEFORE fts_read does * its chdir to the root of a traversal, we can lose -- we need to * chdir into the subdirectory, and we don't know where the current * directory is, so we can't get back so that the upcoming chdir by * fts_read will work. */ if (p->fts_level != FTS_ROOTLEVEL || p->fts_accpath[0] == '/' || ISSET(FTS_NOCHDIR)) return (sp->fts_child = fts_build(sp, instr)); if ((fd = open(".", O_RDONLY, 0)) < 0) return (sp->fts_child = NULL); sp->fts_child = fts_build(sp, instr); if (fchdir(fd)) { (void)close(fd); return (NULL); } (void)close(fd); return (sp->fts_child); } /* * This is the tricky part -- do not casually change *anything* in here. The * idea is to build the linked list of entries that are used by fts_children * and fts_read. There are lots of special cases. * * The real slowdown in walking the tree is the stat calls. If FTS_NOSTAT is * set and it's a physical walk (so that symbolic links can't be directories), * we can do things quickly. First, if it's a 4.4BSD file system, the type * of the file is in the directory entry. Otherwise, we assume that the number * of subdirectories in a node is equal to the number of links to the parent. * The former skips all stat calls. The latter skips stat calls in any leaf * directories and for any files after the subdirectories in the directory have * been found, cutting the stat calls by about 2/3. */ static FTSENT * fts_build(FTS *sp, int type) { struct dirent *dp; FTSENT *p, *head; size_t nitems; FTSENT *cur, *tail; DIR *dirp; int adjust, cderrno, descend, len, level, nlinks, saved_errno, nostat; size_t maxlen; #ifdef FTS_WHITEOUT int oflag; #endif char *cp = NULL; /* pacify gcc */ /* Set current node pointer. */ cur = sp->fts_cur; /* * Open the directory for reading. If this fails, we're done. * If being called from fts_read, set the fts_info field. */ #ifdef FTS_WHITEOUT if (ISSET(FTS_WHITEOUT)) oflag = DTF_NODUP|DTF_REWIND; else oflag = DTF_HIDEW|DTF_NODUP|DTF_REWIND; #else #define __opendir2(path, flag) opendir(path) #endif if ((dirp = __opendir2(cur->fts_accpath, oflag)) == NULL) { if (type == BREAD) { cur->fts_info = FTS_DNR; cur->fts_errno = errno; } return (NULL); } /* * Nlinks is the number of possible entries of type directory in the * directory if we're cheating on stat calls, 0 if we're not doing * any stat calls at all, -1 if we're doing stats on everything. */ if (type == BNAMES) { nlinks = 0; nostat = 1; } else if (ISSET(FTS_NOSTAT) && ISSET(FTS_PHYSICAL)) { nlinks = cur->fts_nlink - (ISSET(FTS_SEEDOT) ? 0 : 2); nostat = 1; } else { nlinks = -1; nostat = 0; } #ifdef notdef (void)printf("nlinks == %d (cur: %d)\n", nlinks, cur->fts_nlink); (void)printf("NOSTAT %d PHYSICAL %d SEEDOT %d\n", ISSET(FTS_NOSTAT), ISSET(FTS_PHYSICAL), ISSET(FTS_SEEDOT)); #endif /* * If we're going to need to stat anything or we want to descend * and stay in the directory, chdir. If this fails we keep going, * but set a flag so we don't chdir after the post-order visit. * We won't be able to stat anything, but we can still return the * names themselves. Note, that since fts_read won't be able to * chdir into the directory, it will have to return different path * names than before, i.e. "a/b" instead of "b". Since the node * has already been visited in pre-order, have to wait until the * post-order visit to return the error. There is a special case * here, if there was nothing to stat then it's not an error to * not be able to stat. This is all fairly nasty. If a program * needed sorted entries or stat information, they had better be * checking FTS_NS on the returned nodes. */ cderrno = 0; if (nlinks || type == BREAD) { if (fts_safe_changedir(sp, cur, -1, cur->fts_accpath)) { if (nlinks && type == BREAD) cur->fts_errno = errno; cur->fts_flags |= FTS_DONTCHDIR; descend = 0; cderrno = errno; } else descend = 1; } else descend = 0; /* * Figure out the max file name length that can be stored in the * current path -- the inner loop allocates more path as necessary. * We really wouldn't have to do the maxlen calculations here, we * could do them in fts_read before returning the path, but it's a * lot easier here since the length is part of the dirent structure. * * If not changing directories set a pointer so that can just append * each new name into the path. */ len = NAPPEND(cur); if (ISSET(FTS_NOCHDIR)) { cp = sp->fts_path + len; *cp++ = '/'; } len++; maxlen = sp->fts_pathlen - len; level = cur->fts_level + 1; /* Read the directory, attaching each entry to the `link' pointer. */ adjust = 0; for (head = tail = NULL, nitems = 0; (dp = readdir(dirp)) != NULL;) { size_t dlen; if (!ISSET(FTS_SEEDOT) && ISDOT(dp->d_name)) continue; dlen = NAMLEN(dp); /* #if !defined(DIRENT_MISSING_D_NAMLEN) dlen = dp->d_namlen; #else dlen = strlen(dp->d_name); #endif */ if ((p = fts_alloc(sp, dp->d_name, dlen)) == NULL) goto mem1; if (dlen >= maxlen) { /* include space for NUL */ if (fts_palloc(sp, len + dlen + 1)) { /* * No more memory for path or structures. Save * errno, free up the current structure and the * structures already allocated. */ mem1: saved_errno = errno; if (p) free(p); fts_lfree(head); (void)closedir(dirp); errno = saved_errno; cur->fts_info = FTS_ERR; SET(FTS_STOP); return (NULL); } adjust = 1; if (ISSET(FTS_NOCHDIR)) cp = sp->fts_path + len; maxlen = sp->fts_pathlen - len; } p->fts_pathlen = len + dlen; p->fts_parent = sp->fts_cur; p->fts_level = level; #ifdef FTS_WHITEOUT if (dp->d_type == DT_WHT) p->fts_flags |= FTS_ISW; #endif if (cderrno) { if (nlinks) { p->fts_info = FTS_NS; p->fts_errno = cderrno; } else p->fts_info = FTS_NSOK; p->fts_accpath = cur->fts_accpath; } else if (nlinks == 0 #ifdef DT_DIR || (nostat && dp->d_type != DT_DIR && dp->d_type != DT_UNKNOWN) #endif ) { p->fts_accpath = ISSET(FTS_NOCHDIR) ? p->fts_path : p->fts_name; p->fts_info = FTS_NSOK; } else { /* Build a file name for fts_stat to stat. */ if (ISSET(FTS_NOCHDIR)) { p->fts_accpath = p->fts_path; memmove(cp, p->fts_name, (size_t)(p->fts_namelen + 1)); } else p->fts_accpath = p->fts_name; /* Stat it. */ p->fts_info = fts_stat(sp, p, 0); /* Decrement link count if applicable. */ if (nlinks > 0 && (p->fts_info == FTS_D || p->fts_info == FTS_DC || p->fts_info == FTS_DOT)) --nlinks; } /* We walk in directory order so "ls -f" doesn't get upset. */ p->fts_link = NULL; if (head == NULL) head = tail = p; else { tail->fts_link = p; tail = p; } ++nitems; } (void)closedir(dirp); /* * If had to realloc the path, adjust the addresses for the rest * of the tree. */ if (adjust) fts_padjust(sp, head); /* * If not changing directories, reset the path back to original * state. */ if (ISSET(FTS_NOCHDIR)) { if (cp - 1 > sp->fts_path) --cp; *cp = '\0'; } /* * If descended after called from fts_children or after called from * fts_read and nothing found, get back. At the root level we use * the saved fd; if one of fts_open()'s arguments is a relative path * to an empty directory, we wind up here with no other way back. If * can't get back, we're done. */ if (descend && (type == BCHILD || !nitems) && (cur->fts_level == FTS_ROOTLEVEL ? FCHDIR(sp, sp->fts_rfd) : fts_safe_changedir(sp, cur->fts_parent, -1, ".."))) { cur->fts_info = FTS_ERR; SET(FTS_STOP); return (NULL); } /* If didn't find anything, return NULL. */ if (!nitems) { if (type == BREAD) cur->fts_info = FTS_DP; return (NULL); } /* Sort the entries. */ if (sp->fts_compar && nitems > 1) head = fts_sort(sp, head, nitems); return (head); } static u_short fts_stat(FTS *sp, FTSENT *p, int follow) { FTSENT *t; dev_t dev; ino_t ino; struct STAT *sbp, sb; int saved_errno; /* If user needs stat info, stat buffer already allocated. */ sbp = ISSET(FTS_NOSTAT) ? &sb : p->fts_statp; #ifdef FTS_WHITEOUT /* check for whiteout */ if (p->fts_flags & FTS_ISW) { if (sbp != &sb) { memset(sbp, '\0', sizeof (*sbp)); sbp->st_mode = S_IFWHT; } return (FTS_W); } #endif /* * If doing a logical walk, or application requested FTS_FOLLOW, do * a stat(2). If that fails, check for a non-existent symlink. If * fail, set the errno from the stat call. */ if (ISSET(FTS_LOGICAL) || follow) { if (stat(p->fts_accpath, sbp)) { saved_errno = errno; if (!lstat(p->fts_accpath, sbp)) { errno = 0; return (FTS_SLNONE); } p->fts_errno = saved_errno; goto err; } } else if (lstat(p->fts_accpath, sbp)) { p->fts_errno = errno; err: memset(sbp, 0, sizeof(struct STAT)); return (FTS_NS); } if (S_ISDIR(sbp->st_mode)) { /* * Set the device/inode. Used to find cycles and check for * crossing mount points. Also remember the link count, used * in fts_build to limit the number of stat calls. It is * understood that these fields are only referenced if fts_info * is set to FTS_D. */ dev = p->fts_dev = sbp->st_dev; ino = p->fts_ino = sbp->st_ino; p->fts_nlink = sbp->st_nlink; if (ISDOT(p->fts_name)) return (FTS_DOT); /* * Cycle detection is done by brute force when the directory * is first encountered. If the tree gets deep enough or the * number of symbolic links to directories is high enough, * something faster might be worthwhile. */ for (t = p->fts_parent; t->fts_level >= FTS_ROOTLEVEL; t = t->fts_parent) if (ino == t->fts_ino && dev == t->fts_dev) { p->fts_cycle = t; return (FTS_DC); } return (FTS_D); } if (S_ISLNK(sbp->st_mode)) return (FTS_SL); if (S_ISREG(sbp->st_mode)) return (FTS_F); return (FTS_DEFAULT); } static FTSENT * fts_sort(FTS *sp, FTSENT *head, size_t nitems) { FTSENT **ap, *p; /* * Construct an array of pointers to the structures and call qsort(3). * Reassemble the array in the order returned by qsort. If unable to * sort for memory reasons, return the directory entries in their * current order. Allocate enough space for the current needs plus * 40 so don't realloc one entry at a time. */ if (nitems > sp->fts_nitems) { FTSENT **new; new = realloc(sp->fts_array, sizeof(FTSENT *) * (nitems + 40)); if (new == 0) return (head); sp->fts_array = new; sp->fts_nitems = nitems + 40; } for (ap = sp->fts_array, p = head; p; p = p->fts_link) *ap++ = p; qsort((void *)sp->fts_array, nitems, sizeof(FTSENT *), (int (*)(const void *, const void *))sp->fts_compar); for (head = *(ap = sp->fts_array); --nitems; ++ap) ap[0]->fts_link = ap[1]; ap[0]->fts_link = NULL; return (head); } static FTSENT * fts_alloc(FTS *sp, const char *name, size_t namelen) { FTSENT *p; #ifndef ALIGNBYTES #define ALIGNBYTES (sizeof(int) - 1) #endif #ifndef ALIGN #define ALIGN(p) (((u_long)(p) + ALIGNBYTES) &~ ALIGNBYTES) #endif size_t len; /* * The file name is a variable length array and no stat structure is * necessary if the user has set the nostat bit. Allocate the FTSENT * structure, the file name and the stat structure in one chunk, but * be careful that the stat structure is reasonably aligned. Since the * fts_name field is declared to be of size 1, the fts_name pointer is * namelen + 2 before the first possible address of the stat structure. */ len = sizeof(FTSENT) + namelen; if (!ISSET(FTS_NOSTAT)) len += sizeof(struct STAT) + ALIGNBYTES; if ((p = malloc(len)) == NULL) return (NULL); if (!ISSET(FTS_NOSTAT)) p->fts_statp = (struct STAT *)ALIGN((u_long)(p->fts_name + namelen + 2)); /* Copy the name plus the trailing NULL. */ memmove(p->fts_name, name, namelen + 1); p->fts_namelen = namelen; p->fts_path = sp->fts_path; p->fts_errno = 0; p->fts_flags = 0; p->fts_instr = FTS_NOINSTR; p->fts_number = 0; p->fts_pointer = NULL; return (p); } static void fts_lfree(FTSENT *head) { FTSENT *p; /* XXX: head may be NULL ? */ /* Free a linked list of structures. */ while ((p = head) != NULL) { head = head->fts_link; free(p); } } static size_t fts_pow2(size_t x) { x--; x |= x>>1; x |= x>>2; x |= x>>4; x |= x>>8; x |= x>>16; #if LONG_BIT > 32 x |= x>>32; #endif #if LONG_BIT > 64 x |= x>>64; #endif x++; return (x); } /* * Allow essentially unlimited paths; find, rm, ls should all work on any tree. * Most systems will allow creation of paths much longer than MAXPATHLEN, even * though the kernel won't resolve them. Round up the new size to a power of 2, * so we don't realloc the path 2 bytes at a time. */ static int fts_palloc(FTS *sp, size_t size) { char *new; #if 1 /* Protect against fts_pathlen overflow. */ if (size > USHRT_MAX + 1) { errno = ENOMEM; return (1); } #endif size = fts_pow2(size); new = realloc(sp->fts_path, size); if (new == 0) return (1); sp->fts_path = new; sp->fts_pathlen = size; return (0); } /* * When the path is realloc'd, have to fix all of the pointers in structures * already returned. */ static void fts_padjust(FTS *sp, FTSENT *head) { FTSENT *p; char *addr; #define ADJUST(p) do { \ if ((p)->fts_accpath != (p)->fts_name) \ (p)->fts_accpath = \ addr + ((p)->fts_accpath - (p)->fts_path); \ (p)->fts_path = addr; \ } while (/*CONSTCOND*/0) addr = sp->fts_path; /* Adjust the current set of children. */ for (p = sp->fts_child; p; p = p->fts_link) ADJUST(p); /* Adjust the rest of the tree, including the current level. */ for (p = head; p->fts_level >= FTS_ROOTLEVEL;) { ADJUST(p); p = p->fts_link ? p->fts_link : p->fts_parent; } } static size_t fts_maxarglen(argv) char * const *argv; { size_t len, max; for (max = 0; *argv; ++argv) if ((len = strlen(*argv)) > max) max = len; return (max + 1); } /* * Change to dir specified by fd or p->fts_accpath without getting * tricked by someone changing the world out from underneath us. * Assumes p->fts_dev and p->fts_ino are filled in. */ static int fts_safe_changedir(sp, p, fd, path) const FTS *sp; const FTSENT *p; int fd; const char *path; { int oldfd = fd, ret = -1; struct STAT sb; if (ISSET(FTS_NOCHDIR)) return 0; if (fd < 0 && (fd = open(path, O_RDONLY)) == -1) return -1; if (fstat(fd, &sb) == -1) goto bail; if (sb.st_ino != p->fts_ino || sb.st_dev != p->fts_dev) { errno = ENOENT; goto bail; } ret = fchdir(fd); bail: if (oldfd < 0) { int save_errno = errno; (void)close(fd); errno = save_errno; } return ret; } nfdump-1.6.23/bin/fts_compat.h000077500000000000000000000122201404501030700161460ustar00rootroot00000000000000/* $Id: fts_compat.h 16 2009-06-19 09:26:19Z haag $ */ /* TNFTPD ORIGINAL: libnetbsd/ftpfts.h */ /* $NetBSD: ftpfts.h,v 1.3 2003/12/15 23:52:02 lukem Exp $ */ /* * Copyright (c) 1989, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)fts.h 8.3 (Berkeley) 8/14/94 */ #ifndef _FTS_H_ #define _FTS_H_ 1 typedef struct { struct _ftsent *fts_cur; /* current node */ struct _ftsent *fts_child; /* linked list of children */ struct _ftsent **fts_array; /* sort array */ dev_t fts_dev; /* starting device # */ char *fts_path; /* path for this descent */ int fts_rfd; /* fd for root */ u_int fts_pathlen; /* sizeof(path) */ u_int fts_nitems; /* elements in the sort array */ int (*fts_compar) /* compare function */ (const struct _ftsent **, const struct _ftsent **); #define FTS_COMFOLLOW 0x001 /* follow command line symlinks */ #define FTS_LOGICAL 0x002 /* logical walk */ #define FTS_NOCHDIR 0x004 /* don't change directories */ #define FTS_NOSTAT 0x008 /* don't get stat info */ #define FTS_PHYSICAL 0x010 /* physical walk */ #define FTS_SEEDOT 0x020 /* return dot and dot-dot */ #define FTS_XDEV 0x040 /* don't cross devices */ #define FTS_WHITEOUT 0x080 /* return whiteout information */ #define FTS_OPTIONMASK 0x0ff /* valid user option mask */ #define FTS_NAMEONLY 0x100 /* (private) child names only */ #define FTS_STOP 0x200 /* (private) unrecoverable error */ int fts_options; /* fts_open options, global flags */ } FTS; typedef struct _ftsent { struct _ftsent *fts_cycle; /* cycle node */ struct _ftsent *fts_parent; /* parent directory */ struct _ftsent *fts_link; /* next file in directory */ long fts_number; /* local numeric value */ void *fts_pointer; /* local address value */ char *fts_accpath; /* access path */ char *fts_path; /* root path */ int fts_errno; /* errno for this node */ int fts_symfd; /* fd for symlink */ u_short fts_pathlen; /* strlen(fts_path) */ u_short fts_namelen; /* strlen(fts_name) */ ino_t fts_ino; /* inode */ dev_t fts_dev; /* device */ nlink_t fts_nlink; /* link count */ #define FTS_ROOTPARENTLEVEL -1 #define FTS_ROOTLEVEL 0 short fts_level; /* depth (-1 to N) */ #define FTS_D 1 /* preorder directory */ #define FTS_DC 2 /* directory that causes cycles */ #define FTS_DEFAULT 3 /* none of the above */ #define FTS_DNR 4 /* unreadable directory */ #define FTS_DOT 5 /* dot or dot-dot */ #define FTS_DP 6 /* postorder directory */ #define FTS_ERR 7 /* error; errno is set */ #define FTS_F 8 /* regular file */ #define FTS_INIT 9 /* initialized only */ #define FTS_NS 10 /* stat(2) failed */ #define FTS_NSOK 11 /* no stat(2) requested */ #define FTS_SL 12 /* symbolic link */ #define FTS_SLNONE 13 /* symbolic link without target */ #define FTS_W 14 /* whiteout object */ u_short fts_info; /* user flags for FTSENT structure */ #define FTS_DONTCHDIR 0x01 /* don't chdir .. to the parent */ #define FTS_SYMFOLLOW 0x02 /* followed a symlink to get here */ #define FTS_ISW 0x04 /* this is a whiteout object */ u_short fts_flags; /* private flags for FTSENT structure */ #define FTS_AGAIN 1 /* read node again */ #define FTS_FOLLOW 2 /* follow symbolic link */ #define FTS_NOINSTR 3 /* no instructions */ #define FTS_SKIP 4 /* discard node */ u_short fts_instr; /* fts_set() instructions */ struct stat *fts_statp; /* stat(2) information */ char fts_name[1]; /* file name */ } FTSENT; FTSENT *fts_children_compat(FTS *, int); int fts_close_compat(FTS *); FTS *fts_open_compat(char * const *, int, int (*)(const FTSENT **, const FTSENT **)); FTSENT *fts_read_compat(FTS *); int fts_set_compat(FTS *, FTSENT *, int); #endif /* !_FTS_H_ */ nfdump-1.6.23/bin/grammar.y000077500000000000000000001724761404501030700155020ustar00rootroot00000000000000/* * Copyright (c) 2016-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ %{ #ifdef HAVE_CONFIG_H #include "config.h" #endif #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "output_util.h" #include "rbtree.h" #include "filter.h" #include "nfdump.h" #include "nffile.h" #include "nftree.h" #include "ipconv.h" #define AnyMask 0xffffffffffffffffLL /* * function prototypes */ static void yyerror(char *msg); static uint32_t ChainHosts(uint64_t *offsets, uint64_t *hostlist, int num_records, int type); static uint64_t VerifyMac(char *s); static int InitSymbols(void); static uint32_t Get_fwd_status_id(char *status); enum { DIR_UNSPEC = 1, SOURCE, DESTINATION, SOURCE_AND_DESTINATION, SOURCE_OR_DESTINATION, DIR_IN, DIR_OUT, IN_SRC, IN_DST, OUT_SRC, OUT_DST, ADJ_PREV, ADJ_NEXT }; enum { IS_START = 0, IS_END }; /* var defs */ extern int lineno; extern char *yytext; extern uint64_t *IPstack; extern uint32_t StartNode; extern uint16_t Extended; extern int (*FilterEngine)(uint32_t *); extern char *FilterFilename; static uint32_t num_ip; static struct fwd_status_def_s { uint32_t id; char *name; } fwd_status_def_list[] = { { 0, "Ukwn"}, // Unknown { 1, "Forw"}, // Normal forwarding { 2, "Frag"}, // Fragmented { 16, "Drop"}, // Drop { 17, "DaclD"}, // Drop ACL deny { 18, "Daclp"}, // Drop ACL drop { 19, "Noroute"}, // Unroutable { 20, "Dadj"}, // Drop Adjacency { 21, "Dfrag"}, // Drop Fragmentation & DF set { 22, "Dbadh"}, // Drop Bad header checksum { 23, "Dbadtlen"}, // Drop Bad total Length { 24, "Dbadhlen"}, // Drop Bad Header Length { 25, "DbadTTL"}, // Drop bad TTL { 26, "Dpolicy"}, // Drop Policer { 27, "Dwred"}, // Drop WRED { 28, "Drpf"}, // Drop RPF { 29, "Dforus"}, // Drop For us { 30, "DbadOf"}, // Drop Bad output interface { 31, "Dhw"}, // Drop Hardware { 128, "Term"}, // Terminate { 129, "Tadj"}, // Terminate Punt Adjacency { 130, "TincAdj"}, // Terminate Incomplete Adjacency { 131, "Tforus"}, // Terminate For us { 0, NULL} // Last entry }; static char **fwd_status = NULL; char yyerror_buff[256]; #define MPLSMAX 0x00ffffff %} %union { uint64_t value; char *s; FilterParam_t param; void *list; } %token ANY IP IF MAC MPLS TOS DIR FLAGS PROTO MASK NET PORT FWDSTAT IN OUT SRC DST EQ LT GT LE GE PREV NEXT %token IDENT ENGINE_TYPE ENGINE_ID AS PACKETS BYTES FLOWS NFVERSION %token PPS BPS BPP DURATION NOT %token IPV4 IPV6 BGPNEXTHOP ROUTER VLAN %token CLIENT SERVER APP LATENCY SYSID %token ASA DENIED XEVENT XNET XPORT INGRESS EGRESS ACL ACE XACE %token NAT ADD EVENT VRF NPORT NIP %token PBLOCK START END STEP SIZE %token STRING REASON %token NUMBER PORTNUM ICMP_TYPE ICMP_CODE %type expr %type dqual term comp acl inout %type iplist ullist %left '+' OR %left '*' AND %left NEGATE %% prog: /* empty */ | expr { StartNode = $1; } ; term: ANY { /* this is an unconditionally true expression, as a filter applies in any case */ $$.self = NewBlock(OffsetProto, 0, 0, CMP_EQ, FUNC_NONE, NULL ); } | IDENT STRING { if ( !ScreenIdentString($2) ) { yyerror("Illegal ident string"); YYABORT; } uint32_t index = AddIdent($2); $$.self = NewBlock(0, 0, index, CMP_IDENT, FUNC_NONE, NULL ); } | IPV4 { $$.self = NewBlock(OffsetRecordFlags, (1LL << ShiftRecordFlags) & MaskRecordFlags, (0LL << ShiftRecordFlags) & MaskRecordFlags, CMP_EQ, FUNC_NONE, NULL); } | IPV6 { $$.self = NewBlock(OffsetRecordFlags, (1LL << ShiftRecordFlags) & MaskRecordFlags, (1LL << ShiftRecordFlags) & MaskRecordFlags, CMP_EQ, FUNC_NONE, NULL); } | PROTO NUMBER { int64_t proto; proto = $2; if ( proto > 255 ) { yyerror("Protocol number > 255"); YYABORT; } if ( proto < 0 ) { yyerror("Unknown protocol"); YYABORT; } $$.self = NewBlock(OffsetProto, MaskProto, (proto << ShiftProto) & MaskProto, CMP_EQ, FUNC_NONE, NULL); } | PROTO STRING { int64_t proto; proto = ProtoNum($2); if ( proto > 255 ) { yyerror("Protocol number > 255"); YYABORT; } if ( proto < 0 ) { yyerror("Unknown protocol"); YYABORT; } $$.self = NewBlock(OffsetProto, MaskProto, (proto << ShiftProto) & MaskProto, CMP_EQ, FUNC_NONE, NULL); } | dqual PACKETS comp NUMBER { switch ( $1.direction ) { case DIR_UNSPEC: case DIR_IN: $$.self = NewBlock(OffsetPackets, MaskPackets, $4, $3.comp, FUNC_NONE, NULL); break; case DIR_OUT: $$.self = NewBlock(OffsetOutPackets, MaskPackets, $4, $3.comp, FUNC_NONE, NULL); break; default: /* should never happen */ yyerror("This token is not expected here!"); YYABORT; } // End of switch } | dqual BYTES comp NUMBER { switch ( $1.direction ) { case DIR_UNSPEC: case DIR_IN: $$.self = NewBlock(OffsetBytes, MaskBytes, $4, $3.comp, FUNC_NONE, NULL); break; case DIR_OUT: $$.self = NewBlock(OffsetOutBytes, MaskBytes, $4, $3.comp, FUNC_NONE, NULL); break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } | FLOWS comp NUMBER { $$.self = NewBlock(OffsetAggrFlows, MaskFlows, $3, $2.comp, FUNC_NONE, NULL); } | PPS comp NUMBER { $$.self = NewBlock(0, AnyMask, $3, $2.comp, FUNC_PPS, NULL); } | BPS comp NUMBER { $$.self = NewBlock(0, AnyMask, $3, $2.comp, FUNC_BPS, NULL); } | BPP comp NUMBER { $$.self = NewBlock(0, AnyMask, $3, $2.comp, FUNC_BPP, NULL); } | DURATION comp NUMBER { $$.self = NewBlock(0, AnyMask, $3, $2.comp, FUNC_DURATION, NULL); } | dqual TOS comp NUMBER { if ( $4 > 255 ) { yyerror("TOS must be 0..255"); YYABORT; } switch ( $1.direction ) { case DIR_UNSPEC: case SOURCE: $$.self = NewBlock(OffsetTos, MaskTos, ($4 << ShiftTos) & MaskTos, $3.comp, FUNC_NONE, NULL); break; case DESTINATION: $$.self = NewBlock(OffsetDstTos, MaskDstTos, ($4 << ShiftDstTos) & MaskDstTos, $3.comp, FUNC_NONE, NULL); break; case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetTos, MaskTos, ($4 << ShiftTos) & MaskTos, $3.comp, FUNC_NONE, NULL), NewBlock(OffsetDstTos, MaskDstTos, ($4 << ShiftDstTos) & MaskDstTos, $3.comp, FUNC_NONE, NULL) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetTos, MaskTos, ($4 << ShiftTos) & MaskTos, $3.comp, FUNC_NONE, NULL), NewBlock(OffsetDstTos, MaskDstTos, ($4 << ShiftDstTos) & MaskDstTos, $3.comp, FUNC_NONE, NULL) ); break; default: yyerror("This token is not expected here!"); YYABORT; } } | FLAGS comp NUMBER { if ( $3 > 63 ) { yyerror("Flags must be 0..63"); YYABORT; } $$.self = NewBlock(OffsetFlags, MaskFlags, ($3 << ShiftFlags) & MaskFlags, $2.comp, FUNC_NONE, NULL); } | NFVERSION comp NUMBER { if ( $3 > 10 ) { yyerror("Netflow version must be <= 10"); YYABORT; } $$.self = NewBlock(OffsetRecordVersion, MaskRecordVersion, ($3 << ShiftRecordVersion) & MaskRecordVersion, $2.comp, FUNC_NONE, NULL); } // handle special case with 'AS' takes as flags. and not AS number | FLAGS AS { uint64_t fl = 0; fl |= 16; fl |= 2; $$.self = NewBlock(OffsetFlags, (fl << ShiftFlags) & MaskFlags, (fl << ShiftFlags) & MaskFlags, CMP_FLAGS, FUNC_NONE, NULL); } | FLAGS STRING { uint64_t fl = 0; int cnt = 0; size_t len = strlen($2); if ( len > 7 ) { yyerror("Too many flags"); YYABORT; } if ( strchr($2, 'F') ) { fl |= 1; cnt++; } if ( strchr($2, 'S') ) { fl |= 2; cnt++; } if ( strchr($2, 'R') ) { fl |= 4; cnt++; } if ( strchr($2, 'P') ) { fl |= 8; cnt++; } if ( strchr($2, 'A') ) { fl |= 16; cnt++; } if ( strchr($2, 'U') ) { fl |= 32; cnt++; } if ( strchr($2, 'X') ) { fl = 63; cnt++; } if ( cnt != len ) { yyerror("Too many flags"); YYABORT; } $$.self = NewBlock(OffsetFlags, (fl << ShiftFlags) & MaskFlags, (fl << ShiftFlags) & MaskFlags, CMP_FLAGS, FUNC_NONE, NULL); } | dqual IP STRING { int af, bytes, ret; ret = parse_ip(&af, $3, IPstack, &bytes, ALLOW_LOOKUP, &num_ip); if ( ret == 0 ) { yyerror("Error parsing IP address."); YYABORT; } // ret == -1 will never happen here, as ALLOW_LOOKUP is set if ( ret == -2 ) { // could not resolv host => 'not any' $$.self = Invert(NewBlock(OffsetProto, 0, 0, CMP_EQ, FUNC_NONE, NULL )); } else { uint64_t offsets[4] = {OffsetSrcIPv6a, OffsetSrcIPv6b, OffsetDstIPv6a, OffsetDstIPv6b }; if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } switch ( $1.direction ) { case SOURCE: case DESTINATION: $$.self = ChainHosts(offsets, IPstack, num_ip, $1.direction); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: { uint32_t src = ChainHosts(offsets, IPstack, num_ip, SOURCE); uint32_t dst = ChainHosts(offsets, IPstack, num_ip, DESTINATION); $$.self = Connect_OR(src, dst); } break; case SOURCE_AND_DESTINATION: { uint32_t src = ChainHosts(offsets, IPstack, num_ip, SOURCE); uint32_t dst = ChainHosts(offsets, IPstack, num_ip, DESTINATION); $$.self = Connect_AND(src, dst); } break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } } | dqual IP IN '[' iplist ']' { switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetSrcIPv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ); break; case DESTINATION: $$.self = NewBlock(OffsetDstIPv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetSrcIPv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ), NewBlock(OffsetDstIPv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetSrcIPv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ), NewBlock(OffsetDstIPv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } } | NEXT IP STRING { int af, bytes, ret; ret = parse_ip(&af, $3, IPstack, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Error parsing IP address."); YYABORT; } if ( ret == -1 ) { yyerror("IP address required - hostname not allowed here."); YYABORT; } // ret == -2 will never happen here, as STRICT_IP is set if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } $$.self = Connect_AND( NewBlock(OffsetNexthopv6b, MaskIPv6, IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetNexthopv6a, MaskIPv6, IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); } | NEXT IP IN '[' iplist ']' { $$.self = NewBlock(OffsetNexthopv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ); } | BGPNEXTHOP IP STRING { int af, bytes, ret; ret = parse_ip(&af, $3, IPstack, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Error parsing IP address."); YYABORT; } if ( ret == -1 ) { yyerror("IP address required - hostname not allowed here."); YYABORT; } // ret == -2 will never happen here, as STRICT_IP is set if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } $$.self = Connect_AND( NewBlock(OffsetBGPNexthopv6b, MaskIPv6, IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetBGPNexthopv6a, MaskIPv6, IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); } | ROUTER IP STRING { int af, bytes, ret; ret = parse_ip(&af, $3, IPstack, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Error parsing IP address."); YYABORT; } if ( ret == -1 ) { yyerror("IP address required - hostname not allowed here."); YYABORT; } // ret == -2 will never happen here, as STRICT_IP is set if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } $$.self = Connect_AND( NewBlock(OffsetRouterv6b, MaskIPv6, IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetRouterv6a, MaskIPv6, IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); } | CLIENT LATENCY comp NUMBER { $$.self = NewBlock(OffsetClientLatency, MaskLatency, $4, $3.comp, FUNC_NONE, NULL); } | SERVER LATENCY comp NUMBER { $$.self = NewBlock(OffsetServerLatency, MaskLatency, $4, $3.comp, FUNC_NONE, NULL); } | APP LATENCY comp NUMBER { $$.self = NewBlock(OffsetAppLatency, MaskLatency, $4, $3.comp, FUNC_NONE, NULL); } | SYSID NUMBER { if ( $2 > 255 ) { yyerror("Router SysID expected between be 1..255"); YYABORT; } $$.self = NewBlock(OffsetExporterSysID, MaskExporterSysID, ($2 << ShiftExporterSysID) & MaskExporterSysID, CMP_EQ, FUNC_NONE, NULL); } | dqual PORT comp NUMBER { if ( $4 > 65535 ) { yyerror("Port outside of range 0..65535"); YYABORT; } switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetPort, MaskSrcPort, ($4 << ShiftSrcPort) & MaskSrcPort, $3.comp, FUNC_NONE, NULL ); break; case DESTINATION: $$.self = NewBlock(OffsetPort, MaskDstPort, ($4 << ShiftDstPort) & MaskDstPort, $3.comp, FUNC_NONE, NULL ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetPort, MaskSrcPort, ($4 << ShiftSrcPort) & MaskSrcPort, $3.comp, FUNC_NONE, NULL ), NewBlock(OffsetPort, MaskDstPort, ($4 << ShiftDstPort) & MaskDstPort, $3.comp, FUNC_NONE, NULL ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetPort, MaskSrcPort, ($4 << ShiftSrcPort) & MaskSrcPort, $3.comp, FUNC_NONE, NULL ), NewBlock(OffsetPort, MaskDstPort, ($4 << ShiftDstPort) & MaskDstPort, $3.comp, FUNC_NONE, NULL ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End switch } | dqual PORT IN PBLOCK { #ifdef NSEL switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetPort, MaskSrcPort, ShiftSrcPort, CMP_EQ, FUNC_PBLOCK, NULL ); break; case DESTINATION: $$.self = NewBlock(OffsetPort, MaskDstPort, ShiftDstPort, CMP_EQ, FUNC_PBLOCK, NULL ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetPort, MaskSrcPort, ShiftSrcPort, CMP_EQ, FUNC_PBLOCK, NULL ), NewBlock(OffsetPort, MaskDstPort, ShiftDstPort, CMP_EQ, FUNC_PBLOCK, NULL ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End switch #else yyerror("NAT filters not available"); YYABORT; #endif } | dqual PORT IN '[' ullist ']' { struct ULongListNode *node; ULongtree_t *root = NULL; if ( $1.direction == DIR_UNSPEC || $1.direction == SOURCE_OR_DESTINATION || $1.direction == SOURCE_AND_DESTINATION ) { // src and/or dst port // we need a second rbtree due to different shifts for src and dst ports root = malloc(sizeof(ULongtree_t)); struct ULongListNode *n; if ( root == NULL) { yyerror("malloc() error"); YYABORT; } RB_INIT(root); RB_FOREACH(node, ULongtree, (ULongtree_t *)$5) { if ( node->value > 65535 ) { yyerror("Port outside of range 0..65535"); YYABORT; } if ((n = malloc(sizeof(struct ULongListNode))) == NULL) { yyerror("malloc() error"); YYABORT; } n->value = (node->value << ShiftDstPort) & MaskDstPort; node->value = (node->value << ShiftSrcPort) & MaskSrcPort; RB_INSERT(ULongtree, root, n); } } switch ( $1.direction ) { case SOURCE: RB_FOREACH(node, ULongtree, (ULongtree_t *)$5) { node->value = (node->value << ShiftSrcPort) & MaskSrcPort; } $$.self = NewBlock(OffsetPort, MaskSrcPort, 0, CMP_ULLIST, FUNC_NONE, (void *)$5 ); break; case DESTINATION: RB_FOREACH(node, ULongtree, (ULongtree_t *)$5) { node->value = (node->value << ShiftDstPort) & MaskDstPort; } $$.self = NewBlock(OffsetPort, MaskDstPort, 0, CMP_ULLIST, FUNC_NONE, (void *)$5 ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetPort, MaskSrcPort, 0, CMP_ULLIST, FUNC_NONE, (void *)$5 ), NewBlock(OffsetPort, MaskDstPort, 0, CMP_ULLIST, FUNC_NONE, (void *)root ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetPort, MaskSrcPort, 0, CMP_ULLIST, FUNC_NONE, (void *)$5 ), NewBlock(OffsetPort, MaskDstPort, 0, CMP_ULLIST, FUNC_NONE, (void *)root ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } | ICMP_TYPE NUMBER { if ( $2 > 255 ) { yyerror("ICMP tpye of range 0..15"); YYABORT; } $$.self = Connect_AND( // imply proto ICMP with a proto ICMP block Connect_OR ( NewBlock(OffsetProto, MaskProto, ((uint64_t)IPPROTO_ICMP << ShiftProto) & MaskProto, CMP_EQ, FUNC_NONE, NULL), NewBlock(OffsetProto, MaskProto, ((uint64_t)IPPROTO_ICMPV6 << ShiftProto) & MaskProto, CMP_EQ, FUNC_NONE, NULL) ), NewBlock(OffsetICMP, MaskICMPtype, ($2 << ShiftICMPtype) & MaskICMPtype, CMP_EQ, FUNC_NONE, NULL ) ); } | ICMP_CODE NUMBER { if ( $2 > 255 ) { yyerror("ICMP code of range 0..15"); YYABORT; } $$.self = Connect_AND( // imply proto ICMP with a proto ICMP block Connect_OR ( NewBlock(OffsetProto, MaskProto, ((uint64_t)IPPROTO_ICMP << ShiftProto) & MaskProto, CMP_EQ, FUNC_NONE, NULL), NewBlock(OffsetProto, MaskProto, ((uint64_t)IPPROTO_ICMPV6 << ShiftProto) & MaskProto, CMP_EQ, FUNC_NONE, NULL) ), NewBlock(OffsetICMP, MaskICMPcode, ($2 << ShiftICMPcode) & MaskICMPcode, CMP_EQ, FUNC_NONE, NULL ) ); } | ENGINE_TYPE comp NUMBER { if ( $3 > 255 ) { yyerror("Engine type of range 0..255"); YYABORT; } $$.self = NewBlock(OffsetRouterID, MaskEngineType, ($3 << ShiftEngineType) & MaskEngineType, $2.comp, FUNC_NONE, NULL); } | ENGINE_ID comp NUMBER { if ( $3 > 255 ) { yyerror("Engine ID of range 0..255"); YYABORT; } $$.self = NewBlock(OffsetRouterID, MaskEngineID, ($3 << ShiftEngineID) & MaskEngineID, $2.comp, FUNC_NONE, NULL); } | ASA EVENT REASON { #ifdef NSEL if ( strncasecmp($3,"ignore", 6) == 0) { $$.self = NewBlock(OffsetConnID, MaskFWevent, ( NSEL_EVENT_IGNORE << ShiftFWevent) & MaskFWevent, CMP_EQ, FUNC_NONE, NULL ); } else if( strncasecmp($3,"create", 6) == 0) { $$.self = NewBlock(OffsetConnID, MaskFWevent, ( NSEL_EVENT_CREATE << ShiftFWevent) & MaskFWevent, CMP_EQ, FUNC_NONE, NULL ); } else if( strncasecmp($3,"term", 4) == 0 || strncasecmp($3,"delete", 6) == 0) { $$.self = NewBlock(OffsetConnID, MaskFWevent, ( NSEL_EVENT_DELETE << ShiftFWevent) & MaskFWevent, CMP_EQ, FUNC_NONE, NULL ); } else if (strncasecmp($3,"deny", 4) == 0) { $$.self = NewBlock(OffsetConnID, MaskFWevent, ( NSEL_EVENT_DENIED << ShiftFWevent) & MaskFWevent, CMP_EQ, FUNC_NONE, NULL ); } else if (strncasecmp($3,"alert", 5) == 0) { $$.self = NewBlock(OffsetConnID, MaskFWevent, ( NSEL_EVENT_ALERT << ShiftFWevent) & MaskFWevent, CMP_EQ, FUNC_NONE, NULL ); } else if (strncasecmp($3,"update", 6) == 0) { $$.self = NewBlock(OffsetConnID, MaskFWevent, ( NSEL_EVENT_UPDATE << ShiftFWevent) & MaskFWevent, CMP_EQ, FUNC_NONE, NULL ); } else { yyerror("Unknown asa event"); YYABORT; } #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | ASA EVENT comp NUMBER { #ifdef NSEL if ( $4 > 255 ) { yyerror("Invalid xevent ID"); YYABORT; } $$.self = NewBlock(OffsetConnID, MaskFWevent, ( $4 << ShiftFWevent) & MaskFWevent, $3.comp, FUNC_NONE, NULL ); #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | ASA EVENT DENIED inout { #ifdef NSEL uint64_t xevent = 0; if ( $4.inout == INGRESS ) { xevent = 1001; } else if ( $4.inout == EGRESS ) { xevent = 1002; } else { yyerror("Invalid inout token"); YYABORT; } $$.self = Connect_AND( NewBlock(OffsetConnID, MaskFWevent, ( NSEL_EVENT_DENIED << ShiftFWevent) & MaskFWevent, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetConnID, MaskFWXevent, ( xevent << ShiftFWXevent) & MaskFWXevent, CMP_EQ, FUNC_NONE, NULL ) ); #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | ASA EVENT DENIED STRING { #ifdef NSEL uint64_t xevent = 0; if( strncasecmp($4,"interface", 9) == 0) { xevent = 1003; } else if( strncasecmp($4,"nosyn", 5) == 0) { xevent = 1004; } else { xevent = (uint64_t)strtol($4, (char **)NULL, 10); if ( (xevent == 0 && errno == EINVAL) || xevent > 65535 ) { yyerror("Invalid xevent ID"); YYABORT; } } $$.self = Connect_AND( NewBlock(OffsetConnID, MaskFWevent, ( NSEL_EVENT_DENIED << ShiftFWevent) & MaskFWevent, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetConnID, MaskFWXevent, ( xevent << ShiftFWXevent) & MaskFWXevent, CMP_EQ, FUNC_NONE, NULL ) ); #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | ASA XEVENT comp NUMBER { #ifdef NSEL if ( $4 > 65535 ) { yyerror("Invalid xevent ID"); YYABORT; } $$.self = NewBlock(OffsetConnID, MaskFWXevent, ( $4 << ShiftFWXevent) & MaskFWXevent, $3.comp, FUNC_NONE, NULL ); #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | dqual XNET STRING '/' NUMBER { #ifdef NSEL int af, bytes, ret; uint64_t mask[2]; ret = parse_ip(&af, $3, IPstack, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } if ( ret == -1 ) { yyerror("IP address required - hostname not allowed here."); YYABORT; } // ret == -2 will never happen here, as STRICT_IP is set if ( $5 > (bytes*8) ) { yyerror("Too many netbits for this IP address"); YYABORT; } if ( af == PF_INET ) { mask[0] = 0xffffffffffffffffLL; mask[1] = 0xffffffffffffffffLL << ( 32 - $5 ); } else { // PF_INET6 if ( $5 > 64 ) { mask[0] = 0xffffffffffffffffLL; mask[1] = 0xffffffffffffffffLL << ( 128 - $5 ); } else { mask[0] = 0xffffffffffffffffLL << ( 64 - $5 ); mask[1] = 0; } } // IP aadresses are stored in network representation mask[0] = mask[0]; mask[1] = mask[1]; IPstack[0] &= mask[0]; IPstack[1] &= mask[1]; switch ( $1.direction ) { case SOURCE: $$.self = Connect_AND( NewBlock(OffsetXLATESRCv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetXLATESRCv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); break; case DESTINATION: $$.self = Connect_AND( NewBlock(OffsetXLATEDSTv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetXLATEDSTv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( Connect_AND( NewBlock(OffsetXLATESRCv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetXLATESRCv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ), Connect_AND( NewBlock(OffsetXLATEDSTv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetXLATEDSTv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( Connect_AND( NewBlock(OffsetXLATESRCv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetXLATESRCv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ), Connect_AND( NewBlock(OffsetXLATEDSTv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetXLATEDSTv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | dqual XPORT comp NUMBER { #ifdef NSEL if ( $4 > 65535 ) { yyerror("Port outside of range 0..65535"); YYABORT; } switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetXLATEPort, MaskXLATESRCPORT, ($4 << ShiftXLATESRCPORT) & MaskXLATESRCPORT, $3.comp, FUNC_NONE, NULL ); break; case DESTINATION: $$.self = NewBlock(OffsetXLATEPort, MaskXLATEDSTPORT, ($4 << ShiftXLATEDSTPORT) & MaskXLATEDSTPORT, $3.comp, FUNC_NONE, NULL ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetXLATEPort, MaskXLATESRCPORT, ($4 << ShiftXLATESRCPORT) & MaskXLATESRCPORT, $3.comp, FUNC_NONE, NULL ), NewBlock(OffsetXLATEPort, MaskXLATEDSTPORT, ($4 << ShiftXLATEDSTPORT) & MaskXLATEDSTPORT, $3.comp, FUNC_NONE, NULL ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetXLATEPort, MaskXLATESRCPORT, ($4 << ShiftXLATESRCPORT) & MaskXLATESRCPORT, $3.comp, FUNC_NONE, NULL ), NewBlock(OffsetXLATEPort, MaskXLATEDSTPORT, ($4 << ShiftXLATEDSTPORT) & MaskXLATEDSTPORT, $3.comp, FUNC_NONE, NULL ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End switch #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | inout acl comp NUMBER { #ifdef NSEL uint64_t offset, mask, shift; if ( $1.inout == INGRESS ) { switch ($2.acl) { case ACL: offset = OffsetIngressAclId; mask = MaskIngressAclId; shift = ShiftIngressAclId; break; case ACE: offset = OffsetIngressAceId; mask = MaskIngressAceId; shift = ShiftIngressAceId; break; case XACE: offset = OffsetIngressGrpId; mask = MaskIngressGrpId; shift = ShiftIngressGrpId; break; default: yyerror("Invalid ACL specifier"); YYABORT; } } else if ( $1.inout == EGRESS && $$.acl == ACL ) { offset = OffsetEgressAclId; mask = MaskEgressAclId; shift = ShiftEgressAclId; } else { yyerror("ingress/egress syntax error"); YYABORT; } $$.self = NewBlock(offset, mask, ($4 << shift) & mask , $3.comp, FUNC_NONE, NULL ); #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | NAT EVENT REASON { #ifdef NSEL if ( strncasecmp($3,"invalid", 7) == 0) { $$.self = NewBlock(OffsetNATevent, MasNATevent, ( NEL_EVENT_INVALID << ShiftNATevent) & MasNATevent, CMP_EQ, FUNC_NONE, NULL ); } else if( strncasecmp($3,"add", 3) == 0 || strncasecmp($3,"create", 6) == 0) { $$.self = NewBlock(OffsetNATevent, MasNATevent, ( NEL_EVENT_ADD << ShiftNATevent) & MasNATevent, CMP_EQ, FUNC_NONE, NULL ); } else if( strncasecmp($3,"delete", 6) == 0) { $$.self = NewBlock(OffsetNATevent, MasNATevent, ( NEL_EVENT_DELETE << ShiftNATevent) & MasNATevent, CMP_EQ, FUNC_NONE, NULL ); } else { yyerror("Unknown nat event"); YYABORT; } #else yyerror("NAT filters not available"); YYABORT; #endif } | NAT EVENT comp NUMBER { #ifdef NSEL if ( $4 > 255 ) { yyerror("Invalid event ID"); YYABORT; } $$.self = NewBlock(OffsetNATevent, MasNATevent, ( $4 << ShiftNATevent) & MasNATevent, $3.comp, FUNC_NONE, NULL ); #else yyerror("NAT filters not available"); YYABORT; #endif } | INGRESS VRF comp NUMBER { #ifdef NSEL if ( $4 > 0xFFFFFFFFLL ) { yyerror("Invalid ingress vrf ID"); YYABORT; } $$.self = NewBlock(OffsetIVRFID, MaskIVRFID, ( $4 << ShiftIVRFID) & MaskIVRFID, $3.comp, FUNC_NONE, NULL ); #else yyerror("NAT filters not available"); YYABORT; #endif } | EGRESS VRF comp NUMBER { #ifdef NSEL if ( $4 > 0xFFFFFFFFLL ) { yyerror("Invalid egress vrf ID"); YYABORT; } $$.self = NewBlock(OffsetEVRFID, MaskEVRFID, ( $4 << ShiftEVRFID) & MaskEVRFID, $3.comp, FUNC_NONE, NULL ); #else yyerror("NAT filters not available"); YYABORT; #endif } | PBLOCK START comp NUMBER { #ifdef NSEL if ( $4 > 65536 ) { yyerror("Invalid port"); YYABORT; } $$.self = NewBlock(OffsetPortBlock, MaskPortBlockStart, ( $4 << ShiftPortBlockStart) & MaskPortBlockStart, $3.comp, FUNC_NONE, NULL ); #else yyerror("NAT filters not available"); YYABORT; #endif } | PBLOCK END comp NUMBER { #ifdef NSEL if ( $4 > 65536 ) { yyerror("Invalid port"); YYABORT; } $$.self = NewBlock(OffsetPortBlock, MaskPortBlockEnd, ( $4 << ShiftPortBlockEnd) & MaskPortBlockEnd, $3.comp, FUNC_NONE, NULL ); #else yyerror("NAT filters not available"); YYABORT; #endif } | PBLOCK STEP comp NUMBER { #ifdef NSEL if ( $4 > 65536 ) { yyerror("Invalid port"); YYABORT; } $$.self = NewBlock(OffsetPortBlock, MaskPortBlockStep, ( $4 << ShiftPortBlockStep) & MaskPortBlockStep, $3.comp, FUNC_NONE, NULL ); #else yyerror("NAT filters not available"); YYABORT; #endif } | PBLOCK SIZE comp NUMBER { #ifdef NSEL if ( $4 > 65536 ) { yyerror("Invalid port"); YYABORT; } $$.self = NewBlock(OffsetPortBlock, MaskPortBlockSize, ( $4 << ShiftPortBlockSize) & MaskPortBlockSize, $3.comp, FUNC_NONE, NULL ); #else yyerror("NAT filters not available"); YYABORT; #endif } | dqual NPORT comp NUMBER { #ifdef NSEL if ( $4 > 65535 ) { yyerror("Port outside of range 0..65535"); YYABORT; } switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetXLATEPort, MaskXLATESRCPORT, ($4 << ShiftXLATESRCPORT) & MaskXLATESRCPORT, $3.comp, FUNC_NONE, NULL ); break; case DESTINATION: $$.self = NewBlock(OffsetXLATEPort, MaskXLATEDSTPORT, ($4 << ShiftXLATEDSTPORT) & MaskXLATEDSTPORT, $3.comp, FUNC_NONE, NULL ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetXLATEPort, MaskXLATESRCPORT, ($4 << ShiftXLATESRCPORT) & MaskXLATESRCPORT, $3.comp, FUNC_NONE, NULL ), NewBlock(OffsetXLATEPort, MaskXLATEDSTPORT, ($4 << ShiftXLATEDSTPORT) & MaskXLATEDSTPORT, $3.comp, FUNC_NONE, NULL ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetXLATEPort, MaskXLATESRCPORT, ($4 << ShiftXLATESRCPORT) & MaskXLATESRCPORT, $3.comp, FUNC_NONE, NULL ), NewBlock(OffsetXLATEPort, MaskXLATEDSTPORT, ($4 << ShiftXLATEDSTPORT) & MaskXLATEDSTPORT, $3.comp, FUNC_NONE, NULL ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End switch #else yyerror("NEL/NAT filters not available"); YYABORT; #endif } | dqual NIP STRING { #ifdef NSEL int af, bytes, ret; ret = parse_ip(&af, $3, IPstack, &bytes, ALLOW_LOOKUP, &num_ip); if ( ret == 0 ) { yyerror("Error parsing IP address."); YYABORT; } // ret == -1 will never happen here, as ALLOW_LOOKUP is set if ( ret == -2 ) { // could not resolv host => 'not any' $$.self = Invert(NewBlock(OffsetProto, 0, 0, CMP_EQ, FUNC_NONE, NULL )); } else { uint64_t offsets[4] = {OffsetXLATESRCv6a, OffsetXLATESRCv6b, OffsetXLATEDSTv6a, OffsetXLATEDSTv6b }; if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } switch ( $1.direction ) { case SOURCE: case DESTINATION: $$.self = ChainHosts(offsets, IPstack, num_ip, $1.direction); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: { uint32_t src = ChainHosts(offsets, IPstack, num_ip, SOURCE); uint32_t dst = ChainHosts(offsets, IPstack, num_ip, DESTINATION); $$.self = Connect_OR(src, dst); } break; case SOURCE_AND_DESTINATION: { uint32_t src = ChainHosts(offsets, IPstack, num_ip, SOURCE); uint32_t dst = ChainHosts(offsets, IPstack, num_ip, DESTINATION); $$.self = Connect_AND(src, dst); } break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | dqual NIP IN '[' iplist ']' { #ifdef NSEL switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetXLATESRCv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ); break; case DESTINATION: $$.self = NewBlock(OffsetXLATEDSTv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetXLATESRCv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ), NewBlock(OffsetXLATEDSTv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetXLATESRCv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ), NewBlock(OffsetXLATEDSTv6a, MaskIPv6, 0 , CMP_IPLIST, FUNC_NONE, (void *)$5 ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } #else yyerror("NSEL/ASA filters not available"); YYABORT; #endif } | dqual AS comp NUMBER { if ( $4 > 0xfFFFFFFF ) { yyerror("AS number of range"); YYABORT; } switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetAS, MaskSrcAS, ($4 << ShiftSrcAS) & MaskSrcAS, $3.comp, FUNC_NONE, NULL ); break; case DESTINATION: $$.self = NewBlock(OffsetAS, MaskDstAS, ($4 << ShiftDstAS) & MaskDstAS, $3.comp, FUNC_NONE, NULL); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetAS, MaskSrcAS, ($4 << ShiftSrcAS) & MaskSrcAS, $3.comp, FUNC_NONE, NULL ), NewBlock(OffsetAS, MaskDstAS, ($4 << ShiftDstAS) & MaskDstAS, $3.comp, FUNC_NONE, NULL) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetAS, MaskSrcAS, ($4 << ShiftSrcAS) & MaskSrcAS, $3.comp, FUNC_NONE, NULL ), NewBlock(OffsetAS, MaskDstAS, ($4 << ShiftDstAS) & MaskDstAS, $3.comp, FUNC_NONE, NULL) ); break; case ADJ_PREV: $$.self = NewBlock(OffsetBGPadj, MaskBGPadjPrev, ($4 << ShiftBGPadjPrev) & MaskBGPadjPrev, $3.comp, FUNC_NONE, NULL ); break; case ADJ_NEXT: $$.self = NewBlock(OffsetBGPadj, MaskBGPadjNext, ($4 << ShiftBGPadjNext) & MaskBGPadjNext, $3.comp, FUNC_NONE, NULL ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } | dqual AS IN '[' ullist ']' { struct ULongListNode *node; ULongtree_t *root = NULL; if ( $1.direction == DIR_UNSPEC || $1.direction == SOURCE_OR_DESTINATION || $1.direction == SOURCE_AND_DESTINATION ) { // src and/or dst AS // we need a second rbtree due to different shifts for src and dst AS root = malloc(sizeof(ULongtree_t)); struct ULongListNode *n; if ( root == NULL) { yyerror("malloc() error"); YYABORT; } RB_INIT(root); RB_FOREACH(node, ULongtree, (ULongtree_t *)$5) { if ( node->value > 0xFFFFFFFFLL ) { yyerror("AS number of range"); YYABORT; } if ((n = malloc(sizeof(struct ULongListNode))) == NULL) { yyerror("malloc() error"); YYABORT; } n->value = (node->value << ShiftDstAS) & MaskDstAS; node->value = (node->value << ShiftSrcAS) & MaskSrcAS; RB_INSERT(ULongtree, root, n); } } switch ( $1.direction ) { case SOURCE: RB_FOREACH(node, ULongtree, (ULongtree_t *)$5) { node->value = (node->value << ShiftSrcAS) & MaskSrcAS; } $$.self = NewBlock(OffsetAS, MaskSrcAS, 0, CMP_ULLIST, FUNC_NONE, (void *)$5 ); break; case DESTINATION: RB_FOREACH(node, ULongtree, (ULongtree_t *)$5) { node->value = (node->value << ShiftDstAS) & MaskDstAS; } $$.self = NewBlock(OffsetAS, MaskDstAS, 0, CMP_ULLIST, FUNC_NONE, (void *)$5 ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetAS, MaskSrcAS, 0, CMP_ULLIST, FUNC_NONE, (void *)$5 ), NewBlock(OffsetAS, MaskDstAS, 0, CMP_ULLIST, FUNC_NONE, (void *)root ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetAS, MaskSrcAS, 0, CMP_ULLIST, FUNC_NONE, (void *)$5 ), NewBlock(OffsetAS, MaskDstAS, 0, CMP_ULLIST, FUNC_NONE, (void *)root ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } } | dqual MASK NUMBER { if ( $3 > 255 ) { yyerror("Mask outside of range 0..255"); YYABORT; } switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetMask, MaskSrcMask, ($3 << ShiftSrcMask) & MaskSrcMask, CMP_EQ, FUNC_NONE, NULL ); break; case DESTINATION: $$.self = NewBlock(OffsetMask, MaskDstMask, ($3 << ShiftDstMask) & MaskDstMask, CMP_EQ, FUNC_NONE, NULL ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetMask, MaskSrcMask, ($3 << ShiftSrcMask) & MaskSrcMask, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetMask, MaskDstMask, ($3 << ShiftDstMask) & MaskDstMask, CMP_EQ, FUNC_NONE, NULL ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetMask, MaskSrcMask, ($3 << ShiftSrcMask) & MaskSrcMask, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetMask, MaskDstMask, ($3 << ShiftDstMask) & MaskDstMask, CMP_EQ, FUNC_NONE, NULL ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End switch } | dqual NET STRING STRING { int af, bytes, ret; uint64_t mask[2]; ret = parse_ip(&af, $3, IPstack, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } if ( ret == -1 ) { yyerror("IP address required - hostname not allowed here."); YYABORT; } // ret == -2 will never happen here, as STRICT_IP is set if ( af != PF_INET ) { yyerror("IP netmask syntax valid only for IPv4"); YYABORT; } if ( bytes != 4 ) { yyerror("Need complete IP address"); YYABORT; } ret = parse_ip(&af, $4, mask, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } if ( ret == -1 ) { yyerror("IP address required - hostname not allowed here."); YYABORT; } // ret == -2 will never happen here, as STRICT_IP is set if ( af != PF_INET || bytes != 4 ) { yyerror("Invalid netmask for IPv4 address"); YYABORT; } IPstack[0] &= mask[0]; IPstack[1] &= mask[1]; switch ( $1.direction ) { case SOURCE: $$.self = Connect_AND( NewBlock(OffsetSrcIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetSrcIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); break; case DESTINATION: $$.self = Connect_AND( NewBlock(OffsetDstIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetDstIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( Connect_AND( NewBlock(OffsetSrcIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetSrcIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ), Connect_AND( NewBlock(OffsetDstIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetDstIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( Connect_AND( NewBlock(OffsetSrcIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetSrcIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ), Connect_AND( NewBlock(OffsetDstIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetDstIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ) ); break; default: /* should never happen */ yyerror("This token is not expected here!"); YYABORT; } // End of switch } | dqual NET STRING '/' NUMBER { int af, bytes, ret; uint64_t mask[2]; ret = parse_ip(&af, $3, IPstack, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } if ( ret == -1 ) { yyerror("IP address required - hostname not allowed here."); YYABORT; } // ret == -2 will never happen here, as STRICT_IP is set if ( $5 > (bytes*8) ) { yyerror("Too many netbits for this IP address"); YYABORT; } if ( af == PF_INET ) { mask[0] = 0xffffffffffffffffLL; mask[1] = 0xffffffffffffffffLL << ( 32 - $5 ); } else { // PF_INET6 if ( $5 > 64 ) { mask[0] = 0xffffffffffffffffLL; mask[1] = 0xffffffffffffffffLL << ( 128 - $5 ); } else { mask[0] = 0xffffffffffffffffLL << ( 64 - $5 ); mask[1] = 0; } } // IP aadresses are stored in network representation mask[0] = mask[0]; mask[1] = mask[1]; IPstack[0] &= mask[0]; IPstack[1] &= mask[1]; switch ( $1.direction ) { case SOURCE: $$.self = Connect_AND( NewBlock(OffsetSrcIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetSrcIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); break; case DESTINATION: $$.self = Connect_AND( NewBlock(OffsetDstIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetDstIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( Connect_AND( NewBlock(OffsetSrcIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetSrcIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ), Connect_AND( NewBlock(OffsetDstIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetDstIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( Connect_AND( NewBlock(OffsetSrcIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetSrcIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ), Connect_AND( NewBlock(OffsetDstIPv6b, mask[1], IPstack[1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetDstIPv6a, mask[0], IPstack[0] , CMP_EQ, FUNC_NONE, NULL ) ) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } | dqual IF NUMBER { if ( $3 > 0xffffffffLL ) { yyerror("Input interface number must 0..2^32"); YYABORT; } switch ( $1.direction ) { case DIR_UNSPEC: $$.self = Connect_OR( NewBlock(OffsetInOut, MaskInput, ($3 << ShiftInput) & MaskInput, CMP_EQ, FUNC_NONE, NULL), NewBlock(OffsetInOut, MaskOutput, ($3 << ShiftOutput) & MaskOutput, CMP_EQ, FUNC_NONE, NULL) ); break; case DIR_IN: $$.self = NewBlock(OffsetInOut, MaskInput, ($3 << ShiftInput) & MaskInput, CMP_EQ, FUNC_NONE, NULL); break; case DIR_OUT: $$.self = NewBlock(OffsetInOut, MaskOutput, ($3 << ShiftOutput) & MaskOutput, CMP_EQ, FUNC_NONE, NULL); break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } | dqual VLAN NUMBER { if ( $3 > 65535 ) { yyerror("VLAN number of range 0..65535"); YYABORT; } switch ( $1.direction ) { case SOURCE: $$.self = NewBlock(OffsetVlan, MaskSrcVlan, ($3 << ShiftSrcVlan) & MaskSrcVlan, CMP_EQ, FUNC_NONE, NULL ); break; case DESTINATION: $$.self = NewBlock(OffsetVlan, MaskDstVlan, ($3 << ShiftDstVlan) & MaskDstVlan, CMP_EQ, FUNC_NONE, NULL); break; case DIR_UNSPEC: case SOURCE_OR_DESTINATION: $$.self = Connect_OR( NewBlock(OffsetVlan, MaskSrcVlan, ($3 << ShiftSrcVlan) & MaskSrcVlan, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetVlan, MaskDstVlan, ($3 << ShiftDstVlan) & MaskDstVlan, CMP_EQ, FUNC_NONE, NULL) ); break; case SOURCE_AND_DESTINATION: $$.self = Connect_AND( NewBlock(OffsetVlan, MaskSrcVlan, ($3 << ShiftSrcVlan) & MaskSrcVlan, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetVlan, MaskDstVlan, ($3 << ShiftDstVlan) & MaskDstVlan, CMP_EQ, FUNC_NONE, NULL) ); break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } | dqual MAC STRING { uint64_t mac = VerifyMac($3); if ( mac == 0 ) { yyerror("Invalid MAC address format"); YYABORT; } switch ( $1.direction ) { case DIR_UNSPEC: { uint32_t in, out; in = Connect_OR( NewBlock(OffsetInSrcMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetInDstMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ) ); out = Connect_OR( NewBlock(OffsetOutSrcMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetOutDstMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ) ); $$.self = Connect_OR(in, out); } break; case DIR_IN: $$.self = Connect_OR( NewBlock(OffsetInSrcMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetInDstMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ) ); break; case DIR_OUT: $$.self = Connect_OR( NewBlock(OffsetOutSrcMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetOutDstMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ) ); break; case SOURCE: $$.self = Connect_OR( NewBlock(OffsetInSrcMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetOutSrcMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ) ); break; case DESTINATION: $$.self = Connect_OR( NewBlock(OffsetInDstMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ), NewBlock(OffsetOutDstMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ) ); break; case IN_SRC: $$.self = NewBlock(OffsetInSrcMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ); break; case IN_DST: $$.self = NewBlock(OffsetInDstMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ); break; case OUT_SRC: $$.self = NewBlock(OffsetOutSrcMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ); break; case OUT_DST: $$.self = NewBlock(OffsetOutDstMAC, MaskMac, mac, CMP_EQ, FUNC_NONE, NULL ); break; break; default: yyerror("This token is not expected here!"); YYABORT; } // End of switch } | MPLS STRING comp NUMBER { if ( $4 > MPLSMAX ) { yyerror("MPLS value out of range"); YYABORT; } // search for label1 - label10 if ( strncasecmp($2, "label", 5) == 0 ) { uint64_t mask; uint32_t offset, shift; char *s = &$2[5]; if ( *s == '\0' ) { yyerror("Missing label number"); YYABORT; } int i = (int)strtol(s, (char **)NULL, 10); switch (i) { case 1: offset = OffsetMPLS12; mask = MaskMPLSlabelOdd; shift = ShiftMPLSlabelOdd; break; case 2: offset = OffsetMPLS12; mask = MaskMPLSlabelEven; shift = ShiftMPLSlabelEven; break; case 3: offset = OffsetMPLS34; mask = MaskMPLSlabelOdd; shift = ShiftMPLSlabelOdd; break; case 4: offset = OffsetMPLS34; mask = MaskMPLSlabelEven; shift = ShiftMPLSlabelEven; break; case 5: offset = OffsetMPLS56; mask = MaskMPLSlabelOdd; shift = ShiftMPLSlabelOdd; break; case 6: offset = OffsetMPLS56; mask = MaskMPLSlabelEven; shift = ShiftMPLSlabelEven; break; case 7: offset = OffsetMPLS78; mask = MaskMPLSlabelOdd; shift = ShiftMPLSlabelOdd; break; case 8: offset = OffsetMPLS78; mask = MaskMPLSlabelEven; shift = ShiftMPLSlabelEven; break; case 9: offset = OffsetMPLS910; mask = MaskMPLSlabelOdd; shift = ShiftMPLSlabelOdd; break; case 10: offset = OffsetMPLS910; mask = MaskMPLSlabelEven; shift = ShiftMPLSlabelEven; break; default: yyerror("MPLS label out of range 1..10"); YYABORT; } $$.self = NewBlock(offset, mask, ($4 << shift) & mask, $3.comp, FUNC_NONE, NULL ); } else if ( strcasecmp($2, "eos") == 0 ) { // match End of Stack label $$.self = NewBlock(0, AnyMask, $4 << 4, $3.comp, FUNC_MPLS_EOS, NULL ); } else if ( strncasecmp($2, "exp", 3) == 0 ) { uint64_t mask; uint32_t offset, shift; char *s = &$2[3]; if ( *s == '\0' ) { yyerror("Missing label number"); YYABORT; } int i = (int)strtol(s, (char **)NULL, 10); if ( $4 > 7 ) { yyerror("MPLS exp value out of range"); YYABORT; } switch (i) { case 1: offset = OffsetMPLS12; mask = MaskMPLSexpOdd; shift = ShiftMPLSexpOdd; break; case 2: offset = OffsetMPLS12; mask = MaskMPLSexpEven; shift = ShiftMPLSexpEven; break; case 3: offset = OffsetMPLS34; mask = MaskMPLSexpOdd; shift = ShiftMPLSexpOdd; break; case 4: offset = OffsetMPLS34; mask = MaskMPLSexpEven; shift = ShiftMPLSexpEven; break; case 5: offset = OffsetMPLS56; mask = MaskMPLSexpOdd; shift = ShiftMPLSexpOdd; break; case 6: offset = OffsetMPLS56; mask = MaskMPLSexpEven; shift = ShiftMPLSexpEven; break; case 7: offset = OffsetMPLS78; mask = MaskMPLSexpOdd; shift = ShiftMPLSexpOdd; break; case 8: offset = OffsetMPLS78; mask = MaskMPLSexpEven; shift = ShiftMPLSexpEven; break; case 9: offset = OffsetMPLS910; mask = MaskMPLSexpOdd; shift = ShiftMPLSexpOdd; break; case 10: offset = OffsetMPLS910; mask = MaskMPLSexpEven; shift = ShiftMPLSexpEven; break; default: yyerror("MPLS label out of range 1..10"); YYABORT; } $$.self = NewBlock(offset, mask, $4 << shift, $3.comp, FUNC_NONE, NULL ); } else { yyerror("Unknown MPLS option"); YYABORT; } } | MPLS ANY NUMBER { uint32_t *opt = malloc(sizeof(uint32_t)); if ( $3 > MPLSMAX ) { yyerror("MPLS value out of range"); YYABORT; } if ( opt == NULL) { yyerror("malloc() error"); YYABORT; } *opt = $3 << 4; $$.self = NewBlock(0, AnyMask, $3 << 4, CMP_EQ, FUNC_MPLS_ANY, opt ); } | FWDSTAT NUMBER { if ( $2 > 255 ) { yyerror("Forwarding status of range 0..255"); YYABORT; } $$.self = NewBlock(OffsetStatus, MaskStatus, ($2 << ShiftStatus) & MaskStatus, CMP_EQ, FUNC_NONE, NULL); } | FWDSTAT STRING { uint64_t id = Get_fwd_status_id($2); if (id == 256 ) { yyerror("Unknown forwarding status"); YYABORT; } $$.self = NewBlock(OffsetStatus, MaskStatus, (id << ShiftStatus) & MaskStatus, CMP_EQ, FUNC_NONE, NULL); } | DIR NUMBER { if ( $2 > 2 ) { yyerror("Flow direction status of range 0, 1"); YYABORT; } $$.self = NewBlock(OffsetDir, MaskDir, ($2 << ShiftDir) & MaskDir, CMP_EQ, FUNC_NONE, NULL); } | DIR inout { uint64_t dir = 0xFF; if ( $2.inout == INGRESS ) dir = 0; else if ( $2.inout == EGRESS ) dir = 1; else { yyerror("Flow direction status of range ingress, egress"); YYABORT; } $$.self = NewBlock(OffsetDir, MaskDir, (dir << ShiftDir) & MaskDir, CMP_EQ, FUNC_NONE, NULL); } /* iplist definition */ iplist: STRING { int i, af, bytes, ret; struct IPListNode *node; IPlist_t *root = malloc(sizeof(IPlist_t)); if ( root == NULL) { yyerror("malloc() error"); YYABORT; } RB_INIT(root); ret = parse_ip(&af, $1, IPstack, &bytes, ALLOW_LOOKUP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } // ret == -1 will never happen here, as ALLOW_LOOKUP is set if ( ret != -2 ) { if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } for ( i=0; iip[0] = IPstack[2*i]; node->ip[1] = IPstack[2*i+1]; node->mask[0] = 0xffffffffffffffffLL; node->mask[1] = 0xffffffffffffffffLL; RB_INSERT(IPtree, root, node); } } $$ = (void *)root; } iplist: STRING '/' NUMBER { int af, bytes, ret; struct IPListNode *node; IPlist_t *root = malloc(sizeof(IPlist_t)); if ( root == NULL) { yyerror("malloc() error"); YYABORT; } RB_INIT(root); ret = parse_ip(&af, $1, IPstack, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } // ret == -1 will never happen here, as ALLOW_LOOKUP is set if ( ret != -2 ) { if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } if ((node = malloc(sizeof(struct IPListNode))) == NULL) { yyerror("malloc() error"); YYABORT; } if ( af == PF_INET ) { node->mask[0] = 0xffffffffffffffffLL; node->mask[1] = 0xffffffffffffffffLL << ( 32 - $3 ); } else { // PF_INET6 if ( $3 > 64 ) { node->mask[0] = 0xffffffffffffffffLL; node->mask[1] = 0xffffffffffffffffLL << ( 128 - $3 ); } else { node->mask[0] = 0xffffffffffffffffLL << ( 64 - $3 ); node->mask[1] = 0; } } node->ip[0] = IPstack[0] & node->mask[0]; node->ip[1] = IPstack[1] & node->mask[1]; RB_INSERT(IPtree, root, node); } $$ = (void *)root; } | iplist STRING { int i, af, bytes, ret; struct IPListNode *node; ret = parse_ip(&af, $2, IPstack, &bytes, ALLOW_LOOKUP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } // ret == - 2 means lookup failure if ( ret != -2 ) { for ( i=0; iip[0] = IPstack[2*i]; node->ip[1] = IPstack[2*i+1]; node->mask[0] = 0xffffffffffffffffLL; node->mask[1] = 0xffffffffffffffffLL; RB_INSERT(IPtree, (IPlist_t *)$$, node); } } } | iplist ',' STRING { int i, af, bytes, ret; struct IPListNode *node; ret = parse_ip(&af, $3, IPstack, &bytes, ALLOW_LOOKUP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } // ret == - 2 means lookup failure if ( ret != -2 ) { for ( i=0; iip[0] = IPstack[2*i]; node->ip[1] = IPstack[2*i+1]; node->mask[0] = 0xffffffffffffffffLL; node->mask[1] = 0xffffffffffffffffLL; RB_INSERT(IPtree, (IPlist_t *)$$, node); } } } | iplist STRING '/' NUMBER { int af, bytes, ret; struct IPListNode *node; ret = parse_ip(&af, $2, IPstack, &bytes, STRICT_IP, &num_ip); if ( ret == 0 ) { yyerror("Invalid IP address"); YYABORT; } if ( af && (( af == PF_INET && bytes != 4 ) || ( af == PF_INET6 && bytes != 16 ))) { yyerror("incomplete IP address"); YYABORT; } // ret == - 2 means lookup failure if ( ret != -2 ) { if ((node = malloc(sizeof(struct IPListNode))) == NULL) { yyerror("malloc() error"); YYABORT; } if ( af == PF_INET ) { node->mask[0] = 0xffffffffffffffffLL; node->mask[1] = 0xffffffffffffffffLL << ( 32 - $4 ); } else { // PF_INET6 if ( $4 > 64 ) { node->mask[0] = 0xffffffffffffffffLL; node->mask[1] = 0xffffffffffffffffLL << ( 128 - $4 ); } else { node->mask[0] = 0xffffffffffffffffLL << ( 64 - $4 ); node->mask[1] = 0; } } node->ip[0] = IPstack[0] & node->mask[0]; node->ip[1] = IPstack[1] & node->mask[1]; RB_INSERT(IPtree, (IPlist_t *)$$, node); } } ; /* ULlist definition */ ullist: NUMBER { struct ULongListNode *node; ULongtree_t *root = malloc(sizeof(ULongtree_t)); if ( root == NULL) { yyerror("malloc() error"); YYABORT; } RB_INIT(root); if ((node = malloc(sizeof(struct ULongListNode))) == NULL) { yyerror("malloc() error"); YYABORT; } node->value = $1; RB_INSERT(ULongtree, root, node); $$ = (void *)root; } | ullist NUMBER { struct ULongListNode *node; if ((node = malloc(sizeof(struct ULongListNode))) == NULL) { yyerror("malloc() error"); YYABORT; } node->value = $2; RB_INSERT(ULongtree, (ULongtree_t *)$$, node); } | ullist ',' NUMBER { struct ULongListNode *node; if ((node = malloc(sizeof(struct ULongListNode))) == NULL) { yyerror("malloc() error"); YYABORT; } node->value = $3; RB_INSERT(ULongtree, (ULongtree_t *)$$, node); } ; /* comparator qualifiers */ comp: { $$.comp = CMP_EQ; } | EQ { $$.comp = CMP_EQ; } | LT { $$.comp = CMP_LT; } | GT { $$.comp = CMP_GT; } | LE { $$.comp = CMP_LE; } | GE { $$.comp = CMP_GE; } ; /* 'direction' qualifiers */ dqual: { $$.direction = DIR_UNSPEC; } | SRC { $$.direction = SOURCE; } | DST { $$.direction = DESTINATION; } | SRC OR DST { $$.direction = SOURCE_OR_DESTINATION; } | DST OR SRC { $$.direction = SOURCE_OR_DESTINATION; } | SRC AND DST { $$.direction = SOURCE_AND_DESTINATION; } | DST AND SRC { $$.direction = SOURCE_AND_DESTINATION; } | IN { $$.direction = DIR_IN; } | OUT { $$.direction = DIR_OUT; } | IN SRC { $$.direction = IN_SRC; } | IN DST { $$.direction = IN_DST; } | OUT SRC { $$.direction = OUT_SRC; } | OUT DST { $$.direction = OUT_DST; } | PREV { $$.direction = ADJ_PREV; } | NEXT { $$.direction = ADJ_NEXT; } ; inout: INGRESS { $$.inout = INGRESS; } | EGRESS { $$.inout = EGRESS; } ; acl: ACL { $$.acl = ACL; } | ACE { $$.acl = ACE; } | XACE { $$.acl = XACE; } ; expr: term { $$ = $1.self; } | expr OR expr { $$ = Connect_OR($1, $3); } | expr AND expr { $$ = Connect_AND($1, $3); } | NOT expr %prec NEGATE { $$ = Invert($2); } | '(' expr ')' { $$ = $2; } | '(' expr ')' '%' STRING { $$ = $2; if ( strlen($5) > 16 ) { yyerror("Error: Maximum 16 chars allowed for flowlabel"); YYABORT; } else { AddLabel($2, $5); } } | '%' STRING '(' expr ')' { $$ = $4; if ( strlen($2) > 16 ) { yyerror("Error: Maximum 16 chars allowed for flowlabel"); YYABORT; } else { AddLabel($4, $2); } } ; %% static void yyerror(char *msg) { if ( FilterFilename ) snprintf(yyerror_buff, 255 ,"File '%s' line %d: %s at '%s'", FilterFilename, lineno, msg, yytext); else snprintf(yyerror_buff, 255, "Line %d: %s at '%s'", lineno, msg, yytext); yyerror_buff[255] = '\0'; fprintf(stderr, "%s\n", yyerror_buff); } /* End of yyerror */ static uint32_t ChainHosts(uint64_t *offsets, uint64_t *hostlist, int num_records, int type) { uint32_t offset_a, offset_b, i, j, block; if ( type == SOURCE ) { offset_a = offsets[0]; offset_b = offsets[1]; } else { offset_a = offsets[2]; offset_b = offsets[3]; } i = 0; block = Connect_AND( NewBlock(offset_b, MaskIPv6, hostlist[i+1] , CMP_EQ, FUNC_NONE, NULL ), NewBlock(offset_a, MaskIPv6, hostlist[i] , CMP_EQ, FUNC_NONE, NULL ) ); i += 2; for ( j=1; j 17 ) return 0; for (i=0; i 255 ) { free(p); return 0; } mac = ( mac << 8 ) | (l & 0xFF ); i++; if ( q ) { r = ++q; q = strchr(r, ':'); } else r = NULL; } if ( i != 6 ) return 0; return mac; } // End of VerifyMac static int InitSymbols(void) { int i; // already initialised? if ( fwd_status ) return 1; // fill fwd status cache table fwd_status = ( char **)calloc(256, sizeof(char *)); if ( !fwd_status ) { fprintf(stderr, "malloc(): %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return 0; } i=0; while ( fwd_status_def_list[i].name ) { uint32_t j = fwd_status_def_list[i].id; fwd_status[j] = fwd_status_def_list[i].name; i++; } return 1; } // End of InitSymbols static uint32_t Get_fwd_status_id(char *status) { int i; if ( !fwd_status && !InitSymbols() ) yyerror("malloc() error"); i = 0; while ( i < 256 ) { if ( fwd_status[i] && strcasecmp(fwd_status[i], status) == 0 ) return i; i++; } return 256; } // End of Get_fwd_status_id nfdump-1.6.23/bin/heapsort_inline.c000077500000000000000000000104021404501030700171650ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ typedef void (*siftElement_t)(SortElement_t *SortElement, uint32_t numbersSize, uint32_t node); static void heapSort(SortElement_t *SortElement, uint32_t array_size, int topN, int direction); static inline void siftDown(SortElement_t *SortElement, uint32_t root, uint32_t bottom); static inline void siftUp(SortElement_t *SortElement, uint32_t numbersSize, uint32_t node); static void heapSort(SortElement_t *SortElement, uint32_t array_size, int topN, int direction) { int32_t i, maxindex; siftElement_t siftElement = direction == DESCENDING ? siftDown : siftUp; for(i = array_size - 1; i >= 0; i--) siftElement(SortElement,array_size,i); /* * we are only interested in the first top N => skip sorting the rest * For topN == 0 -> all flows gets sorted */ if ( (topN >= (array_size - 1)) || topN == 0 ) { maxindex = 0; } else { maxindex = array_size - 1 - topN; } for(i = array_size-1; i > maxindex; i-- ) { SortElement_t temp = SortElement[0]; SortElement[0] = SortElement[i]; SortElement[i] = temp; siftElement(SortElement,i,0); } } // End of heapSort static inline void siftDown(SortElement_t *SortElement, uint32_t numbersSize, uint32_t node) { uint32_t i, parent, child; parent = node; i = parent + 1; while( i != parent ) { i = parent; // Compare with left child node child = 2*i+1; if( (child) < numbersSize && SortElement[child].count > SortElement[parent].count) parent = child; // Compare with right child node child = 2*i+2; if( (child) < numbersSize && SortElement[child].count > SortElement[parent].count) parent = child; if ( i != parent ) { SortElement_t temp = SortElement[i]; SortElement[i] = SortElement[parent]; SortElement[parent] = temp; } } } // End of siftDown static inline void siftUp(SortElement_t *SortElement, uint32_t numbersSize, uint32_t node) { uint32_t i, parent, child; parent = node; i = parent + 1; while( i != parent ) { i = parent; // Compare with left child node child = 2*i+1; if( (child) < numbersSize && SortElement[child].count < SortElement[parent].count) parent = child; // Compare with right child node child = 2*i+2; if( (child) < numbersSize && SortElement[child].count < SortElement[parent].count) parent = child; if ( i != parent ) { SortElement_t temp = SortElement[i]; SortElement[i] = SortElement[parent]; SortElement[parent] = temp; } } } // End of siftUp nfdump-1.6.23/bin/inline.c000066400000000000000000000210001404501030700152510ustar00rootroot00000000000000/* * Copyright (c) 2014, Peter Haag * Copyright (c) 2009, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * * $Author: haag $ * * $Id: inline.c 40 2009-12-16 10:41:44Z haag $ * * $LastChangedRevision: 40 $ * */ static inline uint16_t Get_val16(void *p); static inline uint32_t Get_val24(void *p); static inline uint32_t Get_val32(void *p); static inline uint64_t Get_val40(void *p); static inline uint64_t Get_val48(void *p); static inline uint64_t Get_val56(void *p); static inline uint64_t Get_val64(void *p); static inline uint64_t Get_val(void *p, uint32_t index, uint32_t length); static inline void Put_val16(uint16_t v, void *p); static inline void Put_val24(uint32_t v, void *p); static inline void Put_val32(uint32_t v, void *p); // static inline void Put_val40(uint64_t v, void *p); static inline void Put_val48(uint64_t v, void *p); // static inline void Put_val56(uint64_t v, void *p); static inline void Put_val64(uint64_t v, void *p); static inline uint16_t Get_val16(void *p) { uint8_t *in = (uint8_t *)p; type_mask_t mask; #ifdef WORDS_BIGENDIAN mask.val.val8[0] = in[0]; mask.val.val8[1] = in[1]; #else mask.val.val8[0] = in[1]; mask.val.val8[1] = in[0]; #endif return mask.val.val16[0]; } // End of Get_val16 static inline uint32_t Get_val24(void *p) { uint8_t *in = (uint8_t *)p; type_mask_t mask; #ifdef WORDS_BIGENDIAN mask.val.val8[0] = 0; mask.val.val8[1] = in[0]; mask.val.val8[2] = in[1]; mask.val.val8[3] = in[2]; #else mask.val.val8[0] = in[2]; mask.val.val8[1] = in[1]; mask.val.val8[2] = in[0]; mask.val.val8[3] = 0; #endif return mask.val.val32[0]; } // End of Get_val24 static inline uint32_t Get_val32(void *p) { uint8_t *in = (uint8_t *)p; type_mask_t mask; #ifdef WORDS_BIGENDIAN mask.val.val8[0] = in[0]; mask.val.val8[1] = in[1]; mask.val.val8[2] = in[2]; mask.val.val8[3] = in[3]; #else mask.val.val8[0] = in[3]; mask.val.val8[1] = in[2]; mask.val.val8[2] = in[1]; mask.val.val8[3] = in[0]; #endif return mask.val.val32[0]; } // End of Get_val32 static inline uint64_t Get_val40(void *p) { uint8_t *in = (uint8_t *)p; type_mask_t mask; #ifdef WORDS_BIGENDIAN mask.val.val8[0] = 0; mask.val.val8[1] = 0; mask.val.val8[2] = 0; mask.val.val8[3] = in[0]; mask.val.val8[4] = in[1]; mask.val.val8[5] = in[2]; mask.val.val8[6] = in[3]; mask.val.val8[7] = in[4]; #else mask.val.val8[0] = in[4]; mask.val.val8[1] = in[3]; mask.val.val8[2] = in[2]; mask.val.val8[3] = in[1]; mask.val.val8[4] = in[0]; mask.val.val8[5] = 0; mask.val.val8[6] = 0; mask.val.val8[7] = 0; #endif return mask.val.val64; } // End of Get_val40 static inline uint64_t Get_val48(void *p) { uint8_t *in = (uint8_t *)p; type_mask_t mask; #ifdef WORDS_BIGENDIAN mask.val.val8[0] = 0; mask.val.val8[1] = 0; mask.val.val8[2] = in[0]; mask.val.val8[3] = in[1]; mask.val.val8[4] = in[2]; mask.val.val8[5] = in[3]; mask.val.val8[6] = in[4]; mask.val.val8[7] = in[5]; #else mask.val.val8[0] = in[5]; mask.val.val8[1] = in[4]; mask.val.val8[2] = in[3]; mask.val.val8[3] = in[2]; mask.val.val8[4] = in[1]; mask.val.val8[5] = in[0]; mask.val.val8[6] = 0; mask.val.val8[7] = 0; #endif return mask.val.val64; } // End of Get_val48 static inline uint64_t Get_val56(void *p) { uint8_t *in = (uint8_t *)p; type_mask_t mask; #ifdef WORDS_BIGENDIAN mask.val.val8[0] = 0; mask.val.val8[1] = in[0]; mask.val.val8[2] = in[1]; mask.val.val8[3] = in[2]; mask.val.val8[4] = in[3]; mask.val.val8[5] = in[4]; mask.val.val8[6] = in[5]; mask.val.val8[7] = in[6]; #else mask.val.val8[0] = in[6]; mask.val.val8[1] = in[5]; mask.val.val8[2] = in[4]; mask.val.val8[3] = in[3]; mask.val.val8[4] = in[2]; mask.val.val8[5] = in[1]; mask.val.val8[6] = in[0]; mask.val.val8[7] = 0; #endif return mask.val.val64; } // End of Get_val56 static inline uint64_t Get_val64(void *p) { uint8_t *in = (uint8_t *)p; type_mask_t mask; #ifdef WORDS_BIGENDIAN mask.val.val8[0] = in[0]; mask.val.val8[1] = in[1]; mask.val.val8[2] = in[2]; mask.val.val8[3] = in[3]; mask.val.val8[4] = in[4]; mask.val.val8[5] = in[5]; mask.val.val8[6] = in[6]; mask.val.val8[7] = in[7]; #else mask.val.val8[0] = in[7]; mask.val.val8[1] = in[6]; mask.val.val8[2] = in[5]; mask.val.val8[3] = in[4]; mask.val.val8[4] = in[3]; mask.val.val8[5] = in[2]; mask.val.val8[6] = in[1]; mask.val.val8[7] = in[0]; #endif return mask.val.val64; } // End of Get_val64 static inline uint64_t Get_val(void *p, uint32_t index, uint32_t length) { switch (length) { case 1: return *((uint8_t *)(p + index)); break; case 2: return Get_val16(p + index); break; case 3: return Get_val24(p + index); break; case 4: return Get_val32(p + index); break; case 5: return Get_val40(p + index); break; case 6: return Get_val48(p + index); break; case 7: return Get_val56(p + index); break; case 8: return Get_val64(p + index); break; default: return 0; } return 0; } // End of Get_val static inline void Put_val16(uint16_t v, void *p) { uint8_t *out = (uint8_t *)p; type_mask_t mask; mask.val.val16[0] = v; out[0] = mask.val.val8[0]; out[1] = mask.val.val8[1]; } // End of Put_val16 static inline void Put_val24(uint32_t v, void *p) { uint8_t *out = (uint8_t *)p; type_mask_t mask; mask.val.val32[0] = v; out[0] = mask.val.val8[1]; out[1] = mask.val.val8[2]; out[2] = mask.val.val8[3]; } // End of Put_val24 static inline void Put_val32(uint32_t v, void *p) { uint8_t *out = (uint8_t *)p; type_mask_t mask; mask.val.val32[0] = v; out[0] = mask.val.val8[0]; out[1] = mask.val.val8[1]; out[2] = mask.val.val8[2]; out[3] = mask.val.val8[3]; } // End of Put_val32 /* * not yet used * static inline void Put_val40(uint64_t v, void *p) { uint8_t *out = (uint8_t *)p; type_mask_t mask; mask.val.val64 = v; out[0] = mask.val.val8[3]; out[1] = mask.val.val8[4]; out[2] = mask.val.val8[5]; out[3] = mask.val.val8[6]; out[4] = mask.val.val8[7]; } // End of Put_val40 * */ static inline void Put_val48(uint64_t v, void *p) { uint8_t *out = (uint8_t *)p; type_mask_t mask; mask.val.val64 = v; out[0] = mask.val.val8[2]; out[1] = mask.val.val8[3]; out[2] = mask.val.val8[4]; out[3] = mask.val.val8[5]; out[4] = mask.val.val8[6]; out[5] = mask.val.val8[7]; } // End of Put_val48 /* * not yet used * static inline void Put_val56(uint64_t v, void *p) { uint8_t *out = (uint8_t *)p; type_mask_t mask; mask.val.val64 = v; out[0] = mask.val.val8[1]; out[1] = mask.val.val8[2]; out[2] = mask.val.val8[3]; out[3] = mask.val.val8[4]; out[4] = mask.val.val8[5]; out[5] = mask.val.val8[6]; out[6] = mask.val.val8[7]; } // End of Put_val56 * */ static inline void Put_val64(uint64_t v, void *p) { uint8_t *out = (uint8_t *)p; type_mask_t mask; mask.val.val64 = v; out[0] = mask.val.val8[0]; out[1] = mask.val.val8[1]; out[2] = mask.val.val8[2]; out[3] = mask.val.val8[3]; out[4] = mask.val.val8[4]; out[5] = mask.val.val8[5]; out[6] = mask.val.val8[6]; out[7] = mask.val.val8[7]; } // End of Put_val64 nfdump-1.6.23/bin/ipconv.c000066400000000000000000000221541404501030700153040ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2008-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * * Copyright (c) 1996 by Internet Software Consortium. * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SOFTWARE CONSORTIUM DISCLAIMS * ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL INTERNET SOFTWARE * CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL * DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR * PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS * ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS * SOFTWARE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_RESOLV_H #include #endif #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "ipconv.h" static int parse_ipv4(const char *src, uint32_t *dst, int *bytes); static int parse_ipv6(const char *src, uint64_t *dst, int *bytes); static int lookup_host(const char *hostname, uint64_t *iplist, uint32_t *num_ip ); int parse_ip(int *af, const char *src, uint64_t *dst, int *bytes, int lookup, uint32_t *num_ip ) { char *alpha = "abcdefghijklmnopqrstuvwxzyABCDEFGHIJKLMNOPQRSTUVWXZY"; uint32_t v4addr; int ret; // check for IPv6 address if ( strchr(src, ':') != NULL ) { *af = PF_INET6; // check for alpha chars -> hostname -> lookup } else if ( strpbrk(src, alpha)) { *af = 0; if ( lookup == STRICT_IP ) return -1; else return lookup_host(src, dst, num_ip ); // it's IPv4 } else *af = PF_INET; *num_ip = 1; switch (*af) { case AF_INET: ret = (parse_ipv4(src, &v4addr, bytes)); dst[0] = 0; dst[1] = ntohl(v4addr) & 0xffffffffLL ; return ret; break; case AF_INET6: ret = (parse_ipv6(src, dst, bytes)); dst[0] = ntohll(dst[0]); dst[1] = ntohll(dst[1]); return ret; break; } /* NOTREACHED */ return 0; } static int parse_ipv4(const char *src, uint32_t *dst, int *bytes) { static const char digits[] = "0123456789"; int saw_digit, ch; uint8_t tmp[4], *tp; saw_digit = 0; *bytes = 0; *(tp = tmp) = 0; memset(tmp, 0, sizeof(tmp)); while ((ch = *src++) != '\0') { const char *pch; if ((pch = strchr(digits, ch)) != NULL) { unsigned int new = *tp * 10 + (pch - digits); if (new > 255) return (0); if (! saw_digit) { if (++(*bytes) > 4) return (0); saw_digit = 1; } *tp = new; } else if (ch == '.' && saw_digit) { if (*bytes == 4) return (0); *++tp = 0; saw_digit = 0; if ( !(*src) ) return 0; } else return (0); } memcpy(dst, tmp, sizeof(tmp)); return (1); } static int parse_ipv6(const char *src, uint64_t *dst, int *bytes) { static const char xdigits_l[] = "0123456789abcdef", xdigits_u[] = "0123456789ABCDEF"; uint8_t tmp[16], *tp, *endp, *colonp; const char *xdigits, *curtok; int ch, saw_xdigit; u_int val; memset((tp = tmp), '\0', sizeof(tmp)); endp = tp + sizeof(tmp); colonp = NULL; /* Leading :: requires some special handling. */ if (*src == ':') if (*++src != ':') return (0); curtok = src; saw_xdigit = 0; val = 0; while ((ch = *src++) != '\0') { const char *pch; if ((pch = strchr((xdigits = xdigits_l), ch)) == NULL) pch = strchr((xdigits = xdigits_u), ch); if (pch != NULL) { val <<= 4; val |= (pch - xdigits); if (val > 0xffff) return (0); saw_xdigit = 1; continue; } if (ch == ':') { curtok = src; if (!saw_xdigit) { if (colonp) return (0); colonp = tp; continue; } else if (*src == '\0') { return (0); } if (tp + sizeof(uint16_t) > endp) return (0); *tp++ = (u_char) (val >> 8) & 0xff; *tp++ = (u_char) val & 0xff; saw_xdigit = 0; val = 0; continue; } if (ch == '.' && ((tp + 4) <= endp) && parse_ipv4(curtok, (uint32_t *)tp, bytes) > 0) { tp += 4; saw_xdigit = 0; break; /* '\0' was seen by parse_ipv4(). */ } return (0); } if (saw_xdigit) { if (tp + sizeof(uint16_t) > endp) return (0); *tp++ = (u_char) (val >> 8) & 0xff; *tp++ = (u_char) val & 0xff; } if (colonp != NULL) { /* * Since some memmove()'s erroneously fail to handle * overlapping regions, we'll do the shift by hand. */ const int n = tp - colonp; int i; for (i = 1; i <= n; i++) { endp[- i] = colonp[n - i]; colonp[n - i] = 0; } tp = endp; } *bytes = 16 - ( endp - tp ); memcpy(dst, tmp, sizeof(tmp)); return (1); } static int lookup_host(const char *hostname, uint64_t *iplist, uint32_t *num_ip ) { struct addrinfo hints, *res, *r; int errcode, i, len; char addrstr[128]; char reverse[256]; void *ptr; printf("Resolving %s ...\n", hostname); memset (&hints, 0, sizeof (hints)); hints.ai_family = PF_UNSPEC; hints.ai_socktype = SOCK_STREAM; hints.ai_flags |= AI_CANONNAME; errcode = getaddrinfo (hostname, NULL, &hints, &res); if (errcode != 0) { fprintf(stderr, "Failed to resolve IP address for %s: %s\n", hostname, gai_strerror(errno)); return 0; } // count the number of records found *num_ip = 0; // remember res for later free() r = res; i = 0; while (res) { if ( *num_ip >= MAXHOSTS ) { printf ("Too man IP addresses in DNS response\n"); return 1; } switch (res->ai_family) { case PF_INET: ptr = &(((struct sockaddr_in *) res->ai_addr)->sin_addr); iplist[i++] = 0; iplist[i++] = ntohl(*(uint32_t *)ptr) & 0xffffffffLL ; len = sizeof(struct sockaddr_in); break; case AF_INET6: ptr = &((struct sockaddr_in6 *) res->ai_addr)->sin6_addr; iplist[i++] = ntohll(((uint64_t *)ptr)[0]); iplist[i++] = ntohll(((uint64_t *)ptr)[1]); len = sizeof(struct sockaddr_in6); break; default: { // not handled res = res->ai_next; continue; } } inet_ntop (res->ai_family, ptr, addrstr, 100); addrstr[99] = '\0'; if ( (errcode = getnameinfo(res->ai_addr, len, reverse, sizeof(reverse), NULL,0,0)) != 0 ) { snprintf(reverse, sizeof(reverse)-1, ""); // fprintf(stderr, "Failed to reverse lookup %s: %s\n", addrstr, gai_strerror(errcode)); } printf ("IPv%d address: %s (%s)\n", res->ai_family == PF_INET6 ? 6 : 4, addrstr, reverse ); res = res->ai_next; (*num_ip)++; } freeaddrinfo(r); return 1; } // End of lookup_host int set_nameserver(char *ns) { struct hostent *host; res_init(); host = gethostbyname(ns); if (host == NULL) { (void) fprintf(stderr,"Can not resolv nameserver %s: %s\n", ns, hstrerror(h_errno)); return 0; } (void) memcpy((void *)&_res.nsaddr_list[0].sin_addr, (void *)host->h_addr_list[0], (size_t)host->h_length); _res.nscount = 1; return 1; } // End of set_nameserver /* int main( int argc, char **argv ) { char *s, t[64]; uint64_t anyaddr[2]; uint32_t num_ip; int af, ret, bytes; s = argv[1]; if (argc == 3 && !set_nameserver(argv[2]) ) return 0; lookup_host(s, &num_ip); return 0; ret = parse_ip(&af, s, anyaddr, &bytes); if ( ret != 1 ) { printf("Parse failed!\n"); return 0; } if ( af == PF_INET ) inet_ntop(af, &(((uint32_t *)anyaddr)[3]), t, 64); else inet_ntop(af, anyaddr, t, 64); printf("Convert back: %s => %s %i bytes\n", s, t, bytes); } */ nfdump-1.6.23/bin/ipconv.h000077500000000000000000000041101404501030700153040ustar00rootroot00000000000000/* * Copyright (c) 2017, Peter Haag * Copyright (c) 2014, Peter Haag * Copyright (c) 2009, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _IPCONV_H #define _IPCONV_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif int parse_ip(int *af, const char *src, uint64_t *dst, int *bytes, int lookup, uint32_t *num_ip ); int set_nameserver(char *ns); #define MAXHOSTS 512 #define STRICT_IP 0 #define ALLOW_LOOKUP 1 #endif //_IPCONV_H nfdump-1.6.23/bin/ipfix.c000066400000000000000000002476031404501030700151350ustar00rootroot00000000000000/* * Copyright (c) 2012-2021, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "nfnet.h" #include "output_raw.h" #include "bookkeeper.h" #include "collector.h" #include "exporter.h" #include "ipfix.h" #define GET_FLOWSET_ID(p) (Get_val16(p)) #define GET_FLOWSET_LENGTH(p) (Get_val16((void *)((p) + 2))) #define GET_TEMPLATE_ID(p) (Get_val16(p)) #define GET_TEMPLATE_COUNT(p) (Get_val16((void *)((p) + 2))) #define GET_OPTION_TEMPLATE_ID(p) (Get_val16(p)) #define GET_OPTION_TEMPLATE_FIELD_COUNT(p) (Get_val16((void *)((p) + 2))) #define GET_OPTION_TEMPLATE_SCOPE_FIELD_COUNT(p) (Get_val16((void *)((p) + 4))) #define CHECK_OPTION_DATA(avail, tag) ((tag.offset + tag.length) <= avail) #define DYN_FIELD_LENGTH 65535 /* module limited globals */ /* * sequence element to move data from data input to output * a sequence exists for each IPFIX element */ typedef struct sequence_map_s { /* sequence definition: just move a certain number of bytes -> moveXX set a certain number of output bytes to zero -> zeroXX process input data into appropriate output -> AnyName */ #define nop 0 #define dyn_skip 1 #define move8 2 #define move16 3 #define move32 4 #define move40 5 #define move48 6 #define move56 7 #define move64 8 #define move128 9 #define move32_sampling 10 #define move64_sampling 11 #define move_mac 12 #define move_mpls 13 #define move_flags 14 #define Time64Mili 15 #define TimeDeltaMicro 16 #define TimeMili 17 #define SystemInitTime 18 #define TimeUnix 19 #define saveICMP 20 #define zero8 21 #define zero16 22 #define zero32 23 #define zero64 24 #define zero128 25 uint32_t id; // sequence ID as defined above uint16_t skip_count; // skip this number of bytes in input stream after reading uint16_t type; // Element type uint16_t input_length; // length of input element uint16_t output_offset; // copy final data to this output offset void *stack; // optionally copy data onto this stack } sequence_map_t; /* * the IPFIX template records are processed and * for each template we create a a translation table, which contains * all information required, to transform the data records from * the exporter into nfdump internal data structurs. * All templates are chained in a linked list */ typedef struct input_translation_s { struct input_translation_s *next; // linked list uint32_t flags; // flags for output record time_t updated; // timestamp of last update/refresh uint32_t id; // template ID of exporter domains uint32_t output_record_size; // required size in nfdump format // tmp vars needed while processing the data record int delta_time; // delta micro or absolute ms time stamps uint64_t flow_start; // start time in msec uint64_t flow_end; // end time in msec uint64_t SysUpTime; // System Init Time #160 per record int HasTimeMili; // Exporter sent relative time stamps uint32_t icmpTypeCode; // ICMP type/code in data stream uint64_t packets; // total (in)packets - sampling corrected uint64_t bytes; // total (in)bytes - sampling corrected uint64_t out_packets; // total out packets - sampling corrected uint64_t out_bytes; // total out bytes - sampling corrected uint32_t router_ip_offset; uint32_t received_offset; // extension map infos uint32_t extension_map_changed; // map changed while refreshing? extension_info_t extension_info; // the extension map reflecting this template // sequence map information uint32_t max_number_of_sequences; // max number of sequences for the translate uint32_t number_of_sequences; // number of sequences for the translate sequence_map_t *sequence; // sequence map } input_translation_t; /* * All Obervation Domains from all exporter are stored in a linked list * which uniquely can identify each exporter/Observation Domain */ typedef struct exporterDomain_s { struct exporterDomain_s *next; // linkes list to next exporter // exporter information exporter_info_record_t info; uint64_t packets; // number of packets sent by this exporter uint64_t flows; // number of flow records sent by this exporter uint32_t sequence_failure; // number of sequence failues uint32_t padding_errors; // number of sequence failues // sampler sampler_t *sampler; // sampler info samplerOption_t *samplerOption; // sampler options table info // exporter parameters uint32_t ExportTime; // Current sequence number uint32_t PacketSequence; // statistics uint64_t TemplateRecords; // stat counter uint64_t DataRecords; // stat counter // SysUptime if sent with #160 uint64_t SysUpTime; // in msec optionTag_t SysUpOption; // SysUpTime option information // linked list of all templates sent by this exporter input_translation_t *input_translation_table; // in order to prevent search through all lists keep // the last template we processed as a cache input_translation_t *current_table; } exporterDomain_t; static struct ipfix_element_map_s { uint16_t id; // IPFIX element id uint16_t length; // type of this element ( input length ) uint16_t out_length; // type of this element ( output length ) uint32_t sequence; // uint32_t zero_sequence; // uint16_t extension; // maps into nfdump extension ID } ipfix_element_map[] = { {0, 0, 0}, { IPFIX_octetDeltaCount, _8bytes, _8bytes, move64_sampling, zero64, COMMON_BLOCK }, { IPFIX_octetDeltaCount, _4bytes, _8bytes, move32_sampling, zero64, COMMON_BLOCK }, { IPFIX_packetDeltaCount, _8bytes, _8bytes, move64_sampling, zero64, COMMON_BLOCK }, { IPFIX_packetDeltaCount, _4bytes, _8bytes, move32_sampling, zero64, COMMON_BLOCK }, { IPFIX_octetTotalCount, _8bytes, _8bytes, move64_sampling, zero64, COMMON_BLOCK }, { IPFIX_octetTotalCount, _4bytes, _8bytes, move32_sampling, zero64, COMMON_BLOCK }, { IPFIX_packetTotalCount, _8bytes, _8bytes, move64_sampling, zero64, COMMON_BLOCK }, { IPFIX_packetTotalCount, _4bytes, _8bytes, move32_sampling, zero64, COMMON_BLOCK }, { IPFIX_forwardingStatus, _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { IPFIX_protocolIdentifier, _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { IPFIX_ipClassOfService, _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { IPFIX_tcpControlBits, _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { IPFIX_tcpControlBits, _2bytes, _1byte, move_flags, zero8, COMMON_BLOCK }, { IPFIX_SourceTransportPort, _2bytes, _2bytes, move16, zero16, COMMON_BLOCK }, { IPFIX_SourceIPv4Address, _4bytes, _4bytes, move32, zero32, COMMON_BLOCK }, { IPFIX_SourceIPv4PrefixLength, _1byte, _1byte, move8, zero8, EX_MULIPLE }, { IPFIX_ingressInterface, _4bytes, _4bytes, move32, zero32, EX_IO_SNMP_4 }, { IPFIX_ingressInterface, _2bytes, _2bytes, move16, zero16, EX_IO_SNMP_2 }, { IPFIX_DestinationTransportPort, _2bytes, _2bytes, move16, zero16, COMMON_BLOCK }, { IPFIX_DestinationIPv4Address, _4bytes, _4bytes, move32, zero32, COMMON_BLOCK }, { IPFIX_DestinationIPv4PrefixLength, _1byte, _1byte, move8, zero8, EX_MULIPLE }, { IPFIX_egressInterface, _4bytes, _4bytes, move32, zero32, EX_IO_SNMP_4 }, { IPFIX_egressInterface, _2bytes, _2bytes, move16, zero16, EX_IO_SNMP_2 }, { IPFIX_ipNextHopIPv4Address, _4bytes, _4bytes, move32, zero32, EX_NEXT_HOP_v4 }, { IPFIX_bgpSourceAsNumber, _4bytes, _4bytes, move32, zero32, EX_AS_4 }, { IPFIX_bgpSourceAsNumber, _2bytes, _2bytes, move16, zero16, EX_AS_2 }, { IPFIX_bgpDestinationAsNumber, _4bytes, _4bytes, move32, zero32, EX_AS_4 }, { IPFIX_bgpDestinationAsNumber, _2bytes, _2bytes, move16, zero16, EX_AS_2 }, { IPFIX_bgpNextHopIPv4Address, _4bytes, _4bytes, move32, zero32, EX_NEXT_HOP_BGP_v4}, { IPFIX_flowEndSysUpTime, _4bytes, _4bytes, TimeMili, nop, COMMON_BLOCK}, { IPFIX_flowStartSysUpTime, _4bytes, _4bytes, TimeMili, nop, COMMON_BLOCK }, { IPFIX_postOctetDeltaCount, _8bytes, _8bytes, move64, zero64, EX_OUT_BYTES_8 }, { IPFIX_postOctetDeltaCount, _4bytes, _4bytes, move32, zero32, EX_OUT_BYTES_4 }, { IPFIX_postPacketDeltaCount, _8bytes, _8bytes, move64, zero64, EX_OUT_PKG_8 }, { IPFIX_postPacketDeltaCount, _4bytes, _4bytes, move32, zero32, EX_OUT_PKG_4 }, { IPFIX_SourceIPv6Address, _16bytes, _16bytes, move128, zero128, COMMON_BLOCK }, { IPFIX_DestinationIPv6Address, _16bytes, _16bytes, move128, zero128, COMMON_BLOCK }, { IPFIX_SourceIPv6PrefixLength, _1byte, _1byte, move8, zero8, EX_MULIPLE }, { IPFIX_DestinationIPv6PrefixLength, _1byte, _1byte, move8, zero8, EX_MULIPLE }, { IPFIX_icmpTypeCodeIPv4, _2bytes, _2bytes, saveICMP, nop, COMMON_BLOCK }, { IPFIX_icmpTypeCodeIPv6, _2bytes, _2bytes, saveICMP, nop, COMMON_BLOCK }, { IPFIX_postIpClassOfService, _1byte, _1byte, move8, zero8, EX_MULIPLE }, { IPFIX_SourceMacAddress, _6bytes, _8bytes, move_mac, zero64, EX_MAC_1}, { IPFIX_postDestinationMacAddress, _6bytes, _8bytes, move_mac, zero64, EX_MAC_1}, { IPFIX_vlanId, _2bytes, _2bytes, move16, zero16, EX_VLAN}, { IPFIX_postVlanId, _2bytes, _2bytes, move16, zero16, EX_VLAN}, { IPFIX_flowDirection, _1byte, _1byte, move8, zero8, EX_MULIPLE }, { IPFIX_biflowDirection, _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { IPFIX_flowEndReason, _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { IPFIX_ipNextHopIPv6Address, _16bytes, _16bytes, move128, zero128, EX_NEXT_HOP_v6}, { IPFIX_bgpNextHopIPv6Address, _16bytes, _16bytes, move128, zero128, EX_NEXT_HOP_BGP_v6}, { IPFIX_mplsTopLabelStackSection, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection2, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection3, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection4, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection5, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection6, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection7, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection8, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection9, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_mplsLabelStackSection10, _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { IPFIX_DestinationMacAddress, _6bytes, _8bytes, move_mac, zero64, EX_MAC_2}, { IPFIX_postSourceMacAddress, _6bytes, _8bytes, move_mac, zero64, EX_MAC_2}, { IPFIX_flowStartMilliseconds, _8bytes, _8bytes, Time64Mili, zero64, COMMON_BLOCK}, { IPFIX_flowEndMilliseconds, _8bytes, _8bytes, Time64Mili, zero64, COMMON_BLOCK}, { IPFIX_flowStartSeconds, _4bytes, _4bytes, TimeUnix, zero32, COMMON_BLOCK}, { IPFIX_flowEndSeconds, _4bytes, _4bytes, TimeUnix, zero32, COMMON_BLOCK}, { IPFIX_flowEndMilliseconds, _8bytes, _8bytes, Time64Mili, zero64, COMMON_BLOCK}, { IPFIX_flowStartDeltaMicroseconds, _4bytes, _4bytes, TimeDeltaMicro, zero32, COMMON_BLOCK}, { IPFIX_flowEndDeltaMicroseconds, _4bytes, _4bytes, TimeDeltaMicro, zero32, COMMON_BLOCK}, { IPFIX_SystemInitTimeMiliseconds, _8bytes, _8bytes, SystemInitTime, nop, COMMON_BLOCK}, {0, 0, 0} }; // map for corresponding reverse element, if enterprise ID = IPFIX_ReverseInformationElement static const struct ipfixReverseMap_s { uint16_t ID; // IPFIX element id uint16_t reverseID; // reverse IPFIX element id } ipfixReverseMap[] = { { IPFIX_octetTotalCount, IPFIX_postOctetTotalCount}, { IPFIX_packetTotalCount, IPFIX_postPacketTotalCount}, { IPFIX_octetDeltaCount, IPFIX_postOctetDeltaCount}, { IPFIX_packetDeltaCount, IPFIX_postPacketDeltaCount}, { 0, 0}, }; // cache to be used while parsing a template static struct cache_s { struct element_param_s { uint16_t index; uint16_t found; uint16_t length; } *lookup_info; struct order_s { uint16_t type; uint16_t length; } *input_order; uint32_t input_count; uint32_t max_ipfix_elements; uint32_t *common_extensions; } cache; // module limited globals static int verbose; static uint32_t default_sampling; static uint32_t overwrite_sampling; static uint32_t processed_records; // externals extern uint32_t Max_num_extensions; extern extension_descriptor_t extension_descriptor[]; // prototypes static int HasOptionTable(exporterDomain_t *exporter, uint16_t tableID ); static void InsertSamplerOption(exporterDomain_t *exporter, samplerOption_t *samplerOption); static void InsertSampler(FlowSource_t *fs, exporterDomain_t *exporter, int32_t id, uint16_t mode, uint32_t interval); static input_translation_t *add_translation_table(exporterDomain_t *exporter, uint16_t id); static void remove_translation_table(FlowSource_t *fs, exporterDomain_t *exporter, uint16_t id); static void remove_all_translation_tables(exporterDomain_t *exporter); static exporterDomain_t *GetExporter(FlowSource_t *fs, ipfix_header_t *ipfix_header); static uint32_t MapElement(uint16_t Type, uint16_t Length, uint32_t order, uint32_t EnterpriseNumber); static void PushSequence(input_translation_t *table, uint16_t Type, uint32_t *offset, void *stack); static int compact_input_order(void); static int reorder_sequencer(input_translation_t *table); static void Process_ipfix_templates(exporterDomain_t *exporter, void *flowset_header, uint32_t size_left, FlowSource_t *fs); static void Process_ipfix_template_add(exporterDomain_t *exporter, void *DataPtr, uint32_t size_left, FlowSource_t *fs); static void Process_ipfix_template_withdraw(exporterDomain_t *exporter, void *DataPtr, uint32_t size_left, FlowSource_t *fs); static void Process_ipfix_option_data(exporterDomain_t *exporter, void *data_flowset, FlowSource_t *fs); static void Process_ipfix_data(exporterDomain_t *exporter, uint32_t ExportTime, void *data_flowset, FlowSource_t *fs, input_translation_t *table ); #include "inline.c" #include "nffile_inline.c" int Init_IPFIX(int v, uint32_t sampling, uint32_t overwrite) { int i; verbose = v; default_sampling = sampling; overwrite_sampling = overwrite; cache.lookup_info = (struct element_param_s *)calloc(65536, sizeof(struct element_param_s)); cache.common_extensions = (uint32_t *)malloc((Max_num_extensions+1)*sizeof(uint32_t)); if ( !cache.common_extensions || !cache.lookup_info ) { LogError("Process_ipfix: Panic! malloc(): %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return 0; } // init the helper element table for (i=1; ipfix_element_map[i].id != 0; i++ ) { uint32_t Type = ipfix_element_map[i].id; // multiple same type - save first index only // iterate through same Types afterwards if ( cache.lookup_info[Type].index == 0 ) cache.lookup_info[Type].index = i; } cache.max_ipfix_elements = i; cache.input_order = NULL; LogError("Init IPFIX: Max number of IPFIX tags: %u", cache.max_ipfix_elements); return 1; } // End of Init_IPFIX static int HasOptionTable(exporterDomain_t *exporter, uint16_t tableID ) { samplerOption_t *s; if ( exporter->SysUpOption.length ) return 1; s = exporter->samplerOption; while ( s && s->tableID != tableID ) s = s->next; dbg_printf("Has option table: %s\n", s == NULL ? "not found" : "found"); return s != NULL; } // End of HasOptionTable static exporterDomain_t *GetExporter(FlowSource_t *fs, ipfix_header_t *ipfix_header) { #define IP_STRING_LEN 40 char ipstr[IP_STRING_LEN]; exporterDomain_t **e = (exporterDomain_t **)&(fs->exporter_data); uint32_t ObservationDomain = ntohl(ipfix_header->ObservationDomain); while ( *e ) { if ( (*e)->info.id == ObservationDomain && (*e)->info.version == 10 && (*e)->info.ip.V6[0] == fs->ip.V6[0] && (*e)->info.ip.V6[1] == fs->ip.V6[1]) return *e; e = &((*e)->next); } if ( fs->sa_family == AF_INET ) { uint32_t _ip = htonl(fs->ip.V4); inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr)); } else if ( fs->sa_family == AF_INET6 ) { uint64_t _ip[2]; _ip[0] = htonll(fs->ip.V6[0]); _ip[1] = htonll(fs->ip.V6[1]); inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr)); } else { strncpy(ipstr, "", IP_STRING_LEN); } // nothing found *e = (exporterDomain_t *)malloc(sizeof(exporterDomain_t)); if ( !(*e)) { LogError("Process_ipfix: Panic! malloc() %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return NULL; } memset((void *)(*e), 0, sizeof(exporterDomain_t)); (*e)->info.header.type = ExporterInfoRecordType; (*e)->info.header.size = sizeof(exporter_info_record_t); (*e)->info.id = ObservationDomain; (*e)->info.ip = fs->ip; (*e)->info.sa_family = fs->sa_family; (*e)->info.version = 10; (*e)->info.sysid = 0; (*e)->TemplateRecords = 0; (*e)->DataRecords = 0; (*e)->sequence_failure = 0; (*e)->padding_errors = 0; (*e)->next = NULL; (*e)->sampler = NULL; FlushInfoExporter(fs, &((*e)->info)); dbg_printf("[%u] New exporter: SysID: %u, Observation domain %u from: %s:%u\n", ObservationDomain, (*e)->info.sysid, ObservationDomain, ipstr, fs->port); LogInfo("Process_ipfix: New exporter: SysID: %u, Observation domain %u from: %s\n", (*e)->info.sysid, ObservationDomain, ipstr); return (*e); } // End of GetExporter static uint32_t MapElement(uint16_t Type, uint16_t Length, uint32_t order, uint32_t EnterpriseNumber) { cache.input_order[order].type = SKIP_ELEMENT; cache.input_order[order].length = Length; switch ( EnterpriseNumber ) { case 0: // no Enterprise value break; case 6871: // yaf CERT Coordination Centre dbg_printf(" Skip yaf CERT Coordination Centre\n"); return 0; break; case IPFIX_ReverseInformationElement: for (int i=0; ipfixReverseMap[i].ID != 0; i++ ) { if ( ipfixReverseMap[i].ID == Type ) { Type = ipfixReverseMap[i].reverseID; dbg_printf(" Reverse mapped element type: %u\n", Type); break; } } break; default: dbg_printf(" Skip enterprise id: %u\n", EnterpriseNumber); return 0; } int index = cache.lookup_info[Type].index; if ( index ) { while ( index && ipfix_element_map[index].id == Type ) { if ( Length == ipfix_element_map[index].length ) { cache.input_order[order].type = Type; cache.lookup_info[Type].found = 1; cache.lookup_info[Type].length = Length; cache.lookup_info[Type].index = index; dbg_printf("found extension %u for type: %u, input length: %u output length: %u Extension: %u\n", ipfix_element_map[index].extension, ipfix_element_map[index].id, ipfix_element_map[index].length, ipfix_element_map[index].out_length, ipfix_element_map[index].extension); return ipfix_element_map[index].extension; } index++; } dbg_printf("Skip known element type: %u, Unknown length: %u\n", Type, Length); } else { dbg_printf("Skip unknown element type: %u, Length: %u\n", Type, Length); } return 0; } // End of MapElement static input_translation_t *GetTranslationTable(exporterDomain_t *exporter, uint16_t id) { input_translation_t *table; if ( exporter->current_table && ( exporter->current_table->id == id ) ) return exporter->current_table; table = exporter->input_translation_table; while ( table ) { if ( table->id == id ) { exporter->current_table = table; return table; } table = table->next; } dbg_printf("[%u] Get translation table %u: %s\n", exporter->info.id, id, table == NULL ? "not found" : "found"); exporter->current_table = table; return table; } // End of GetTranslationTable static input_translation_t *add_translation_table(exporterDomain_t *exporter, uint16_t id) { input_translation_t **table; table = &(exporter->input_translation_table); while ( *table ) { table = &((*table)->next); } // Allocate enough space for all potential ipfix tags, which we support // so template refreshing may change the table size without danger of overflowing *table = calloc(1, sizeof(input_translation_t)); if ( !(*table) ) { LogError("Process_ipfix: Panic! calloc() %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return NULL; } (*table)->max_number_of_sequences = 0; (*table)->number_of_sequences = 0; (*table)->sequence = NULL; (*table)->id = id; (*table)->next = NULL; dbg_printf("[%u] Get new translation table %u\n", exporter->info.id, id); return *table; } // End of add_translation_table static void remove_translation_table(FlowSource_t *fs, exporterDomain_t *exporter, uint16_t id) { input_translation_t *table, *parent; LogInfo("Process_ipfix: [%u] Withdraw template id: %i", exporter->info.id, id); parent = NULL; table = exporter->input_translation_table; while ( table && ( table->id != id ) ) { parent = table; table = table->next; } if ( table == NULL ) { LogError("Process_ipfix: [%u] Withdraw template id: %i. translation table not found", exporter->info.id, id); return; } dbg_printf("\n[%u] Withdraw template ID: %u\n", exporter->info.id, table->id); // clear table cache, if this is the table to delete if (exporter->current_table == table) exporter->current_table = NULL; if ( parent ) { // remove table from list parent->next = table->next; } else { // last table removed exporter->input_translation_table = NULL; } RemoveExtensionMap(fs, table->extension_info.map); free(table->sequence); free(table->extension_info.map); free(table); } // End of remove_translation_table static void remove_all_translation_tables(exporterDomain_t *exporter) { input_translation_t *table, *next; LogInfo("Process_ipfix: Withdraw all templates from observation domain %u\n", exporter->info.id); table = exporter->input_translation_table; while ( table ) { next = table->next; dbg_printf("\n[%u] Withdraw template ID: %u\n", exporter->info.id, table->id); free(table->sequence); free(table->extension_info.map); free(table); table = next; } // clear references exporter->input_translation_table = NULL; exporter->current_table = NULL; } // End of remove_all_translation_tables static int CheckSequenceMap(input_translation_t *table) { if ( table->number_of_sequences < table->max_number_of_sequences ) return 1; dbg_printf("Extend sequence map %u -> ", table->max_number_of_sequences); void *p = realloc(table->sequence, (table->max_number_of_sequences + 1) * sizeof(sequence_map_t)); if ( !p ) { LogError("Process_ipfix: realloc() at %s line %d: %s", __FILE__, __LINE__, strerror (errno)); dbg_printf("\nProcess_ipfix: realloc() at %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return 0; } table->sequence = p; table->max_number_of_sequences += 1; dbg_printf("%u\n", table->max_number_of_sequences); return 1; } // End of CheckSequenceMap static void PushSequence(input_translation_t *table, uint16_t Type, uint32_t *offset, void *stack) { uint32_t i = table->number_of_sequences; uint32_t index = cache.lookup_info[Type].index; if ( !CheckSequenceMap(table) ) return; table->sequence[i].skip_count = 0; table->sequence[i].type = Type; if ( cache.lookup_info[Type].found ) { table->sequence[i].id = ipfix_element_map[index].sequence; table->sequence[i].output_offset = offset ? *offset : 0; table->sequence[i].stack = stack; table->sequence[i].input_length = cache.lookup_info[Type].length; } else { table->sequence[i].id = ipfix_element_map[index].zero_sequence; table->sequence[i].output_offset = offset ? *offset : 0; table->sequence[i].stack = NULL; } dbg_printf("Push: sequence: %u, Type: %u, in length: %u, out length: %u, id: %u, out offset: %u\n", i, Type, ipfix_element_map[index].length, ipfix_element_map[index].out_length, table->sequence[i].id, table->sequence[i].output_offset); table->number_of_sequences++; if ( offset ) (*offset) += ipfix_element_map[index].out_length; } // End of PushSequence static int compact_input_order(void) { int i; dbg_printf("\nCompacting element input order: %u elements\n", cache.input_count); for ( i=0; i< cache.input_count; i++ ) { dbg_printf("%i: type: %u, length: %u\n", i, cache.input_order[i].type, cache.input_order[i].length); if ( (cache.input_order[i].type == SKIP_ELEMENT) && (cache.input_order[i].length == DYN_FIELD_LENGTH) ) { dbg_printf("Dynamic length field: %u\n", cache.input_order[i].type); continue; } while ( (i+1) < cache.input_count && (cache.input_order[i].type == SKIP_ELEMENT) && (cache.input_order[i].length != DYN_FIELD_LENGTH) && (cache.input_order[i+1].type == SKIP_ELEMENT) && (cache.input_order[i+1].length != DYN_FIELD_LENGTH)) { // merge multiple skipped fix length elements into 1 hole sequence int j; dbg_printf("%i: type: %u, length: %u\n", i+1, cache.input_order[i+1].type, cache.input_order[i+1].length); dbg_printf("Merge order %u and %u\n", i, i+1); // type set 0 to mark skip sequence cache.input_order[i].type = SKIP_ELEMENT; cache.input_order[i].length += cache.input_order[i+1].length; // shift up lower entries - fill for ( j=i+1; (j+1)sequence; #ifdef DEVEL printf("\nReorder Sequencer. Sequence steps: %u\n", table->number_of_sequences); for ( i=0; inumber_of_sequences; i++ ) { printf("Order: %i, Sequence: %u, Type: %u, Input length: %u, Output offset: %u, Skip Count: %u\n", i, sequence[i].id, sequence[i].type, sequence[i].input_length, sequence[i].output_offset, sequence[i].skip_count); } #endif n = 0; // index into sequencer table // reorder Sequencer steps, so they follow input order // insert skip counts if needed for ( i=0; inumber_of_sequences-1; j>=n; j-- ) { sequence[j+1] = sequence[j]; } // slot n now available - create nop sequence with skip count if ( cache.input_order[i].length == DYN_FIELD_LENGTH ) { sequence[n].id = dyn_skip; sequence[n].skip_count = 0; } else { sequence[n].id = nop; sequence[n].skip_count = cache.input_order[i].length; } sequence[n].type = SKIP_ELEMENT; sequence[n].input_length = 0; sequence[n].stack = NULL; table->number_of_sequences++; dbg_printf("Insert skip sequence in slot: %u, skip count: %u, dyn: %u\n", n, sequence[n].skip_count, cache.input_order[i].length == DYN_FIELD_LENGTH ? 1 : 0); } else { sequence[n-1].skip_count += cache.input_order[i].length; dbg_printf("Merge skip count: %u into previous sequence: %u\n", cache.input_order[i].length, n-1); continue; } } else { // reorder input element if needed if ( sequence[n].type != cache.input_order[i].type ) { sequence_map_t _s; int j = n+1; while ( sequence[j].type != cache.input_order[i].type && j < table->number_of_sequences ) j++; // must never happen! if ( j == table->number_of_sequences ) { // skip this field if ( n == 0 ) { // XXX fix this return 0; } else { sequence[n-1].skip_count += cache.input_order[i].length; dbg_printf("Merge skip count: %u into previous sequence: %u\n", cache.input_order[i].length, n-1); continue; } return 0; } // swap slots _s = sequence[n]; sequence[n] = sequence[j]; sequence[j] = _s; dbg_printf("Swap slots %u <-> %u\n", n, j); } else { dbg_printf("In order slot %u\n", n); } } // advance sequence slot n++; } #ifdef DEVEL printf("\nReordered Sequencer. Sequence steps: %u\n", table->number_of_sequences); for ( int i=0; inumber_of_sequences; i++ ) { printf("Order: %i, Sequence: %u, Type: %u, Input length: %u, Output offset: %u, Skip Count: %u\n", i, sequence[i].id, sequence[i].type, sequence[i].input_length, sequence[i].output_offset, sequence[i].skip_count); } printf("\n"); #endif return 1; } // End of reorder_sequencer static input_translation_t *setup_translation_table (exporterDomain_t *exporter, uint16_t id) { input_translation_t *table; extension_map_t *extension_map; uint32_t i, ipv6, offset, next_extension; size_t size_required; ipv6 = 0; table = GetTranslationTable(exporter, id); if ( !table ) { LogInfo("Process_ipfix: [%u] Add template %u", exporter->info.id, id); table = add_translation_table(exporter, id); if ( !table ) { return NULL; } // Add an extension map // The number of extensions for this template is currently unknown // Allocate enough space for all configured extensions - some may be unused later // make sure memory is 4byte alligned size_required = Max_num_extensions * sizeof(uint16_t) + sizeof(extension_map_t); size_required = (size_required + 3) &~(size_t)3; extension_map = malloc(size_required); if ( !extension_map ) { LogError("Process_ipfix: Panic! malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return NULL; } extension_map->type = ExtensionMapType; // Set size to an empty table - will be adapted later extension_map->size = sizeof(extension_map_t); extension_map->map_id = INIT_ID; // packed record size still unknown at this point - will be added later extension_map->extension_size = 0; table->extension_info.map = extension_map; table->extension_map_changed = 1; table->number_of_sequences = 0; } else { extension_map = table->extension_info.map; // reset size/extension size - it's refreshed automatically extension_map->size = sizeof(extension_map_t); extension_map->extension_size = 0; free((void *)table->sequence); dbg_printf("[%u] Refresh template %u\n", exporter->info.id, id); } // Add new sequence table->sequence = calloc(cache.max_ipfix_elements, sizeof(sequence_map_t)); if ( !table->sequence ) { LogError("Process_ipfix: Panic! malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return NULL; } table->number_of_sequences = 0; table->max_number_of_sequences = cache.max_ipfix_elements; table->updated = time(NULL); // IPFIX only has 64bit counters table->flags = 0; SetFlag(table->flags, FLAG_PKG_64); SetFlag(table->flags, FLAG_BYTES_64); table->delta_time = 0; table->icmpTypeCode = 0; table->router_ip_offset = 0; table->received_offset = 0; dbg_printf("[%u] Build sequence table %u\n", exporter->info.id, id); // fill table table->id = id; /* * common data block: The common record is expected in the output stream. If not available * in the template, fill values with 0 */ // All required extensions // The order we Push all ipfix elements, must corresponde to the structure of the common record // followed by all available extension in the extension map offset = BYTE_OFFSET_first; if ( cache.lookup_info[IPFIX_flowStartDeltaMicroseconds].found ) { PushSequence( table, IPFIX_flowStartDeltaMicroseconds, NULL, &table->flow_start); PushSequence( table, IPFIX_flowEndDeltaMicroseconds, NULL, &table->flow_end); offset = BYTE_OFFSET_first + 8; table->delta_time = 1; dbg_printf("Time stamp: flow start/end delta microseconds: %u/%u\n", IPFIX_flowStartDeltaMicroseconds, IPFIX_flowEndDeltaMicroseconds); } else if ( cache.lookup_info[IPFIX_flowStartMilliseconds].found ) { PushSequence( table, IPFIX_flowStartMilliseconds, NULL, &table->flow_start); PushSequence( table, IPFIX_flowEndMilliseconds, NULL, &table->flow_end); offset = BYTE_OFFSET_first + 8; dbg_printf("Time stamp: flow start/end absolute milliseconds: %u/%u\n", IPFIX_flowStartMilliseconds, IPFIX_flowEndMilliseconds); } else if ( cache.lookup_info[IPFIX_flowStartSysUpTime].found ) { PushSequence( table, IPFIX_flowStartSysUpTime, NULL, &table->flow_start); PushSequence( table, IPFIX_flowEndSysUpTime, NULL, &table->flow_end); PushSequence( table, IPFIX_SystemInitTimeMiliseconds, NULL, &table->SysUpTime); offset = BYTE_OFFSET_first + 8; dbg_printf("Time stamp: flow start/end relative milliseconds: %u/%u\n", IPFIX_flowStartSysUpTime, IPFIX_flowEndSysUpTime); } else if ( cache.lookup_info[IPFIX_flowStartSeconds].found ) { PushSequence( table, IPFIX_flowStartSeconds, NULL, &table->flow_start); PushSequence( table, IPFIX_flowEndSeconds, NULL, &table->flow_end); offset = BYTE_OFFSET_first + 8; dbg_printf("Time stamp: flow start/end absolute seconds: %u/%u\n", IPFIX_flowStartSeconds, IPFIX_flowEndSeconds); }else { dbg_printf("Time stamp: No known format found\n"); offset = BYTE_OFFSET_first + 8; } PushSequence( table, IPFIX_forwardingStatus, &offset, NULL); PushSequence( table, IPFIX_tcpControlBits, &offset, NULL); PushSequence( table, IPFIX_protocolIdentifier, &offset, NULL); PushSequence( table, IPFIX_ipClassOfService, &offset, NULL); PushSequence( table, IPFIX_SourceTransportPort, &offset, NULL); PushSequence( table, IPFIX_DestinationTransportPort, &offset, NULL); // skip exporter_sysid offset += 2; PushSequence( table, IPFIX_biflowDirection, &offset, NULL); PushSequence( table, IPFIX_flowEndReason, &offset, NULL); /* IP address record * This record is expected in the output stream. If not available * in the template, assume empty v4 address. */ if ( cache.lookup_info[IPFIX_SourceIPv4Address].found ) { // IPv4 addresses PushSequence( table, IPFIX_SourceIPv4Address, &offset, NULL); PushSequence( table, IPFIX_DestinationIPv4Address, &offset, NULL); } else if ( cache.lookup_info[IPFIX_SourceIPv6Address].found ) { // IPv6 addresses PushSequence( table, IPFIX_SourceIPv6Address, &offset, NULL); PushSequence( table, IPFIX_DestinationIPv6Address, &offset, NULL); // mark IPv6 SetFlag(table->flags, FLAG_IPV6_ADDR); ipv6 = 1; } else { // should not happen, assume empty IPv4 addresses, zero PushSequence( table, IPFIX_SourceIPv4Address, &offset, NULL); PushSequence( table, IPFIX_DestinationIPv4Address, &offset, NULL); } // decide between Delta or Total counters - prefer Total if available if ( cache.lookup_info[IPFIX_packetTotalCount].found ) PushSequence( table, IPFIX_packetTotalCount, &offset, &table->packets); else PushSequence( table, IPFIX_packetDeltaCount, &offset, &table->packets); SetFlag(table->flags, FLAG_PKG_64); if ( cache.lookup_info[IPFIX_octetTotalCount].found ) PushSequence( table, IPFIX_octetTotalCount, &offset, &table->bytes); else PushSequence( table, IPFIX_octetDeltaCount, &offset, &table->bytes); SetFlag(table->flags, FLAG_BYTES_64); // Optional extensions next_extension = 0; for (i=4; extension_descriptor[i].id; i++ ) { uint32_t map_index = i; if ( cache.common_extensions[i] == 0 ) continue; switch(i) { case EX_IO_SNMP_2: PushSequence( table, IPFIX_ingressInterface, &offset, NULL); PushSequence( table, IPFIX_egressInterface, &offset, NULL); break; case EX_IO_SNMP_4: PushSequence( table, IPFIX_ingressInterface, &offset, NULL); PushSequence( table, IPFIX_egressInterface, &offset, NULL); break; case EX_AS_2: PushSequence( table, IPFIX_bgpSourceAsNumber, &offset, NULL); PushSequence( table, IPFIX_bgpDestinationAsNumber, &offset, NULL); break; case EX_AS_4: PushSequence( table, IPFIX_bgpSourceAsNumber, &offset, NULL); PushSequence( table, IPFIX_bgpDestinationAsNumber, &offset, NULL); break; case EX_MULIPLE: PushSequence( table, IPFIX_postIpClassOfService, &offset, NULL); PushSequence( table, IPFIX_flowDirection, &offset, NULL); if ( ipv6 ) { // IPv6 PushSequence( table, IPFIX_SourceIPv6PrefixLength, &offset, NULL); PushSequence( table, IPFIX_DestinationIPv6PrefixLength, &offset, NULL); } else { // IPv4 PushSequence( table, IPFIX_SourceIPv4PrefixLength, &offset, NULL); PushSequence( table, IPFIX_DestinationIPv4PrefixLength, &offset, NULL); } break; case EX_NEXT_HOP_v4: PushSequence( table, IPFIX_ipNextHopIPv4Address, &offset, NULL); break; case EX_NEXT_HOP_v6: PushSequence( table, IPFIX_ipNextHopIPv6Address, &offset, NULL); SetFlag(table->flags, FLAG_IPV6_NH); break; case EX_NEXT_HOP_BGP_v4: PushSequence( table, IPFIX_bgpNextHopIPv4Address, &offset, NULL); break; case EX_NEXT_HOP_BGP_v6: PushSequence( table, IPFIX_bgpNextHopIPv6Address, &offset, NULL); SetFlag(table->flags, FLAG_IPV6_NHB); break; case EX_VLAN: PushSequence( table, IPFIX_vlanId, &offset, NULL); PushSequence( table, IPFIX_postVlanId, &offset, NULL); break; case EX_OUT_PKG_4: PushSequence( table, IPFIX_postPacketDeltaCount, &offset, NULL); break; case EX_OUT_PKG_8: PushSequence( table, IPFIX_postPacketDeltaCount, &offset, NULL); break; case EX_OUT_BYTES_4: PushSequence( table, IPFIX_postOctetDeltaCount, &offset, NULL); break; case EX_OUT_BYTES_8: PushSequence( table, IPFIX_postOctetDeltaCount, &offset, NULL); break; case EX_AGGR_FLOWS_8: break; case EX_MAC_1: PushSequence( table, IPFIX_SourceMacAddress, &offset, NULL); PushSequence( table, IPFIX_postDestinationMacAddress, &offset, NULL); break; case EX_MAC_2: PushSequence( table, IPFIX_DestinationMacAddress, &offset, NULL); PushSequence( table, IPFIX_postSourceMacAddress, &offset, NULL); break; case EX_MPLS: PushSequence( table, IPFIX_mplsTopLabelStackSection, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection2, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection3, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection4, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection5, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection6, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection7, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection8, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection9, &offset, NULL); PushSequence( table, IPFIX_mplsLabelStackSection10, &offset, NULL); break; case EX_ROUTER_IP_v4: case EX_ROUTER_IP_v6: if ( exporter->info.sa_family == PF_INET6 ) { table->router_ip_offset = offset; dbg_printf("Router IPv6: offset: %u, olen: %u\n", offset, 16 ); // not an entry for the translateion table. // but reserve space in the output record for IPv6 offset += 16; SetFlag(table->flags, FLAG_IPV6_EXP); map_index = EX_ROUTER_IP_v6; } else { table->router_ip_offset = offset; dbg_printf("Router IPv4: offset: %u, olen: %u\n", offset, 4 ); // not an entry for the translateion table. // but reserve space in the output record for IPv4 offset += 4; ClearFlag(table->flags, FLAG_IPV6_EXP); map_index = EX_ROUTER_IP_v4; } break; case EX_ROUTER_ID: // no value in ipfix break; case EX_RECEIVED: table->received_offset = offset; dbg_printf("Received offset: %u\n", offset); offset += 8; break; } extension_map->size += sizeof(uint16_t); extension_map->extension_size += extension_descriptor[map_index].size; // found extension in map_index must be the same as in map - otherwise map is dirty if ( extension_map->ex_id[next_extension] != map_index ) { // dirty map - needs to be refreshed in output stream extension_map->ex_id[next_extension] = map_index; table->extension_map_changed = 1; } next_extension++; } extension_map->ex_id[next_extension++] = 0; // make sure map is aligned if ( extension_map->size & 0x3 ) { extension_map->ex_id[next_extension] = 0; extension_map->size = ( extension_map->size + 3 ) &~ 0x3; } table->output_record_size = offset; // for netflow historical reason, ICMP type/code goes into dst port field // remember offset, for decoding if ( cache.lookup_info[IPFIX_icmpTypeCodeIPv4].found && cache.lookup_info[IPFIX_icmpTypeCodeIPv4].length == 2 ) { PushSequence( table, IPFIX_icmpTypeCodeIPv4, NULL, &table->icmpTypeCode); } if ( cache.lookup_info[IPFIX_icmpTypeCodeIPv6].found && cache.lookup_info[IPFIX_icmpTypeCodeIPv6].length == 2 ) { PushSequence( table, IPFIX_icmpTypeCodeIPv6, NULL, &table->icmpTypeCode); } #ifdef DEVEL if ( table->extension_map_changed ) { printf("Extension Map id=%u changed!\n", extension_map->map_id); } else { printf("[%u] template %u unchanged\n", exporter->info.id, id); } printf("Process_ipfix: Check extension map: id: %d, size: %u, extension_size: %u\n", extension_map->map_id, extension_map->size, extension_map->extension_size); { int i; for (i=0; inumber_of_sequences; i++ ) { printf("Sequence %i: id: %u, Type: %u, Length: %u, Output offset: %u, stack: %llu\n", i, table->sequence[i].id, table->sequence[i].type, table->sequence[i].input_length, table->sequence[i].output_offset, (unsigned long long)table->sequence[i].stack); } printf("Flags: 0x%x output record size: %u\n", table->flags, table->output_record_size); } PrintExtensionMap(extension_map); #endif return table; } // End of setup_translation_table static void InsertSampler(FlowSource_t *fs, exporterDomain_t *exporter, int32_t id, uint16_t mode, uint32_t interval) { sampler_t *sampler; dbg_printf("[%u] Insert Sampler: Exporter is 0x%llu\n", exporter->info.id, (long long unsigned)exporter); if ( !exporter->sampler ) { // no samplers so far sampler = (sampler_t *)malloc(sizeof(sampler_t)); if ( !sampler ) { LogError( "Process_ipfix: Panic! malloc(): %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return; } sampler->info.header.type = SamplerInfoRecordype; sampler->info.header.size = sizeof(sampler_info_record_t); sampler->info.exporter_sysid = exporter->info.sysid; sampler->info.id = id; sampler->info.mode = mode; sampler->info.interval = interval; sampler->next = NULL; exporter->sampler = sampler; FlushInfoSampler(fs, &(sampler->info)); LogInfo( "Add new sampler: ID: %i, mode: %u, interval: %u\n", id, mode, interval); dbg_printf("Add new sampler: ID: %i, mode: %u, interval: %u\n", id, mode, interval); } else { sampler = exporter->sampler; while ( sampler ) { // test for update of existing sampler if ( sampler->info.id == id ) { // found same sampler id - update record dbg_printf("Update existing sampler id: %i, mode: %u, interval: %u\n", id, mode, interval); // we update only on changes if ( mode != sampler->info.mode || interval != sampler->info.interval ) { FlushInfoSampler(fs, &(sampler->info)); sampler->info.mode = mode; sampler->info.interval = interval; LogInfo( "Update existing sampler id: %i, mode: %u, interval: %u\n", id, mode, interval); } else { dbg_printf("Sampler unchanged!\n"); } break; } // test for end of chain if ( sampler->next == NULL ) { // end of sampler chain - insert new sampler sampler->next = (sampler_t *)malloc(sizeof(sampler_t)); if ( !sampler->next ) { LogError( "Process_ipfix: Panic! malloc(): %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return; } sampler = sampler->next; sampler->info.header.type = SamplerInfoRecordype; sampler->info.header.size = sizeof(sampler_info_record_t); sampler->info.exporter_sysid = exporter->info.sysid; sampler->info.id = id; sampler->info.mode = mode; sampler->info.interval = interval; sampler->next = NULL; FlushInfoSampler(fs, &(sampler->info)); LogInfo( "Append new sampler: ID: %u, mode: %u, interval: %u\n", id, mode, interval); dbg_printf("Append new sampler: ID: %u, mode: %u, interval: %u\n", id, mode, interval); break; } // advance sampler = sampler->next; } } } // End of InsertSampler static void InsertSamplerOption(exporterDomain_t *exporter, samplerOption_t *samplerOption) { samplerOption_t *s, *parent; parent = NULL; s = exporter->samplerOption; while (s) { if ( s->tableID == samplerOption->tableID ) { // table already known to us - update data dbg_printf("Found existing sampling info in template %i\n", samplerOption->tableID); break; } parent = s; s = s->next; } if ( s != NULL ) { // existing entry // replace existing table dbg_printf("Replace existing sampler table ID %i\n", samplerOption->tableID); if ( parent ) { parent->next = samplerOption; } else { exporter->samplerOption = samplerOption; } samplerOption->next = s->next; free(s); s = NULL; } else { // new entry dbg_printf("New sampling table ID %i\n", samplerOption->tableID); // push new sampling table samplerOption->next = exporter->samplerOption; exporter->samplerOption = samplerOption; } dbg_printf("Update/Insert sampler table id: %u flags: 0x%x - sampler ID: %u/%u, mode: %u/%u, interval: %u/%u\n", samplerOption->tableID, samplerOption->flags, samplerOption->id.offset, samplerOption->id.length, samplerOption->mode.offset, samplerOption->mode.length, samplerOption->interval.offset, samplerOption->interval.length); } // End of InsertSamplerOption static void Process_ipfix_templates(exporterDomain_t *exporter, void *flowset_header, uint32_t size_left, FlowSource_t *fs) { ipfix_template_record_t *ipfix_template_record; void *DataPtr; uint32_t count; size_left -= 4; // subtract message header DataPtr = flowset_header + 4; ipfix_template_record = (ipfix_template_record_t *)DataPtr; // uint32_t id = ntohs(ipfix_template_record->TemplateID); count = ntohs(ipfix_template_record->FieldCount); if ( count == 0 ) { // withdraw template Process_ipfix_template_withdraw(exporter, DataPtr, size_left, fs); } else { // refresh/add templates Process_ipfix_template_add(exporter, DataPtr, size_left, fs); } } // End of Process_ipfix_templates static void Process_ipfix_template_add(exporterDomain_t *exporter, void *DataPtr, uint32_t size_left, FlowSource_t *fs) { input_translation_t *translation_table; ipfix_template_record_t *ipfix_template_record; ipfix_template_elements_std_t *NextElement; int i; // a template flowset can contain multiple records ( templates ) while ( size_left ) { uint32_t table_id, count, size_required; uint32_t num_extensions = 0; if ( size_left < 4 ) { LogError("Process_ipfix [%u] Template size error at %s line %u" , exporter->info.id, __FILE__, __LINE__, strerror (errno)); size_left = 0; continue; } // map next record. ipfix_template_record = (ipfix_template_record_t *)DataPtr; size_left -= 4; table_id = ntohs(ipfix_template_record->TemplateID); count = ntohs(ipfix_template_record->FieldCount); dbg_printf("\n[%u] Template ID: %u\n", exporter->info.id, table_id); dbg_printf("FieldCount: %u buffersize: %u\n", count, size_left); // prepare // clear helper tables memset((void *)cache.common_extensions, 0, (Max_num_extensions+1)*sizeof(uint32_t)); memset((void *)cache.lookup_info, 0, 65536 * sizeof(struct element_param_s)); for (i=1; ipfix_element_map[i].id != 0; i++ ) { uint32_t Type = ipfix_element_map[i].id; if ( ipfix_element_map[i].id == ipfix_element_map[i-1].id ) continue; cache.lookup_info[Type].index = i; // other elements cleard be memset } cache.input_order = calloc(count, sizeof(struct order_s)); if ( !cache.input_order ) { LogError("Process_ipfix: Panic! malloc(): %s line %d: %s", __FILE__, __LINE__, strerror (errno)); size_left = 0; continue; } cache.input_count = count; // assume all elements in template are std elements. correct this value, if we find an enterprise element size_required = 4*count; if ( size_left < size_required ) { // if we fail this check, this flowset must be skipped. LogError("Process_ipfix: [%u] Not enough data for template elements! required: %i, left: %u", exporter->info.id, size_required, size_left); dbg_printf("ERROR: Not enough data for template elements! required: %i, left: %u", size_required, size_left); return; } // process all elements in this record NextElement = (ipfix_template_elements_std_t *)ipfix_template_record->elements; for ( i=0; iType); Length = ntohs(NextElement->Length); Enterprise = Type & 0x8000 ? 1 : 0; Type = Type & 0x7FFF; uint32_t EnterpriseNumber = 0; if ( Enterprise ) { ipfix_template_elements_e_t *e = (ipfix_template_elements_e_t *)NextElement; size_required += 4; // ad 4 for enterprise value if ( size_left < size_required ) { LogError("Process_ipfix: [%u] Not enough data for template elements! required: %i, left: %u", exporter->info.id, size_required, size_left); dbg_printf("ERROR: Not enough data for template elements! required: %i, left: %u", size_required, size_left); return; } EnterpriseNumber = ntohl(e->EnterpriseNumber); if ( EnterpriseNumber == IPFIX_ReverseInformationElement ) { dbg_printf(" [%i] Enterprise: 1, Type: %u, Length %u Reverse Information Element: %u\n", i, Type, Length, ntohl(e->EnterpriseNumber)); } else { dbg_printf(" [%i] Enterprise: 1, Type: %u, Length %u EnterpriseNumber: %u\n", i, Type, Length, ntohl(e->EnterpriseNumber)); } e++; NextElement = (ipfix_template_elements_std_t *)e; } else { dbg_printf(" [%i] Enterprise: 0, Type: %u, Length %u\n", i, Type, Length); NextElement++; } // do we store this extension? enabled != 0 // more than 1 ipfix tag may map to an extension - so count this extension once only uint32_t ext_id = MapElement(Type, Length, i, EnterpriseNumber); if ( ext_id && extension_descriptor[ext_id].enabled ) { if ( cache.common_extensions[ext_id] == 0 ) { cache.common_extensions[ext_id] = 1; num_extensions++; } } } dbg_printf("Processed: %u, num_extensions found: %u\n", size_required, num_extensions); // compact input order and reorder sequencer if ( num_extensions && compact_input_order() ) { // valid template with common inout fields // as the router IP address extension is not part announced in a template, we need to deal with it here if ( extension_descriptor[EX_ROUTER_IP_v4].enabled ) { if ( cache.common_extensions[EX_ROUTER_IP_v4] == 0 ) { cache.common_extensions[EX_ROUTER_IP_v4] = 1; num_extensions++; } dbg_printf("Add sending router IP address (%s) => Extension: %u\n", fs->sa_family == PF_INET6 ? "ipv6" : "ipv4", EX_ROUTER_IP_v4); } // XXX for now, we do not store router ID in IPFIX extension_descriptor[EX_ROUTER_ID].enabled = 0; /* // as the router IP address extension is not part announced in a template, we need to deal with it here if ( extension_descriptor[EX_ROUTER_ID].enabled ) { if ( cache.common_extensions[EX_ROUTER_ID] == 0 ) { cache.common_extensions[EX_ROUTER_ID] = 1; num_extensions++; } dbg_printf("Force add router ID (engine type/ID), Extension: %u\n", EX_ROUTER_ID); } */ // received time if ( extension_descriptor[EX_RECEIVED].enabled ) { if ( cache.common_extensions[EX_RECEIVED] == 0 ) { cache.common_extensions[EX_RECEIVED] = 1; num_extensions++; } dbg_printf("Force add packet received time, Extension: %u\n", EX_RECEIVED); } #ifdef DEVEL { int i; for (i=4; extension_descriptor[i].id; i++ ) { if ( cache.common_extensions[i] ) { printf("Enabled extension: %i\n", i); } } } #endif translation_table = setup_translation_table(exporter, table_id); if (translation_table->extension_map_changed ) { // refresh he map in the ouput buffer dbg_printf("Translation Table changed! Add extension map ID: %i\n", translation_table->extension_info.map->map_id); AddExtensionMap(fs, translation_table->extension_info.map); translation_table->extension_map_changed = 0; dbg_printf("Translation Table added! map ID: %i\n", translation_table->extension_info.map->map_id); } if ( !reorder_sequencer(translation_table) ) { LogError("Process_ipfix: [%u] Failed to reorder sequencer. Remove table id: %u", exporter->info.id, table_id); remove_translation_table(fs, exporter, table_id); } } else { dbg_printf("Template does not contain any common fields - skip\n"); } // update size left of this flowset size_left -= size_required; DataPtr = DataPtr + size_required+4; // +4 for header if ( size_left < 4 ) { // pading dbg_printf("Skip %u bytes padding\n", size_left); size_left = 0; } free(cache.input_order); cache.input_order = NULL; } } // End of Process_ipfix_template_add static void Process_ipfix_template_withdraw(exporterDomain_t *exporter, void *DataPtr, uint32_t size_left, FlowSource_t *fs) { ipfix_template_record_t *ipfix_template_record; // a template flowset can contain multiple records ( templates ) while ( size_left ) { uint32_t id; if ( size_left < 4 ) { LogError("Process_ipfix [%u] Template withdraw size error at %s line %u" , exporter->info.id, __FILE__, __LINE__, strerror (errno)); size_left = 0; continue; } // map next record. ipfix_template_record = (ipfix_template_record_t *)DataPtr; size_left -= 4; id = ntohs(ipfix_template_record->TemplateID); // count = ntohs(ipfix_template_record->FieldCount); if ( id == IPFIX_TEMPLATE_FLOWSET_ID ) { // withdraw all templates remove_all_translation_tables(exporter); ReInitExtensionMapList(fs); } else { remove_translation_table(fs, exporter, id); } DataPtr = DataPtr + 4; if ( size_left < 4 ) { // pading dbg_printf("Skip %u bytes padding\n", size_left); size_left = 0; } } } // End of Process_ipfix_template_withdraw static void Process_ipfix_option_templates(exporterDomain_t *exporter, void *option_template_flowset, FlowSource_t *fs) { uint8_t *option_template; uint32_t size_left, size_required; // uint32_t nr_scopes, nr_options; uint16_t tableID, field_count, scope_field_count, offset; samplerOption_t *samplerOption; size_left = GET_FLOWSET_LENGTH(option_template_flowset) - 4; // -4 for flowset header -> id and length if ( size_left < 6 ) { LogError("Process_ipfix: [%u] option template length error: size left %u too small for an options template", exporter->info.id, size_left); return; } option_template = option_template_flowset + 4; tableID = GET_OPTION_TEMPLATE_ID(option_template); field_count = GET_OPTION_TEMPLATE_FIELD_COUNT(option_template); scope_field_count = GET_OPTION_TEMPLATE_SCOPE_FIELD_COUNT(option_template); option_template += 6; size_left -= 6; dbg_printf("Decode Option Template. tableID: %u, field count: %u, scope field count: %u\n", tableID, field_count, scope_field_count); if ( scope_field_count == 0 ) { LogError("Process_ipfx: [%u] scope field count error: length must not be zero", exporter->info.id); dbg_printf("scope field count error: length must not be zero\n"); return; } size_required = 2 * field_count * sizeof(uint16_t); dbg_printf("Size left: %u, size required: %u\n", size_left, size_required); if ( size_left < size_required ) { LogError("Process_ipfix: [%u] option template length error: size left %u too small for %u scopes length and %u options length", exporter->info.id, size_left, field_count, scope_field_count); dbg_printf("option template length error: size left %u too small for field_count %u\n", size_left, field_count); return; } if ( scope_field_count == 0 ) { LogError("Process_ipfxi: [%u] scope field count error: length must not be zero", exporter->info.id); return; } samplerOption = (samplerOption_t *)malloc(sizeof(samplerOption_t)); if ( !samplerOption ) { LogError("Error malloc(): %s in %s:%d", strerror (errno), __FILE__, __LINE__); return; } memset((void *)samplerOption, 0, sizeof(samplerOption_t)); samplerOption->tableID = tableID; int i; offset = 0; for ( i=0; iinfo.id, __FILE__, __LINE__, strerror (errno)); return; } id = Get_val16(option_template); option_template += 2; length = Get_val16(option_template); option_template += 2; size_left -= 4; Enterprise = id & 0x8000 ? 1 : 0; if ( Enterprise ) { size_required += 4; if ( size_left < 4 ) { LogError("Process_ipfix: [%u] option template length error: size left %u too small", exporter->info.id, size_left); dbg_printf("option template length error: size left %u too small\n", size_left); return; } option_template += 4; size_left -= 4; dbg_printf(" [%i] Enterprise: 1, scope id: %u, scope length %u enterprise value: %u\n", i, id, length, Get_val32(option_template)); } else { dbg_printf(" [%i] Enterprise: 0, scope id: %u, scope length %u\n", i, id, length); } offset += length; } for ( ;iinfo.id, size_left); dbg_printf("option template length error: size left %u too small\n", size_left); return; } enterprise_value = Get_val32(option_template); option_template += 4; size_left -= 4; dbg_printf(" [%i] Enterprise: 1, option type: %u, option length %u enterprise value: %u\n", i, type, length, enterprise_value); } else { dbg_printf(" [%i] Enterprise: 0, option type: %u, option length %u\n", i, type, length); } switch (type) { // general sampling case IPFIX_samplingInterval: // #34 samplerOption->interval.length = length; samplerOption->interval.offset = offset; SetFlag(samplerOption->flags, STDSAMPLING34); dbg_printf("Std Sampling found. length: %u, offset: %u\n", length, offset); break; case IPFIX_samplingAlgorithm: // #35 samplerOption->mode.length = length; samplerOption->mode.offset = offset; SetFlag(samplerOption->flags, STDSAMPLING35); break; // individual samplers case IPFIX_samplerId: // #48 depricated - fall through case IPFIX_selectorId: // #302 samplerOption->id.length = length; samplerOption->id.offset = offset; SetFlag(samplerOption->flags, SAMPLER302); break; case IPFIX_samplerMode: // #49 depricated - fall through case IPFIX_selectorAlgorithm: // #304 samplerOption->mode.length = length; samplerOption->mode.offset = offset; SetFlag(samplerOption->flags, SAMPLER304); break; case IPFIX_samplerRandomInterval: // #50 depricated - fall through case IPFIX_samplingPacketInterval: // #305 samplerOption->interval.length = length; samplerOption->interval.offset = offset; SetFlag(samplerOption->flags, SAMPLER305); break; // SysUpTime information case IPFIX_SystemInitTimeMiliseconds: exporter->SysUpOption.length = length; exporter->SysUpOption.offset = offset; break; } offset += length; } if ( (samplerOption->flags & SAMPLERMASK ) != 0) { dbg_printf("[%u] Sampler information found\n", exporter->info.id); InsertSamplerOption(exporter, samplerOption); } else if ( (samplerOption->flags & STDMASK ) != 0) { dbg_printf("[%u] Std sampling information found\n", exporter->info.id); InsertSamplerOption(exporter, samplerOption); } else { free(samplerOption); dbg_printf("[%u] No Sampling information found\n", exporter->info.id); } if ( exporter->SysUpOption.length ) { dbg_printf("[%u] SysupTime information found, offset: %u\n", exporter->info.id, exporter->SysUpOption.offset); } processed_records++; } // End of Process_ipfix_option_templates static void Process_ipfix_data(exporterDomain_t *exporter, uint32_t ExportTime, void *data_flowset, FlowSource_t *fs, input_translation_t *table ){ uint64_t sampling_rate; uint32_t size_left; uint8_t *in, *out; int i; char *string; size_left = GET_FLOWSET_LENGTH(data_flowset) - 4; // -4 for data flowset header -> id and length // map input buffer as a byte array in = (uint8_t *)(data_flowset + 4); // skip flowset header dbg_printf("[%u] Process data flowset size: %u\n", exporter->info.id, size_left); // Check if sampling is announced sampling_rate = 1; sampler_t *sampler = exporter->sampler; while ( sampler && sampler->info.id != -1 ) sampler = sampler->next; if ( sampler ) { sampling_rate = sampler->info.interval; dbg_printf("[%u] Std sampling available for this flow source: Rate: %llu\n", exporter->info.id, (long long unsigned)sampling_rate); } else { sampling_rate = default_sampling; dbg_printf("[%u] No Sampling record found\n", exporter->info.id); } if ( overwrite_sampling > 0 ) { sampling_rate = overwrite_sampling; dbg_printf("[%u] Hard overwrite sampling rate: %llu\n", exporter->info.id, (long long unsigned)sampling_rate); } if ( sampling_rate != 1 ) SetFlag(table->flags, FLAG_SAMPLED); while (size_left) { int input_offset; common_record_t *data_record; if ( size_left < 4 ) { // rounding pads size_left = 0; continue; } // check for enough space in output buffer if ( !CheckBufferSpace(fs->nffile, table->output_record_size) ) { // this should really never occur, because the buffer gets flushed ealier LogError("Process_ipfix: output buffer size error. Abort ipfix record processing"); dbg_printf("Process_ipfix: output buffer size error. Abort ipfix record processing"); return; } processed_records++; exporter->PacketSequence++; // map file record to output buffer data_record = (common_record_t *)fs->nffile->buff_ptr; // map output buffer as a byte array out = (uint8_t *)data_record; dbg_printf("[%u] Process data record: %u addr: %llu, buffer size_left: %u\n", exporter->info.id, processed_records, (long long unsigned)((ptrdiff_t)in - (ptrdiff_t)data_flowset), size_left); // fill the data record data_record->flags = table->flags; data_record->size = table->output_record_size; data_record->type = CommonRecordType; data_record->ext_map = table->extension_info.map->map_id; data_record->exporter_sysid = exporter->info.sysid; data_record->nfversion = 10; table->flow_start = 0; table->flow_end = 0; table->SysUpTime = 0; table->packets = 0; table->bytes = 0; table->out_packets = 0; table->out_bytes = 0; input_offset = 0; // apply copy and processing sequence for ( i=0; inumber_of_sequences; i++ ) { int output_offset = table->sequence[i].output_offset; void *stack = table->sequence[i].stack; if ( input_offset > size_left ) { // overrun LogError("Process ipfix: buffer overrun!! input_offset: %i > size left data buffer: %u", input_offset, size_left); dbg_printf("Buffer overrun!! input_offset: %i > size left data buffer: %u\n", input_offset, size_left); return; } switch (table->sequence[i].id) { case nop: break; case dyn_skip: { uint16_t skip = in[input_offset]; if ( skip < 255 ) { input_offset += (skip+1); } else { skip = Get_val16((void *)&in[input_offset+1]); input_offset += (skip+3); } } break; case move8: out[output_offset] = in[input_offset]; break; case move16: *((uint16_t *)&out[output_offset]) = Get_val16((void *)&in[input_offset]); break; case move32: *((uint32_t *)&out[output_offset]) = Get_val32((void *)&in[input_offset]); break; case move40: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val40((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move48: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val48((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move56: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val56((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move64: { type_mask_t t; t.val.val64 = Get_val64((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move128: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val64((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; t.val.val64 = Get_val64((void *)&in[input_offset+8]); *((uint32_t *)&out[output_offset+8]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+12]) = t.val.val32[1]; } break; case move32_sampling: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val32((void *)&in[input_offset]); t.val.val64 *= sampling_rate; *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; *(uint64_t *)stack = t.val.val64; } break; case move64_sampling: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val64((void *)&in[input_offset]); t.val.val64 *= sampling_rate; *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; *(uint64_t *)stack = t.val.val64; } break; case Time64Mili: { uint64_t DateMiliseconds = Get_val64((void *)&in[input_offset]); *(uint64_t *)stack = DateMiliseconds; } break; case TimeUnix: { uint64_t DateSeconds = Get_val32((void *)&in[input_offset]); *(uint64_t *)stack = DateSeconds *1000LL; } break; case TimeDeltaMicro: { uint64_t DeltaMicroSec = Get_val32((void *)&in[input_offset]); *(uint64_t *)stack = ((1000000LL * (uint64_t)ExportTime) - DeltaMicroSec) / 1000LL; } break; case SystemInitTime: { uint64_t SysUpTime = Get_val64((void *)&in[input_offset]); *(uint64_t *)stack = SysUpTime; } break; case TimeMili: *(uint32_t *)stack = Get_val32((void *)&in[input_offset]); table->HasTimeMili = 1; break; case saveICMP: *(uint32_t *)stack = Get_val16((void *)&in[input_offset]); break; case move_mpls: *((uint32_t *)&out[output_offset]) = Get_val24((void *)&in[input_offset]); break; case move_flags: { uint16_t flags = Get_val16((void *)&in[input_offset]); // cut upper byte out[output_offset] = flags & 0xFF; } break; case move_mac: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val48((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case zero8: out[output_offset] = 0; break; case zero16: *((uint16_t *)&out[output_offset]) = 0; break; case zero32: *((uint32_t *)&out[output_offset]) = 0; break; case zero64: *((uint64_t *)&out[output_offset]) = 0; break; case zero128: *((uint64_t *)&out[output_offset]) = 0; *((uint64_t *)&out[output_offset+8]) = 0; break; default: LogError("Process_ipfix: Software bug! Unknown Sequence: %u. at %s line %d", table->sequence[i].id, __FILE__, __LINE__); dbg_printf("Software bug! Unknown Sequence: %u. at %s line %d\n", table->sequence[i].id, __FILE__, __LINE__); } input_offset += (table->sequence[i].input_length + table->sequence[i].skip_count); } // for netflow historical reason, ICMP type/code goes into dst port field if ( data_record->prot == IPPROTO_ICMP || data_record->prot == IPPROTO_ICMPV6 ) { if ( table->icmpTypeCode ) { data_record->srcport = 0; data_record->dstport = table->icmpTypeCode; // data_record->dstport = Get_val16((void *)&in[table->ICMP_offset]); } } // check, if we need to store the packet received time if ( table->received_offset ) { type_mask_t t; t.val.val64 = (uint64_t)((uint64_t)fs->received.tv_sec * 1000LL) + (uint64_t)((uint64_t)fs->received.tv_usec / 1000LL); *((uint32_t *)&out[table->received_offset]) = t.val.val32[0]; *((uint32_t *)&out[table->received_offset+4]) = t.val.val32[1]; } // record sysuptime overwrites option template sysuptime if ( table->SysUpTime && table->HasTimeMili ) { dbg_printf("Calculate first/last from record SysUpTime\n"); table->flow_start += table->SysUpTime; table->flow_end += table->SysUpTime; } else if ( exporter->SysUpTime && table->HasTimeMili ) { dbg_printf("Calculate first/last from option SysUpTime\n"); table->flow_start += exporter->SysUpTime; table->flow_end += exporter->SysUpTime; } // split first/last time into epoch/msec values data_record->first = table->flow_start / 1000; data_record->msec_first = table->flow_start % 1000; data_record->last = table->flow_end / 1000; data_record->msec_last = table->flow_end % 1000; /* cross check for invalid date/time records: * if either date is < 820454400 === 1.1.1996 (year of invention of netflow) * invalidates set start/end to 0 */ if ( data_record->first < 820454400 || data_record->last < 820454400 ) { dbg_printf("Zero date < 19960101\n"); data_record->first = 0; data_record->msec_first = 0; data_record->last = 0; data_record->msec_last = 0; table->flow_start = 0; table->flow_end = 0; } // update first_seen, last_seen if ( table->flow_start < fs->first_seen ) fs->first_seen = table->flow_start; if ( table->flow_end > fs->last_seen ) fs->last_seen = table->flow_end; dbg_printf("msecFrist: %llu\n", table->flow_start); dbg_printf("msecLast : %llu\n", table->flow_end); // check if we need to record the router IP address if ( table->router_ip_offset ) { int output_offset = table->router_ip_offset; if ( exporter->info.sa_family == PF_INET6 ) { // 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs type_mask_t t; t.val.val64 = exporter->info.ip.V6[0]; *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; t.val.val64 = exporter->info.ip.V6[1]; *((uint32_t *)&out[output_offset+8]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+12]) = t.val.val32[1]; } else { *((uint32_t *)&out[output_offset]) = exporter->info.ip.V4; } } switch (data_record->prot ) { // switch protocol of case IPPROTO_ICMP: fs->nffile->stat_record->numflows_icmp++; fs->nffile->stat_record->numpackets_icmp += table->packets; fs->nffile->stat_record->numbytes_icmp += table->bytes; fs->nffile->stat_record->numpackets_icmp += table->out_packets; fs->nffile->stat_record->numbytes_icmp += table->out_bytes; break; case IPPROTO_TCP: fs->nffile->stat_record->numflows_tcp++; fs->nffile->stat_record->numpackets_tcp += table->packets; fs->nffile->stat_record->numbytes_tcp += table->bytes; fs->nffile->stat_record->numpackets_tcp += table->out_packets; fs->nffile->stat_record->numbytes_tcp += table->out_bytes; break; case IPPROTO_UDP: fs->nffile->stat_record->numflows_udp++; fs->nffile->stat_record->numpackets_udp += table->packets; fs->nffile->stat_record->numbytes_udp += table->bytes; fs->nffile->stat_record->numpackets_udp += table->out_packets; fs->nffile->stat_record->numbytes_udp += table->out_bytes; break; default: fs->nffile->stat_record->numflows_other++; fs->nffile->stat_record->numpackets_other += table->packets; fs->nffile->stat_record->numbytes_other += table->bytes; fs->nffile->stat_record->numpackets_other += table->out_packets; fs->nffile->stat_record->numbytes_other += table->out_bytes; } exporter->flows++; fs->nffile->stat_record->numflows++; fs->nffile->stat_record->numpackets += table->packets; fs->nffile->stat_record->numbytes += table->bytes; fs->nffile->stat_record->numpackets += table->out_packets; fs->nffile->stat_record->numbytes += table->out_bytes; if ( verbose ) { master_record_t master_record; memset((void *)&master_record, 0, sizeof(master_record_t)); ExpandRecord_v2((common_record_t *)data_record, &(table->extension_info), &(exporter->info), &master_record); flow_record_to_raw(&master_record, &string, 0); printf("%s\n", string); } fs->nffile->block_header->size += data_record->size; fs->nffile->block_header->NumRecords++; fs->nffile->buff_ptr = (common_record_t *)((pointer_addr_t)data_record + data_record->size); // advance input dbg_printf("Adjust input stream offset: %u\n", input_offset); if ( input_offset > size_left ) { // overrun LogError("Process ipfix: buffer overrun!! input_offset: %i > size left data buffer: %u\n", input_offset, size_left); return; } size_left -= input_offset; in += input_offset; // buffer size sanity check if ( fs->nffile->block_header->size > BUFFSIZE ) { // should never happen LogError("### Software error ###: %s line %d", __FILE__, __LINE__); LogError("Process ipfix: Output buffer overflow! Flush buffer and skip records."); LogError("Buffer size: %u > %u", fs->nffile->block_header->size, BUFFSIZE); // reset buffer fs->nffile->block_header->size = 0; fs->nffile->block_header->NumRecords = 0; fs->nffile->buff_ptr = (void *)((pointer_addr_t)fs->nffile->block_header + sizeof(data_block_header_t) ); return; } } } // End of Process_ipfix_data static void Process_ipfix_option_data(exporterDomain_t *exporter, void *data_flowset, FlowSource_t *fs) { uint32_t tableID = GET_FLOWSET_ID(data_flowset); uint32_t size_left = GET_FLOWSET_LENGTH(data_flowset) - 4; // -4 for data flowset header -> id and length dbg_printf("[%u] Process option data flowset size: %u\n", exporter->info.id, size_left); // map input buffer as a byte array uint8_t *in = (uint8_t *)(data_flowset + 4); // skip flowset header if ( exporter->SysUpOption.length ) { if (CHECK_OPTION_DATA(size_left, exporter->SysUpOption)) { exporter->SysUpTime = Get_val(in, exporter->SysUpOption.offset, exporter->SysUpOption.length); dbg_printf("Found SysUpTime option data\n"); dbg_printf("Extracted SysUpTime : %llu\n", exporter->SysUpTime); } else { LogError("Process_ipfix_option: %s line %d: Not enough data for option data", __FILE__, __LINE__); return; } } else { dbg_printf("No SysUpTime option data found\n"); } samplerOption_t *samplerOption = exporter->samplerOption; while ( samplerOption && samplerOption->tableID != tableID ) samplerOption = samplerOption->next; if ( !samplerOption ) { dbg_printf("No sampler option info\n"); return; } dbg_printf("sampler option found\n"); if ( (samplerOption->flags & SAMPLERMASK ) != 0) { int32_t id; uint16_t mode; uint32_t interval; if ( CHECK_OPTION_DATA(size_left, samplerOption->id) && CHECK_OPTION_DATA(size_left, samplerOption->mode) && CHECK_OPTION_DATA(size_left, samplerOption->interval)) { id = Get_val(in, samplerOption->id.offset, samplerOption->id.length); mode = Get_val(in, samplerOption->mode.offset, samplerOption->mode.length); interval = Get_val(in, samplerOption->interval.offset, samplerOption->interval.length); } else { LogError("Process_ipfix_option: %s line %d: Not enough data for option data", __FILE__, __LINE__); return; } InsertSampler(fs, exporter, id, mode, interval); dbg_printf("Extracted Sampler data:\n"); dbg_printf("Sampler ID : %u\n", id); dbg_printf("Sampler mode : %u\n", mode); dbg_printf("Sampler interval: %u\n", interval); } if ( (samplerOption->flags & STDMASK ) != 0) { int32_t id; uint16_t mode; uint32_t interval; id = -1; if ( CHECK_OPTION_DATA(size_left, samplerOption->mode) && CHECK_OPTION_DATA(size_left, samplerOption->interval)) { mode = Get_val(in, samplerOption->mode.offset, samplerOption->mode.length); interval = Get_val(in, samplerOption->interval.offset, samplerOption->interval.length); } else { LogError("Process_ipfix_option: %s line %d: Not enough data for option data", __FILE__, __LINE__); return; } InsertSampler(fs, exporter, id, mode, interval); dbg_printf("Extracted Std Sampler data:\n"); dbg_printf("Sampler ID : %i\n", id); dbg_printf("Sampler algorithm: %u\n", mode); dbg_printf("Sampler interval : %u\n", interval); dbg_printf("Set std sampler: algorithm: %u, interval: %u\n", mode, interval); } processed_records++; } // End of Process_ipfix_option_data void Process_IPFIX(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs) { exporterDomain_t *exporter; ssize_t size_left; uint32_t ExportTime, Sequence, flowset_length; ipfix_header_t *ipfix_header; void *flowset_header; #ifdef DEVEL static uint32_t packet_cntr = 0; uint32_t ObservationDomain; #endif size_left = in_buff_cnt; if ( size_left < IPFIX_HEADER_LENGTH ) { LogError("Process_ipfix: Too little data for ipfix packet: '%lli'", (long long)size_left); return; } ipfix_header = (ipfix_header_t *)in_buff; ExportTime = ntohl(ipfix_header->ExportTime); Sequence = ntohl(ipfix_header->LastSequence); #ifdef DEVEL ObservationDomain = ntohl(ipfix_header->ObservationDomain); packet_cntr++; printf("Next packet: %u\n", packet_cntr); #endif exporter = GetExporter(fs, ipfix_header); if ( !exporter ) { LogError("Process_ipfix: Exporter NULL: Abort ipfix record processing"); return; } exporter->packets++; //exporter->PacketSequence = Sequence; flowset_header = (void *)ipfix_header + IPFIX_HEADER_LENGTH; size_left -= IPFIX_HEADER_LENGTH; dbg_printf("\n[%u] process packet: %u, exported: %s, TemplateRecords: %llu, DataRecords: %llu, buffer: %li \n", ObservationDomain, packet_cntr, UNIX2ISO(ExportTime), (long long unsigned)exporter->TemplateRecords, (long long unsigned)exporter->DataRecords, size_left); dbg_printf("[%u] Sequence: %u\n", ObservationDomain, Sequence); // sequence check // 2^32 wrap is handled automatically as both counters overflow if ( Sequence != exporter->PacketSequence ) { if ( exporter->DataRecords != 0 ) { // sync sequence on first data record without error report fs->nffile->stat_record->sequence_failure++; exporter->sequence_failure++; dbg_printf("[%u] Sequence check failed: last seq: %u, seq %u\n", exporter->info.id, Sequence, exporter->PacketSequence); /* maybee to noise on buggy exporters LogError("Process_ipfix [%u] Sequence error: last seq: %u, seq %u\n", info.id, exporter->LastSequence, Sequence); */ } else { dbg_printf("[%u] Sync Sequence: %u\n", exporter->info.id, Sequence); } exporter->PacketSequence = Sequence; } else { dbg_printf("[%u] Sequence check ok\n", exporter->info.id); } // iterate over all set flowset_length = 0; while (size_left) { uint16_t flowset_id; if ( size_left && size_left < 4 ) { size_left = 0; continue; } flowset_header = flowset_header + flowset_length; flowset_id = GET_FLOWSET_ID(flowset_header); flowset_length = GET_FLOWSET_LENGTH(flowset_header); dbg_printf("Process_ipfix: Next flowset id %u, length %u.\n", flowset_id, flowset_length); if ( flowset_length == 0 ) { /* this should never happen, as 4 is an empty flowset and smaller is an illegal flowset anyway ... if it happends, we can't determine the next flowset, so skip the entire export packet */ LogError("Process_ipfix: flowset zero length error."); dbg_printf("Process_ipfix: flowset zero length error.\n"); return; } // possible padding if ( flowset_length <= 4 ) { size_left = 0; continue; } if ( flowset_length > size_left ) { LogError("Process_ipfix: flowset length error. Expected bytes: %u > buffersize: %lli", flowset_length, (long long)size_left); size_left = 0; continue; } switch (flowset_id) { case IPFIX_TEMPLATE_FLOWSET_ID: // Process_ipfix_templates(exporter, flowset_header, fs); exporter->TemplateRecords++; dbg_printf("Process template flowset, length: %u\n", flowset_length); Process_ipfix_templates(exporter, flowset_header, flowset_length, fs); break; case IPFIX_OPTIONS_FLOWSET_ID: // option_flowset = (option_template_flowset_t *)flowset_header; exporter->TemplateRecords++; dbg_printf("Process option template flowset, length: %u\n", flowset_length); Process_ipfix_option_templates(exporter, flowset_header, fs); break; default: { if ( flowset_id < IPFIX_MIN_RECORD_FLOWSET_ID ) { dbg_printf("Invalid flowset id: %u. Skip flowset\n", flowset_id); LogError("Process_ipfix: Invalid flowset id: %u. Skip flowset", flowset_id); } else { input_translation_t *table; dbg_printf("Process data flowset, length: %u\n", flowset_length); table = GetTranslationTable(exporter, flowset_id); if ( table ) { Process_ipfix_data(exporter, ExportTime, flowset_header, fs, table); exporter->DataRecords++; } else if ( HasOptionTable(exporter, flowset_id) ) { Process_ipfix_option_data(exporter, flowset_header, fs); } else { // maybe a flowset with option data dbg_printf("Process ipfix: [%u] No table for id %u -> Skip record\n", exporter->info.id, flowset_id); } } } } // End of switch // next record size_left -= flowset_length; } // End of while } // End of Process_IPFIX nfdump-1.6.23/bin/ipfix.h000066400000000000000000000260071404501030700151330ustar00rootroot00000000000000/* * Copyright (c) 2009-2021, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _IPFIX_H #define _IPFIX_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include "collector.h" /* reference: http://tools.ietf.org/html/draft-ietf-ipfix-protocol-rfc5101bis-00 */ typedef struct ipfix_header { uint16_t Version; // set to 10 for IPFIX uint16_t Length; // Total length incl. this header. up to 65535 bytes uint32_t ExportTime; // UNIC epoch export Time of flow. uint32_t LastSequence; // Incremental sequence counter modulo 2^32 of all IPFIX Data Records uint32_t ObservationDomain; // identifier , unique to the exporter } ipfix_header_t; #define IPFIX_HEADER_LENGTH sizeof(ipfix_header_t) /* Message Header Field Descriptions: Version Version of Flow Record format exported in this message. The value of this field is 0x000a for the current version, incrementing by one the version used in the NetFlow services export version 9 [RFC3954]. Length Total length of the IPFIX Message, measured in octets, including Message Header and Set(s). Export Time Time at which the IPFIX Message Header leaves the Exporter, expressed in seconds since the UNIX epoch of 1 January 1970 at 00:00 UTC, encoded as an unsigned 32-bit integer. Sequence Number Incremental sequence counter modulo 2^32 of all IPFIX Data Records sent on this PR-SCTP stream from the current Observation Domain by the Exporting Process. Check the specific meaning of this field in the subsections of Section 10 when UDP or TCP is selected as the transport protocol. This value SHOULD be used by the Collecting Process to identify whether any IPFIX Data Records have been missed. Template and Options Template Records do not increase the Sequence Number. Observation Domain ID A 32-bit identifier of the Observation Domain that is locally unique to the Exporting Process. The Exporting Process uses the Observation Domain ID to uniquely identify to the Collecting Process the Observation Domain that metered the Flows. It is RECOMMENDED that this identifier also be unique per IPFIX Device. Collecting Processes SHOULD use the Transport Session and the Observation Domain ID field to separate different export streams originating from the same Exporter. The Observation Domain ID SHOULD be 0 when no specific Observation Domain ID is relevant for the entire IPFIX Message, for example, when exporting the Exporting Process Statistics, or in case of a hierarchy of Collectors when aggregated Data Records are exported. */ /* set format: A Set has the format shown in Figure H. The record types can be either Template Records, Options Template Records, or Data Records. The record types MUST NOT be mixed within a Set. +--------------------------------------------------+ | Set Header | +--------------------------------------------------+ | record | +--------------------------------------------------+ | record | +--------------------------------------------------+ ... +--------------------------------------------------+ | record | +--------------------------------------------------+ | Padding (opt.) | +--------------------------------------------------+ */ typedef struct set_header_s { uint16_t SetID; // SetIDs: // 0, 1: not used // 2: Template Set // 3: Options Template Set #define IPFIX_TEMPLATE_FLOWSET_ID 2 #define IPFIX_OPTIONS_FLOWSET_ID 3 #define IPFIX_MIN_RECORD_FLOWSET_ID 256 uint16_t Length; // Length of bytes incl. this header uint32_t records[1]; // pointer to records } set_header_t; /* Template Record Format The format of the Template Record is shown in Figure J. It consists of a Template Record Header and one or more Field Specifiers. The definition of the Field Specifiers is given in Figure G above. +--------------------------------------------------+ | Template Record Header | +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ ... +--------------------------------------------------+ | Field Specifier | +--------------------------------------------------+ */ typedef struct ipfix_template_record_s { uint16_t TemplateID; // Template ID: // 0-255 reserved for Template Sets, Options Template Sets, // and other reserved Sets yet to be created. // 256-65535 Template IDs of Data Sets uint16_t FieldCount; uint32_t elements[1]; } ipfix_template_record_t; /* Standard Information Elements 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| Information Element ident. | Field Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ */ typedef struct ipfix_template_elements_std_s { uint16_t Type; uint16_t Length; } ipfix_template_elements_std_t; /* enterprise-specific Information Elements 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| Information Element ident. | Field Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ */ typedef struct ipfix_template_elements_e_s { uint16_t Type; uint16_t Length; uint32_t EnterpriseNumber; } ipfix_template_elements_e_t; #define _1byte 1 #define _2bytes 2 #define _3bytes 3 #define _4bytes 4 #define _6bytes 6 #define _8bytes 8 #define _16bytes 16 #define SKIP_ELEMENT 0 // IPFIX std element definitios // Flowset record types #define IPFIX_octetDeltaCount 1 #define IPFIX_packetDeltaCount 2 // reserved 3 #define IPFIX_FLOWS_AGGR 3 #define IPFIX_protocolIdentifier 4 #define IPFIX_ipClassOfService 5 #define IPFIX_tcpControlBits 6 #define IPFIX_SourceTransportPort 7 #define IPFIX_SourceIPv4Address 8 #define IPFIX_SourceIPv4PrefixLength 9 #define IPFIX_ingressInterface 10 #define IPFIX_DestinationTransportPort 11 #define IPFIX_DestinationIPv4Address 12 #define IPFIX_DestinationIPv4PrefixLength 13 #define IPFIX_egressInterface 14 #define IPFIX_ipNextHopIPv4Address 15 #define IPFIX_bgpSourceAsNumber 16 #define IPFIX_bgpDestinationAsNumber 17 #define IPFIX_bgpNextHopIPv4Address 18 #define IPFIX_flowEndSysUpTime 21 #define IPFIX_flowStartSysUpTime 22 #define IPFIX_postOctetDeltaCount 23 #define IPFIX_postPacketDeltaCount 24 #define IPFIX_SourceIPv6Address 27 #define IPFIX_DestinationIPv6Address 28 #define IPFIX_SourceIPv6PrefixLength 29 #define IPFIX_DestinationIPv6PrefixLength 30 #define IPFIX_flowLabelIPv6 31 #define IPFIX_icmpTypeCodeIPv4 32 // deprecated elements for sampling #define IPFIX_samplingInterval 34 #define IPFIX_samplingAlgorithm 35 // depricated 38, 39 #define IPFIX_samplerId 48 #define IPFIX_samplerMode 49 #define IPFIX_samplerRandomInterval 50 // #define IPFIX_MIN_TTL 52 // #define IPFIX_MAX_TTL 53 // #define IPFIX_IPV4_IDENT 54 #define IPFIX_postIpClassOfService 55 #define IPFIX_SourceMacAddress 56 #define IPFIX_postDestinationMacAddress 57 #define IPFIX_vlanId 58 #define IPFIX_postVlanId 59 #define IPFIX_flowDirection 61 #define IPFIX_ipNextHopIPv6Address 62 #define IPFIX_bgpNextHopIPv6Address 63 #define IPFIX_mplsTopLabelStackSection 70 #define IPFIX_mplsLabelStackSection2 71 #define IPFIX_mplsLabelStackSection3 72 #define IPFIX_mplsLabelStackSection4 73 #define IPFIX_mplsLabelStackSection5 74 #define IPFIX_mplsLabelStackSection6 75 #define IPFIX_mplsLabelStackSection7 76 #define IPFIX_mplsLabelStackSection8 77 #define IPFIX_mplsLabelStackSection9 78 #define IPFIX_mplsLabelStackSection10 79 #define IPFIX_DestinationMacAddress 80 #define IPFIX_postSourceMacAddress 81 #define IPFIX_octetTotalCount 85 #define IPFIX_packetTotalCount 86 #define IPFIX_forwardingStatus 89 #define IPFIX_flowEndReason 136 #define IPFIX_icmpTypeCodeIPv6 139 #define IPFIX_flowStartSeconds 150 #define IPFIX_flowEndSeconds 151 #define IPFIX_flowStartMilliseconds 152 #define IPFIX_flowEndMilliseconds 153 #define IPFIX_flowStartDeltaMicroseconds 158 #define IPFIX_flowEndDeltaMicroseconds 159 #define IPFIX_SystemInitTimeMiliseconds 160 #define IPFIX_postOctetTotalCount 171 #define IPFIX_postPacketTotalCount 172 #define IPFIX_biflowDirection 239 // sampling #define IPFIX_selectorId 302 #define IPFIX_selectorAlgorithm 304 #define IPFIX_samplingPacketInterval 305 // Private Enterprise Numbers #define IPFIX_ReverseInformationElement 29305 /* prototypes */ int Init_IPFIX(int v, uint32_t sampling, uint32_t overwrite); void Process_IPFIX(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs); #endif //_IPFIX_H 1 nfdump-1.6.23/bin/ipfrag.c000066400000000000000000000236371404501030700152650ustar00rootroot00000000000000/* * Copyright (c) 2014-2020, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #ifdef HAVE_NETINET_IN_SYSTM_H #include #endif #ifdef HAVE_NETINET_IN_H #include #endif #include #include #include #include #include #include #include "util.h" #include "rbtree.h" #include "ipfrag.h" #define KEYLEN (offsetof(IPFragNode_t,data_size) - offsetof(IPFragNode_t, src_addr)) static int IPFragNodeCMP(struct IPFragNode *e1, struct IPFragNode *e2); static struct IPFragNode *New_frag_node(void); static void Free_node(struct IPFragNode *node, int free_data); static void Remove_node(struct IPFragNode *node, int free_data); // Insert the IP RB tree code here RB_GENERATE(IPFragTree, IPFragNode, entry, IPFragNodeCMP); static IPFragTree_t *IPFragTree; static uint32_t NumFragments; static time_t lastExpire = 0; static int IPFragNodeCMP(struct IPFragNode *e1, struct IPFragNode *e2) { uint32_t *a = &e1->src_addr; uint32_t *b = &e2->src_addr; int i; // 2 x sizeof(uint32_t) (8) + frag_offset == 12 i = memcmp((void *)a, (void *)b, KEYLEN ); return i; } // End of IPFragNodeCMP static struct IPFragNode *New_frag_node(void) { struct IPFragNode *node; node = calloc(1, sizeof(struct IPFragNode)); if ( !node ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return NULL; } node->data = calloc(1, IP_MAXPACKET); if ( !node->data ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); free(node); return NULL; } node->eod = node->data; node->data_size = 0; node->holes = calloc(1, sizeof(hole_t)); if ( !node->holes ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); free(node->data); free(node); return NULL; } node->holes->next = NULL; node->holes->first = 0; node->holes->last = IP_MAXPACKET; NumFragments++; return node; } // End of New_frag_node static void Free_node(struct IPFragNode *node, int free_data) { hole_t *hole, *h; hole = node->holes; while (hole) { h = hole->next; free(hole); hole = h; } if (free_data) free(node->data); free(node); NumFragments--; } // End of Free_node static void Remove_node(struct IPFragNode *node, int free_data) { struct IPFragNode *n; n = RB_REMOVE(IPFragTree, IPFragTree, node); if ( n ) { Free_node(n, free_data); } // else - node not in tree } // End of Remove_node int IPFragTree_init(void) { IPFragTree = calloc(1, sizeof(IPFragTree_t)); if ( !IPFragTree ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return 0; } RB_INIT(IPFragTree); NumFragments = 0; dbg_printf("IPFrag key len: %lu\n", KEYLEN); return 1; } // End of IPFragTree_init void IPFragTree_free(void) { struct IPFragNode *node, *nxt; nxt = NULL; for (node = RB_MIN(IPFragTree, IPFragTree); node != NULL; node = nxt) { nxt = RB_NEXT(IPFragTree, IPFragTree, node); RB_REMOVE(IPFragTree, IPFragTree, node); Free_node(node, 1); } free(IPFragTree); IPFragTree = NULL; NumFragments = 0; } // End of IPFragTree_free static void IPFragTree_expire(time_t when) { struct IPFragNode *node, *nxt; uint32_t expireCnt = 0; nxt = NULL; for (node = RB_MIN(IPFragTree, IPFragTree); node != NULL; node = nxt) { nxt = RB_NEXT(IPFragTree, IPFragTree, node); if ( (when - node->last) > 15 ) { RB_REMOVE(IPFragTree, IPFragTree, node); Free_node(node, 1); expireCnt++; } } dbg_printf("Expired %u incomplete IP fragments, total fragments: %u\n", expireCnt, NumFragments); if ( expireCnt ) LogInfo("Expired %u incomplete IP fragments, total fragments: %u", expireCnt, NumFragments); } // End of IPFragTree_expire void *IPFrag_tree_Update(time_t when, uint32_t src, uint32_t dst, uint32_t ident, uint32_t *length, uint32_t ip_off, void *data) { struct IPFragNode FindNode, *n; hole_t *hole, *h, *hole_parent; uint16_t more_fragments, first, last, max; dbg(int found_hole;) char src_s[16], dst_s[16]; if ( (when - lastExpire ) > 10 ) { IPFragTree_expire(when); lastExpire = when; } #ifdef DEVEL inet_ntop(AF_INET, &src, src_s, 16); inet_ntop(AF_INET, &dst, dst_s, 16); printf("Update %s - %s\n", src_s, dst_s); #endif FindNode.src_addr = src; FindNode.dst_addr = dst; FindNode.ident = ident; FindNode.last = 0; n = RB_FIND(IPFragTree, IPFragTree, &FindNode); if ( !n ) { n = New_frag_node(); n->src_addr = src; n->dst_addr = dst; n->ident = ident; n->last = when; if ( RB_INSERT(IPFragTree, IPFragTree, n) ) { // must never happen LogError("Node insert returned existing node - Software error in %s line %d", __FILE__, __LINE__); } } hole = n->holes; hole_parent = NULL; first = (ip_off & IP_OFFMASK) << 3; more_fragments = (ip_off & IP_MF) != 0 ? 1 : 0; last = first + *length - 1; if ( last > IP_MAXPACKET ) { LogError("Fragment assembly error: last > IP_MAXPACKET"); LogError("Fragment assembly: first: %u, last: %u, MF: %u\n", first, last, more_fragments); return NULL; } // last fragment - sets max offset dbg(found_hole = 0;) max = more_fragments == 0 ? last : 0; dbg_printf("Fragment assembly: first: %u, last: %u, MF: %u, ID: %x\n", first, last, more_fragments, ident); while (hole) { uint16_t hole_last; if ( max ) { dbg_printf("max in last fragment: %u\n", max); // last fragment offset/length if ( hole->last == IP_MAXPACKET ) { // last fragment has max size if ( max >= hole->first ) { dbg_printf("set max of last fragment: %u\n", max); hole->last = max; } else { inet_ntop(AF_INET, &src, src_s, 16); inet_ntop(AF_INET, &dst, dst_s, 16); LogError("last fragment offset error - teardrop attack?? SRC: %s, DST: %s", src_s, dst_s); } } } dbg_printf("Check Hole: first: %u, last: %u\n", hole->first, hole->last); if ( first > hole->last ) { hole_parent = hole; hole = hole->next; dbg_printf("Fragment right outside hole\n"); continue; } if ( last < hole->first ) { hole_parent = hole; hole = hole->next; dbg_printf("Fragment left outside hole\n"); continue; } // check for overlapping - cut off overlap if ( last > hole->last ) { dbg_printf("Truncate right overlapping fragment: %u -> %u\n", last, hole->last); last = hole->last; } if ( first < hole->first ) { dbg_printf("Truncate left overlapping fragment: %u -> %u\n", first, hole->first); first = hole->first; } if ( first > last ) { LogInfo("fragment error first %u >= last %u", first, last); return NULL; } // fragment fits into hole dbg(found_hole = 1;) if ( last > n->data_size ) n->data_size = last; hole_last = hole->last; if ( first == hole->first ) { dbg_printf("Fragment matches first\n"); // fragment fits at beginning of hole if ( last == hole->last ) { dbg_printf("Fragment matches last\n"); // fragment fits completly into hole - delete hole if ( hole_parent ) { hole_parent->next = hole->next; } else { n->holes = hole->next; } free(hole); hole = NULL; } else { // fragment smaller than hole dbg_printf("Fragment smaller than hole\n"); hole->first = last+1; } } else { // fragment start within hole dbg_printf("Fragment inside hole\n"); hole->last = first - 1; if ( last < hole_last ) { // fragment ends within hole - add another hole h = malloc(sizeof(hole_t)); if ( !h ) { LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) ); return NULL; } h->first = last + 1; h->last = hole_last; h->next = n->holes; n->holes = h; } } memcpy(n->data + first, data, *length); break; } #ifdef DEVEL if ( !found_hole ) { hole_t *h = n->holes; dbg_printf("No space in fragment list for: first: %u, last: %u\n", first, last); while ( h ) { dbg_printf("first: %u,last: %u\n", h->first, h->last); h = h->next; } } #endif if ( n->holes == NULL ) { void *data = n->data; n->data_size++; *length = n->data_size; Remove_node(n, 0); dbg_printf("Defragmentation complete - size: %u\n", n->data_size); return data; } else { return NULL; } } // End of IPFrag_tree_Update uint32_t IPFragEntries() { return NumFragments; } // End of IPFragEntries nfdump-1.6.23/bin/ipfrag.h000066400000000000000000000050641404501030700152640ustar00rootroot00000000000000/* * Copyright (c) 2017, Peter Haag * Copyright (c) 2014, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _IPFRAG_H #define _IPFRAG_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include #include "rbtree.h" typedef struct hole_s { struct hole_s *next; uint32_t first; uint32_t last; } hole_t; struct IPFragNode { // tree RB_ENTRY(IPFragNode) entry; // flow key // IP addr uint32_t src_addr; uint32_t dst_addr; uint32_t ident; // End of flow key uint32_t data_size; time_t last; // packet data void *data; void *eod; hole_t *holes; }; typedef struct IPFragNode IPFragNode_t; /* flow tree type */ typedef RB_HEAD(IPFragTree, IPFragNode) IPFragTree_t; // Insert the RB prototypes here RB_PROTOTYPE(IPFragTree, IPFragNode, entry, IPFragNodeCMP); int IPFragTree_init(void); void IPFragTree_free(void); void *IPFrag_tree_Update(time_t when, uint32_t src, uint32_t dst, uint32_t ident, uint32_t *length, uint32_t ip_off, void *data); uint32_t IPFragEntries(void); #endif nfdump-1.6.23/bin/launch.c000066400000000000000000000265001404501030700152570ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifdef HAVE_CONFIG_H #include "config.h" #endif #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "nfstatfile.h" #include "bookkeeper.h" #ifdef HAVE_FTS_H # include #else # include "fts_compat.h" #define fts_children fts_children_compat #define fts_close fts_close_compat #define fts_open fts_open_compat #define fts_read fts_read_compat #define fts_set fts_set_compat #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "expire.h" #include "collector.h" #include "launch.h" #define DEVEL 1 static int done, launch, child_exit; static void SignalHandler(int signal); static char *cmd_expand(srecord_t *InfoRecord, char *ident, char *datadir, char *process); static void cmd_parse(char *buf, char **args); static void cmd_execute(char **args); static void do_expire(char *datadir); #define MAXARGS 256 #define MAXCMDLEN 4096 static void SignalHandler(int signal) { switch (signal) { case SIGTERM: // in case the process will not terminate, we // kill the process directly after the 2nd TERM signal if ( done > 1 ) exit(234); done++; break; case SIGHUP: launch = 1; break; case SIGCHLD: child_exit++; break; } } /* End of IntHandler */ /* * Expand % placeholders in command string * expand the memory needed in the command string and replace placeholders * prevent endless expansion */ static char *cmd_expand(srecord_t *InfoRecord, char *ident, char *datadir, char *process) { char *q, *s, tmp[16]; int i; q = strdup(process); if ( !q ) { LogError("strdup() error in %s:%i: %s", __FILE__, __LINE__ , strerror(errno)); return NULL; } i = 0; while ( q[i] ) { if ( (q[i] == '%') && q[i+1] ) { // replace the %x var switch ( q[i+1] ) { case 'd' : s = datadir; break; case 'f' : s = InfoRecord->fname; break; case 't' : s = InfoRecord->tstring; break; case 'u' : snprintf(tmp, 16, "%lli", (long long)InfoRecord->tstamp); tmp[15] = 0; s = tmp; break; case 'i' : s = ident; break; default: LogError("Unknown format token '%%%c'", q[i+1]); s = NULL; } if ( s ) { q = realloc(q, strlen(q) + strlen(s)); if ( !q ) { LogError("realloc() error in %s:%i: %s", __FILE__, __LINE__ , strerror(errno)); return NULL; } // sanity check if ( strlen(q) > MAXCMDLEN ) { LogError("command expand error in %s:%i: cmd line too long", __FILE__, __LINE__); return NULL; } memmove(&q[i] + strlen(s), &q[i+2], strlen(&q[i+2]) + 1); // include trailing '0' in memmove memcpy(&q[i], s, strlen(s)); } } i++; } return q; } // End of cmd_expand /* * split the command in buf into individual arguments. */ static void cmd_parse(char *buf, char **args) { int i, argnum; i = argnum = 0; while ( (i < MAXCMDLEN) && (buf[i] != 0) ) { /* * Strip whitespace. Use nulls, so * that the previous argument is terminated * automatically. */ while ( (i < MAXCMDLEN) && ((buf[i] == ' ') || (buf[i] == '\t'))) buf[i++] = 0; /* * Save the argument. */ if ( argnum < MAXARGS ) args[argnum++] = &(buf[i]); /* * Skip over the argument. */ while ( (i < MAXCMDLEN) && ((buf[i] != 0) && (buf[i] != ' ') && (buf[i] != '\t'))) i++; } if ( argnum < MAXARGS ) args[argnum] = NULL; if ( (i >= MAXCMDLEN) || (argnum >= MAXARGS) ) { // for safety reason, disable the command args[0] = NULL; LogError("Launcher: Unable to parse command: '%s'", buf); } } // End of cmd_parse /* * cmd_execute * spawn a child process and execute the program. */ static void cmd_execute(char **args) { int pid; // Get a child process. LogInfo("Launcher: fork child."); if ((pid = fork()) < 0) { LogError("Can't fork: %s", strerror(errno)); return; } if (pid == 0) { // we are the child execvp(*args, args); LogError("Can't execvp: %s: %s", args[0], strerror(errno)); _exit(1); } // we are the parent LogInfo("Launcher: child exec done."); /* empty */ } // End of cmd_execute static void do_expire(char *datadir) { bookkeeper_t *books; dirstat_t *dirstat, oldstat; int ret, bookkeeper_stat, do_rescan; LogInfo("Run expire on '%s'", datadir); do_rescan = 0; ret = ReadStatInfo(datadir, &dirstat, CREATE_AND_LOCK); switch (ret) { case STATFILE_OK: break; case ERR_NOSTATFILE: dirstat->low_water = 95; case FORCE_REBUILD: LogInfo("Force rebuild stat record"); do_rescan = 1; break; case ERR_FAIL: LogError("expire failed: can't read stat record"); return; /* not reached */ break; default: LogError("expire failed: unexpected return code %i reading stat record", ret); return; /* not reached */ } bookkeeper_stat = AccessBookkeeper(&books, datadir); if ( do_rescan ) { RescanDir(datadir, dirstat); if ( bookkeeper_stat == BOOKKEEPER_OK ) { ClearBooks(books, NULL); // release the books below } } if ( bookkeeper_stat == BOOKKEEPER_OK ) { bookkeeper_t tmp_books; ClearBooks(books, &tmp_books); UpdateBookStat(dirstat, &tmp_books); ReleaseBookkeeper(books, DETACH_ONLY); } else { LogError("Error %i: can't access book keeping records", ret); } LogInfo("Limits: Filesize %s, Lifetime %s, Watermark: %llu%%\n", dirstat->max_size ? ScaleValue(dirstat->max_size) : "", dirstat->max_lifetime ? ScaleTime(dirstat->max_lifetime) : "", (unsigned long long)dirstat->low_water); LogInfo("Current size: %s, Current lifetime: %s, Number of files: %llu", ScaleValue(dirstat->filesize), ScaleTime(dirstat->last - dirstat->first), (unsigned long long)dirstat->numfiles); oldstat = *dirstat; if ( dirstat->max_size || dirstat->max_lifetime ) ExpireDir(datadir, dirstat, dirstat->max_size, dirstat->max_lifetime, 0); WriteStatInfo(dirstat); if ( (oldstat.numfiles - dirstat->numfiles) > 0 ) { LogInfo("expire completed"); LogInfo(" expired files: %llu", (unsigned long long)(oldstat.numfiles - dirstat->numfiles)); LogInfo(" expired time slot: %s", ScaleTime(dirstat->first - oldstat.first)); LogInfo(" expired file size: %s", ScaleValue(oldstat.filesize - dirstat->filesize)); LogInfo("New size: %s, New lifetime: %s, Number of files: %llu", ScaleValue(dirstat->filesize), ScaleTime(dirstat->last - dirstat->first), (unsigned long long)dirstat->numfiles); } else { LogInfo("expire completed - nothing to expire."); } ReleaseStatInfo(dirstat); } // End of do_expire void launcher (void *commbuff, FlowSource_t *FlowSource, char *process, int expire) { FlowSource_t *fs; struct sigaction act; char *args[MAXARGS]; int pid, stat; srecord_t *InfoRecord; InfoRecord = (srecord_t *)commbuff; LogInfo("Launcher: Startup. auto-expire %s", expire ? "enabled" : "off" ); done = launch = child_exit = 0; // process may be NULL, if we only expire data files if ( process ) { char *cmd = NULL; srecord_t TestRecord; // check for valid command expansion strncpy(TestRecord.fname, "test", MAXPATHLEN-1); TestRecord.fname[MAXPATHLEN-1] = 0; strncpy(TestRecord.tstring, "20190711084500", MAXTIMESTRING-1); TestRecord.tstring[MAXTIMESTRING-1] = 0; TestRecord.tstamp = 1; // checkk valid command expansion fs = FlowSource; cmd = cmd_expand(&TestRecord, fs->Ident, fs->datadir, process); if ( cmd == NULL ) { LogError("Launcher: ident: %s, Unable to expand command: '%s'", fs->Ident, process); exit(255); } free(cmd); } /* Signal handling */ memset((void *)&act,0,sizeof(struct sigaction)); act.sa_handler = SignalHandler; sigemptyset(&act.sa_mask); act.sa_flags = 0; sigaction(SIGCHLD, &act, NULL); // child process terminated sigaction(SIGTERM, &act, NULL); // we are done sigaction(SIGINT, &act, NULL); // we are done sigaction(SIGHUP, &act, NULL); // run command while ( !done ) { // sleep until we get signaled dbg_printf("Launcher: Sleeping"); select(0, NULL, NULL, NULL, NULL); dbg_printf("Launcher: Wakeup"); if ( launch ) { // SIGHUP launch = 0; if ( process ) { char *cmd = NULL; fs = FlowSource; while ( fs ) { // Expand % placeholders cmd = cmd_expand(InfoRecord, fs->Ident, fs->datadir, process); if ( cmd == NULL ) { LogError("Launcher: ident: %s, Unable to expand command: '%s'", fs->Ident, process); continue; } dbg_printf("Launcher: ident: %s run command: '%s'", fs->Ident, cmd); // prepare args array cmd_parse(cmd, args); if ( args[0] ) cmd_execute(args); // do not flood the system with new processes sleep(1); // else cmd_parse already reported the error free(cmd); fs = fs->next; } } fs = FlowSource; while ( fs ) { if ( expire ) do_expire(fs->datadir); fs = fs->next; } } if ( child_exit ) { LogInfo("launcher child exit %d children.", child_exit); while ( (pid = waitpid (-1, &stat, WNOHANG)) > 0 ) { if ( WIFEXITED(stat) ) { LogInfo("launcher child %i exit status: %i", pid, WEXITSTATUS(stat)); } if ( WIFSIGNALED(stat) ) { LogError("launcher child %i died due to signal %i", pid, WTERMSIG(stat)); } child_exit--; } LogInfo("launcher waiting children done. %d children", child_exit); child_exit = 0; } if ( done ) { LogInfo("Launcher: Terminating."); } } waitpid (-1, &stat, 0); // we are done LogInfo("Launcher: exit."); } // End of launcher nfdump-1.6.23/bin/launch.h000066400000000000000000000042621404501030700152650ustar00rootroot00000000000000/* * Copyright (c) 2009-2019, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _LAUNCH_H #define _LAUNCH_H 1 #include "config.h" #include #include #include "collector.h" #include "nffile.h" #define MAXTIMESTRING 64 typedef struct srecord_s { char fname[MAXPATHLEN]; // %f file name char tstring[MAXTIMESTRING]; // %t 14 needed for YYYYmmddHHMMSS + opt. timezone time_t tstamp; // UNIX time stamp int failed; // in case of an error } srecord_t; void launcher (void *commbuff, FlowSource_t *FlowSource, char *process, int expire); #endif //_LAUNCH_H nfdump-1.6.23/bin/lz4.c000077500000000000000000001705531404501030700145310ustar00rootroot00000000000000/* LZ4 - Fast LZ compression algorithm Copyright (C) 2011-2017, Yann Collet. BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - LZ4 homepage : http://www.lz4.org - LZ4 source repository : https://github.com/lz4/lz4 */ /*-************************************ * Tuning parameters **************************************/ /* * LZ4_HEAPMODE : * Select how default compression functions will allocate memory for their hash table, * in memory stack (0:default, fastest), or in memory heap (1:requires malloc()). */ #ifndef LZ4_HEAPMODE # define LZ4_HEAPMODE 0 #endif /* * ACCELERATION_DEFAULT : * Select "acceleration" for LZ4_compress_fast() when parameter value <= 0 */ #define ACCELERATION_DEFAULT 1 /*-************************************ * CPU Feature Detection **************************************/ /* LZ4_FORCE_MEMORY_ACCESS * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable. * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal. * The below switch allow to select different access method for improved performance. * Method 0 (default) : use `memcpy()`. Safe and portable. * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable). * This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`. * Method 2 : direct access. This method is portable but violate C standard. * It can generate buggy code on targets which assembly generation depends on alignment. * But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6) * See https://fastcompression.blogspot.fr/2015/08/accessing-unaligned-memory.html for details. * Prefer these methods in priority order (0 > 1 > 2) */ #ifndef LZ4_FORCE_MEMORY_ACCESS /* can be defined externally */ # if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) ) # define LZ4_FORCE_MEMORY_ACCESS 2 # elif defined(__INTEL_COMPILER) || defined(__GNUC__) # define LZ4_FORCE_MEMORY_ACCESS 1 # endif #endif /* * LZ4_FORCE_SW_BITCOUNT * Define this parameter if your target system or compiler does not support hardware bit count */ #if defined(_MSC_VER) && defined(_WIN32_WCE) /* Visual Studio for Windows CE does not support Hardware bit count */ # define LZ4_FORCE_SW_BITCOUNT #endif /* * register is ignored when the code built with a C++-17 compiler * Remove the keyword when built with C++-17 to silent the warning */ #if defined(__cplusplus) && __cplusplus > 201402L # define REGISTER #else # define REGISTER register #endif /*-************************************ * Dependency **************************************/ #include "lz4.h" /* see also "memory routines" below */ /*-************************************ * Compiler Options **************************************/ #ifdef _MSC_VER /* Visual Studio */ # include # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ # pragma warning(disable : 4293) /* disable: C4293: too large shift (32-bits) */ #endif /* _MSC_VER */ #ifndef LZ4_FORCE_INLINE # ifdef _MSC_VER /* Visual Studio */ # define LZ4_FORCE_INLINE static __forceinline # else # if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ # ifdef __GNUC__ # define LZ4_FORCE_INLINE static inline __attribute__((always_inline)) # else # define LZ4_FORCE_INLINE static inline # endif # else # define LZ4_FORCE_INLINE static # endif /* __STDC_VERSION__ */ # endif /* _MSC_VER */ #endif /* LZ4_FORCE_INLINE */ /* LZ4_FORCE_O2_GCC_PPC64LE and LZ4_FORCE_O2_INLINE_GCC_PPC64LE * Gcc on ppc64le generates an unrolled SIMDized loop for LZ4_wildCopy, * together with a simple 8-byte copy loop as a fall-back path. * However, this optimization hurts the decompression speed by >30%, * because the execution does not go to the optimized loop * for typical compressible data, and all of the preamble checks * before going to the fall-back path become useless overhead. * This optimization happens only with the -O3 flag, and -O2 generates * a simple 8-byte copy loop. * With gcc on ppc64le, all of the LZ4_decompress_* and LZ4_wildCopy * functions are annotated with __attribute__((optimize("O2"))), * and also LZ4_wildCopy is forcibly inlined, so that the O2 attribute * of LZ4_wildCopy does not affect the compression speed. */ #if defined(__PPC64__) && defined(__LITTLE_ENDIAN__) && defined(__GNUC__) # define LZ4_FORCE_O2_GCC_PPC64LE __attribute__((optimize("O2"))) # define LZ4_FORCE_O2_INLINE_GCC_PPC64LE __attribute__((optimize("O2"))) LZ4_FORCE_INLINE #else # define LZ4_FORCE_O2_GCC_PPC64LE # define LZ4_FORCE_O2_INLINE_GCC_PPC64LE static #endif #if (defined(__GNUC__) && (__GNUC__ >= 3)) || (defined(__INTEL_COMPILER) && (__INTEL_COMPILER >= 800)) || defined(__clang__) # define expect(expr,value) (__builtin_expect ((expr),(value)) ) #else # define expect(expr,value) (expr) #endif #define likely(expr) expect((expr) != 0, 1) #define unlikely(expr) expect((expr) != 0, 0) /*-************************************ * Memory routines **************************************/ #include /* malloc, calloc, free */ #define ALLOCATOR(n,s) calloc(n,s) #define FREEMEM free #include /* memset, memcpy */ #define MEM_INIT memset /*-************************************ * Basic Types **************************************/ #if defined(__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) # include typedef uint8_t BYTE; typedef uint16_t U16; typedef uint32_t U32; typedef int32_t S32; typedef uint64_t U64; typedef uintptr_t uptrval; #else typedef unsigned char BYTE; typedef unsigned short U16; typedef unsigned int U32; typedef signed int S32; typedef unsigned long long U64; typedef size_t uptrval; /* generally true, except OpenVMS-64 */ #endif #if defined(__x86_64__) typedef U64 reg_t; /* 64-bits in x32 mode */ #else typedef size_t reg_t; /* 32-bits in x32 mode */ #endif /*-************************************ * Reading and writing into memory **************************************/ static unsigned LZ4_isLittleEndian(void) { const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */ return one.c[0]; } #if defined(LZ4_FORCE_MEMORY_ACCESS) && (LZ4_FORCE_MEMORY_ACCESS==2) /* lie to the compiler about data alignment; use with caution */ static U16 LZ4_read16(const void* memPtr) { return *(const U16*) memPtr; } static U32 LZ4_read32(const void* memPtr) { return *(const U32*) memPtr; } static reg_t LZ4_read_ARCH(const void* memPtr) { return *(const reg_t*) memPtr; } static void LZ4_write16(void* memPtr, U16 value) { *(U16*)memPtr = value; } static void LZ4_write32(void* memPtr, U32 value) { *(U32*)memPtr = value; } #elif defined(LZ4_FORCE_MEMORY_ACCESS) && (LZ4_FORCE_MEMORY_ACCESS==1) /* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */ /* currently only defined for gcc and icc */ typedef union { U16 u16; U32 u32; reg_t uArch; } __attribute__((packed)) unalign; static U16 LZ4_read16(const void* ptr) { return ((const unalign*)ptr)->u16; } static U32 LZ4_read32(const void* ptr) { return ((const unalign*)ptr)->u32; } static reg_t LZ4_read_ARCH(const void* ptr) { return ((const unalign*)ptr)->uArch; } static void LZ4_write16(void* memPtr, U16 value) { ((unalign*)memPtr)->u16 = value; } static void LZ4_write32(void* memPtr, U32 value) { ((unalign*)memPtr)->u32 = value; } #else /* safe and portable access through memcpy() */ static U16 LZ4_read16(const void* memPtr) { U16 val; memcpy(&val, memPtr, sizeof(val)); return val; } static U32 LZ4_read32(const void* memPtr) { U32 val; memcpy(&val, memPtr, sizeof(val)); return val; } static reg_t LZ4_read_ARCH(const void* memPtr) { reg_t val; memcpy(&val, memPtr, sizeof(val)); return val; } static void LZ4_write16(void* memPtr, U16 value) { memcpy(memPtr, &value, sizeof(value)); } static void LZ4_write32(void* memPtr, U32 value) { memcpy(memPtr, &value, sizeof(value)); } #endif /* LZ4_FORCE_MEMORY_ACCESS */ static U16 LZ4_readLE16(const void* memPtr) { if (LZ4_isLittleEndian()) { return LZ4_read16(memPtr); } else { const BYTE* p = (const BYTE*)memPtr; return (U16)((U16)p[0] + (p[1]<<8)); } } static void LZ4_writeLE16(void* memPtr, U16 value) { if (LZ4_isLittleEndian()) { LZ4_write16(memPtr, value); } else { BYTE* p = (BYTE*)memPtr; p[0] = (BYTE) value; p[1] = (BYTE)(value>>8); } } static void LZ4_copy8(void* dst, const void* src) { memcpy(dst,src,8); } /* customized variant of memcpy, which can overwrite up to 8 bytes beyond dstEnd */ LZ4_FORCE_O2_INLINE_GCC_PPC64LE void LZ4_wildCopy(void* dstPtr, const void* srcPtr, void* dstEnd) { BYTE* d = (BYTE*)dstPtr; const BYTE* s = (const BYTE*)srcPtr; BYTE* const e = (BYTE*)dstEnd; do { LZ4_copy8(d,s); d+=8; s+=8; } while (d=1) # include #else # ifndef assert # define assert(condition) ((void)0) # endif #endif #define LZ4_STATIC_ASSERT(c) { enum { LZ4_static_assert = 1/(int)(!!(c)) }; } /* use only *after* variable declarations */ #if defined(LZ4_DEBUG) && (LZ4_DEBUG>=2) # include static int g_debuglog_enable = 1; # define DEBUGLOG(l, ...) { \ if ((g_debuglog_enable) && (l<=LZ4_DEBUG)) { \ fprintf(stderr, __FILE__ ": "); \ fprintf(stderr, __VA_ARGS__); \ fprintf(stderr, " \n"); \ } } #else # define DEBUGLOG(l, ...) {} /* disabled */ #endif /*-************************************ * Common functions **************************************/ static unsigned LZ4_NbCommonBytes (REGISTER reg_t val) { if (LZ4_isLittleEndian()) { if (sizeof(val)==8) { # if defined(_MSC_VER) && defined(_WIN64) && !defined(LZ4_FORCE_SW_BITCOUNT) unsigned long r = 0; _BitScanForward64( &r, (U64)val ); return (int)(r>>3); # elif (defined(__clang__) || (defined(__GNUC__) && (__GNUC__>=3))) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_ctzll((U64)val) >> 3); # else static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 }; return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58]; # endif } else /* 32 bits */ { # if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT) unsigned long r; _BitScanForward( &r, (U32)val ); return (int)(r>>3); # elif (defined(__clang__) || (defined(__GNUC__) && (__GNUC__>=3))) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_ctz((U32)val) >> 3); # else static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 }; return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27]; # endif } } else /* Big Endian CPU */ { if (sizeof(val)==8) { /* 64-bits */ # if defined(_MSC_VER) && defined(_WIN64) && !defined(LZ4_FORCE_SW_BITCOUNT) unsigned long r = 0; _BitScanReverse64( &r, val ); return (unsigned)(r>>3); # elif (defined(__clang__) || (defined(__GNUC__) && (__GNUC__>=3))) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_clzll((U64)val) >> 3); # else static const U32 by32 = sizeof(val)*4; /* 32 on 64 bits (goal), 16 on 32 bits. Just to avoid some static analyzer complaining about shift by 32 on 32-bits target. Note that this code path is never triggered in 32-bits mode. */ unsigned r; if (!(val>>by32)) { r=4; } else { r=0; val>>=by32; } if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; } r += (!val); return r; # endif } else /* 32 bits */ { # if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT) unsigned long r = 0; _BitScanReverse( &r, (unsigned long)val ); return (unsigned)(r>>3); # elif (defined(__clang__) || (defined(__GNUC__) && (__GNUC__>=3))) && !defined(LZ4_FORCE_SW_BITCOUNT) return (__builtin_clz((U32)val) >> 3); # else unsigned r; if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; } r += (!val); return r; # endif } } } #define STEPSIZE sizeof(reg_t) LZ4_FORCE_INLINE unsigned LZ4_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* pInLimit) { const BYTE* const pStart = pIn; if (likely(pIn < pInLimit-(STEPSIZE-1))) { reg_t const diff = LZ4_read_ARCH(pMatch) ^ LZ4_read_ARCH(pIn); if (!diff) { pIn+=STEPSIZE; pMatch+=STEPSIZE; } else { return LZ4_NbCommonBytes(diff); } } while (likely(pIn < pInLimit-(STEPSIZE-1))) { reg_t const diff = LZ4_read_ARCH(pMatch) ^ LZ4_read_ARCH(pIn); if (!diff) { pIn+=STEPSIZE; pMatch+=STEPSIZE; continue; } pIn += LZ4_NbCommonBytes(diff); return (unsigned)(pIn - pStart); } if ((STEPSIZE==8) && (pIn<(pInLimit-3)) && (LZ4_read32(pMatch) == LZ4_read32(pIn))) { pIn+=4; pMatch+=4; } if ((pIn<(pInLimit-1)) && (LZ4_read16(pMatch) == LZ4_read16(pIn))) { pIn+=2; pMatch+=2; } if ((pIn compression run slower on incompressible data */ /*-************************************ * Local Structures and types **************************************/ typedef enum { notLimited = 0, limitedOutput = 1 } limitedOutput_directive; typedef enum { byPtr, byU32, byU16 } tableType_t; typedef enum { noDict = 0, withPrefix64k, usingExtDict } dict_directive; typedef enum { noDictIssue = 0, dictSmall } dictIssue_directive; typedef enum { endOnOutputSize = 0, endOnInputSize = 1 } endCondition_directive; typedef enum { full = 0, partial = 1 } earlyEnd_directive; /*-************************************ * Local Utils **************************************/ int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; } const char* LZ4_versionString(void) { return LZ4_VERSION_STRING; } int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); } int LZ4_sizeofState() { return LZ4_STREAMSIZE; } /*-****************************** * Compression functions ********************************/ static U32 LZ4_hash4(U32 sequence, tableType_t const tableType) { if (tableType == byU16) return ((sequence * 2654435761U) >> ((MINMATCH*8)-(LZ4_HASHLOG+1))); else return ((sequence * 2654435761U) >> ((MINMATCH*8)-LZ4_HASHLOG)); } static U32 LZ4_hash5(U64 sequence, tableType_t const tableType) { static const U64 prime5bytes = 889523592379ULL; static const U64 prime8bytes = 11400714785074694791ULL; const U32 hashLog = (tableType == byU16) ? LZ4_HASHLOG+1 : LZ4_HASHLOG; if (LZ4_isLittleEndian()) return (U32)(((sequence << 24) * prime5bytes) >> (64 - hashLog)); else return (U32)(((sequence >> 24) * prime8bytes) >> (64 - hashLog)); } LZ4_FORCE_INLINE U32 LZ4_hashPosition(const void* const p, tableType_t const tableType) { if ((sizeof(reg_t)==8) && (tableType != byU16)) return LZ4_hash5(LZ4_read_ARCH(p), tableType); return LZ4_hash4(LZ4_read32(p), tableType); } static void LZ4_putPositionOnHash(const BYTE* p, U32 h, void* tableBase, tableType_t const tableType, const BYTE* srcBase) { switch (tableType) { case byPtr: { const BYTE** hashTable = (const BYTE**)tableBase; hashTable[h] = p; return; } case byU32: { U32* hashTable = (U32*) tableBase; hashTable[h] = (U32)(p-srcBase); return; } case byU16: { U16* hashTable = (U16*) tableBase; hashTable[h] = (U16)(p-srcBase); return; } } } LZ4_FORCE_INLINE void LZ4_putPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase) { U32 const h = LZ4_hashPosition(p, tableType); LZ4_putPositionOnHash(p, h, tableBase, tableType, srcBase); } static const BYTE* LZ4_getPositionOnHash(U32 h, void* tableBase, tableType_t tableType, const BYTE* srcBase) { if (tableType == byPtr) { const BYTE** hashTable = (const BYTE**) tableBase; return hashTable[h]; } if (tableType == byU32) { const U32* const hashTable = (U32*) tableBase; return hashTable[h] + srcBase; } { const U16* const hashTable = (U16*) tableBase; return hashTable[h] + srcBase; } /* default, to ensure a return */ } LZ4_FORCE_INLINE const BYTE* LZ4_getPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase) { U32 const h = LZ4_hashPosition(p, tableType); return LZ4_getPositionOnHash(h, tableBase, tableType, srcBase); } /** LZ4_compress_generic() : inlined, to ensure branches are decided at compilation time */ LZ4_FORCE_INLINE int LZ4_compress_generic( LZ4_stream_t_internal* const cctx, const char* const source, char* const dest, const int inputSize, const int maxOutputSize, const limitedOutput_directive outputLimited, const tableType_t tableType, const dict_directive dict, const dictIssue_directive dictIssue, const U32 acceleration) { const BYTE* ip = (const BYTE*) source; const BYTE* base; const BYTE* lowLimit; const BYTE* const lowRefLimit = ip - cctx->dictSize; const BYTE* const dictionary = cctx->dictionary; const BYTE* const dictEnd = dictionary + cctx->dictSize; const ptrdiff_t dictDelta = dictEnd - (const BYTE*)source; const BYTE* anchor = (const BYTE*) source; const BYTE* const iend = ip + inputSize; const BYTE* const mflimit = iend - MFLIMIT; const BYTE* const matchlimit = iend - LASTLITERALS; BYTE* op = (BYTE*) dest; BYTE* const olimit = op + maxOutputSize; U32 forwardH; /* Init conditions */ if ((U32)inputSize > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported inputSize, too large (or negative) */ switch(dict) { case noDict: default: base = (const BYTE*)source; lowLimit = (const BYTE*)source; break; case withPrefix64k: base = (const BYTE*)source - cctx->currentOffset; lowLimit = (const BYTE*)source - cctx->dictSize; break; case usingExtDict: base = (const BYTE*)source - cctx->currentOffset; lowLimit = (const BYTE*)source; break; } if ((tableType == byU16) && (inputSize>=LZ4_64Klimit)) return 0; /* Size too large (not within 64K limit) */ if (inputSizehashTable, tableType, base); ip++; forwardH = LZ4_hashPosition(ip, tableType); /* Main Loop */ for ( ; ; ) { ptrdiff_t refDelta = 0; const BYTE* match; BYTE* token; /* Find a match */ { const BYTE* forwardIp = ip; unsigned step = 1; unsigned searchMatchNb = acceleration << LZ4_skipTrigger; do { U32 const h = forwardH; ip = forwardIp; forwardIp += step; step = (searchMatchNb++ >> LZ4_skipTrigger); if (unlikely(forwardIp > mflimit)) goto _last_literals; match = LZ4_getPositionOnHash(h, cctx->hashTable, tableType, base); if (dict==usingExtDict) { if (match < (const BYTE*)source) { refDelta = dictDelta; lowLimit = dictionary; } else { refDelta = 0; lowLimit = (const BYTE*)source; } } forwardH = LZ4_hashPosition(forwardIp, tableType); LZ4_putPositionOnHash(ip, h, cctx->hashTable, tableType, base); } while ( ((dictIssue==dictSmall) ? (match < lowRefLimit) : 0) || ((tableType==byU16) ? 0 : (match + MAX_DISTANCE < ip)) || (LZ4_read32(match+refDelta) != LZ4_read32(ip)) ); } /* Catch up */ while (((ip>anchor) & (match+refDelta > lowLimit)) && (unlikely(ip[-1]==match[refDelta-1]))) { ip--; match--; } /* Encode Literals */ { unsigned const litLength = (unsigned)(ip - anchor); token = op++; if ((outputLimited) && /* Check output buffer overflow */ (unlikely(op + litLength + (2 + 1 + LASTLITERALS) + (litLength/255) > olimit))) return 0; if (litLength >= RUN_MASK) { int len = (int)litLength-RUN_MASK; *token = (RUN_MASK<= 255 ; len-=255) *op++ = 255; *op++ = (BYTE)len; } else *token = (BYTE)(litLength< matchlimit) limit = matchlimit; matchCode = LZ4_count(ip+MINMATCH, match+MINMATCH, limit); ip += MINMATCH + matchCode; if (ip==limit) { unsigned const more = LZ4_count(ip, (const BYTE*)source, matchlimit); matchCode += more; ip += more; } } else { matchCode = LZ4_count(ip+MINMATCH, match+MINMATCH, matchlimit); ip += MINMATCH + matchCode; } if ( outputLimited && /* Check output buffer overflow */ (unlikely(op + (1 + LASTLITERALS) + (matchCode>>8) > olimit)) ) return 0; if (matchCode >= ML_MASK) { *token += ML_MASK; matchCode -= ML_MASK; LZ4_write32(op, 0xFFFFFFFF); while (matchCode >= 4*255) { op+=4; LZ4_write32(op, 0xFFFFFFFF); matchCode -= 4*255; } op += matchCode / 255; *op++ = (BYTE)(matchCode % 255); } else *token += (BYTE)(matchCode); } anchor = ip; /* Test end of chunk */ if (ip > mflimit) break; /* Fill table */ LZ4_putPosition(ip-2, cctx->hashTable, tableType, base); /* Test next position */ match = LZ4_getPosition(ip, cctx->hashTable, tableType, base); if (dict==usingExtDict) { if (match < (const BYTE*)source) { refDelta = dictDelta; lowLimit = dictionary; } else { refDelta = 0; lowLimit = (const BYTE*)source; } } LZ4_putPosition(ip, cctx->hashTable, tableType, base); if ( ((dictIssue==dictSmall) ? (match>=lowRefLimit) : 1) && (match+MAX_DISTANCE>=ip) && (LZ4_read32(match+refDelta)==LZ4_read32(ip)) ) { token=op++; *token=0; goto _next_match; } /* Prepare next loop */ forwardH = LZ4_hashPosition(++ip, tableType); } _last_literals: /* Encode Last Literals */ { size_t const lastRun = (size_t)(iend - anchor); if ( (outputLimited) && /* Check output buffer overflow */ ((op - (BYTE*)dest) + lastRun + 1 + ((lastRun+255-RUN_MASK)/255) > (U32)maxOutputSize) ) return 0; if (lastRun >= RUN_MASK) { size_t accumulator = lastRun - RUN_MASK; *op++ = RUN_MASK << ML_BITS; for(; accumulator >= 255 ; accumulator-=255) *op++ = 255; *op++ = (BYTE) accumulator; } else { *op++ = (BYTE)(lastRun<internal_donotuse; LZ4_resetStream((LZ4_stream_t*)state); if (acceleration < 1) acceleration = ACCELERATION_DEFAULT; if (maxOutputSize >= LZ4_compressBound(inputSize)) { if (inputSize < LZ4_64Klimit) return LZ4_compress_generic(ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, acceleration); else return LZ4_compress_generic(ctx, source, dest, inputSize, 0, notLimited, (sizeof(void*)==8) ? byU32 : byPtr, noDict, noDictIssue, acceleration); } else { if (inputSize < LZ4_64Klimit) return LZ4_compress_generic(ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); else return LZ4_compress_generic(ctx, source, dest, inputSize, maxOutputSize, limitedOutput, (sizeof(void*)==8) ? byU32 : byPtr, noDict, noDictIssue, acceleration); } } int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { #if (LZ4_HEAPMODE) void* ctxPtr = ALLOCATOR(1, sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */ #else LZ4_stream_t ctx; void* const ctxPtr = &ctx; #endif int const result = LZ4_compress_fast_extState(ctxPtr, source, dest, inputSize, maxOutputSize, acceleration); #if (LZ4_HEAPMODE) FREEMEM(ctxPtr); #endif return result; } int LZ4_compress_default(const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_fast(source, dest, inputSize, maxOutputSize, 1); } /* hidden debug function */ /* strangely enough, gcc generates faster code when this function is uncommented, even if unused */ int LZ4_compress_fast_force(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { LZ4_stream_t ctx; LZ4_resetStream(&ctx); if (inputSize < LZ4_64Klimit) return LZ4_compress_generic(&ctx.internal_donotuse, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); else return LZ4_compress_generic(&ctx.internal_donotuse, source, dest, inputSize, maxOutputSize, limitedOutput, sizeof(void*)==8 ? byU32 : byPtr, noDict, noDictIssue, acceleration); } /*-****************************** * *_destSize() variant ********************************/ static int LZ4_compress_destSize_generic( LZ4_stream_t_internal* const ctx, const char* const src, char* const dst, int* const srcSizePtr, const int targetDstSize, const tableType_t tableType) { const BYTE* ip = (const BYTE*) src; const BYTE* base = (const BYTE*) src; const BYTE* lowLimit = (const BYTE*) src; const BYTE* anchor = ip; const BYTE* const iend = ip + *srcSizePtr; const BYTE* const mflimit = iend - MFLIMIT; const BYTE* const matchlimit = iend - LASTLITERALS; BYTE* op = (BYTE*) dst; BYTE* const oend = op + targetDstSize; BYTE* const oMaxLit = op + targetDstSize - 2 /* offset */ - 8 /* because 8+MINMATCH==MFLIMIT */ - 1 /* token */; BYTE* const oMaxMatch = op + targetDstSize - (LASTLITERALS + 1 /* token */); BYTE* const oMaxSeq = oMaxLit - 1 /* token */; U32 forwardH; /* Init conditions */ if (targetDstSize < 1) return 0; /* Impossible to store anything */ if ((U32)*srcSizePtr > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size, too large (or negative) */ if ((tableType == byU16) && (*srcSizePtr>=LZ4_64Klimit)) return 0; /* Size too large (not within 64K limit) */ if (*srcSizePtrhashTable, tableType, base); ip++; forwardH = LZ4_hashPosition(ip, tableType); /* Main Loop */ for ( ; ; ) { const BYTE* match; BYTE* token; /* Find a match */ { const BYTE* forwardIp = ip; unsigned step = 1; unsigned searchMatchNb = 1 << LZ4_skipTrigger; do { U32 h = forwardH; ip = forwardIp; forwardIp += step; step = (searchMatchNb++ >> LZ4_skipTrigger); if (unlikely(forwardIp > mflimit)) goto _last_literals; match = LZ4_getPositionOnHash(h, ctx->hashTable, tableType, base); forwardH = LZ4_hashPosition(forwardIp, tableType); LZ4_putPositionOnHash(ip, h, ctx->hashTable, tableType, base); } while ( ((tableType==byU16) ? 0 : (match + MAX_DISTANCE < ip)) || (LZ4_read32(match) != LZ4_read32(ip)) ); } /* Catch up */ while ((ip>anchor) && (match > lowLimit) && (unlikely(ip[-1]==match[-1]))) { ip--; match--; } /* Encode Literal length */ { unsigned litLength = (unsigned)(ip - anchor); token = op++; if (op + ((litLength+240)/255) + litLength > oMaxLit) { /* Not enough space for a last match */ op--; goto _last_literals; } if (litLength>=RUN_MASK) { unsigned len = litLength - RUN_MASK; *token=(RUN_MASK<= 255 ; len-=255) *op++ = 255; *op++ = (BYTE)len; } else *token = (BYTE)(litLength< oMaxMatch) { /* Match description too long : reduce it */ matchLength = (15-1) + (oMaxMatch-op) * 255; } ip += MINMATCH + matchLength; if (matchLength>=ML_MASK) { *token += ML_MASK; matchLength -= ML_MASK; while (matchLength >= 255) { matchLength-=255; *op++ = 255; } *op++ = (BYTE)matchLength; } else *token += (BYTE)(matchLength); } anchor = ip; /* Test end of block */ if (ip > mflimit) break; if (op > oMaxSeq) break; /* Fill table */ LZ4_putPosition(ip-2, ctx->hashTable, tableType, base); /* Test next position */ match = LZ4_getPosition(ip, ctx->hashTable, tableType, base); LZ4_putPosition(ip, ctx->hashTable, tableType, base); if ( (match+MAX_DISTANCE>=ip) && (LZ4_read32(match)==LZ4_read32(ip)) ) { token=op++; *token=0; goto _next_match; } /* Prepare next loop */ forwardH = LZ4_hashPosition(++ip, tableType); } _last_literals: /* Encode Last Literals */ { size_t lastRunSize = (size_t)(iend - anchor); if (op + 1 /* token */ + ((lastRunSize+240)/255) /* litLength */ + lastRunSize /* literals */ > oend) { /* adapt lastRunSize to fill 'dst' */ lastRunSize = (oend-op) - 1; lastRunSize -= (lastRunSize+240)/255; } ip = anchor + lastRunSize; if (lastRunSize >= RUN_MASK) { size_t accumulator = lastRunSize - RUN_MASK; *op++ = RUN_MASK << ML_BITS; for(; accumulator >= 255 ; accumulator-=255) *op++ = 255; *op++ = (BYTE) accumulator; } else { *op++ = (BYTE)(lastRunSize<= LZ4_compressBound(*srcSizePtr)) { /* compression success is guaranteed */ return LZ4_compress_fast_extState(state, src, dst, *srcSizePtr, targetDstSize, 1); } else { if (*srcSizePtr < LZ4_64Klimit) return LZ4_compress_destSize_generic(&state->internal_donotuse, src, dst, srcSizePtr, targetDstSize, byU16); else return LZ4_compress_destSize_generic(&state->internal_donotuse, src, dst, srcSizePtr, targetDstSize, sizeof(void*)==8 ? byU32 : byPtr); } } int LZ4_compress_destSize(const char* src, char* dst, int* srcSizePtr, int targetDstSize) { #if (LZ4_HEAPMODE) LZ4_stream_t* ctx = (LZ4_stream_t*)ALLOCATOR(1, sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */ #else LZ4_stream_t ctxBody; LZ4_stream_t* ctx = &ctxBody; #endif int result = LZ4_compress_destSize_extState(ctx, src, dst, srcSizePtr, targetDstSize); #if (LZ4_HEAPMODE) FREEMEM(ctx); #endif return result; } /*-****************************** * Streaming functions ********************************/ LZ4_stream_t* LZ4_createStream(void) { LZ4_stream_t* lz4s = (LZ4_stream_t*)ALLOCATOR(8, LZ4_STREAMSIZE_U64); LZ4_STATIC_ASSERT(LZ4_STREAMSIZE >= sizeof(LZ4_stream_t_internal)); /* A compilation error here means LZ4_STREAMSIZE is not large enough */ LZ4_resetStream(lz4s); return lz4s; } void LZ4_resetStream (LZ4_stream_t* LZ4_stream) { DEBUGLOG(4, "LZ4_resetStream"); MEM_INIT(LZ4_stream, 0, sizeof(LZ4_stream_t)); } int LZ4_freeStream (LZ4_stream_t* LZ4_stream) { if (!LZ4_stream) return 0; /* support free on NULL */ FREEMEM(LZ4_stream); return (0); } #define HASH_UNIT sizeof(reg_t) int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) { LZ4_stream_t_internal* dict = &LZ4_dict->internal_donotuse; const BYTE* p = (const BYTE*)dictionary; const BYTE* const dictEnd = p + dictSize; const BYTE* base; if ((dict->initCheck) || (dict->currentOffset > 1 GB)) /* Uninitialized structure, or reuse overflow */ LZ4_resetStream(LZ4_dict); if (dictSize < (int)HASH_UNIT) { dict->dictionary = NULL; dict->dictSize = 0; return 0; } if ((dictEnd - p) > 64 KB) p = dictEnd - 64 KB; dict->currentOffset += 64 KB; base = p - dict->currentOffset; dict->dictionary = p; dict->dictSize = (U32)(dictEnd - p); dict->currentOffset += dict->dictSize; while (p <= dictEnd-HASH_UNIT) { LZ4_putPosition(p, dict->hashTable, byU32, base); p+=3; } return dict->dictSize; } static void LZ4_renormDictT(LZ4_stream_t_internal* LZ4_dict, const BYTE* src) { if ((LZ4_dict->currentOffset > 0x80000000) || ((uptrval)LZ4_dict->currentOffset > (uptrval)src)) { /* address space overflow */ /* rescale hash table */ U32 const delta = LZ4_dict->currentOffset - 64 KB; const BYTE* dictEnd = LZ4_dict->dictionary + LZ4_dict->dictSize; int i; for (i=0; ihashTable[i] < delta) LZ4_dict->hashTable[i]=0; else LZ4_dict->hashTable[i] -= delta; } LZ4_dict->currentOffset = 64 KB; if (LZ4_dict->dictSize > 64 KB) LZ4_dict->dictSize = 64 KB; LZ4_dict->dictionary = dictEnd - LZ4_dict->dictSize; } } int LZ4_compress_fast_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) { LZ4_stream_t_internal* streamPtr = &LZ4_stream->internal_donotuse; const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize; const BYTE* smallest = (const BYTE*) source; if (streamPtr->initCheck) return 0; /* Uninitialized structure detected */ if ((streamPtr->dictSize>0) && (smallest>dictEnd)) smallest = dictEnd; LZ4_renormDictT(streamPtr, smallest); if (acceleration < 1) acceleration = ACCELERATION_DEFAULT; /* Check overlapping input/dictionary space */ { const BYTE* sourceEnd = (const BYTE*) source + inputSize; if ((sourceEnd > streamPtr->dictionary) && (sourceEnd < dictEnd)) { streamPtr->dictSize = (U32)(dictEnd - sourceEnd); if (streamPtr->dictSize > 64 KB) streamPtr->dictSize = 64 KB; if (streamPtr->dictSize < 4) streamPtr->dictSize = 0; streamPtr->dictionary = dictEnd - streamPtr->dictSize; } } /* prefix mode : source data follows dictionary */ if (dictEnd == (const BYTE*)source) { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) result = LZ4_compress_generic(streamPtr, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, dictSmall, acceleration); else result = LZ4_compress_generic(streamPtr, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, noDictIssue, acceleration); streamPtr->dictSize += (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; return result; } /* external dictionary mode */ { int result; if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) result = LZ4_compress_generic(streamPtr, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, dictSmall, acceleration); else result = LZ4_compress_generic(streamPtr, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, noDictIssue, acceleration); streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; return result; } } /* Hidden debug function, to force external dictionary mode */ int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* dest, int inputSize) { LZ4_stream_t_internal* streamPtr = &LZ4_dict->internal_donotuse; int result; const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize; const BYTE* smallest = dictEnd; if (smallest > (const BYTE*) source) smallest = (const BYTE*) source; LZ4_renormDictT(streamPtr, smallest); result = LZ4_compress_generic(streamPtr, source, dest, inputSize, 0, notLimited, byU32, usingExtDict, noDictIssue, 1); streamPtr->dictionary = (const BYTE*)source; streamPtr->dictSize = (U32)inputSize; streamPtr->currentOffset += (U32)inputSize; return result; } /*! LZ4_saveDict() : * If previously compressed data block is not guaranteed to remain available at its memory location, * save it into a safer place (char* safeBuffer). * Note : you don't need to call LZ4_loadDict() afterwards, * dictionary is immediately usable, you can therefore call LZ4_compress_fast_continue(). * Return : saved dictionary size in bytes (necessarily <= dictSize), or 0 if error. */ int LZ4_saveDict (LZ4_stream_t* LZ4_dict, char* safeBuffer, int dictSize) { LZ4_stream_t_internal* const dict = &LZ4_dict->internal_donotuse; const BYTE* const previousDictEnd = dict->dictionary + dict->dictSize; if ((U32)dictSize > 64 KB) dictSize = 64 KB; /* useless to define a dictionary > 64 KB */ if ((U32)dictSize > dict->dictSize) dictSize = dict->dictSize; memmove(safeBuffer, previousDictEnd - dictSize, dictSize); dict->dictionary = (const BYTE*)safeBuffer; dict->dictSize = (U32)dictSize; return dictSize; } /*-***************************** * Decompression functions *******************************/ /*! LZ4_decompress_generic() : * This generic decompression function covers all use cases. * It shall be instantiated several times, using different sets of directives. * Note that it is important for performance that this function really get inlined, * in order to remove useless branches during compilation optimization. */ LZ4_FORCE_O2_GCC_PPC64LE LZ4_FORCE_INLINE int LZ4_decompress_generic( const char* const src, char* const dst, int srcSize, int outputSize, /* If endOnInput==endOnInputSize, this value is `dstCapacity` */ int endOnInput, /* endOnOutputSize, endOnInputSize */ int partialDecoding, /* full, partial */ int targetOutputSize, /* only used if partialDecoding==partial */ int dict, /* noDict, withPrefix64k, usingExtDict */ const BYTE* const lowPrefix, /* always <= dst, == dst when no prefix */ const BYTE* const dictStart, /* only if dict==usingExtDict */ const size_t dictSize /* note : = 0 if noDict */ ) { const BYTE* ip = (const BYTE*) src; const BYTE* const iend = ip + srcSize; BYTE* op = (BYTE*) dst; BYTE* const oend = op + outputSize; BYTE* cpy; BYTE* oexit = op + targetOutputSize; const BYTE* const dictEnd = (const BYTE*)dictStart + dictSize; const unsigned inc32table[8] = {0, 1, 2, 1, 0, 4, 4, 4}; const int dec64table[8] = {0, 0, 0, -1, -4, 1, 2, 3}; const int safeDecode = (endOnInput==endOnInputSize); const int checkOffset = ((safeDecode) && (dictSize < (int)(64 KB))); /* Special cases */ if ((partialDecoding) && (oexit > oend-MFLIMIT)) oexit = oend-MFLIMIT; /* targetOutputSize too high => just decode everything */ if ((endOnInput) && (unlikely(outputSize==0))) return ((srcSize==1) && (*ip==0)) ? 0 : -1; /* Empty output buffer */ if ((!endOnInput) && (unlikely(outputSize==0))) return (*ip==0?1:-1); /* Main Loop : decode sequences */ while (1) { size_t length; const BYTE* match; size_t offset; unsigned const token = *ip++; /* shortcut for common case : * in most circumstances, we expect to decode small matches (<= 18 bytes) separated by few literals (<= 14 bytes). * this shortcut was tested on x86 and x64, where it improves decoding speed. * it has not yet been benchmarked on ARM, Power, mips, etc. */ if (((ip + 14 /*maxLL*/ + 2 /*offset*/ <= iend) & (op + 14 /*maxLL*/ + 18 /*maxML*/ <= oend)) & ((token < (15<> ML_BITS; size_t const off = LZ4_readLE16(ip+ll); const BYTE* const matchPtr = op + ll - off; /* pointer underflow risk ? */ if ((off >= 18) /* do not deal with overlapping matches */ & (matchPtr >= lowPrefix)) { size_t const ml = (token & ML_MASK) + MINMATCH; memcpy(op, ip, 16); op += ll; ip += ll + 2 /*offset*/; memcpy(op, matchPtr, 18); op += ml; continue; } } /* decode literal length */ if ((length=(token>>ML_BITS)) == RUN_MASK) { unsigned s; do { s = *ip++; length += s; } while ( likely(endOnInput ? ip(partialDecoding?oexit:oend-MFLIMIT)) || (ip+length>iend-(2+1+LASTLITERALS))) ) || ((!endOnInput) && (cpy>oend-WILDCOPYLENGTH)) ) { if (partialDecoding) { if (cpy > oend) goto _output_error; /* Error : write attempt beyond end of output buffer */ if ((endOnInput) && (ip+length > iend)) goto _output_error; /* Error : read attempt beyond end of input buffer */ } else { if ((!endOnInput) && (cpy != oend)) goto _output_error; /* Error : block decoding must stop exactly there */ if ((endOnInput) && ((ip+length != iend) || (cpy > oend))) goto _output_error; /* Error : input must be consumed */ } memcpy(op, ip, length); ip += length; op += length; break; /* Necessarily EOF, due to parsing restrictions */ } LZ4_wildCopy(op, ip, cpy); ip += length; op = cpy; /* get offset */ offset = LZ4_readLE16(ip); ip+=2; match = op - offset; if ((checkOffset) && (unlikely(match + dictSize < lowPrefix))) goto _output_error; /* Error : offset outside buffers */ LZ4_write32(op, (U32)offset); /* costs ~1%; silence an msan warning when offset==0 */ /* get matchlength */ length = token & ML_MASK; if (length == ML_MASK) { unsigned s; do { s = *ip++; if ((endOnInput) && (ip > iend-LASTLITERALS)) goto _output_error; length += s; } while (s==255); if ((safeDecode) && unlikely((uptrval)(op)+length<(uptrval)op)) goto _output_error; /* overflow detection */ } length += MINMATCH; /* check external dictionary */ if ((dict==usingExtDict) && (match < lowPrefix)) { if (unlikely(op+length > oend-LASTLITERALS)) goto _output_error; /* doesn't respect parsing restriction */ if (length <= (size_t)(lowPrefix-match)) { /* match can be copied as a single segment from external dictionary */ memmove(op, dictEnd - (lowPrefix-match), length); op += length; } else { /* match encompass external dictionary and current block */ size_t const copySize = (size_t)(lowPrefix-match); size_t const restSize = length - copySize; memcpy(op, dictEnd - copySize, copySize); op += copySize; if (restSize > (size_t)(op-lowPrefix)) { /* overlap copy */ BYTE* const endOfMatch = op + restSize; const BYTE* copyFrom = lowPrefix; while (op < endOfMatch) *op++ = *copyFrom++; } else { memcpy(op, lowPrefix, restSize); op += restSize; } } continue; } /* copy match within block */ cpy = op + length; if (unlikely(offset<8)) { op[0] = match[0]; op[1] = match[1]; op[2] = match[2]; op[3] = match[3]; match += inc32table[offset]; memcpy(op+4, match, 4); match -= dec64table[offset]; } else { LZ4_copy8(op, match); match+=8; } op += 8; if (unlikely(cpy>oend-12)) { BYTE* const oCopyLimit = oend-(WILDCOPYLENGTH-1); if (cpy > oend-LASTLITERALS) goto _output_error; /* Error : last LASTLITERALS bytes must be literals (uncompressed) */ if (op < oCopyLimit) { LZ4_wildCopy(op, match, oCopyLimit); match += oCopyLimit - op; op = oCopyLimit; } while (op16) LZ4_wildCopy(op+8, match+8, cpy); } op = cpy; /* correction */ } /* end of decoding */ if (endOnInput) return (int) (((char*)op)-dst); /* Nb of output bytes decoded */ else return (int) (((const char*)ip)-src); /* Nb of input bytes read */ /* Overflow error detected */ _output_error: return (int) (-(((const char*)ip)-src))-1; } LZ4_FORCE_O2_GCC_PPC64LE int LZ4_decompress_safe(const char* source, char* dest, int compressedSize, int maxDecompressedSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxDecompressedSize, endOnInputSize, full, 0, noDict, (BYTE*)dest, NULL, 0); } LZ4_FORCE_O2_GCC_PPC64LE int LZ4_decompress_safe_partial(const char* source, char* dest, int compressedSize, int targetOutputSize, int maxDecompressedSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxDecompressedSize, endOnInputSize, partial, targetOutputSize, noDict, (BYTE*)dest, NULL, 0); } LZ4_FORCE_O2_GCC_PPC64LE int LZ4_decompress_fast(const char* source, char* dest, int originalSize) { return LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, withPrefix64k, (BYTE*)(dest - 64 KB), NULL, 64 KB); } /*===== streaming decompression functions =====*/ LZ4_streamDecode_t* LZ4_createStreamDecode(void) { LZ4_streamDecode_t* lz4s = (LZ4_streamDecode_t*) ALLOCATOR(1, sizeof(LZ4_streamDecode_t)); return lz4s; } int LZ4_freeStreamDecode (LZ4_streamDecode_t* LZ4_stream) { if (!LZ4_stream) return 0; /* support free on NULL */ FREEMEM(LZ4_stream); return 0; } /*! * LZ4_setStreamDecode() : * Use this function to instruct where to find the dictionary. * This function is not necessary if previous data is still available where it was decoded. * Loading a size of 0 is allowed (same effect as no dictionary). * Return : 1 if OK, 0 if error */ int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dictionary, int dictSize) { LZ4_streamDecode_t_internal* lz4sd = &LZ4_streamDecode->internal_donotuse; lz4sd->prefixSize = (size_t) dictSize; lz4sd->prefixEnd = (const BYTE*) dictionary + dictSize; lz4sd->externalDict = NULL; lz4sd->extDictSize = 0; return 1; } /* *_continue() : These decoding functions allow decompression of multiple blocks in "streaming" mode. Previously decoded blocks must still be available at the memory position where they were decoded. If it's not possible, save the relevant part of decoded data into a safe buffer, and indicate where it stands using LZ4_setStreamDecode() */ LZ4_FORCE_O2_GCC_PPC64LE int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int compressedSize, int maxOutputSize) { LZ4_streamDecode_t_internal* lz4sd = &LZ4_streamDecode->internal_donotuse; int result; if (lz4sd->prefixEnd == (BYTE*)dest) { result = LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, usingExtDict, lz4sd->prefixEnd - lz4sd->prefixSize, lz4sd->externalDict, lz4sd->extDictSize); if (result <= 0) return result; lz4sd->prefixSize += result; lz4sd->prefixEnd += result; } else { lz4sd->extDictSize = lz4sd->prefixSize; lz4sd->externalDict = lz4sd->prefixEnd - lz4sd->extDictSize; result = LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, usingExtDict, (BYTE*)dest, lz4sd->externalDict, lz4sd->extDictSize); if (result <= 0) return result; lz4sd->prefixSize = result; lz4sd->prefixEnd = (BYTE*)dest + result; } return result; } LZ4_FORCE_O2_GCC_PPC64LE int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int originalSize) { LZ4_streamDecode_t_internal* lz4sd = &LZ4_streamDecode->internal_donotuse; int result; if (lz4sd->prefixEnd == (BYTE*)dest) { result = LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, usingExtDict, lz4sd->prefixEnd - lz4sd->prefixSize, lz4sd->externalDict, lz4sd->extDictSize); if (result <= 0) return result; lz4sd->prefixSize += originalSize; lz4sd->prefixEnd += originalSize; } else { lz4sd->extDictSize = lz4sd->prefixSize; lz4sd->externalDict = lz4sd->prefixEnd - lz4sd->extDictSize; result = LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, usingExtDict, (BYTE*)dest, lz4sd->externalDict, lz4sd->extDictSize); if (result <= 0) return result; lz4sd->prefixSize = originalSize; lz4sd->prefixEnd = (BYTE*)dest + originalSize; } return result; } /* Advanced decoding functions : *_usingDict() : These decoding functions work the same as "_continue" ones, the dictionary must be explicitly provided within parameters */ LZ4_FORCE_O2_GCC_PPC64LE LZ4_FORCE_INLINE int LZ4_decompress_usingDict_generic(const char* source, char* dest, int compressedSize, int maxOutputSize, int safe, const char* dictStart, int dictSize) { if (dictSize==0) return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, noDict, (BYTE*)dest, NULL, 0); if (dictStart+dictSize == dest) { if (dictSize >= (int)(64 KB - 1)) return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, withPrefix64k, (BYTE*)dest-64 KB, NULL, 0); return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, noDict, (BYTE*)dest-dictSize, NULL, 0); } return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, usingExtDict, (BYTE*)dest, (const BYTE*)dictStart, dictSize); } LZ4_FORCE_O2_GCC_PPC64LE int LZ4_decompress_safe_usingDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize) { return LZ4_decompress_usingDict_generic(source, dest, compressedSize, maxOutputSize, 1, dictStart, dictSize); } LZ4_FORCE_O2_GCC_PPC64LE int LZ4_decompress_fast_usingDict(const char* source, char* dest, int originalSize, const char* dictStart, int dictSize) { return LZ4_decompress_usingDict_generic(source, dest, 0, originalSize, 0, dictStart, dictSize); } /* debug function */ LZ4_FORCE_O2_GCC_PPC64LE int LZ4_decompress_safe_forceExtDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, usingExtDict, (BYTE*)dest, (const BYTE*)dictStart, dictSize); } /*=************************************************* * Obsolete Functions ***************************************************/ /* obsolete compression functions */ int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_default(source, dest, inputSize, maxOutputSize); } int LZ4_compress(const char* source, char* dest, int inputSize) { return LZ4_compress_default(source, dest, inputSize, LZ4_compressBound(inputSize)); } int LZ4_compress_limitedOutput_withState (void* state, const char* src, char* dst, int srcSize, int dstSize) { return LZ4_compress_fast_extState(state, src, dst, srcSize, dstSize, 1); } int LZ4_compress_withState (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_fast_extState(state, src, dst, srcSize, LZ4_compressBound(srcSize), 1); } int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_fast_continue(LZ4_stream, src, dst, srcSize, maxDstSize, 1); } int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) { return LZ4_compress_fast_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize), 1); } /* These function names are deprecated and should no longer be used. They are only provided here for compatibility with older user programs. - LZ4_uncompress is totally equivalent to LZ4_decompress_fast - LZ4_uncompress_unknownOutputSize is totally equivalent to LZ4_decompress_safe */ int LZ4_uncompress (const char* source, char* dest, int outputSize) { return LZ4_decompress_fast(source, dest, outputSize); } int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize) { return LZ4_decompress_safe(source, dest, isize, maxOutputSize); } /* Obsolete Streaming functions */ int LZ4_sizeofStreamState() { return LZ4_STREAMSIZE; } static void LZ4_init(LZ4_stream_t* lz4ds, BYTE* base) { MEM_INIT(lz4ds, 0, sizeof(LZ4_stream_t)); lz4ds->internal_donotuse.bufferStart = base; } int LZ4_resetStreamState(void* state, char* inputBuffer) { if ((((uptrval)state) & 3) != 0) return 1; /* Error : pointer is not aligned on 4-bytes boundary */ LZ4_init((LZ4_stream_t*)state, (BYTE*)inputBuffer); return 0; } void* LZ4_create (char* inputBuffer) { LZ4_stream_t* lz4ds = (LZ4_stream_t*)ALLOCATOR(8, sizeof(LZ4_stream_t)); LZ4_init (lz4ds, (BYTE*)inputBuffer); return lz4ds; } char* LZ4_slideInputBuffer (void* LZ4_Data) { LZ4_stream_t_internal* ctx = &((LZ4_stream_t*)LZ4_Data)->internal_donotuse; int dictSize = LZ4_saveDict((LZ4_stream_t*)LZ4_Data, (char*)ctx->bufferStart, 64 KB); return (char*)(ctx->bufferStart + dictSize); } /* Obsolete streaming decompression functions */ int LZ4_decompress_safe_withPrefix64k(const char* source, char* dest, int compressedSize, int maxOutputSize) { return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, withPrefix64k, (BYTE*)dest - 64 KB, NULL, 64 KB); } int LZ4_decompress_fast_withPrefix64k(const char* source, char* dest, int originalSize) { return LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, withPrefix64k, (BYTE*)dest - 64 KB, NULL, 64 KB); } #endif /* LZ4_COMMONDEFS_ONLY */ nfdump-1.6.23/bin/lz4.h000077500000000000000000000602241404501030700145270ustar00rootroot00000000000000/* * LZ4 - Fast LZ compression algorithm * Header File * Copyright (C) 2011-2017, Yann Collet. BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - LZ4 homepage : http://www.lz4.org - LZ4 source repository : https://github.com/lz4/lz4 */ #if defined (__cplusplus) extern "C" { #endif #ifndef LZ4_H_2983827168210 #define LZ4_H_2983827168210 /* --- Dependency --- */ #include /* size_t */ /** Introduction LZ4 is lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems. The LZ4 compression library provides in-memory compression and decompression functions. Compression can be done in: - a single step (described as Simple Functions) - a single step, reusing a context (described in Advanced Functions) - unbounded multiple steps (described as Streaming compression) lz4.h provides block compression functions. It gives full buffer control to user. Decompressing an lz4-compressed block also requires metadata (such as compressed size). Each application is free to encode such metadata in whichever way it wants. An additional format, called LZ4 frame specification (doc/lz4_Frame_format.md), take care of encoding standard metadata alongside LZ4-compressed blocks. If your application requires interoperability, it's recommended to use it. A library is provided to take care of it, see lz4frame.h. */ /*^*************************************************************** * Export parameters *****************************************************************/ /* * LZ4_DLL_EXPORT : * Enable exporting of functions when building a Windows DLL * LZ4LIB_VISIBILITY : * Control library symbols visibility. */ #ifndef LZ4LIB_VISIBILITY # if defined(__GNUC__) && (__GNUC__ >= 4) # define LZ4LIB_VISIBILITY __attribute__ ((visibility ("default"))) # else # define LZ4LIB_VISIBILITY # endif #endif #if defined(LZ4_DLL_EXPORT) && (LZ4_DLL_EXPORT==1) # define LZ4LIB_API __declspec(dllexport) LZ4LIB_VISIBILITY #elif defined(LZ4_DLL_IMPORT) && (LZ4_DLL_IMPORT==1) # define LZ4LIB_API __declspec(dllimport) LZ4LIB_VISIBILITY /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/ #else # define LZ4LIB_API LZ4LIB_VISIBILITY #endif /*------ Version ------*/ #define LZ4_VERSION_MAJOR 1 /* for breaking interface changes */ #define LZ4_VERSION_MINOR 8 /* for new (non-breaking) interface capabilities */ #define LZ4_VERSION_RELEASE 0 /* for tweaks, bug-fixes, or development */ #define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE) #define LZ4_LIB_VERSION LZ4_VERSION_MAJOR.LZ4_VERSION_MINOR.LZ4_VERSION_RELEASE #define LZ4_QUOTE(str) #str #define LZ4_EXPAND_AND_QUOTE(str) LZ4_QUOTE(str) #define LZ4_VERSION_STRING LZ4_EXPAND_AND_QUOTE(LZ4_LIB_VERSION) LZ4LIB_API int LZ4_versionNumber (void); /**< library version number; to be used when checking dll version */ LZ4LIB_API const char* LZ4_versionString (void); /**< library version string; to be used when checking dll version */ /*-************************************ * Tuning parameter **************************************/ /*! * LZ4_MEMORY_USAGE : * Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.) * Increasing memory usage improves compression ratio * Reduced memory usage can improve speed, due to cache effect * Default value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */ #ifndef LZ4_MEMORY_USAGE # define LZ4_MEMORY_USAGE 14 #endif /*-************************************ * Simple Functions **************************************/ /*! LZ4_compress_default() : Compresses 'srcSize' bytes from buffer 'src' into already allocated 'dst' buffer of size 'dstCapacity'. Compression is guaranteed to succeed if 'dstCapacity' >= LZ4_compressBound(srcSize). It also runs faster, so it's a recommended setting. If the function cannot compress 'src' into a limited 'dst' budget, compression stops *immediately*, and the function result is zero. As a consequence, 'dst' content is not valid. This function never writes outside 'dst' buffer, nor read outside 'source' buffer. srcSize : supported max value is LZ4_MAX_INPUT_VALUE dstCapacity : full or partial size of buffer 'dst' (which must be already allocated) return : the number of bytes written into buffer 'dst' (necessarily <= dstCapacity) or 0 if compression fails */ LZ4LIB_API int LZ4_compress_default(const char* src, char* dst, int srcSize, int dstCapacity); /*! LZ4_decompress_safe() : compressedSize : is the exact complete size of the compressed block. dstCapacity : is the size of destination buffer, which must be already allocated. return : the number of bytes decompressed into destination buffer (necessarily <= dstCapacity) If destination buffer is not large enough, decoding will stop and output an error code (negative value). If the source stream is detected malformed, the function will stop decoding and return a negative result. This function is protected against buffer overflow exploits, including malicious data packets. It never writes outside output buffer, nor reads outside input buffer. */ LZ4LIB_API int LZ4_decompress_safe (const char* src, char* dst, int compressedSize, int dstCapacity); /*-************************************ * Advanced Functions **************************************/ #define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */ #define LZ4_COMPRESSBOUND(isize) ((unsigned)(isize) > (unsigned)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16) /*! LZ4_compressBound() : Provides the maximum size that LZ4 compression may output in a "worst case" scenario (input data not compressible) This function is primarily useful for memory allocation purposes (destination buffer size). Macro LZ4_COMPRESSBOUND() is also provided for compilation-time evaluation (stack memory allocation for example). Note that LZ4_compress_default() compress faster when dest buffer size is >= LZ4_compressBound(srcSize) inputSize : max supported value is LZ4_MAX_INPUT_SIZE return : maximum output size in a "worst case" scenario or 0, if input size is too large ( > LZ4_MAX_INPUT_SIZE) */ LZ4LIB_API int LZ4_compressBound(int inputSize); /*! LZ4_compress_fast() : Same as LZ4_compress_default(), but allows to select an "acceleration" factor. The larger the acceleration value, the faster the algorithm, but also the lesser the compression. It's a trade-off. It can be fine tuned, with each successive value providing roughly +~3% to speed. An acceleration value of "1" is the same as regular LZ4_compress_default() Values <= 0 will be replaced by ACCELERATION_DEFAULT (see lz4.c), which is 1. */ LZ4LIB_API int LZ4_compress_fast (const char* src, char* dst, int srcSize, int dstCapacity, int acceleration); /*! LZ4_compress_fast_extState() : Same compression function, just using an externally allocated memory space to store compression state. Use LZ4_sizeofState() to know how much memory must be allocated, and allocate it on 8-bytes boundaries (using malloc() typically). Then, provide it as 'void* state' to compression function. */ LZ4LIB_API int LZ4_sizeofState(void); LZ4LIB_API int LZ4_compress_fast_extState (void* state, const char* src, char* dst, int srcSize, int dstCapacity, int acceleration); /*! LZ4_compress_destSize() : Reverse the logic : compresses as much data as possible from 'src' buffer into already allocated buffer 'dst' of size 'targetDestSize'. This function either compresses the entire 'src' content into 'dst' if it's large enough, or fill 'dst' buffer completely with as much data as possible from 'src'. *srcSizePtr : will be modified to indicate how many bytes where read from 'src' to fill 'dst'. New value is necessarily <= old value. return : Nb bytes written into 'dst' (necessarily <= targetDestSize) or 0 if compression fails */ LZ4LIB_API int LZ4_compress_destSize (const char* src, char* dst, int* srcSizePtr, int targetDstSize); /*! LZ4_decompress_fast() : (unsafe!!) originalSize : is the original uncompressed size return : the number of bytes read from the source buffer (in other words, the compressed size) If the source stream is detected malformed, the function will stop decoding and return a negative result. Destination buffer must be already allocated. Its size must be >= 'originalSize' bytes. note : This function respects memory boundaries for *properly formed* compressed data. It is a bit faster than LZ4_decompress_safe(). However, it does not provide any protection against intentionally modified data stream (malicious input). Use this function in trusted environment only (data to decode comes from a trusted source). */ LZ4LIB_API int LZ4_decompress_fast (const char* src, char* dst, int originalSize); /*! LZ4_decompress_safe_partial() : This function decompress a compressed block of size 'srcSize' at position 'src' into destination buffer 'dst' of size 'dstCapacity'. The function will decompress a minimum of 'targetOutputSize' bytes, and stop after that. However, it's not accurate, and may write more than 'targetOutputSize' (but <= dstCapacity). @return : the number of bytes decoded in the destination buffer (necessarily <= dstCapacity) Note : this number can be < 'targetOutputSize' should the compressed block contain less data. Always control how many bytes were decoded. If the source stream is detected malformed, the function will stop decoding and return a negative result. This function never writes outside of output buffer, and never reads outside of input buffer. It is therefore protected against malicious data packets. */ LZ4LIB_API int LZ4_decompress_safe_partial (const char* src, char* dst, int srcSize, int targetOutputSize, int dstCapacity); /*-********************************************* * Streaming Compression Functions ***********************************************/ typedef union LZ4_stream_u LZ4_stream_t; /* incomplete type (defined later) */ /*! LZ4_createStream() and LZ4_freeStream() : * LZ4_createStream() will allocate and initialize an `LZ4_stream_t` structure. * LZ4_freeStream() releases its memory. */ LZ4LIB_API LZ4_stream_t* LZ4_createStream(void); LZ4LIB_API int LZ4_freeStream (LZ4_stream_t* streamPtr); /*! LZ4_resetStream() : * An LZ4_stream_t structure can be allocated once and re-used multiple times. * Use this function to start compressing a new stream. */ LZ4LIB_API void LZ4_resetStream (LZ4_stream_t* streamPtr); /*! LZ4_loadDict() : * Use this function to load a static dictionary into LZ4_stream_t. * Any previous data will be forgotten, only 'dictionary' will remain in memory. * Loading a size of 0 is allowed, and is the same as reset. * @return : dictionary size, in bytes (necessarily <= 64 KB) */ LZ4LIB_API int LZ4_loadDict (LZ4_stream_t* streamPtr, const char* dictionary, int dictSize); /*! LZ4_compress_fast_continue() : * Compress content into 'src' using data from previously compressed blocks, improving compression ratio. * 'dst' buffer must be already allocated. * If dstCapacity >= LZ4_compressBound(srcSize), compression is guaranteed to succeed, and runs faster. * * Important : Up to 64KB of previously compressed data is assumed to remain present and unmodified in memory ! * Special 1 : If input buffer is a double-buffer, it can have any size, including < 64 KB. * Special 2 : If input buffer is a ring-buffer, it can have any size, including < 64 KB. * * @return : size of compressed block * or 0 if there is an error (typically, compressed data cannot fit into 'dst') * After an error, the stream status is invalid, it can only be reset or freed. */ LZ4LIB_API int LZ4_compress_fast_continue (LZ4_stream_t* streamPtr, const char* src, char* dst, int srcSize, int dstCapacity, int acceleration); /*! LZ4_saveDict() : * If previously compressed data block is not guaranteed to remain available at its current memory location, * save it into a safer place (char* safeBuffer). * Note : it's not necessary to call LZ4_loadDict() after LZ4_saveDict(), dictionary is immediately usable. * @return : saved dictionary size in bytes (necessarily <= dictSize), or 0 if error. */ LZ4LIB_API int LZ4_saveDict (LZ4_stream_t* streamPtr, char* safeBuffer, int dictSize); // add missing prototypes int LZ4_compress_fast_force(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration); int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* dest, int inputSize); int LZ4_decompress_safe_forceExtDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize); /*-********************************************** * Streaming Decompression Functions * Bufferless synchronous API ************************************************/ typedef union LZ4_streamDecode_u LZ4_streamDecode_t; /* incomplete type (defined later) */ /*! LZ4_createStreamDecode() and LZ4_freeStreamDecode() : * creation / destruction of streaming decompression tracking structure. * A tracking structure can be re-used multiple times sequentially. */ LZ4LIB_API LZ4_streamDecode_t* LZ4_createStreamDecode(void); LZ4LIB_API int LZ4_freeStreamDecode (LZ4_streamDecode_t* LZ4_stream); /*! LZ4_setStreamDecode() : * An LZ4_streamDecode_t structure can be allocated once and re-used multiple times. * Use this function to start decompression of a new stream of blocks. * A dictionary can optionnally be set. Use NULL or size 0 for a simple reset order. * @return : 1 if OK, 0 if error */ LZ4LIB_API int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dictionary, int dictSize); /*! LZ4_decompress_*_continue() : * These decoding functions allow decompression of consecutive blocks in "streaming" mode. * A block is an unsplittable entity, it must be presented entirely to a decompression function. * Decompression functions only accept one block at a time. * Previously decoded blocks *must* remain available at the memory position where they were decoded (up to 64 KB). * * Special : if application sets a ring buffer for decompression, it must respect one of the following conditions : * - Exactly same size as encoding buffer, with same update rule (block boundaries at same positions) * In which case, the decoding & encoding ring buffer can have any size, including very small ones ( < 64 KB). * - Larger than encoding buffer, by a minimum of maxBlockSize more bytes. * maxBlockSize is implementation dependent. It's the maximum size of any single block. * In which case, encoding and decoding buffers do not need to be synchronized, * and encoding ring buffer can have any size, including small ones ( < 64 KB). * - _At least_ 64 KB + 8 bytes + maxBlockSize. * In which case, encoding and decoding buffers do not need to be synchronized, * and encoding ring buffer can have any size, including larger than decoding buffer. * Whenever these conditions are not possible, save the last 64KB of decoded data into a safe buffer, * and indicate where it is saved using LZ4_setStreamDecode() before decompressing next block. */ LZ4LIB_API int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* src, char* dst, int srcSize, int dstCapacity); LZ4LIB_API int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* src, char* dst, int originalSize); /*! LZ4_decompress_*_usingDict() : * These decoding functions work the same as * a combination of LZ4_setStreamDecode() followed by LZ4_decompress_*_continue() * They are stand-alone, and don't need an LZ4_streamDecode_t structure. */ LZ4LIB_API int LZ4_decompress_safe_usingDict (const char* src, char* dst, int srcSize, int dstCapcity, const char* dictStart, int dictSize); LZ4LIB_API int LZ4_decompress_fast_usingDict (const char* src, char* dst, int originalSize, const char* dictStart, int dictSize); /*^********************************************** * !!!!!! STATIC LINKING ONLY !!!!!! ***********************************************/ /*-************************************ * Private definitions ************************************** * Do not use these definitions. * They are exposed to allow static allocation of `LZ4_stream_t` and `LZ4_streamDecode_t`. * Using these definitions will expose code to API and/or ABI break in future versions of the library. **************************************/ #define LZ4_HASHLOG (LZ4_MEMORY_USAGE-2) #define LZ4_HASHTABLESIZE (1 << LZ4_MEMORY_USAGE) #define LZ4_HASH_SIZE_U32 (1 << LZ4_HASHLOG) /* required as macro for static allocation */ #if defined(__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) #include typedef struct { uint32_t hashTable[LZ4_HASH_SIZE_U32]; uint32_t currentOffset; uint32_t initCheck; const uint8_t* dictionary; uint8_t* bufferStart; /* obsolete, used for slideInputBuffer */ uint32_t dictSize; } LZ4_stream_t_internal; typedef struct { const uint8_t* externalDict; size_t extDictSize; const uint8_t* prefixEnd; size_t prefixSize; } LZ4_streamDecode_t_internal; #else typedef struct { unsigned int hashTable[LZ4_HASH_SIZE_U32]; unsigned int currentOffset; unsigned int initCheck; const unsigned char* dictionary; unsigned char* bufferStart; /* obsolete, used for slideInputBuffer */ unsigned int dictSize; } LZ4_stream_t_internal; typedef struct { const unsigned char* externalDict; size_t extDictSize; const unsigned char* prefixEnd; size_t prefixSize; } LZ4_streamDecode_t_internal; #endif /*! * LZ4_stream_t : * information structure to track an LZ4 stream. * init this structure before first use. * note : only use in association with static linking ! * this definition is not API/ABI safe, * it may change in a future version ! */ #define LZ4_STREAMSIZE_U64 ((1 << (LZ4_MEMORY_USAGE-3)) + 4) #define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U64 * sizeof(unsigned long long)) union LZ4_stream_u { unsigned long long table[LZ4_STREAMSIZE_U64]; LZ4_stream_t_internal internal_donotuse; } ; /* previously typedef'd to LZ4_stream_t */ /*! * LZ4_streamDecode_t : * information structure to track an LZ4 stream during decompression. * init this structure using LZ4_setStreamDecode (or memset()) before first use * note : only use in association with static linking ! * this definition is not API/ABI safe, * and may change in a future version ! */ #define LZ4_STREAMDECODESIZE_U64 4 #define LZ4_STREAMDECODESIZE (LZ4_STREAMDECODESIZE_U64 * sizeof(unsigned long long)) union LZ4_streamDecode_u { unsigned long long table[LZ4_STREAMDECODESIZE_U64]; LZ4_streamDecode_t_internal internal_donotuse; } ; /* previously typedef'd to LZ4_streamDecode_t */ /*-************************************ * Obsolete Functions **************************************/ /*! Deprecation warnings Should deprecation warnings be a problem, it is generally possible to disable them, typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual. Otherwise, it's also possible to define LZ4_DISABLE_DEPRECATE_WARNINGS */ #ifdef LZ4_DISABLE_DEPRECATE_WARNINGS # define LZ4_DEPRECATED(message) /* disable deprecation warnings */ #else # define LZ4_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) # if defined(__clang__) /* clang doesn't handle mixed C++11 and CNU attributes */ # define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) # elif defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */ # define LZ4_DEPRECATED(message) [[deprecated(message)]] # elif (LZ4_GCC_VERSION >= 405) # define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) # elif (LZ4_GCC_VERSION >= 301) # define LZ4_DEPRECATED(message) __attribute__((deprecated)) # elif defined(_MSC_VER) # define LZ4_DEPRECATED(message) __declspec(deprecated(message)) # else # pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler") # define LZ4_DEPRECATED(message) # endif #endif /* LZ4_DISABLE_DEPRECATE_WARNINGS */ /* Obsolete compression functions */ LZ4LIB_API LZ4_DEPRECATED("use LZ4_compress_default() instead") int LZ4_compress (const char* source, char* dest, int sourceSize); LZ4LIB_API LZ4_DEPRECATED("use LZ4_compress_default() instead") int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize); LZ4LIB_API LZ4_DEPRECATED("use LZ4_compress_fast_extState() instead") int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize); LZ4LIB_API LZ4_DEPRECATED("use LZ4_compress_fast_extState() instead") int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); LZ4LIB_API LZ4_DEPRECATED("use LZ4_compress_fast_continue() instead") int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize); LZ4LIB_API LZ4_DEPRECATED("use LZ4_compress_fast_continue() instead") int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); /* Obsolete decompression functions */ LZ4LIB_API LZ4_DEPRECATED("use LZ4_decompress_fast() instead") int LZ4_uncompress (const char* source, char* dest, int outputSize); LZ4LIB_API LZ4_DEPRECATED("use LZ4_decompress_safe() instead") int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); /* Obsolete streaming functions; use new streaming interface whenever possible */ LZ4LIB_API LZ4_DEPRECATED("use LZ4_createStream() instead") void* LZ4_create (char* inputBuffer); LZ4LIB_API LZ4_DEPRECATED("use LZ4_createStream() instead") int LZ4_sizeofStreamState(void); LZ4LIB_API LZ4_DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void* state, char* inputBuffer); LZ4LIB_API LZ4_DEPRECATED("use LZ4_saveDict() instead") char* LZ4_slideInputBuffer (void* state); /* Obsolete streaming decoding functions */ LZ4LIB_API LZ4_DEPRECATED("use LZ4_decompress_safe_usingDict() instead") int LZ4_decompress_safe_withPrefix64k (const char* src, char* dst, int compressedSize, int maxDstSize); LZ4LIB_API LZ4_DEPRECATED("use LZ4_decompress_fast_usingDict() instead") int LZ4_decompress_fast_withPrefix64k (const char* src, char* dst, int originalSize); #endif /* LZ4_H_2983827168210 */ #if defined (__cplusplus) } #endif nfdump-1.6.23/bin/lzoconf.h000077500000000000000000000372061404501030700154740ustar00rootroot00000000000000/* lzoconf.h -- configuration of the LZO data compression library This file is part of the LZO real-time data compression library. Copyright (C) 1996-2017 Markus Franz Xaver Johannes Oberhumer All Rights Reserved. The LZO library is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. The LZO library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with the LZO library; see the file COPYING. If not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Markus F.X.J. Oberhumer http://www.oberhumer.com/opensource/lzo/ */ #ifndef __LZOCONF_H_INCLUDED #define __LZOCONF_H_INCLUDED 1 #define LZO_VERSION 0x20a0 /* 2.10 */ #define LZO_VERSION_STRING "2.10" #define LZO_VERSION_DATE "Mar 01 2017" /* internal Autoconf configuration file - only used when building LZO */ #if defined(LZO_HAVE_CONFIG_H) # include #endif #include #include /*********************************************************************** // LZO requires a conforming ************************************************************************/ #if !defined(CHAR_BIT) || (CHAR_BIT != 8) # error "invalid CHAR_BIT" #endif #if !defined(UCHAR_MAX) || !defined(USHRT_MAX) || !defined(UINT_MAX) || !defined(ULONG_MAX) # error "check your compiler installation" #endif #if (USHRT_MAX < 1) || (UINT_MAX < 1) || (ULONG_MAX < 1) # error "your limits.h macros are broken" #endif /* get OS and architecture defines */ #ifndef __LZODEFS_H_INCLUDED #include #endif #ifdef __cplusplus extern "C" { #endif /*********************************************************************** // some core defines ************************************************************************/ /* memory checkers */ #if !defined(__LZO_CHECKER) # if defined(__BOUNDS_CHECKING_ON) # define __LZO_CHECKER 1 # elif defined(__CHECKER__) # define __LZO_CHECKER 1 # elif defined(__INSURE__) # define __LZO_CHECKER 1 # elif defined(__PURIFY__) # define __LZO_CHECKER 1 # endif #endif /*********************************************************************** // integral and pointer types ************************************************************************/ /* lzo_uint must match size_t */ #if !defined(LZO_UINT_MAX) # if (LZO_ABI_LLP64) # if (LZO_OS_WIN64) typedef unsigned __int64 lzo_uint; typedef __int64 lzo_int; # define LZO_TYPEOF_LZO_INT LZO_TYPEOF___INT64 # else typedef lzo_ullong_t lzo_uint; typedef lzo_llong_t lzo_int; # define LZO_TYPEOF_LZO_INT LZO_TYPEOF_LONG_LONG # endif # define LZO_SIZEOF_LZO_INT 8 # define LZO_UINT_MAX 0xffffffffffffffffull # define LZO_INT_MAX 9223372036854775807LL # define LZO_INT_MIN (-1LL - LZO_INT_MAX) # elif (LZO_ABI_IP32L64) /* MIPS R5900 */ typedef unsigned int lzo_uint; typedef int lzo_int; # define LZO_SIZEOF_LZO_INT LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_INT LZO_TYPEOF_INT # define LZO_UINT_MAX UINT_MAX # define LZO_INT_MAX INT_MAX # define LZO_INT_MIN INT_MIN # elif (ULONG_MAX >= LZO_0xffffffffL) typedef unsigned long lzo_uint; typedef long lzo_int; # define LZO_SIZEOF_LZO_INT LZO_SIZEOF_LONG # define LZO_TYPEOF_LZO_INT LZO_TYPEOF_LONG # define LZO_UINT_MAX ULONG_MAX # define LZO_INT_MAX LONG_MAX # define LZO_INT_MIN LONG_MIN # else # error "lzo_uint" # endif #endif /* The larger type of lzo_uint and lzo_uint32_t. */ #if (LZO_SIZEOF_LZO_INT >= 4) # define lzo_xint lzo_uint #else # define lzo_xint lzo_uint32_t #endif typedef int lzo_bool; /* sanity checks */ LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int) == LZO_SIZEOF_LZO_INT) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) == LZO_SIZEOF_LZO_INT) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_xint) >= sizeof(lzo_uint)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_xint) >= sizeof(lzo_uint32_t)) #ifndef __LZO_MMODEL #define __LZO_MMODEL /*empty*/ #endif /* no typedef here because of const-pointer issues */ #define lzo_bytep unsigned char __LZO_MMODEL * #define lzo_charp char __LZO_MMODEL * #define lzo_voidp void __LZO_MMODEL * #define lzo_shortp short __LZO_MMODEL * #define lzo_ushortp unsigned short __LZO_MMODEL * #define lzo_intp lzo_int __LZO_MMODEL * #define lzo_uintp lzo_uint __LZO_MMODEL * #define lzo_xintp lzo_xint __LZO_MMODEL * #define lzo_voidpp lzo_voidp __LZO_MMODEL * #define lzo_bytepp lzo_bytep __LZO_MMODEL * #define lzo_int8_tp lzo_int8_t __LZO_MMODEL * #define lzo_uint8_tp lzo_uint8_t __LZO_MMODEL * #define lzo_int16_tp lzo_int16_t __LZO_MMODEL * #define lzo_uint16_tp lzo_uint16_t __LZO_MMODEL * #define lzo_int32_tp lzo_int32_t __LZO_MMODEL * #define lzo_uint32_tp lzo_uint32_t __LZO_MMODEL * #if defined(lzo_int64_t) #define lzo_int64_tp lzo_int64_t __LZO_MMODEL * #define lzo_uint64_tp lzo_uint64_t __LZO_MMODEL * #endif /* Older LZO versions used to support ancient systems and memory models * such as 16-bit MSDOS with __huge pointers or Cray PVP, but these * obsolete configurations are not supported any longer. */ #if defined(__LZO_MMODEL_HUGE) #error "__LZO_MMODEL_HUGE memory model is unsupported" #endif #if (LZO_MM_PVP) #error "LZO_MM_PVP memory model is unsupported" #endif #if (LZO_SIZEOF_INT < 4) #error "LZO_SIZEOF_INT < 4 is unsupported" #endif #if (__LZO_UINTPTR_T_IS_POINTER) #error "__LZO_UINTPTR_T_IS_POINTER is unsupported" #endif LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(int) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) >= 4) /* Strange configurations where sizeof(lzo_uint) != sizeof(size_t) should * work but have not received much testing lately, so be strict here. */ LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) == sizeof(size_t)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) == sizeof(ptrdiff_t)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) == sizeof(lzo_uintptr_t)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(void *) == sizeof(lzo_uintptr_t)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(char *) == sizeof(lzo_uintptr_t)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long *) == sizeof(lzo_uintptr_t)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(void *) == sizeof(lzo_voidp)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(char *) == sizeof(lzo_bytep)) /*********************************************************************** // function types ************************************************************************/ /* name mangling */ #if !defined(__LZO_EXTERN_C) # ifdef __cplusplus # define __LZO_EXTERN_C extern "C" # else # define __LZO_EXTERN_C extern # endif #endif /* calling convention */ #if !defined(__LZO_CDECL) # define __LZO_CDECL __lzo_cdecl #endif /* DLL export information */ #if !defined(__LZO_EXPORT1) # define __LZO_EXPORT1 /*empty*/ #endif #if !defined(__LZO_EXPORT2) # define __LZO_EXPORT2 /*empty*/ #endif /* __cdecl calling convention for public C and assembly functions */ #if !defined(LZO_PUBLIC) # define LZO_PUBLIC(r) __LZO_EXPORT1 r __LZO_EXPORT2 __LZO_CDECL #endif #if !defined(LZO_EXTERN) # define LZO_EXTERN(r) __LZO_EXTERN_C LZO_PUBLIC(r) #endif #if !defined(LZO_PRIVATE) # define LZO_PRIVATE(r) static r __LZO_CDECL #endif /* function types */ typedef int (__LZO_CDECL *lzo_compress_t) ( const lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp dst_len, lzo_voidp wrkmem ); typedef int (__LZO_CDECL *lzo_decompress_t) ( const lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp dst_len, lzo_voidp wrkmem ); typedef int (__LZO_CDECL *lzo_optimize_t) ( lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp dst_len, lzo_voidp wrkmem ); typedef int (__LZO_CDECL *lzo_compress_dict_t)(const lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp dst_len, lzo_voidp wrkmem, const lzo_bytep dict, lzo_uint dict_len ); typedef int (__LZO_CDECL *lzo_decompress_dict_t)(const lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp dst_len, lzo_voidp wrkmem, const lzo_bytep dict, lzo_uint dict_len ); /* Callback interface. Currently only the progress indicator ("nprogress") * is used, but this may change in a future release. */ struct lzo_callback_t; typedef struct lzo_callback_t lzo_callback_t; #define lzo_callback_p lzo_callback_t __LZO_MMODEL * /* malloc & free function types */ typedef lzo_voidp (__LZO_CDECL *lzo_alloc_func_t) (lzo_callback_p self, lzo_uint items, lzo_uint size); typedef void (__LZO_CDECL *lzo_free_func_t) (lzo_callback_p self, lzo_voidp ptr); /* a progress indicator callback function */ typedef void (__LZO_CDECL *lzo_progress_func_t) (lzo_callback_p, lzo_uint, lzo_uint, int); struct lzo_callback_t { /* custom allocators (set to 0 to disable) */ lzo_alloc_func_t nalloc; /* [not used right now] */ lzo_free_func_t nfree; /* [not used right now] */ /* a progress indicator callback function (set to 0 to disable) */ lzo_progress_func_t nprogress; /* INFO: the first parameter "self" of the nalloc/nfree/nprogress * callbacks points back to this struct, so you are free to store * some extra info in the following variables. */ lzo_voidp user1; lzo_xint user2; lzo_xint user3; }; /*********************************************************************** // error codes and prototypes ************************************************************************/ /* Error codes for the compression/decompression functions. Negative * values are errors, positive values will be used for special but * normal events. */ #define LZO_E_OK 0 #define LZO_E_ERROR (-1) #define LZO_E_OUT_OF_MEMORY (-2) /* [lzo_alloc_func_t failure] */ #define LZO_E_NOT_COMPRESSIBLE (-3) /* [not used right now] */ #define LZO_E_INPUT_OVERRUN (-4) #define LZO_E_OUTPUT_OVERRUN (-5) #define LZO_E_LOOKBEHIND_OVERRUN (-6) #define LZO_E_EOF_NOT_FOUND (-7) #define LZO_E_INPUT_NOT_CONSUMED (-8) #define LZO_E_NOT_YET_IMPLEMENTED (-9) /* [not used right now] */ #define LZO_E_INVALID_ARGUMENT (-10) #define LZO_E_INVALID_ALIGNMENT (-11) /* pointer argument is not properly aligned */ #define LZO_E_OUTPUT_NOT_CONSUMED (-12) #define LZO_E_INTERNAL_ERROR (-99) #ifndef lzo_sizeof_dict_t # define lzo_sizeof_dict_t ((unsigned)sizeof(lzo_bytep)) #endif /* lzo_init() should be the first function you call. * Check the return code ! * * lzo_init() is a macro to allow checking that the library and the * compiler's view of various types are consistent. */ #define lzo_init() __lzo_init_v2(LZO_VERSION,(int)sizeof(short),(int)sizeof(int),\ (int)sizeof(long),(int)sizeof(lzo_uint32_t),(int)sizeof(lzo_uint),\ (int)lzo_sizeof_dict_t,(int)sizeof(char *),(int)sizeof(lzo_voidp),\ (int)sizeof(lzo_callback_t)) LZO_EXTERN(int) __lzo_init_v2(unsigned,int,int,int,int,int,int,int,int,int); /* version functions (useful for shared libraries) */ LZO_EXTERN(unsigned) lzo_version(void); LZO_EXTERN(const char *) lzo_version_string(void); LZO_EXTERN(const char *) lzo_version_date(void); LZO_EXTERN(const lzo_charp) _lzo_version_string(void); LZO_EXTERN(const lzo_charp) _lzo_version_date(void); /* string functions */ LZO_EXTERN(int) lzo_memcmp(const lzo_voidp a, const lzo_voidp b, lzo_uint len); LZO_EXTERN(lzo_voidp) lzo_memcpy(lzo_voidp dst, const lzo_voidp src, lzo_uint len); LZO_EXTERN(lzo_voidp) lzo_memmove(lzo_voidp dst, const lzo_voidp src, lzo_uint len); LZO_EXTERN(lzo_voidp) lzo_memset(lzo_voidp buf, int c, lzo_uint len); /* checksum functions */ LZO_EXTERN(lzo_uint32_t) lzo_adler32(lzo_uint32_t c, const lzo_bytep buf, lzo_uint len); LZO_EXTERN(lzo_uint32_t) lzo_crc32(lzo_uint32_t c, const lzo_bytep buf, lzo_uint len); LZO_EXTERN(const lzo_uint32_tp) lzo_get_crc32_table(void); /* misc. */ LZO_EXTERN(int) _lzo_config_check(void); typedef union { lzo_voidp a00; lzo_bytep a01; lzo_uint a02; lzo_xint a03; lzo_uintptr_t a04; void *a05; unsigned char *a06; unsigned long a07; size_t a08; ptrdiff_t a09; #if defined(lzo_int64_t) lzo_uint64_t a10; #endif } lzo_align_t; /* align a char pointer on a boundary that is a multiple of 'size' */ LZO_EXTERN(unsigned) __lzo_align_gap(const lzo_voidp p, lzo_uint size); #define LZO_PTR_ALIGN_UP(p,size) \ ((p) + (lzo_uint) __lzo_align_gap((const lzo_voidp)(p),(lzo_uint)(size))) /*********************************************************************** // deprecated macros - only for backward compatibility ************************************************************************/ /* deprecated - use 'lzo_bytep' instead of 'lzo_byte *' */ #define lzo_byte unsigned char /* deprecated type names */ #define lzo_int32 lzo_int32_t #define lzo_uint32 lzo_uint32_t #define lzo_int32p lzo_int32_t __LZO_MMODEL * #define lzo_uint32p lzo_uint32_t __LZO_MMODEL * #define LZO_INT32_MAX LZO_INT32_C(2147483647) #define LZO_UINT32_MAX LZO_UINT32_C(4294967295) #if defined(lzo_int64_t) #define lzo_int64 lzo_int64_t #define lzo_uint64 lzo_uint64_t #define lzo_int64p lzo_int64_t __LZO_MMODEL * #define lzo_uint64p lzo_uint64_t __LZO_MMODEL * #define LZO_INT64_MAX LZO_INT64_C(9223372036854775807) #define LZO_UINT64_MAX LZO_UINT64_C(18446744073709551615) #endif /* deprecated types */ typedef union { lzo_bytep a; lzo_uint b; } __lzo_pu_u; typedef union { lzo_bytep a; lzo_uint32_t b; } __lzo_pu32_u; /* deprecated defines */ #if !defined(LZO_SIZEOF_LZO_UINT) # define LZO_SIZEOF_LZO_UINT LZO_SIZEOF_LZO_INT #endif #if defined(LZO_CFG_COMPAT) #define __LZOCONF_H 1 #if defined(LZO_ARCH_I086) # define __LZO_i386 1 #elif defined(LZO_ARCH_I386) # define __LZO_i386 1 #endif #if defined(LZO_OS_DOS16) # define __LZO_DOS 1 # define __LZO_DOS16 1 #elif defined(LZO_OS_DOS32) # define __LZO_DOS 1 #elif defined(LZO_OS_WIN16) # define __LZO_WIN 1 # define __LZO_WIN16 1 #elif defined(LZO_OS_WIN32) # define __LZO_WIN 1 #endif #define __LZO_CMODEL /*empty*/ #define __LZO_DMODEL /*empty*/ #define __LZO_ENTRY __LZO_CDECL #define LZO_EXTERN_CDECL LZO_EXTERN #define LZO_ALIGN LZO_PTR_ALIGN_UP #define lzo_compress_asm_t lzo_compress_t #define lzo_decompress_asm_t lzo_decompress_t #endif /* LZO_CFG_COMPAT */ #ifdef __cplusplus } /* extern "C" */ #endif #endif /* already included */ /* vim:set ts=4 sw=4 et: */ nfdump-1.6.23/bin/lzodefs.h000077500000000000000000003704711404501030700154740ustar00rootroot00000000000000/* lzodefs.h -- architecture, OS and compiler specific defines This file is part of the LZO real-time data compression library. Copyright (C) 1996-2017 Markus Franz Xaver Johannes Oberhumer All Rights Reserved. The LZO library is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. The LZO library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with the LZO library; see the file COPYING. If not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Markus F.X.J. Oberhumer http://www.oberhumer.com/opensource/lzo/ */ #ifndef __LZODEFS_H_INCLUDED #define __LZODEFS_H_INCLUDED 1 #if defined(__CYGWIN32__) && !defined(__CYGWIN__) # define __CYGWIN__ __CYGWIN32__ #endif #if 1 && defined(__INTERIX) && defined(__GNUC__) && !defined(_ALL_SOURCE) # define _ALL_SOURCE 1 #endif #if defined(__mips__) && defined(__R5900__) # if !defined(__LONG_MAX__) # define __LONG_MAX__ 9223372036854775807L # endif #endif #if 0 #elif !defined(__LZO_LANG_OVERRIDE) #if (defined(__clang__) || defined(__GNUC__)) && defined(__ASSEMBLER__) # if (__ASSEMBLER__+0) <= 0 # error "__ASSEMBLER__" # else # define LZO_LANG_ASSEMBLER 1 # endif #elif defined(__cplusplus) # if (__cplusplus+0) <= 0 # error "__cplusplus" # elif (__cplusplus < 199711L) # define LZO_LANG_CXX 1 # elif defined(_MSC_VER) && defined(_MSVC_LANG) && (_MSVC_LANG+0 >= 201402L) && 1 # define LZO_LANG_CXX _MSVC_LANG # else # define LZO_LANG_CXX __cplusplus # endif # define LZO_LANG_CPLUSPLUS LZO_LANG_CXX #else # if defined(__STDC_VERSION__) && (__STDC_VERSION__+0 >= 199409L) # define LZO_LANG_C __STDC_VERSION__ # else # define LZO_LANG_C 1 # endif #endif #endif #if !defined(LZO_CFG_NO_DISABLE_WUNDEF) #if defined(__ARMCC_VERSION) # pragma diag_suppress 193 #elif defined(__clang__) && defined(__clang_minor__) # pragma clang diagnostic ignored "-Wundef" #elif defined(__INTEL_COMPILER) # pragma warning(disable: 193) #elif defined(__KEIL__) && defined(__C166__) # pragma warning disable = 322 #elif defined(__GNUC__) && defined(__GNUC_MINOR__) && !defined(__PATHSCALE__) # if ((__GNUC__-0) >= 5 || ((__GNUC__-0) == 4 && (__GNUC_MINOR__-0) >= 2)) # pragma GCC diagnostic ignored "-Wundef" # endif #elif defined(_MSC_VER) && !defined(__clang__) && !defined(__INTEL_COMPILER) && !defined(__MWERKS__) # if ((_MSC_VER-0) >= 1300) # pragma warning(disable: 4668) # endif #endif #endif #if 0 && defined(__POCC__) && defined(_WIN32) # if (__POCC__ >= 400) # pragma warn(disable: 2216) # endif #endif #if 0 && defined(__WATCOMC__) # if (__WATCOMC__ >= 1050) && (__WATCOMC__ < 1060) # pragma warning 203 9 # endif #endif #if defined(__BORLANDC__) && defined(__MSDOS__) && !defined(__FLAT__) # pragma option -h #endif #if !(LZO_CFG_NO_DISABLE_WCRTNONSTDC) #ifndef _CRT_NONSTDC_NO_DEPRECATE #define _CRT_NONSTDC_NO_DEPRECATE 1 #endif #ifndef _CRT_NONSTDC_NO_WARNINGS #define _CRT_NONSTDC_NO_WARNINGS 1 #endif #ifndef _CRT_SECURE_NO_DEPRECATE #define _CRT_SECURE_NO_DEPRECATE 1 #endif #ifndef _CRT_SECURE_NO_WARNINGS #define _CRT_SECURE_NO_WARNINGS 1 #endif #endif #if 0 #define LZO_0xffffUL 0xfffful #define LZO_0xffffffffUL 0xfffffffful #else #define LZO_0xffffUL 65535ul #define LZO_0xffffffffUL 4294967295ul #endif #define LZO_0xffffL LZO_0xffffUL #define LZO_0xffffffffL LZO_0xffffffffUL #if (LZO_0xffffL == LZO_0xffffffffL) # error "your preprocessor is broken 1" #endif #if (16ul * 16384ul != 262144ul) # error "your preprocessor is broken 2" #endif #if 0 #if (32767 >= 4294967295ul) # error "your preprocessor is broken 3" #endif #if (65535u >= 4294967295ul) # error "your preprocessor is broken 4" #endif #endif #if defined(__COUNTER__) # ifndef LZO_CFG_USE_COUNTER # define LZO_CFG_USE_COUNTER 1 # endif #else # undef LZO_CFG_USE_COUNTER #endif #if (UINT_MAX == LZO_0xffffL) #if defined(__ZTC__) && defined(__I86__) && !defined(__OS2__) # if !defined(MSDOS) # define MSDOS 1 # endif # if !defined(_MSDOS) # define _MSDOS 1 # endif #elif 0 && defined(__VERSION) && defined(MB_LEN_MAX) # if (__VERSION == 520) && (MB_LEN_MAX == 1) # if !defined(__AZTEC_C__) # define __AZTEC_C__ __VERSION # endif # if !defined(__DOS__) # define __DOS__ 1 # endif # endif #endif #endif #if (UINT_MAX == LZO_0xffffL) #if defined(_MSC_VER) && defined(M_I86HM) # define ptrdiff_t long # define _PTRDIFF_T_DEFINED 1 #endif #endif #if (UINT_MAX == LZO_0xffffL) # undef __LZO_RENAME_A # undef __LZO_RENAME_B # if defined(__AZTEC_C__) && defined(__DOS__) # define __LZO_RENAME_A 1 # elif defined(_MSC_VER) && defined(MSDOS) # if (_MSC_VER < 600) # define __LZO_RENAME_A 1 # elif (_MSC_VER < 700) # define __LZO_RENAME_B 1 # endif # elif defined(__TSC__) && defined(__OS2__) # define __LZO_RENAME_A 1 # elif defined(__MSDOS__) && defined(__TURBOC__) && (__TURBOC__ < 0x0410) # define __LZO_RENAME_A 1 # elif defined(__PACIFIC__) && defined(DOS) # if !defined(__far) # define __far far # endif # if !defined(__near) # define __near near # endif # endif # if defined(__LZO_RENAME_A) # if !defined(__cdecl) # define __cdecl cdecl # endif # if !defined(__far) # define __far far # endif # if !defined(__huge) # define __huge huge # endif # if !defined(__near) # define __near near # endif # if !defined(__pascal) # define __pascal pascal # endif # if !defined(__huge) # define __huge huge # endif # elif defined(__LZO_RENAME_B) # if !defined(__cdecl) # define __cdecl _cdecl # endif # if !defined(__far) # define __far _far # endif # if !defined(__huge) # define __huge _huge # endif # if !defined(__near) # define __near _near # endif # if !defined(__pascal) # define __pascal _pascal # endif # elif (defined(__PUREC__) || defined(__TURBOC__)) && defined(__TOS__) # if !defined(__cdecl) # define __cdecl cdecl # endif # if !defined(__pascal) # define __pascal pascal # endif # endif # undef __LZO_RENAME_A # undef __LZO_RENAME_B #endif #if (UINT_MAX == LZO_0xffffL) #if defined(__AZTEC_C__) && defined(__DOS__) # define LZO_BROKEN_CDECL_ALT_SYNTAX 1 #elif defined(_MSC_VER) && defined(MSDOS) # if (_MSC_VER < 600) # define LZO_BROKEN_INTEGRAL_CONSTANTS 1 # endif # if (_MSC_VER < 700) # define LZO_BROKEN_INTEGRAL_PROMOTION 1 # define LZO_BROKEN_SIZEOF 1 # endif #elif defined(__PACIFIC__) && defined(DOS) # define LZO_BROKEN_INTEGRAL_CONSTANTS 1 #elif defined(__TURBOC__) && defined(__MSDOS__) # if (__TURBOC__ < 0x0150) # define LZO_BROKEN_CDECL_ALT_SYNTAX 1 # define LZO_BROKEN_INTEGRAL_CONSTANTS 1 # define LZO_BROKEN_INTEGRAL_PROMOTION 1 # endif # if (__TURBOC__ < 0x0200) # define LZO_BROKEN_SIZEOF 1 # endif # if (__TURBOC__ < 0x0400) && defined(__cplusplus) # define LZO_BROKEN_CDECL_ALT_SYNTAX 1 # endif #elif (defined(__PUREC__) || defined(__TURBOC__)) && defined(__TOS__) # define LZO_BROKEN_CDECL_ALT_SYNTAX 1 # define LZO_BROKEN_SIZEOF 1 #endif #endif #if defined(__WATCOMC__) && (__WATCOMC__ < 900) # define LZO_BROKEN_INTEGRAL_CONSTANTS 1 #endif #if defined(_CRAY) && defined(_CRAY1) # define LZO_BROKEN_SIGNED_RIGHT_SHIFT 1 #endif #define LZO_PP_STRINGIZE(x) #x #define LZO_PP_MACRO_EXPAND(x) LZO_PP_STRINGIZE(x) #define LZO_PP_CONCAT0() /*empty*/ #define LZO_PP_CONCAT1(a) a #define LZO_PP_CONCAT2(a,b) a ## b #define LZO_PP_CONCAT3(a,b,c) a ## b ## c #define LZO_PP_CONCAT4(a,b,c,d) a ## b ## c ## d #define LZO_PP_CONCAT5(a,b,c,d,e) a ## b ## c ## d ## e #define LZO_PP_CONCAT6(a,b,c,d,e,f) a ## b ## c ## d ## e ## f #define LZO_PP_CONCAT7(a,b,c,d,e,f,g) a ## b ## c ## d ## e ## f ## g #define LZO_PP_ECONCAT0() LZO_PP_CONCAT0() #define LZO_PP_ECONCAT1(a) LZO_PP_CONCAT1(a) #define LZO_PP_ECONCAT2(a,b) LZO_PP_CONCAT2(a,b) #define LZO_PP_ECONCAT3(a,b,c) LZO_PP_CONCAT3(a,b,c) #define LZO_PP_ECONCAT4(a,b,c,d) LZO_PP_CONCAT4(a,b,c,d) #define LZO_PP_ECONCAT5(a,b,c,d,e) LZO_PP_CONCAT5(a,b,c,d,e) #define LZO_PP_ECONCAT6(a,b,c,d,e,f) LZO_PP_CONCAT6(a,b,c,d,e,f) #define LZO_PP_ECONCAT7(a,b,c,d,e,f,g) LZO_PP_CONCAT7(a,b,c,d,e,f,g) #define LZO_PP_EMPTY /*empty*/ #define LZO_PP_EMPTY0() /*empty*/ #define LZO_PP_EMPTY1(a) /*empty*/ #define LZO_PP_EMPTY2(a,b) /*empty*/ #define LZO_PP_EMPTY3(a,b,c) /*empty*/ #define LZO_PP_EMPTY4(a,b,c,d) /*empty*/ #define LZO_PP_EMPTY5(a,b,c,d,e) /*empty*/ #define LZO_PP_EMPTY6(a,b,c,d,e,f) /*empty*/ #define LZO_PP_EMPTY7(a,b,c,d,e,f,g) /*empty*/ #if 1 #define LZO_CPP_STRINGIZE(x) #x #define LZO_CPP_MACRO_EXPAND(x) LZO_CPP_STRINGIZE(x) #define LZO_CPP_CONCAT2(a,b) a ## b #define LZO_CPP_CONCAT3(a,b,c) a ## b ## c #define LZO_CPP_CONCAT4(a,b,c,d) a ## b ## c ## d #define LZO_CPP_CONCAT5(a,b,c,d,e) a ## b ## c ## d ## e #define LZO_CPP_CONCAT6(a,b,c,d,e,f) a ## b ## c ## d ## e ## f #define LZO_CPP_CONCAT7(a,b,c,d,e,f,g) a ## b ## c ## d ## e ## f ## g #define LZO_CPP_ECONCAT2(a,b) LZO_CPP_CONCAT2(a,b) #define LZO_CPP_ECONCAT3(a,b,c) LZO_CPP_CONCAT3(a,b,c) #define LZO_CPP_ECONCAT4(a,b,c,d) LZO_CPP_CONCAT4(a,b,c,d) #define LZO_CPP_ECONCAT5(a,b,c,d,e) LZO_CPP_CONCAT5(a,b,c,d,e) #define LZO_CPP_ECONCAT6(a,b,c,d,e,f) LZO_CPP_CONCAT6(a,b,c,d,e,f) #define LZO_CPP_ECONCAT7(a,b,c,d,e,f,g) LZO_CPP_CONCAT7(a,b,c,d,e,f,g) #endif #define __LZO_MASK_GEN(o,b) (((((o) << ((b)-((b)!=0))) - (o)) << 1) + (o)*((b)!=0)) #if 1 && defined(__cplusplus) # if !defined(__STDC_CONSTANT_MACROS) # define __STDC_CONSTANT_MACROS 1 # endif # if !defined(__STDC_LIMIT_MACROS) # define __STDC_LIMIT_MACROS 1 # endif #endif #if defined(__cplusplus) # define LZO_EXTERN_C extern "C" # define LZO_EXTERN_C_BEGIN extern "C" { # define LZO_EXTERN_C_END } #else # define LZO_EXTERN_C extern # define LZO_EXTERN_C_BEGIN /*empty*/ # define LZO_EXTERN_C_END /*empty*/ #endif #if !defined(__LZO_OS_OVERRIDE) #if (LZO_OS_FREESTANDING) # define LZO_INFO_OS "freestanding" #elif (LZO_OS_EMBEDDED) # define LZO_INFO_OS "embedded" #elif 1 && defined(__IAR_SYSTEMS_ICC__) # define LZO_OS_EMBEDDED 1 # define LZO_INFO_OS "embedded" #elif defined(__CYGWIN__) && defined(__GNUC__) # define LZO_OS_CYGWIN 1 # define LZO_INFO_OS "cygwin" #elif defined(__EMX__) && defined(__GNUC__) # define LZO_OS_EMX 1 # define LZO_INFO_OS "emx" #elif defined(__BEOS__) # define LZO_OS_BEOS 1 # define LZO_INFO_OS "beos" #elif defined(__Lynx__) # define LZO_OS_LYNXOS 1 # define LZO_INFO_OS "lynxos" #elif defined(__OS400__) # define LZO_OS_OS400 1 # define LZO_INFO_OS "os400" #elif defined(__QNX__) # define LZO_OS_QNX 1 # define LZO_INFO_OS "qnx" #elif defined(__BORLANDC__) && defined(__DPMI32__) && (__BORLANDC__ >= 0x0460) # define LZO_OS_DOS32 1 # define LZO_INFO_OS "dos32" #elif defined(__BORLANDC__) && defined(__DPMI16__) # define LZO_OS_DOS16 1 # define LZO_INFO_OS "dos16" #elif defined(__ZTC__) && defined(DOS386) # define LZO_OS_DOS32 1 # define LZO_INFO_OS "dos32" #elif defined(__OS2__) || defined(__OS2V2__) # if (UINT_MAX == LZO_0xffffL) # define LZO_OS_OS216 1 # define LZO_INFO_OS "os216" # elif (UINT_MAX == LZO_0xffffffffL) # define LZO_OS_OS2 1 # define LZO_INFO_OS "os2" # else # error "check your limits.h header" # endif #elif defined(__WIN64__) || defined(_WIN64) || defined(WIN64) # define LZO_OS_WIN64 1 # define LZO_INFO_OS "win64" #elif defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__WINDOWS_386__) # define LZO_OS_WIN32 1 # define LZO_INFO_OS "win32" #elif defined(__MWERKS__) && defined(__INTEL__) # define LZO_OS_WIN32 1 # define LZO_INFO_OS "win32" #elif defined(__WINDOWS__) || defined(_WINDOWS) || defined(_Windows) # if (UINT_MAX == LZO_0xffffL) # define LZO_OS_WIN16 1 # define LZO_INFO_OS "win16" # elif (UINT_MAX == LZO_0xffffffffL) # define LZO_OS_WIN32 1 # define LZO_INFO_OS "win32" # else # error "check your limits.h header" # endif #elif defined(__DOS__) || defined(__MSDOS__) || defined(_MSDOS) || defined(MSDOS) || (defined(__PACIFIC__) && defined(DOS)) # if (UINT_MAX == LZO_0xffffL) # define LZO_OS_DOS16 1 # define LZO_INFO_OS "dos16" # elif (UINT_MAX == LZO_0xffffffffL) # define LZO_OS_DOS32 1 # define LZO_INFO_OS "dos32" # else # error "check your limits.h header" # endif #elif defined(__WATCOMC__) # if defined(__NT__) && (UINT_MAX == LZO_0xffffL) # define LZO_OS_DOS16 1 # define LZO_INFO_OS "dos16" # elif defined(__NT__) && (__WATCOMC__ < 1100) # define LZO_OS_WIN32 1 # define LZO_INFO_OS "win32" # elif defined(__linux__) || defined(__LINUX__) # define LZO_OS_POSIX 1 # define LZO_INFO_OS "posix" # else # error "please specify a target using the -bt compiler option" # endif #elif defined(__palmos__) # define LZO_OS_PALMOS 1 # define LZO_INFO_OS "palmos" #elif defined(__TOS__) || defined(__atarist__) # define LZO_OS_TOS 1 # define LZO_INFO_OS "tos" #elif defined(macintosh) && !defined(__arm__) && !defined(__i386__) && !defined(__ppc__) && !defined(__x64_64__) # define LZO_OS_MACCLASSIC 1 # define LZO_INFO_OS "macclassic" #elif defined(__VMS) # define LZO_OS_VMS 1 # define LZO_INFO_OS "vms" #elif (defined(__mips__) && defined(__R5900__)) || defined(__MIPS_PSX2__) # define LZO_OS_CONSOLE 1 # define LZO_OS_CONSOLE_PS2 1 # define LZO_INFO_OS "console" # define LZO_INFO_OS_CONSOLE "ps2" #elif defined(__mips__) && defined(__psp__) # define LZO_OS_CONSOLE 1 # define LZO_OS_CONSOLE_PSP 1 # define LZO_INFO_OS "console" # define LZO_INFO_OS_CONSOLE "psp" #else # define LZO_OS_POSIX 1 # define LZO_INFO_OS "posix" #endif #if (LZO_OS_POSIX) # if defined(_AIX) || defined(__AIX__) || defined(__aix__) # define LZO_OS_POSIX_AIX 1 # define LZO_INFO_OS_POSIX "aix" # elif defined(__FreeBSD__) # define LZO_OS_POSIX_FREEBSD 1 # define LZO_INFO_OS_POSIX "freebsd" # elif defined(__hpux__) || defined(__hpux) # define LZO_OS_POSIX_HPUX 1 # define LZO_INFO_OS_POSIX "hpux" # elif defined(__INTERIX) # define LZO_OS_POSIX_INTERIX 1 # define LZO_INFO_OS_POSIX "interix" # elif defined(__IRIX__) || defined(__irix__) # define LZO_OS_POSIX_IRIX 1 # define LZO_INFO_OS_POSIX "irix" # elif defined(__linux__) || defined(__linux) || defined(__LINUX__) # define LZO_OS_POSIX_LINUX 1 # define LZO_INFO_OS_POSIX "linux" # elif defined(__APPLE__) && defined(__MACH__) # if ((__ENVIRONMENT_IPHONE_OS_VERSION_MIN_REQUIRED__-0) >= 20000) # define LZO_OS_POSIX_DARWIN 1040 # define LZO_INFO_OS_POSIX "darwin_iphone" # elif ((__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__-0) >= 1040) # define LZO_OS_POSIX_DARWIN __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ # define LZO_INFO_OS_POSIX "darwin" # else # define LZO_OS_POSIX_DARWIN 1 # define LZO_INFO_OS_POSIX "darwin" # endif # define LZO_OS_POSIX_MACOSX LZO_OS_POSIX_DARWIN # elif defined(__minix__) || defined(__minix) # define LZO_OS_POSIX_MINIX 1 # define LZO_INFO_OS_POSIX "minix" # elif defined(__NetBSD__) # define LZO_OS_POSIX_NETBSD 1 # define LZO_INFO_OS_POSIX "netbsd" # elif defined(__OpenBSD__) # define LZO_OS_POSIX_OPENBSD 1 # define LZO_INFO_OS_POSIX "openbsd" # elif defined(__osf__) # define LZO_OS_POSIX_OSF 1 # define LZO_INFO_OS_POSIX "osf" # elif defined(__solaris__) || defined(__sun) # if defined(__SVR4) || defined(__svr4__) # define LZO_OS_POSIX_SOLARIS 1 # define LZO_INFO_OS_POSIX "solaris" # else # define LZO_OS_POSIX_SUNOS 1 # define LZO_INFO_OS_POSIX "sunos" # endif # elif defined(__ultrix__) || defined(__ultrix) # define LZO_OS_POSIX_ULTRIX 1 # define LZO_INFO_OS_POSIX "ultrix" # elif defined(_UNICOS) # define LZO_OS_POSIX_UNICOS 1 # define LZO_INFO_OS_POSIX "unicos" # else # define LZO_OS_POSIX_UNKNOWN 1 # define LZO_INFO_OS_POSIX "unknown" # endif #endif #endif #if (LZO_OS_DOS16 || LZO_OS_OS216 || LZO_OS_WIN16) # if (UINT_MAX != LZO_0xffffL) # error "unexpected configuration - check your compiler defines" # endif # if (ULONG_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif #endif #if (LZO_OS_DOS32 || LZO_OS_OS2 || LZO_OS_WIN32 || LZO_OS_WIN64) # if (UINT_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif # if (ULONG_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif #endif #if defined(CIL) && defined(_GNUCC) && defined(__GNUC__) # define LZO_CC_CILLY 1 # define LZO_INFO_CC "Cilly" # if defined(__CILLY__) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__CILLY__) # else # define LZO_INFO_CCVER "unknown" # endif #elif 0 && defined(SDCC) && defined(__VERSION__) && !defined(__GNUC__) # define LZO_CC_SDCC 1 # define LZO_INFO_CC "sdcc" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(SDCC) #elif defined(__PATHSCALE__) && defined(__PATHCC_PATCHLEVEL__) # define LZO_CC_PATHSCALE (__PATHCC__ * 0x10000L + (__PATHCC_MINOR__-0) * 0x100 + (__PATHCC_PATCHLEVEL__-0)) # define LZO_INFO_CC "Pathscale C" # define LZO_INFO_CCVER __PATHSCALE__ # if defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # define LZO_CC_PATHSCALE_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # endif #elif defined(__INTEL_COMPILER) && ((__INTEL_COMPILER-0) > 0) # define LZO_CC_INTELC __INTEL_COMPILER # define LZO_INFO_CC "Intel C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__INTEL_COMPILER) # if defined(_MSC_VER) && ((_MSC_VER-0) > 0) # define LZO_CC_INTELC_MSC _MSC_VER # elif defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # define LZO_CC_INTELC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # endif #elif defined(__POCC__) && defined(_WIN32) # define LZO_CC_PELLESC 1 # define LZO_INFO_CC "Pelles C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__POCC__) #elif defined(__ARMCC_VERSION) && defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # if defined(__GNUC_PATCHLEVEL__) # define LZO_CC_ARMCC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # else # define LZO_CC_ARMCC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100) # endif # define LZO_CC_ARMCC __ARMCC_VERSION # define LZO_INFO_CC "ARM C Compiler" # define LZO_INFO_CCVER __VERSION__ #elif defined(__clang__) && defined(__c2__) && defined(__c2_version__) && defined(_MSC_VER) # define LZO_CC_CLANG (__clang_major__ * 0x10000L + (__clang_minor__-0) * 0x100 + (__clang_patchlevel__-0)) # define LZO_CC_CLANG_C2 _MSC_VER # define LZO_CC_CLANG_VENDOR_MICROSOFT 1 # define LZO_INFO_CC "clang/c2" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__c2_version__) #elif defined(__clang__) && defined(__llvm__) && defined(__VERSION__) # if defined(__clang_major__) && defined(__clang_minor__) && defined(__clang_patchlevel__) # define LZO_CC_CLANG (__clang_major__ * 0x10000L + (__clang_minor__-0) * 0x100 + (__clang_patchlevel__-0)) # else # define LZO_CC_CLANG 0x010000L # endif # if defined(_MSC_VER) && ((_MSC_VER-0) > 0) # define LZO_CC_CLANG_MSC _MSC_VER # elif defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # define LZO_CC_CLANG_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # endif # if defined(__APPLE_CC__) # define LZO_CC_CLANG_VENDOR_APPLE 1 # define LZO_INFO_CC "clang/apple" # else # define LZO_CC_CLANG_VENDOR_LLVM 1 # define LZO_INFO_CC "clang" # endif # if defined(__clang_version__) # define LZO_INFO_CCVER __clang_version__ # else # define LZO_INFO_CCVER __VERSION__ # endif #elif defined(__llvm__) && defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # if defined(__GNUC_PATCHLEVEL__) # define LZO_CC_LLVM_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # else # define LZO_CC_LLVM_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100) # endif # define LZO_CC_LLVM LZO_CC_LLVM_GNUC # define LZO_INFO_CC "llvm-gcc" # define LZO_INFO_CCVER __VERSION__ #elif defined(__ACK__) && defined(_ACK) # define LZO_CC_ACK 1 # define LZO_INFO_CC "Amsterdam Compiler Kit C" # define LZO_INFO_CCVER "unknown" #elif defined(__ARMCC_VERSION) && !defined(__GNUC__) # define LZO_CC_ARMCC __ARMCC_VERSION # define LZO_CC_ARMCC_ARMCC __ARMCC_VERSION # define LZO_INFO_CC "ARM C Compiler" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__ARMCC_VERSION) #elif defined(__AZTEC_C__) # define LZO_CC_AZTECC 1 # define LZO_INFO_CC "Aztec C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__AZTEC_C__) #elif defined(__CODEGEARC__) # define LZO_CC_CODEGEARC 1 # define LZO_INFO_CC "CodeGear C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__CODEGEARC__) #elif defined(__BORLANDC__) # define LZO_CC_BORLANDC 1 # define LZO_INFO_CC "Borland C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__BORLANDC__) #elif defined(_CRAYC) && defined(_RELEASE) # define LZO_CC_CRAYC 1 # define LZO_INFO_CC "Cray C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(_RELEASE) #elif defined(__DMC__) && defined(__SC__) # define LZO_CC_DMC 1 # define LZO_INFO_CC "Digital Mars C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__DMC__) #elif defined(__DECC) # define LZO_CC_DECC 1 # define LZO_INFO_CC "DEC C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__DECC) #elif (defined(__ghs) || defined(__ghs__)) && defined(__GHS_VERSION_NUMBER) && ((__GHS_VERSION_NUMBER-0) > 0) # define LZO_CC_GHS 1 # define LZO_INFO_CC "Green Hills C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__GHS_VERSION_NUMBER) # if defined(_MSC_VER) && ((_MSC_VER-0) > 0) # define LZO_CC_GHS_MSC _MSC_VER # elif defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # define LZO_CC_GHS_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # endif #elif defined(__HIGHC__) # define LZO_CC_HIGHC 1 # define LZO_INFO_CC "MetaWare High C" # define LZO_INFO_CCVER "unknown" #elif defined(__HP_aCC) && ((__HP_aCC-0) > 0) # define LZO_CC_HPACC __HP_aCC # define LZO_INFO_CC "HP aCC" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__HP_aCC) #elif defined(__IAR_SYSTEMS_ICC__) # define LZO_CC_IARC 1 # define LZO_INFO_CC "IAR C" # if defined(__VER__) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__VER__) # else # define LZO_INFO_CCVER "unknown" # endif #elif defined(__IBMC__) && ((__IBMC__-0) > 0) # define LZO_CC_IBMC __IBMC__ # define LZO_INFO_CC "IBM C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__IBMC__) #elif defined(__IBMCPP__) && ((__IBMCPP__-0) > 0) # define LZO_CC_IBMC __IBMCPP__ # define LZO_INFO_CC "IBM C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__IBMCPP__) #elif defined(__KEIL__) && defined(__C166__) # define LZO_CC_KEILC 1 # define LZO_INFO_CC "Keil C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__C166__) #elif defined(__LCC__) && defined(_WIN32) && defined(__LCCOPTIMLEVEL) # define LZO_CC_LCCWIN32 1 # define LZO_INFO_CC "lcc-win32" # define LZO_INFO_CCVER "unknown" #elif defined(__LCC__) # define LZO_CC_LCC 1 # define LZO_INFO_CC "lcc" # if defined(__LCC_VERSION__) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__LCC_VERSION__) # else # define LZO_INFO_CCVER "unknown" # endif #elif defined(__MWERKS__) && ((__MWERKS__-0) > 0) # define LZO_CC_MWERKS __MWERKS__ # define LZO_INFO_CC "Metrowerks C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__MWERKS__) #elif (defined(__NDPC__) || defined(__NDPX__)) && defined(__i386) # define LZO_CC_NDPC 1 # define LZO_INFO_CC "Microway NDP C" # define LZO_INFO_CCVER "unknown" #elif defined(__PACIFIC__) # define LZO_CC_PACIFICC 1 # define LZO_INFO_CC "Pacific C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__PACIFIC__) #elif defined(__PGI) && defined(__PGIC__) && defined(__PGIC_MINOR__) # if defined(__PGIC_PATCHLEVEL__) # define LZO_CC_PGI (__PGIC__ * 0x10000L + (__PGIC_MINOR__-0) * 0x100 + (__PGIC_PATCHLEVEL__-0)) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__PGIC__) "." LZO_PP_MACRO_EXPAND(__PGIC_MINOR__) "." LZO_PP_MACRO_EXPAND(__PGIC_PATCHLEVEL__) # else # define LZO_CC_PGI (__PGIC__ * 0x10000L + (__PGIC_MINOR__-0) * 0x100) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__PGIC__) "." LZO_PP_MACRO_EXPAND(__PGIC_MINOR__) ".0" # endif # define LZO_INFO_CC "Portland Group PGI C" #elif defined(__PGI) && (defined(__linux__) || defined(__WIN32__)) # define LZO_CC_PGI 1 # define LZO_INFO_CC "Portland Group PGI C" # define LZO_INFO_CCVER "unknown" #elif defined(__PUREC__) && defined(__TOS__) # define LZO_CC_PUREC 1 # define LZO_INFO_CC "Pure C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__PUREC__) #elif defined(__SC__) && defined(__ZTC__) # define LZO_CC_SYMANTECC 1 # define LZO_INFO_CC "Symantec C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__SC__) #elif defined(__SUNPRO_C) # define LZO_INFO_CC "SunPro C" # if ((__SUNPRO_C-0) > 0) # define LZO_CC_SUNPROC __SUNPRO_C # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__SUNPRO_C) # else # define LZO_CC_SUNPROC 1 # define LZO_INFO_CCVER "unknown" # endif #elif defined(__SUNPRO_CC) # define LZO_INFO_CC "SunPro C" # if ((__SUNPRO_CC-0) > 0) # define LZO_CC_SUNPROC __SUNPRO_CC # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__SUNPRO_CC) # else # define LZO_CC_SUNPROC 1 # define LZO_INFO_CCVER "unknown" # endif #elif defined(__TINYC__) # define LZO_CC_TINYC 1 # define LZO_INFO_CC "Tiny C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__TINYC__) #elif defined(__TSC__) # define LZO_CC_TOPSPEEDC 1 # define LZO_INFO_CC "TopSpeed C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__TSC__) #elif defined(__WATCOMC__) # define LZO_CC_WATCOMC 1 # define LZO_INFO_CC "Watcom C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__WATCOMC__) #elif defined(__TURBOC__) # define LZO_CC_TURBOC 1 # define LZO_INFO_CC "Turbo C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__TURBOC__) #elif defined(__ZTC__) # define LZO_CC_ZORTECHC 1 # define LZO_INFO_CC "Zortech C" # if ((__ZTC__-0) == 0x310) # define LZO_INFO_CCVER "0x310" # else # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__ZTC__) # endif #elif defined(__GNUC__) && defined(__VERSION__) # if defined(__GNUC_MINOR__) && defined(__GNUC_PATCHLEVEL__) # define LZO_CC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # elif defined(__GNUC_MINOR__) # define LZO_CC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100) # else # define LZO_CC_GNUC (__GNUC__ * 0x10000L) # endif # define LZO_INFO_CC "gcc" # define LZO_INFO_CCVER __VERSION__ #elif defined(_MSC_VER) && ((_MSC_VER-0) > 0) # define LZO_CC_MSC _MSC_VER # define LZO_INFO_CC "Microsoft C" # if defined(_MSC_FULL_VER) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(_MSC_VER) "." LZO_PP_MACRO_EXPAND(_MSC_FULL_VER) # else # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(_MSC_VER) # endif #else # define LZO_CC_UNKNOWN 1 # define LZO_INFO_CC "unknown" # define LZO_INFO_CCVER "unknown" #endif #if (LZO_CC_GNUC) && defined(__OPEN64__) # if defined(__OPENCC__) && defined(__OPENCC_MINOR__) && defined(__OPENCC_PATCHLEVEL__) # define LZO_CC_OPEN64 (__OPENCC__ * 0x10000L + (__OPENCC_MINOR__-0) * 0x100 + (__OPENCC_PATCHLEVEL__-0)) # define LZO_CC_OPEN64_GNUC LZO_CC_GNUC # endif #endif #if (LZO_CC_GNUC) && defined(__PCC__) # if defined(__PCC__) && defined(__PCC_MINOR__) && defined(__PCC_MINORMINOR__) # define LZO_CC_PCC (__PCC__ * 0x10000L + (__PCC_MINOR__-0) * 0x100 + (__PCC_MINORMINOR__-0)) # define LZO_CC_PCC_GNUC LZO_CC_GNUC # endif #endif #if 0 && (LZO_CC_MSC && (_MSC_VER >= 1200)) && !defined(_MSC_FULL_VER) # error "LZO_CC_MSC: _MSC_FULL_VER is not defined" #endif #if !defined(__LZO_ARCH_OVERRIDE) && !(LZO_ARCH_GENERIC) && defined(_CRAY) # if (UINT_MAX > LZO_0xffffffffL) && defined(_CRAY) # if defined(_CRAYMPP) || defined(_CRAYT3D) || defined(_CRAYT3E) # define LZO_ARCH_CRAY_MPP 1 # elif defined(_CRAY1) # define LZO_ARCH_CRAY_PVP 1 # endif # endif #endif #if !defined(__LZO_ARCH_OVERRIDE) #if (LZO_ARCH_GENERIC) # define LZO_INFO_ARCH "generic" #elif (LZO_OS_DOS16 || LZO_OS_OS216 || LZO_OS_WIN16) # define LZO_ARCH_I086 1 # define LZO_INFO_ARCH "i086" #elif defined(__aarch64__) || defined(_M_ARM64) # define LZO_ARCH_ARM64 1 # define LZO_INFO_ARCH "arm64" #elif defined(__alpha__) || defined(__alpha) || defined(_M_ALPHA) # define LZO_ARCH_ALPHA 1 # define LZO_INFO_ARCH "alpha" #elif (LZO_ARCH_CRAY_MPP) && (defined(_CRAYT3D) || defined(_CRAYT3E)) # define LZO_ARCH_ALPHA 1 # define LZO_INFO_ARCH "alpha" #elif defined(__amd64__) || defined(__x86_64__) || defined(_M_AMD64) # define LZO_ARCH_AMD64 1 # define LZO_INFO_ARCH "amd64" #elif defined(__arm__) || defined(_M_ARM) # define LZO_ARCH_ARM 1 # define LZO_INFO_ARCH "arm" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICCARM__) # define LZO_ARCH_ARM 1 # define LZO_INFO_ARCH "arm" #elif (UINT_MAX <= LZO_0xffffL) && defined(__AVR__) # define LZO_ARCH_AVR 1 # define LZO_INFO_ARCH "avr" #elif defined(__avr32__) || defined(__AVR32__) # define LZO_ARCH_AVR32 1 # define LZO_INFO_ARCH "avr32" #elif defined(__bfin__) # define LZO_ARCH_BLACKFIN 1 # define LZO_INFO_ARCH "blackfin" #elif (UINT_MAX == LZO_0xffffL) && defined(__C166__) # define LZO_ARCH_C166 1 # define LZO_INFO_ARCH "c166" #elif defined(__cris__) # define LZO_ARCH_CRIS 1 # define LZO_INFO_ARCH "cris" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICCEZ80__) # define LZO_ARCH_EZ80 1 # define LZO_INFO_ARCH "ez80" #elif defined(__H8300__) || defined(__H8300H__) || defined(__H8300S__) || defined(__H8300SX__) # define LZO_ARCH_H8300 1 # define LZO_INFO_ARCH "h8300" #elif defined(__hppa__) || defined(__hppa) # define LZO_ARCH_HPPA 1 # define LZO_INFO_ARCH "hppa" #elif defined(__386__) || defined(__i386__) || defined(__i386) || defined(_M_IX86) || defined(_M_I386) # define LZO_ARCH_I386 1 # define LZO_ARCH_IA32 1 # define LZO_INFO_ARCH "i386" #elif (LZO_CC_ZORTECHC && defined(__I86__)) # define LZO_ARCH_I386 1 # define LZO_ARCH_IA32 1 # define LZO_INFO_ARCH "i386" #elif (LZO_OS_DOS32 && LZO_CC_HIGHC) && defined(_I386) # define LZO_ARCH_I386 1 # define LZO_ARCH_IA32 1 # define LZO_INFO_ARCH "i386" #elif defined(__ia64__) || defined(__ia64) || defined(_M_IA64) # define LZO_ARCH_IA64 1 # define LZO_INFO_ARCH "ia64" #elif (UINT_MAX == LZO_0xffffL) && defined(__m32c__) # define LZO_ARCH_M16C 1 # define LZO_INFO_ARCH "m16c" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICCM16C__) # define LZO_ARCH_M16C 1 # define LZO_INFO_ARCH "m16c" #elif defined(__m32r__) # define LZO_ARCH_M32R 1 # define LZO_INFO_ARCH "m32r" #elif (LZO_OS_TOS) || defined(__m68k__) || defined(__m68000__) || defined(__mc68000__) || defined(__mc68020__) || defined(_M_M68K) # define LZO_ARCH_M68K 1 # define LZO_INFO_ARCH "m68k" #elif (UINT_MAX == LZO_0xffffL) && defined(__C251__) # define LZO_ARCH_MCS251 1 # define LZO_INFO_ARCH "mcs251" #elif (UINT_MAX == LZO_0xffffL) && defined(__C51__) # define LZO_ARCH_MCS51 1 # define LZO_INFO_ARCH "mcs51" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICC8051__) # define LZO_ARCH_MCS51 1 # define LZO_INFO_ARCH "mcs51" #elif defined(__mips__) || defined(__mips) || defined(_MIPS_ARCH) || defined(_M_MRX000) # define LZO_ARCH_MIPS 1 # define LZO_INFO_ARCH "mips" #elif (UINT_MAX == LZO_0xffffL) && defined(__MSP430__) # define LZO_ARCH_MSP430 1 # define LZO_INFO_ARCH "msp430" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICC430__) # define LZO_ARCH_MSP430 1 # define LZO_INFO_ARCH "msp430" #elif defined(__powerpc__) || defined(__powerpc) || defined(__ppc__) || defined(__PPC__) || defined(_M_PPC) || defined(_ARCH_PPC) || defined(_ARCH_PWR) # define LZO_ARCH_POWERPC 1 # define LZO_INFO_ARCH "powerpc" #elif defined(__powerpc64__) || defined(__powerpc64) || defined(__ppc64__) || defined(__PPC64__) # define LZO_ARCH_POWERPC 1 # define LZO_INFO_ARCH "powerpc" #elif defined(__powerpc64le__) || defined(__powerpc64le) || defined(__ppc64le__) || defined(__PPC64LE__) # define LZO_ARCH_POWERPC 1 # define LZO_INFO_ARCH "powerpc" #elif defined(__riscv) # define LZO_ARCH_RISCV 1 # define LZO_INFO_ARCH "riscv" #elif defined(__s390__) || defined(__s390) || defined(__s390x__) || defined(__s390x) # define LZO_ARCH_S390 1 # define LZO_INFO_ARCH "s390" #elif defined(__sh__) || defined(_M_SH) # define LZO_ARCH_SH 1 # define LZO_INFO_ARCH "sh" #elif defined(__sparc__) || defined(__sparc) || defined(__sparcv8) # define LZO_ARCH_SPARC 1 # define LZO_INFO_ARCH "sparc" #elif defined(__SPU__) # define LZO_ARCH_SPU 1 # define LZO_INFO_ARCH "spu" #elif (UINT_MAX == LZO_0xffffL) && defined(__z80) # define LZO_ARCH_Z80 1 # define LZO_INFO_ARCH "z80" #elif (LZO_ARCH_CRAY_PVP) # if defined(_CRAYSV1) # define LZO_ARCH_CRAY_SV1 1 # define LZO_INFO_ARCH "cray_sv1" # elif (_ADDR64) # define LZO_ARCH_CRAY_T90 1 # define LZO_INFO_ARCH "cray_t90" # elif (_ADDR32) # define LZO_ARCH_CRAY_YMP 1 # define LZO_INFO_ARCH "cray_ymp" # else # define LZO_ARCH_CRAY_XMP 1 # define LZO_INFO_ARCH "cray_xmp" # endif #else # define LZO_ARCH_UNKNOWN 1 # define LZO_INFO_ARCH "unknown" #endif #endif #if !defined(LZO_ARCH_ARM_THUMB2) #if (LZO_ARCH_ARM) # if defined(__thumb__) || defined(__thumb) || defined(_M_THUMB) # if defined(__thumb2__) # define LZO_ARCH_ARM_THUMB2 1 # elif 1 && defined(__TARGET_ARCH_THUMB) && ((__TARGET_ARCH_THUMB)+0 >= 4) # define LZO_ARCH_ARM_THUMB2 1 # elif 1 && defined(_MSC_VER) && defined(_M_THUMB) && ((_M_THUMB)+0 >= 7) # define LZO_ARCH_ARM_THUMB2 1 # endif # endif #endif #endif #if (LZO_ARCH_ARM_THUMB2) # undef LZO_INFO_ARCH # define LZO_INFO_ARCH "arm_thumb2" #endif #if 1 && (LZO_ARCH_UNKNOWN) && (LZO_OS_DOS32 || LZO_OS_OS2) # error "FIXME - missing define for CPU architecture" #endif #if 1 && (LZO_ARCH_UNKNOWN) && (LZO_OS_WIN32) # error "FIXME - missing LZO_OS_WIN32 define for CPU architecture" #endif #if 1 && (LZO_ARCH_UNKNOWN) && (LZO_OS_WIN64) # error "FIXME - missing LZO_OS_WIN64 define for CPU architecture" #endif #if (LZO_OS_OS216 || LZO_OS_WIN16) # define LZO_ARCH_I086PM 1 #elif 1 && (LZO_OS_DOS16 && defined(BLX286)) # define LZO_ARCH_I086PM 1 #elif 1 && (LZO_OS_DOS16 && defined(DOSX286)) # define LZO_ARCH_I086PM 1 #elif 1 && (LZO_OS_DOS16 && LZO_CC_BORLANDC && defined(__DPMI16__)) # define LZO_ARCH_I086PM 1 #endif #if (LZO_ARCH_AMD64 && !LZO_ARCH_X64) # define LZO_ARCH_X64 1 #elif (!LZO_ARCH_AMD64 && LZO_ARCH_X64) && defined(__LZO_ARCH_OVERRIDE) # define LZO_ARCH_AMD64 1 #endif #if (LZO_ARCH_ARM64 && !LZO_ARCH_AARCH64) # define LZO_ARCH_AARCH64 1 #elif (!LZO_ARCH_ARM64 && LZO_ARCH_AARCH64) && defined(__LZO_ARCH_OVERRIDE) # define LZO_ARCH_ARM64 1 #endif #if (LZO_ARCH_I386 && !LZO_ARCH_X86) # define LZO_ARCH_X86 1 #elif (!LZO_ARCH_I386 && LZO_ARCH_X86) && defined(__LZO_ARCH_OVERRIDE) # define LZO_ARCH_I386 1 #endif #if (LZO_ARCH_AMD64 && !LZO_ARCH_X64) || (!LZO_ARCH_AMD64 && LZO_ARCH_X64) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM64 && !LZO_ARCH_AARCH64) || (!LZO_ARCH_ARM64 && LZO_ARCH_AARCH64) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_I386 && !LZO_ARCH_X86) || (!LZO_ARCH_I386 && LZO_ARCH_X86) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM_THUMB1 && !LZO_ARCH_ARM) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM_THUMB2 && !LZO_ARCH_ARM) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM_THUMB1 && LZO_ARCH_ARM_THUMB2) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_I086PM && !LZO_ARCH_I086) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_I086) # if (UINT_MAX != LZO_0xffffL) # error "unexpected configuration - check your compiler defines" # endif # if (ULONG_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif #endif #if (LZO_ARCH_I386) # if (UINT_MAX != LZO_0xffffL) && defined(__i386_int16__) # error "unexpected configuration - check your compiler defines" # endif # if (UINT_MAX != LZO_0xffffffffL) && !defined(__i386_int16__) # error "unexpected configuration - check your compiler defines" # endif # if (ULONG_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif #endif #if (LZO_ARCH_AMD64 || LZO_ARCH_I386) # if !defined(LZO_TARGET_FEATURE_SSE2) # if defined(__SSE2__) # define LZO_TARGET_FEATURE_SSE2 1 # elif defined(_MSC_VER) && (defined(_M_IX86_FP) && ((_M_IX86_FP)+0 >= 2)) # define LZO_TARGET_FEATURE_SSE2 1 # elif (LZO_CC_INTELC_MSC || LZO_CC_MSC) && defined(_M_AMD64) # define LZO_TARGET_FEATURE_SSE2 1 # endif # endif # if !defined(LZO_TARGET_FEATURE_SSSE3) # if (LZO_TARGET_FEATURE_SSE2) # if defined(__SSSE3__) # define LZO_TARGET_FEATURE_SSSE3 1 # elif defined(_MSC_VER) && defined(__AVX__) # define LZO_TARGET_FEATURE_SSSE3 1 # endif # endif # endif # if !defined(LZO_TARGET_FEATURE_SSE4_2) # if (LZO_TARGET_FEATURE_SSSE3) # if defined(__SSE4_2__) # define LZO_TARGET_FEATURE_SSE4_2 1 # endif # endif # endif # if !defined(LZO_TARGET_FEATURE_AVX) # if (LZO_TARGET_FEATURE_SSSE3) # if defined(__AVX__) # define LZO_TARGET_FEATURE_AVX 1 # endif # endif # endif # if !defined(LZO_TARGET_FEATURE_AVX2) # if (LZO_TARGET_FEATURE_AVX) # if defined(__AVX2__) # define LZO_TARGET_FEATURE_AVX2 1 # endif # endif # endif #endif #if (LZO_TARGET_FEATURE_SSSE3 && !(LZO_TARGET_FEATURE_SSE2)) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_TARGET_FEATURE_SSE4_2 && !(LZO_TARGET_FEATURE_SSSE3)) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_TARGET_FEATURE_AVX && !(LZO_TARGET_FEATURE_SSSE3)) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_TARGET_FEATURE_AVX2 && !(LZO_TARGET_FEATURE_AVX)) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM) # if !defined(LZO_TARGET_FEATURE_NEON) # if defined(__ARM_NEON) && ((__ARM_NEON)+0) # define LZO_TARGET_FEATURE_NEON 1 # elif 1 && defined(__ARM_NEON__) && ((__ARM_NEON__)+0) # define LZO_TARGET_FEATURE_NEON 1 # elif 1 && defined(__TARGET_FEATURE_NEON) && ((__TARGET_FEATURE_NEON)+0) # define LZO_TARGET_FEATURE_NEON 1 # endif # endif #elif (LZO_ARCH_ARM64) # if !defined(LZO_TARGET_FEATURE_NEON) # if 1 # define LZO_TARGET_FEATURE_NEON 1 # endif # endif #endif #if 0 #elif !defined(__LZO_MM_OVERRIDE) #if (LZO_ARCH_I086) #if (UINT_MAX != LZO_0xffffL) # error "unexpected configuration - check your compiler defines" #endif #if defined(__TINY__) || defined(M_I86TM) || defined(_M_I86TM) # define LZO_MM_TINY 1 #elif defined(__HUGE__) || defined(_HUGE_) || defined(M_I86HM) || defined(_M_I86HM) # define LZO_MM_HUGE 1 #elif defined(__SMALL__) || defined(M_I86SM) || defined(_M_I86SM) || defined(SMALL_MODEL) # define LZO_MM_SMALL 1 #elif defined(__MEDIUM__) || defined(M_I86MM) || defined(_M_I86MM) # define LZO_MM_MEDIUM 1 #elif defined(__COMPACT__) || defined(M_I86CM) || defined(_M_I86CM) # define LZO_MM_COMPACT 1 #elif defined(__LARGE__) || defined(M_I86LM) || defined(_M_I86LM) || defined(LARGE_MODEL) # define LZO_MM_LARGE 1 #elif (LZO_CC_AZTECC) # if defined(_LARGE_CODE) && defined(_LARGE_DATA) # define LZO_MM_LARGE 1 # elif defined(_LARGE_CODE) # define LZO_MM_MEDIUM 1 # elif defined(_LARGE_DATA) # define LZO_MM_COMPACT 1 # else # define LZO_MM_SMALL 1 # endif #elif (LZO_CC_ZORTECHC && defined(__VCM__)) # define LZO_MM_LARGE 1 #else # error "unknown LZO_ARCH_I086 memory model" #endif #if (LZO_OS_DOS16 || LZO_OS_OS216 || LZO_OS_WIN16) #define LZO_HAVE_MM_HUGE_PTR 1 #define LZO_HAVE_MM_HUGE_ARRAY 1 #if (LZO_MM_TINY) # undef LZO_HAVE_MM_HUGE_ARRAY #endif #if (LZO_CC_AZTECC || LZO_CC_PACIFICC || LZO_CC_ZORTECHC) # undef LZO_HAVE_MM_HUGE_PTR # undef LZO_HAVE_MM_HUGE_ARRAY #elif (LZO_CC_DMC || LZO_CC_SYMANTECC) # undef LZO_HAVE_MM_HUGE_ARRAY #elif (LZO_CC_MSC && defined(_QC)) # undef LZO_HAVE_MM_HUGE_ARRAY # if (_MSC_VER < 600) # undef LZO_HAVE_MM_HUGE_PTR # endif #elif (LZO_CC_TURBOC && (__TURBOC__ < 0x0295)) # undef LZO_HAVE_MM_HUGE_ARRAY #endif #if (LZO_ARCH_I086PM) && !(LZO_HAVE_MM_HUGE_PTR) # if (LZO_OS_DOS16) # error "unexpected configuration - check your compiler defines" # elif (LZO_CC_ZORTECHC) # else # error "unexpected configuration - check your compiler defines" # endif #endif #if defined(__cplusplus) extern "C" { #endif #if (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0200)) extern void __near __cdecl _AHSHIFT(void); # define LZO_MM_AHSHIFT ((unsigned) _AHSHIFT) #elif (LZO_CC_DMC || LZO_CC_SYMANTECC || LZO_CC_ZORTECHC) extern void __near __cdecl _AHSHIFT(void); # define LZO_MM_AHSHIFT ((unsigned) _AHSHIFT) #elif (LZO_CC_MSC || LZO_CC_TOPSPEEDC) extern void __near __cdecl _AHSHIFT(void); # define LZO_MM_AHSHIFT ((unsigned) _AHSHIFT) #elif (LZO_CC_TURBOC && (__TURBOC__ >= 0x0295)) extern void __near __cdecl _AHSHIFT(void); # define LZO_MM_AHSHIFT ((unsigned) _AHSHIFT) #elif ((LZO_CC_AZTECC || LZO_CC_PACIFICC || LZO_CC_TURBOC) && LZO_OS_DOS16) # define LZO_MM_AHSHIFT 12 #elif (LZO_CC_WATCOMC) extern unsigned char _HShift; # define LZO_MM_AHSHIFT ((unsigned) _HShift) #else # error "FIXME - implement LZO_MM_AHSHIFT" #endif #if defined(__cplusplus) } #endif #endif #elif (LZO_ARCH_C166) #if !defined(__MODEL__) # error "FIXME - LZO_ARCH_C166 __MODEL__" #elif ((__MODEL__) == 0) # define LZO_MM_SMALL 1 #elif ((__MODEL__) == 1) # define LZO_MM_SMALL 1 #elif ((__MODEL__) == 2) # define LZO_MM_LARGE 1 #elif ((__MODEL__) == 3) # define LZO_MM_TINY 1 #elif ((__MODEL__) == 4) # define LZO_MM_XTINY 1 #elif ((__MODEL__) == 5) # define LZO_MM_XSMALL 1 #else # error "FIXME - LZO_ARCH_C166 __MODEL__" #endif #elif (LZO_ARCH_MCS251) #if !defined(__MODEL__) # error "FIXME - LZO_ARCH_MCS251 __MODEL__" #elif ((__MODEL__) == 0) # define LZO_MM_SMALL 1 #elif ((__MODEL__) == 2) # define LZO_MM_LARGE 1 #elif ((__MODEL__) == 3) # define LZO_MM_TINY 1 #elif ((__MODEL__) == 4) # define LZO_MM_XTINY 1 #elif ((__MODEL__) == 5) # define LZO_MM_XSMALL 1 #else # error "FIXME - LZO_ARCH_MCS251 __MODEL__" #endif #elif (LZO_ARCH_MCS51) #if !defined(__MODEL__) # error "FIXME - LZO_ARCH_MCS51 __MODEL__" #elif ((__MODEL__) == 1) # define LZO_MM_SMALL 1 #elif ((__MODEL__) == 2) # define LZO_MM_LARGE 1 #elif ((__MODEL__) == 3) # define LZO_MM_TINY 1 #elif ((__MODEL__) == 4) # define LZO_MM_XTINY 1 #elif ((__MODEL__) == 5) # define LZO_MM_XSMALL 1 #else # error "FIXME - LZO_ARCH_MCS51 __MODEL__" #endif #elif (LZO_ARCH_CRAY_PVP) # define LZO_MM_PVP 1 #else # define LZO_MM_FLAT 1 #endif #if (LZO_MM_COMPACT) # define LZO_INFO_MM "compact" #elif (LZO_MM_FLAT) # define LZO_INFO_MM "flat" #elif (LZO_MM_HUGE) # define LZO_INFO_MM "huge" #elif (LZO_MM_LARGE) # define LZO_INFO_MM "large" #elif (LZO_MM_MEDIUM) # define LZO_INFO_MM "medium" #elif (LZO_MM_PVP) # define LZO_INFO_MM "pvp" #elif (LZO_MM_SMALL) # define LZO_INFO_MM "small" #elif (LZO_MM_TINY) # define LZO_INFO_MM "tiny" #else # error "unknown memory model" #endif #endif #if !defined(__lzo_gnuc_extension__) #if (LZO_CC_GNUC >= 0x020800ul) # define __lzo_gnuc_extension__ __extension__ #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_gnuc_extension__ __extension__ #elif (LZO_CC_IBMC >= 600) # define __lzo_gnuc_extension__ __extension__ #endif #endif #if !defined(__lzo_gnuc_extension__) # define __lzo_gnuc_extension__ /*empty*/ #endif #if !defined(lzo_has_builtin) #if (LZO_CC_CLANG) && defined(__has_builtin) # define lzo_has_builtin __has_builtin #endif #endif #if !defined(lzo_has_builtin) # define lzo_has_builtin(x) 0 #endif #if !defined(lzo_has_attribute) #if (LZO_CC_CLANG) && defined(__has_attribute) # define lzo_has_attribute __has_attribute #endif #endif #if !defined(lzo_has_attribute) # define lzo_has_attribute(x) 0 #endif #if !defined(lzo_has_declspec_attribute) #if (LZO_CC_CLANG) && defined(__has_declspec_attribute) # define lzo_has_declspec_attribute __has_declspec_attribute #endif #endif #if !defined(lzo_has_declspec_attribute) # define lzo_has_declspec_attribute(x) 0 #endif #if !defined(lzo_has_feature) #if (LZO_CC_CLANG) && defined(__has_feature) # define lzo_has_feature __has_feature #endif #endif #if !defined(lzo_has_feature) # define lzo_has_feature(x) 0 #endif #if !defined(lzo_has_extension) #if (LZO_CC_CLANG) && defined(__has_extension) # define lzo_has_extension __has_extension #elif (LZO_CC_CLANG) && defined(__has_feature) # define lzo_has_extension __has_feature #endif #endif #if !defined(lzo_has_extension) # define lzo_has_extension(x) 0 #endif #if !defined(LZO_CFG_USE_NEW_STYLE_CASTS) && defined(__cplusplus) && 0 # if (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020800ul)) # define LZO_CFG_USE_NEW_STYLE_CASTS 0 # elif (LZO_CC_INTELC && (__INTEL_COMPILER < 1200)) # define LZO_CFG_USE_NEW_STYLE_CASTS 0 # else # define LZO_CFG_USE_NEW_STYLE_CASTS 1 # endif #endif #if !defined(LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_CFG_USE_NEW_STYLE_CASTS 0 #endif #if !defined(__cplusplus) # if defined(LZO_CFG_USE_NEW_STYLE_CASTS) # undef LZO_CFG_USE_NEW_STYLE_CASTS # endif # define LZO_CFG_USE_NEW_STYLE_CASTS 0 #endif #if !defined(LZO_REINTERPRET_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_REINTERPRET_CAST(t,e) (reinterpret_cast (e)) # endif #endif #if !defined(LZO_REINTERPRET_CAST) # define LZO_REINTERPRET_CAST(t,e) ((t) (e)) #endif #if !defined(LZO_STATIC_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_STATIC_CAST(t,e) (static_cast (e)) # endif #endif #if !defined(LZO_STATIC_CAST) # define LZO_STATIC_CAST(t,e) ((t) (e)) #endif #if !defined(LZO_STATIC_CAST2) # define LZO_STATIC_CAST2(t1,t2,e) LZO_STATIC_CAST(t1, LZO_STATIC_CAST(t2, e)) #endif #if !defined(LZO_UNCONST_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_UNCONST_CAST(t,e) (const_cast (e)) # elif (LZO_HAVE_MM_HUGE_PTR) # define LZO_UNCONST_CAST(t,e) ((t) (e)) # elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNCONST_CAST(t,e) ((t) ((void *) ((lzo_uintptr_t) ((const void *) (e))))) # endif #endif #if !defined(LZO_UNCONST_CAST) # define LZO_UNCONST_CAST(t,e) ((t) ((void *) ((const void *) (e)))) #endif #if !defined(LZO_UNCONST_VOLATILE_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_UNCONST_VOLATILE_CAST(t,e) (const_cast (e)) # elif (LZO_HAVE_MM_HUGE_PTR) # define LZO_UNCONST_VOLATILE_CAST(t,e) ((t) (e)) # elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNCONST_VOLATILE_CAST(t,e) ((t) ((volatile void *) ((lzo_uintptr_t) ((volatile const void *) (e))))) # endif #endif #if !defined(LZO_UNCONST_VOLATILE_CAST) # define LZO_UNCONST_VOLATILE_CAST(t,e) ((t) ((volatile void *) ((volatile const void *) (e)))) #endif #if !defined(LZO_UNVOLATILE_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_UNVOLATILE_CAST(t,e) (const_cast (e)) # elif (LZO_HAVE_MM_HUGE_PTR) # define LZO_UNVOLATILE_CAST(t,e) ((t) (e)) # elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNVOLATILE_CAST(t,e) ((t) ((void *) ((lzo_uintptr_t) ((volatile void *) (e))))) # endif #endif #if !defined(LZO_UNVOLATILE_CAST) # define LZO_UNVOLATILE_CAST(t,e) ((t) ((void *) ((volatile void *) (e)))) #endif #if !defined(LZO_UNVOLATILE_CONST_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_UNVOLATILE_CONST_CAST(t,e) (const_cast (e)) # elif (LZO_HAVE_MM_HUGE_PTR) # define LZO_UNVOLATILE_CONST_CAST(t,e) ((t) (e)) # elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNVOLATILE_CONST_CAST(t,e) ((t) ((const void *) ((lzo_uintptr_t) ((volatile const void *) (e))))) # endif #endif #if !defined(LZO_UNVOLATILE_CONST_CAST) # define LZO_UNVOLATILE_CONST_CAST(t,e) ((t) ((const void *) ((volatile const void *) (e)))) #endif #if !defined(LZO_PCAST) # if (LZO_HAVE_MM_HUGE_PTR) # define LZO_PCAST(t,e) ((t) (e)) # endif #endif #if !defined(LZO_PCAST) # define LZO_PCAST(t,e) LZO_STATIC_CAST(t, LZO_STATIC_CAST(void *, e)) #endif #if !defined(LZO_CCAST) # if (LZO_HAVE_MM_HUGE_PTR) # define LZO_CCAST(t,e) ((t) (e)) # endif #endif #if !defined(LZO_CCAST) # define LZO_CCAST(t,e) LZO_STATIC_CAST(t, LZO_STATIC_CAST(const void *, e)) #endif #if !defined(LZO_ICONV) # define LZO_ICONV(t,e) LZO_STATIC_CAST(t, e) #endif #if !defined(LZO_ICAST) # define LZO_ICAST(t,e) LZO_STATIC_CAST(t, e) #endif #if !defined(LZO_ITRUNC) # define LZO_ITRUNC(t,e) LZO_STATIC_CAST(t, e) #endif #if !defined(__lzo_cte) # if (LZO_CC_MSC || LZO_CC_WATCOMC) # define __lzo_cte(e) ((void)0,(e)) # elif 1 # define __lzo_cte(e) ((void)0,(e)) # endif #endif #if !defined(__lzo_cte) # define __lzo_cte(e) (e) #endif #if !defined(LZO_BLOCK_BEGIN) # define LZO_BLOCK_BEGIN do { # define LZO_BLOCK_END } while __lzo_cte(0) #endif #if !defined(LZO_UNUSED) # if (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0600)) # define LZO_UNUSED(var) ((void) &var) # elif (LZO_CC_BORLANDC || LZO_CC_HIGHC || LZO_CC_NDPC || LZO_CC_PELLESC || LZO_CC_TURBOC) # define LZO_UNUSED(var) if (&var) ; else # elif (LZO_CC_CLANG && (LZO_CC_CLANG >= 0x030200ul)) # define LZO_UNUSED(var) ((void) &var) # elif (LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNUSED(var) ((void) var) # elif (LZO_CC_MSC && (_MSC_VER < 900)) # define LZO_UNUSED(var) if (&var) ; else # elif (LZO_CC_KEILC) # define LZO_UNUSED(var) {extern int lzo_unused__[1-2*!(sizeof(var)>0)]; (void)lzo_unused__;} # elif (LZO_CC_PACIFICC) # define LZO_UNUSED(var) ((void) sizeof(var)) # elif (LZO_CC_WATCOMC) && defined(__cplusplus) # define LZO_UNUSED(var) ((void) var) # else # define LZO_UNUSED(var) ((void) &var) # endif #endif #if !defined(LZO_UNUSED_RESULT) # define LZO_UNUSED_RESULT(var) LZO_UNUSED(var) #endif #if !defined(LZO_UNUSED_FUNC) # if (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0600)) # define LZO_UNUSED_FUNC(func) ((void) func) # elif (LZO_CC_BORLANDC || LZO_CC_NDPC || LZO_CC_TURBOC) # define LZO_UNUSED_FUNC(func) if (func) ; else # elif (LZO_CC_CLANG || LZO_CC_LLVM) # define LZO_UNUSED_FUNC(func) ((void) &func) # elif (LZO_CC_MSC && (_MSC_VER < 900)) # define LZO_UNUSED_FUNC(func) if (func) ; else # elif (LZO_CC_MSC) # define LZO_UNUSED_FUNC(func) ((void) &func) # elif (LZO_CC_KEILC || LZO_CC_PELLESC) # define LZO_UNUSED_FUNC(func) {extern int lzo_unused__[1-2*!(sizeof((int)func)>0)]; (void)lzo_unused__;} # else # define LZO_UNUSED_FUNC(func) ((void) func) # endif #endif #if !defined(LZO_UNUSED_LABEL) # if (LZO_CC_CLANG >= 0x020800ul) # define LZO_UNUSED_LABEL(l) (__lzo_gnuc_extension__ ((void) ((const void *) &&l))) # elif (LZO_CC_ARMCC || LZO_CC_CLANG || LZO_CC_INTELC || LZO_CC_WATCOMC) # define LZO_UNUSED_LABEL(l) if __lzo_cte(0) goto l # else # define LZO_UNUSED_LABEL(l) switch (0) case 1:goto l # endif #endif #if !defined(LZO_DEFINE_UNINITIALIZED_VAR) # if 0 # define LZO_DEFINE_UNINITIALIZED_VAR(type,var,init) type var # elif 0 && (LZO_CC_GNUC) # define LZO_DEFINE_UNINITIALIZED_VAR(type,var,init) type var = var # else # define LZO_DEFINE_UNINITIALIZED_VAR(type,var,init) type var = init # endif #endif #if !defined(__lzo_inline) #if (LZO_CC_TURBOC && (__TURBOC__ <= 0x0295)) #elif defined(__cplusplus) # define __lzo_inline inline #elif defined(__STDC_VERSION__) && (__STDC_VERSION__-0 >= 199901L) # define __lzo_inline inline #elif (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0550)) # define __lzo_inline __inline #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CILLY || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || LZO_CC_PGI) # define __lzo_inline __inline__ #elif (LZO_CC_DMC) # define __lzo_inline __inline #elif (LZO_CC_GHS) # define __lzo_inline __inline__ #elif (LZO_CC_IBMC >= 600) # define __lzo_inline __inline__ #elif (LZO_CC_INTELC) # define __lzo_inline __inline #elif (LZO_CC_MWERKS && (__MWERKS__ >= 0x2405)) # define __lzo_inline __inline #elif (LZO_CC_MSC && (_MSC_VER >= 900)) # define __lzo_inline __inline #elif (LZO_CC_SUNPROC >= 0x5100) # define __lzo_inline __inline__ #endif #endif #if defined(__lzo_inline) # ifndef __lzo_HAVE_inline # define __lzo_HAVE_inline 1 # endif #else # define __lzo_inline /*empty*/ #endif #if !defined(__lzo_forceinline) #if (LZO_CC_GNUC >= 0x030200ul) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_IBMC >= 700) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 450)) # define __lzo_forceinline __forceinline #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 800)) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_MSC && (_MSC_VER >= 1200)) # define __lzo_forceinline __forceinline #elif (LZO_CC_PGI >= 0x0d0a00ul) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_SUNPROC >= 0x5100) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #endif #endif #if defined(__lzo_forceinline) # ifndef __lzo_HAVE_forceinline # define __lzo_HAVE_forceinline 1 # endif #else # define __lzo_forceinline __lzo_inline #endif #if !defined(__lzo_noinline) #if 1 && (LZO_ARCH_I386) && (LZO_CC_GNUC >= 0x040000ul) && (LZO_CC_GNUC < 0x040003ul) # define __lzo_noinline __attribute__((__noinline__,__used__)) #elif (LZO_CC_GNUC >= 0x030200ul) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_IBMC >= 700) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 600)) # define __lzo_noinline __declspec(noinline) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 800)) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_noinline __declspec(noinline) #elif (LZO_CC_MWERKS && (__MWERKS__ >= 0x3200) && (LZO_OS_WIN32 || LZO_OS_WIN64)) # if defined(__cplusplus) # else # define __lzo_noinline __declspec(noinline) # endif #elif (LZO_CC_PGI >= 0x0d0a00ul) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_SUNPROC >= 0x5100) # define __lzo_noinline __attribute__((__noinline__)) #endif #endif #if defined(__lzo_noinline) # ifndef __lzo_HAVE_noinline # define __lzo_HAVE_noinline 1 # endif #else # define __lzo_noinline /*empty*/ #endif #if (__lzo_HAVE_forceinline || __lzo_HAVE_noinline) && !(__lzo_HAVE_inline) # error "unexpected configuration - check your compiler defines" #endif #if !defined(__lzo_static_inline) #if (LZO_CC_IBMC) # define __lzo_static_inline __lzo_gnuc_extension__ static __lzo_inline #endif #endif #if !defined(__lzo_static_inline) # define __lzo_static_inline static __lzo_inline #endif #if !defined(__lzo_static_forceinline) #if (LZO_CC_IBMC) # define __lzo_static_forceinline __lzo_gnuc_extension__ static __lzo_forceinline #endif #endif #if !defined(__lzo_static_forceinline) # define __lzo_static_forceinline static __lzo_forceinline #endif #if !defined(__lzo_static_noinline) #if (LZO_CC_IBMC) # define __lzo_static_noinline __lzo_gnuc_extension__ static __lzo_noinline #endif #endif #if !defined(__lzo_static_noinline) # define __lzo_static_noinline static __lzo_noinline #endif #if !defined(__lzo_c99_extern_inline) #if defined(__GNUC_GNU_INLINE__) # define __lzo_c99_extern_inline __lzo_inline #elif defined(__GNUC_STDC_INLINE__) # define __lzo_c99_extern_inline extern __lzo_inline #elif defined(__STDC_VERSION__) && (__STDC_VERSION__-0 >= 199901L) # define __lzo_c99_extern_inline extern __lzo_inline #endif #if !defined(__lzo_c99_extern_inline) && (__lzo_HAVE_inline) # define __lzo_c99_extern_inline __lzo_inline #endif #endif #if defined(__lzo_c99_extern_inline) # ifndef __lzo_HAVE_c99_extern_inline # define __lzo_HAVE_c99_extern_inline 1 # endif #else # define __lzo_c99_extern_inline /*empty*/ #endif #if !defined(__lzo_may_alias) #if (LZO_CC_GNUC >= 0x030400ul) # define __lzo_may_alias __attribute__((__may_alias__)) #elif (LZO_CC_CLANG >= 0x020900ul) # define __lzo_may_alias __attribute__((__may_alias__)) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 1210)) && 0 # define __lzo_may_alias __attribute__((__may_alias__)) #elif (LZO_CC_PGI >= 0x0d0a00ul) && 0 # define __lzo_may_alias __attribute__((__may_alias__)) #endif #endif #if defined(__lzo_may_alias) # ifndef __lzo_HAVE_may_alias # define __lzo_HAVE_may_alias 1 # endif #else # define __lzo_may_alias /*empty*/ #endif #if !defined(__lzo_noreturn) #if (LZO_CC_GNUC >= 0x020700ul) # define __lzo_noreturn __attribute__((__noreturn__)) #elif (LZO_CC_IBMC >= 700) # define __lzo_noreturn __attribute__((__noreturn__)) #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 450)) # define __lzo_noreturn __declspec(noreturn) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 600)) # define __lzo_noreturn __attribute__((__noreturn__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_noreturn __attribute__((__noreturn__)) #elif (LZO_CC_MSC && (_MSC_VER >= 1200)) # define __lzo_noreturn __declspec(noreturn) #elif (LZO_CC_PGI >= 0x0d0a00ul) # define __lzo_noreturn __attribute__((__noreturn__)) #endif #endif #if defined(__lzo_noreturn) # ifndef __lzo_HAVE_noreturn # define __lzo_HAVE_noreturn 1 # endif #else # define __lzo_noreturn /*empty*/ #endif #if !defined(__lzo_nothrow) #if (LZO_CC_GNUC >= 0x030300ul) # define __lzo_nothrow __attribute__((__nothrow__)) #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 450)) && defined(__cplusplus) # define __lzo_nothrow __declspec(nothrow) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 900)) # define __lzo_nothrow __attribute__((__nothrow__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_nothrow __attribute__((__nothrow__)) #elif (LZO_CC_MSC && (_MSC_VER >= 1200)) && defined(__cplusplus) # define __lzo_nothrow __declspec(nothrow) #endif #endif #if defined(__lzo_nothrow) # ifndef __lzo_HAVE_nothrow # define __lzo_HAVE_nothrow 1 # endif #else # define __lzo_nothrow /*empty*/ #endif #if !defined(__lzo_restrict) #if (LZO_CC_GNUC >= 0x030400ul) # define __lzo_restrict __restrict__ #elif (LZO_CC_IBMC >= 800) && !defined(__cplusplus) # define __lzo_restrict __restrict__ #elif (LZO_CC_IBMC >= 1210) # define __lzo_restrict __restrict__ #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 600)) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 600)) # define __lzo_restrict __restrict__ #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM) # define __lzo_restrict __restrict__ #elif (LZO_CC_MSC && (_MSC_VER >= 1400)) # define __lzo_restrict __restrict #elif (LZO_CC_PGI >= 0x0d0a00ul) # define __lzo_restrict __restrict__ #endif #endif #if defined(__lzo_restrict) # ifndef __lzo_HAVE_restrict # define __lzo_HAVE_restrict 1 # endif #else # define __lzo_restrict /*empty*/ #endif #if !defined(__lzo_alignof) #if (LZO_CC_ARMCC || LZO_CC_CILLY || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || LZO_CC_PGI) # define __lzo_alignof(e) __alignof__(e) #elif (LZO_CC_GHS) && !defined(__cplusplus) # define __lzo_alignof(e) __alignof__(e) #elif (LZO_CC_IBMC >= 600) # define __lzo_alignof(e) (__lzo_gnuc_extension__ __alignof__(e)) #elif (LZO_CC_INTELC && (__INTEL_COMPILER >= 700)) # define __lzo_alignof(e) __alignof__(e) #elif (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_alignof(e) __alignof(e) #elif (LZO_CC_SUNPROC >= 0x5100) # define __lzo_alignof(e) __alignof__(e) #endif #endif #if defined(__lzo_alignof) # ifndef __lzo_HAVE_alignof # define __lzo_HAVE_alignof 1 # endif #endif #if !defined(__lzo_struct_packed) #if (LZO_CC_CLANG && (LZO_CC_CLANG < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020700ul)) #elif (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_PCC && (LZO_CC_PCC < 0x010100ul)) #elif (LZO_CC_SUNPROC && (LZO_CC_SUNPROC < 0x5110)) && !defined(__cplusplus) #elif (LZO_CC_GNUC >= 0x030400ul) && !(LZO_CC_PCC_GNUC) && (LZO_ARCH_AMD64 || LZO_ARCH_I386) # define __lzo_struct_packed(s) struct s { # define __lzo_struct_packed_end() } __attribute__((__gcc_struct__,__packed__)); # define __lzo_struct_packed_ma_end() } __lzo_may_alias __attribute__((__gcc_struct__,__packed__)); #elif (LZO_CC_ARMCC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || (LZO_CC_PGI >= 0x0d0a00ul) || (LZO_CC_SUNPROC >= 0x5100)) # define __lzo_struct_packed(s) struct s { # define __lzo_struct_packed_end() } __attribute__((__packed__)); # define __lzo_struct_packed_ma_end() } __lzo_may_alias __attribute__((__packed__)); #elif (LZO_CC_IBMC >= 700) # define __lzo_struct_packed(s) __lzo_gnuc_extension__ struct s { # define __lzo_struct_packed_end() } __attribute__((__packed__)); # define __lzo_struct_packed_ma_end() } __lzo_may_alias __attribute__((__packed__)); #elif (LZO_CC_INTELC_MSC) || (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_struct_packed(s) __pragma(pack(push,1)) struct s { # define __lzo_struct_packed_end() } __pragma(pack(pop)); #elif (LZO_CC_WATCOMC && (__WATCOMC__ >= 900)) # define __lzo_struct_packed(s) _Packed struct s { # define __lzo_struct_packed_end() }; #endif #endif #if defined(__lzo_struct_packed) && !defined(__lzo_struct_packed_ma) # define __lzo_struct_packed_ma(s) __lzo_struct_packed(s) #endif #if defined(__lzo_struct_packed_end) && !defined(__lzo_struct_packed_ma_end) # define __lzo_struct_packed_ma_end() __lzo_struct_packed_end() #endif #if !defined(__lzo_byte_struct) #if defined(__lzo_struct_packed) # define __lzo_byte_struct(s,n) __lzo_struct_packed(s) unsigned char a[n]; __lzo_struct_packed_end() # define __lzo_byte_struct_ma(s,n) __lzo_struct_packed_ma(s) unsigned char a[n]; __lzo_struct_packed_ma_end() #elif (LZO_CC_CILLY || LZO_CC_CLANG || LZO_CC_PGI || (LZO_CC_SUNPROC >= 0x5100)) # define __lzo_byte_struct(s,n) struct s { unsigned char a[n]; } __attribute__((__packed__)); # define __lzo_byte_struct_ma(s,n) struct s { unsigned char a[n]; } __lzo_may_alias __attribute__((__packed__)); #endif #endif #if defined(__lzo_byte_struct) && !defined(__lzo_byte_struct_ma) # define __lzo_byte_struct_ma(s,n) __lzo_byte_struct(s,n) #endif #if !defined(__lzo_struct_align16) && (__lzo_HAVE_alignof) #if (LZO_CC_GNUC && (LZO_CC_GNUC < 0x030000ul)) #elif (LZO_CC_CLANG && (LZO_CC_CLANG < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_CILLY || LZO_CC_PCC) #elif (LZO_CC_INTELC_MSC) || (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_struct_align16(s) struct __declspec(align(16)) s { # define __lzo_struct_align16_end() }; # define __lzo_struct_align32(s) struct __declspec(align(32)) s { # define __lzo_struct_align32_end() }; # define __lzo_struct_align64(s) struct __declspec(align(64)) s { # define __lzo_struct_align64_end() }; #elif (LZO_CC_ARMCC || LZO_CC_CLANG || LZO_CC_GNUC || (LZO_CC_IBMC >= 700) || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_struct_align16(s) struct s { # define __lzo_struct_align16_end() } __attribute__((__aligned__(16))); # define __lzo_struct_align32(s) struct s { # define __lzo_struct_align32_end() } __attribute__((__aligned__(32))); # define __lzo_struct_align64(s) struct s { # define __lzo_struct_align64_end() } __attribute__((__aligned__(64))); #endif #endif #if !defined(__lzo_union_um) #if (LZO_CC_CLANG && (LZO_CC_CLANG < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020700ul)) #elif (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER < 810)) #elif (LZO_CC_PCC && (LZO_CC_PCC < 0x010100ul)) #elif (LZO_CC_SUNPROC && (LZO_CC_SUNPROC < 0x5110)) && !defined(__cplusplus) #elif (LZO_CC_ARMCC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || (LZO_CC_PGI >= 0x0d0a00ul) || (LZO_CC_SUNPROC >= 0x5100)) # define __lzo_union_am(s) union s { # define __lzo_union_am_end() } __lzo_may_alias; # define __lzo_union_um(s) union s { # define __lzo_union_um_end() } __lzo_may_alias __attribute__((__packed__)); #elif (LZO_CC_IBMC >= 700) # define __lzo_union_am(s) __lzo_gnuc_extension__ union s { # define __lzo_union_am_end() } __lzo_may_alias; # define __lzo_union_um(s) __lzo_gnuc_extension__ union s { # define __lzo_union_um_end() } __lzo_may_alias __attribute__((__packed__)); #elif (LZO_CC_INTELC_MSC) || (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_union_um(s) __pragma(pack(push,1)) union s { # define __lzo_union_um_end() } __pragma(pack(pop)); #elif (LZO_CC_WATCOMC && (__WATCOMC__ >= 900)) # define __lzo_union_um(s) _Packed union s { # define __lzo_union_um_end() }; #endif #endif #if !defined(__lzo_union_am) # define __lzo_union_am(s) union s { # define __lzo_union_am_end() }; #endif #if !defined(__lzo_constructor) #if (LZO_CC_GNUC >= 0x030400ul) # define __lzo_constructor __attribute__((__constructor__,__used__)) #elif (LZO_CC_GNUC >= 0x020700ul) # define __lzo_constructor __attribute__((__constructor__)) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 800)) # define __lzo_constructor __attribute__((__constructor__,__used__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_constructor __attribute__((__constructor__)) #endif #endif #if defined(__lzo_constructor) # ifndef __lzo_HAVE_constructor # define __lzo_HAVE_constructor 1 # endif #endif #if !defined(__lzo_destructor) #if (LZO_CC_GNUC >= 0x030400ul) # define __lzo_destructor __attribute__((__destructor__,__used__)) #elif (LZO_CC_GNUC >= 0x020700ul) # define __lzo_destructor __attribute__((__destructor__)) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 800)) # define __lzo_destructor __attribute__((__destructor__,__used__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_destructor __attribute__((__destructor__)) #endif #endif #if defined(__lzo_destructor) # ifndef __lzo_HAVE_destructor # define __lzo_HAVE_destructor 1 # endif #endif #if (__lzo_HAVE_destructor) && !(__lzo_HAVE_constructor) # error "unexpected configuration - check your compiler defines" #endif #if !defined(__lzo_likely) && !defined(__lzo_unlikely) #if (LZO_CC_GNUC >= 0x030200ul) # define __lzo_likely(e) (__builtin_expect(!!(e),1)) # define __lzo_unlikely(e) (__builtin_expect(!!(e),0)) #elif (LZO_CC_IBMC >= 1010) # define __lzo_likely(e) (__builtin_expect(!!(e),1)) # define __lzo_unlikely(e) (__builtin_expect(!!(e),0)) #elif (LZO_CC_INTELC && (__INTEL_COMPILER >= 800)) # define __lzo_likely(e) (__builtin_expect(!!(e),1)) # define __lzo_unlikely(e) (__builtin_expect(!!(e),0)) #elif (LZO_CC_CLANG && LZO_CC_CLANG_C2) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_likely(e) (__builtin_expect(!!(e),1)) # define __lzo_unlikely(e) (__builtin_expect(!!(e),0)) #endif #endif #if defined(__lzo_likely) # ifndef __lzo_HAVE_likely # define __lzo_HAVE_likely 1 # endif #else # define __lzo_likely(e) (e) #endif #if defined(__lzo_very_likely) # ifndef __lzo_HAVE_very_likely # define __lzo_HAVE_very_likely 1 # endif #else # define __lzo_very_likely(e) __lzo_likely(e) #endif #if defined(__lzo_unlikely) # ifndef __lzo_HAVE_unlikely # define __lzo_HAVE_unlikely 1 # endif #else # define __lzo_unlikely(e) (e) #endif #if defined(__lzo_very_unlikely) # ifndef __lzo_HAVE_very_unlikely # define __lzo_HAVE_very_unlikely 1 # endif #else # define __lzo_very_unlikely(e) __lzo_unlikely(e) #endif #if !defined(__lzo_loop_forever) # if (LZO_CC_IBMC) # define __lzo_loop_forever() LZO_BLOCK_BEGIN for (;;) { ; } LZO_BLOCK_END # else # define __lzo_loop_forever() do { ; } while __lzo_cte(1) # endif #endif #if !defined(__lzo_unreachable) #if (LZO_CC_CLANG && (LZO_CC_CLANG >= 0x020800ul)) && lzo_has_builtin(__builtin_unreachable) # define __lzo_unreachable() __builtin_unreachable(); #elif (LZO_CC_GNUC >= 0x040500ul) # define __lzo_unreachable() __builtin_unreachable(); #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 1300)) && 1 # define __lzo_unreachable() __builtin_unreachable(); #endif #endif #if defined(__lzo_unreachable) # ifndef __lzo_HAVE_unreachable # define __lzo_HAVE_unreachable 1 # endif #else # if 0 # define __lzo_unreachable() ((void)0); # else # define __lzo_unreachable() __lzo_loop_forever(); # endif #endif #if !defined(lzo_unused_funcs_impl) # if 1 && (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || (LZO_CC_GNUC >= 0x020700ul) || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || LZO_CC_PGI) # define lzo_unused_funcs_impl(r,f) static r __attribute__((__unused__)) f # elif 1 && (LZO_CC_BORLANDC || LZO_CC_GNUC) # define lzo_unused_funcs_impl(r,f) static r f # else # define lzo_unused_funcs_impl(r,f) __lzo_static_forceinline r f # endif #endif #ifndef __LZO_CTA_NAME #if (LZO_CFG_USE_COUNTER) # define __LZO_CTA_NAME(a) LZO_PP_ECONCAT2(a,__COUNTER__) #else # define __LZO_CTA_NAME(a) LZO_PP_ECONCAT2(a,__LINE__) #endif #endif #if !defined(LZO_COMPILE_TIME_ASSERT_HEADER) # if (LZO_CC_AZTECC || LZO_CC_ZORTECHC) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN extern int __LZO_CTA_NAME(lzo_cta__)[1-!(e)]; LZO_EXTERN_C_END # elif (LZO_CC_DMC || LZO_CC_SYMANTECC) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN extern int __LZO_CTA_NAME(lzo_cta__)[1u-2*!(e)]; LZO_EXTERN_C_END # elif (LZO_CC_TURBOC && (__TURBOC__ == 0x0295)) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN extern int __LZO_CTA_NAME(lzo_cta__)[1-!(e)]; LZO_EXTERN_C_END # elif (LZO_CC_CLANG && (LZO_CC_CLANG < 0x020900ul)) && defined(__cplusplus) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN int __LZO_CTA_NAME(lzo_cta_f__)(int [1-2*!(e)]); LZO_EXTERN_C_END # elif (LZO_CC_GNUC) && defined(__CHECKER__) && defined(__SPARSE_CHECKER__) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN enum {__LZO_CTA_NAME(lzo_cta_e__)=1/!!(e)} __attribute__((__unused__)); LZO_EXTERN_C_END # else # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN extern int __LZO_CTA_NAME(lzo_cta__)[1-2*!(e)]; LZO_EXTERN_C_END # endif #endif #if !defined(LZO_COMPILE_TIME_ASSERT) # if (LZO_CC_AZTECC) # define LZO_COMPILE_TIME_ASSERT(e) {typedef int __LZO_CTA_NAME(lzo_cta_t__)[1-!(e)];} # elif (LZO_CC_CLANG && (LZO_CC_CLANG >= 0x030000ul)) # define LZO_COMPILE_TIME_ASSERT(e) {typedef int __LZO_CTA_NAME(lzo_cta_t__)[1-2*!(e)] __attribute__((__unused__));} # elif (LZO_CC_DMC || LZO_CC_PACIFICC || LZO_CC_SYMANTECC || LZO_CC_ZORTECHC) # define LZO_COMPILE_TIME_ASSERT(e) switch(0) case 1:case !(e):break; # elif (LZO_CC_GNUC) && defined(__CHECKER__) && defined(__SPARSE_CHECKER__) # define LZO_COMPILE_TIME_ASSERT(e) {(void) (0/!!(e));} # elif (LZO_CC_GNUC >= 0x040700ul) && (LZO_CFG_USE_COUNTER) && defined(__cplusplus) # define LZO_COMPILE_TIME_ASSERT(e) {enum {__LZO_CTA_NAME(lzo_cta_e__)=1/!!(e)} __attribute__((__unused__));} # elif (LZO_CC_GNUC >= 0x040700ul) # define LZO_COMPILE_TIME_ASSERT(e) {typedef int __LZO_CTA_NAME(lzo_cta_t__)[1-2*!(e)] __attribute__((__unused__));} # elif (LZO_CC_MSC && (_MSC_VER < 900)) # define LZO_COMPILE_TIME_ASSERT(e) switch(0) case 1:case !(e):break; # elif (LZO_CC_TURBOC && (__TURBOC__ == 0x0295)) # define LZO_COMPILE_TIME_ASSERT(e) switch(0) case 1:case !(e):break; # else # define LZO_COMPILE_TIME_ASSERT(e) {typedef int __LZO_CTA_NAME(lzo_cta_t__)[1-2*!(e)];} # endif #endif #if (LZO_LANG_ASSEMBLER) # undef LZO_COMPILE_TIME_ASSERT_HEADER # define LZO_COMPILE_TIME_ASSERT_HEADER(e) /*empty*/ #else LZO_COMPILE_TIME_ASSERT_HEADER(1 == 1) #if defined(__cplusplus) extern "C" { LZO_COMPILE_TIME_ASSERT_HEADER(2 == 2) } #endif LZO_COMPILE_TIME_ASSERT_HEADER(3 == 3) #endif #if (LZO_ARCH_I086 || LZO_ARCH_I386) && (LZO_OS_DOS16 || LZO_OS_DOS32 || LZO_OS_OS2 || LZO_OS_OS216 || LZO_OS_WIN16 || LZO_OS_WIN32 || LZO_OS_WIN64) # if (LZO_CC_GNUC || LZO_CC_HIGHC || LZO_CC_NDPC || LZO_CC_PACIFICC) # elif (LZO_CC_DMC || LZO_CC_SYMANTECC || LZO_CC_ZORTECHC) # define __lzo_cdecl __cdecl # define __lzo_cdecl_atexit /*empty*/ # define __lzo_cdecl_main __cdecl # if (LZO_OS_OS2 && (LZO_CC_DMC || LZO_CC_SYMANTECC)) # define __lzo_cdecl_qsort __pascal # elif (LZO_OS_OS2 && (LZO_CC_ZORTECHC)) # define __lzo_cdecl_qsort _stdcall # else # define __lzo_cdecl_qsort __cdecl # endif # elif (LZO_CC_WATCOMC) # define __lzo_cdecl __cdecl # else # define __lzo_cdecl __cdecl # define __lzo_cdecl_atexit __cdecl # define __lzo_cdecl_main __cdecl # define __lzo_cdecl_qsort __cdecl # endif # if (LZO_CC_GNUC || LZO_CC_HIGHC || LZO_CC_NDPC || LZO_CC_PACIFICC || LZO_CC_WATCOMC) # elif (LZO_OS_OS2 && (LZO_CC_DMC || LZO_CC_SYMANTECC)) # define __lzo_cdecl_sighandler __pascal # elif (LZO_OS_OS2 && (LZO_CC_ZORTECHC)) # define __lzo_cdecl_sighandler _stdcall # elif (LZO_CC_MSC && (_MSC_VER >= 1400)) && defined(_M_CEE_PURE) # define __lzo_cdecl_sighandler __clrcall # elif (LZO_CC_MSC && (_MSC_VER >= 600 && _MSC_VER < 700)) # if defined(_DLL) # define __lzo_cdecl_sighandler _far _cdecl _loadds # elif defined(_MT) # define __lzo_cdecl_sighandler _far _cdecl # else # define __lzo_cdecl_sighandler _cdecl # endif # else # define __lzo_cdecl_sighandler __cdecl # endif #elif (LZO_ARCH_I386) && (LZO_CC_WATCOMC) # define __lzo_cdecl __cdecl #elif (LZO_ARCH_M68K && LZO_OS_TOS && (LZO_CC_PUREC || LZO_CC_TURBOC)) # define __lzo_cdecl cdecl #endif #if !defined(__lzo_cdecl) # define __lzo_cdecl /*empty*/ #endif #if !defined(__lzo_cdecl_atexit) # define __lzo_cdecl_atexit /*empty*/ #endif #if !defined(__lzo_cdecl_main) # define __lzo_cdecl_main /*empty*/ #endif #if !defined(__lzo_cdecl_qsort) # define __lzo_cdecl_qsort /*empty*/ #endif #if !defined(__lzo_cdecl_sighandler) # define __lzo_cdecl_sighandler /*empty*/ #endif #if !defined(__lzo_cdecl_va) # define __lzo_cdecl_va __lzo_cdecl #endif #if !(LZO_CFG_NO_WINDOWS_H) #if !defined(LZO_HAVE_WINDOWS_H) #if (LZO_OS_CYGWIN || (LZO_OS_EMX && defined(__RSXNT__)) || LZO_OS_WIN32 || LZO_OS_WIN64) # if (LZO_CC_WATCOMC && (__WATCOMC__ < 1000)) # elif ((LZO_OS_WIN32 && defined(__PW32__)) && (LZO_CC_GNUC && (LZO_CC_GNUC < 0x030000ul))) # elif ((LZO_OS_CYGWIN || defined(__MINGW32__)) && (LZO_CC_GNUC && (LZO_CC_GNUC < 0x025f00ul))) # else # define LZO_HAVE_WINDOWS_H 1 # endif #endif #endif #endif #define LZO_SIZEOF_CHAR 1 #ifndef LZO_SIZEOF_SHORT #if defined(SIZEOF_SHORT) # define LZO_SIZEOF_SHORT (SIZEOF_SHORT) #elif defined(__SIZEOF_SHORT__) # define LZO_SIZEOF_SHORT (__SIZEOF_SHORT__) #endif #endif #ifndef LZO_SIZEOF_INT #if defined(SIZEOF_INT) # define LZO_SIZEOF_INT (SIZEOF_INT) #elif defined(__SIZEOF_INT__) # define LZO_SIZEOF_INT (__SIZEOF_INT__) #endif #endif #ifndef LZO_SIZEOF_LONG #if defined(SIZEOF_LONG) # define LZO_SIZEOF_LONG (SIZEOF_LONG) #elif defined(__SIZEOF_LONG__) # define LZO_SIZEOF_LONG (__SIZEOF_LONG__) #endif #endif #ifndef LZO_SIZEOF_LONG_LONG #if defined(SIZEOF_LONG_LONG) # define LZO_SIZEOF_LONG_LONG (SIZEOF_LONG_LONG) #elif defined(__SIZEOF_LONG_LONG__) # define LZO_SIZEOF_LONG_LONG (__SIZEOF_LONG_LONG__) #endif #endif #ifndef LZO_SIZEOF___INT16 #if defined(SIZEOF___INT16) # define LZO_SIZEOF___INT16 (SIZEOF___INT16) #endif #endif #ifndef LZO_SIZEOF___INT32 #if defined(SIZEOF___INT32) # define LZO_SIZEOF___INT32 (SIZEOF___INT32) #endif #endif #ifndef LZO_SIZEOF___INT64 #if defined(SIZEOF___INT64) # define LZO_SIZEOF___INT64 (SIZEOF___INT64) #endif #endif #ifndef LZO_SIZEOF_VOID_P #if defined(SIZEOF_VOID_P) # define LZO_SIZEOF_VOID_P (SIZEOF_VOID_P) #elif defined(__SIZEOF_POINTER__) # define LZO_SIZEOF_VOID_P (__SIZEOF_POINTER__) #endif #endif #ifndef LZO_SIZEOF_SIZE_T #if defined(SIZEOF_SIZE_T) # define LZO_SIZEOF_SIZE_T (SIZEOF_SIZE_T) #elif defined(__SIZEOF_SIZE_T__) # define LZO_SIZEOF_SIZE_T (__SIZEOF_SIZE_T__) #endif #endif #ifndef LZO_SIZEOF_PTRDIFF_T #if defined(SIZEOF_PTRDIFF_T) # define LZO_SIZEOF_PTRDIFF_T (SIZEOF_PTRDIFF_T) #elif defined(__SIZEOF_PTRDIFF_T__) # define LZO_SIZEOF_PTRDIFF_T (__SIZEOF_PTRDIFF_T__) #endif #endif #define __LZO_LSR(x,b) (((x)+0ul) >> (b)) #if !defined(LZO_SIZEOF_SHORT) # if (LZO_ARCH_CRAY_PVP) # define LZO_SIZEOF_SHORT 8 # elif (USHRT_MAX == LZO_0xffffL) # define LZO_SIZEOF_SHORT 2 # elif (__LZO_LSR(USHRT_MAX,7) == 1) # define LZO_SIZEOF_SHORT 1 # elif (__LZO_LSR(USHRT_MAX,15) == 1) # define LZO_SIZEOF_SHORT 2 # elif (__LZO_LSR(USHRT_MAX,31) == 1) # define LZO_SIZEOF_SHORT 4 # elif (__LZO_LSR(USHRT_MAX,63) == 1) # define LZO_SIZEOF_SHORT 8 # elif (__LZO_LSR(USHRT_MAX,127) == 1) # define LZO_SIZEOF_SHORT 16 # else # error "LZO_SIZEOF_SHORT" # endif #endif LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_SHORT == sizeof(short)) #if !defined(LZO_SIZEOF_INT) # if (LZO_ARCH_CRAY_PVP) # define LZO_SIZEOF_INT 8 # elif (UINT_MAX == LZO_0xffffL) # define LZO_SIZEOF_INT 2 # elif (UINT_MAX == LZO_0xffffffffL) # define LZO_SIZEOF_INT 4 # elif (__LZO_LSR(UINT_MAX,7) == 1) # define LZO_SIZEOF_INT 1 # elif (__LZO_LSR(UINT_MAX,15) == 1) # define LZO_SIZEOF_INT 2 # elif (__LZO_LSR(UINT_MAX,31) == 1) # define LZO_SIZEOF_INT 4 # elif (__LZO_LSR(UINT_MAX,63) == 1) # define LZO_SIZEOF_INT 8 # elif (__LZO_LSR(UINT_MAX,127) == 1) # define LZO_SIZEOF_INT 16 # else # error "LZO_SIZEOF_INT" # endif #endif LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_INT == sizeof(int)) #if !defined(LZO_SIZEOF_LONG) # if (ULONG_MAX == LZO_0xffffffffL) # define LZO_SIZEOF_LONG 4 # elif (__LZO_LSR(ULONG_MAX,7) == 1) # define LZO_SIZEOF_LONG 1 # elif (__LZO_LSR(ULONG_MAX,15) == 1) # define LZO_SIZEOF_LONG 2 # elif (__LZO_LSR(ULONG_MAX,31) == 1) # define LZO_SIZEOF_LONG 4 # elif (__LZO_LSR(ULONG_MAX,39) == 1) # define LZO_SIZEOF_LONG 5 # elif (__LZO_LSR(ULONG_MAX,63) == 1) # define LZO_SIZEOF_LONG 8 # elif (__LZO_LSR(ULONG_MAX,127) == 1) # define LZO_SIZEOF_LONG 16 # else # error "LZO_SIZEOF_LONG" # endif #endif LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_LONG == sizeof(long)) #if !defined(LZO_SIZEOF_LONG_LONG) && !defined(LZO_SIZEOF___INT64) #if (LZO_SIZEOF_LONG > 0 && LZO_SIZEOF_LONG < 8) # if defined(__LONG_MAX__) && defined(__LONG_LONG_MAX__) # if (LZO_CC_GNUC >= 0x030300ul) # if ((__LONG_MAX__-0) == (__LONG_LONG_MAX__-0)) # define LZO_SIZEOF_LONG_LONG LZO_SIZEOF_LONG # elif (__LZO_LSR(__LONG_LONG_MAX__,30) == 1) # define LZO_SIZEOF_LONG_LONG 4 # endif # endif # endif #endif #endif #if !defined(LZO_SIZEOF_LONG_LONG) && !defined(LZO_SIZEOF___INT64) #if (LZO_SIZEOF_LONG > 0 && LZO_SIZEOF_LONG < 8) #if (LZO_ARCH_I086 && LZO_CC_DMC) #elif (LZO_CC_CILLY) && defined(__GNUC__) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_SIZEOF_LONG_LONG 8 #elif ((LZO_OS_WIN32 || LZO_OS_WIN64 || defined(_WIN32)) && LZO_CC_MSC && (_MSC_VER >= 1400)) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_OS_WIN64 || defined(_WIN64)) # define LZO_SIZEOF___INT64 8 #elif (LZO_ARCH_I386 && (LZO_CC_DMC)) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_ARCH_I386 && (LZO_CC_SYMANTECC && (__SC__ >= 0x700))) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_ARCH_I386 && (LZO_CC_INTELC && defined(__linux__))) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_ARCH_I386 && (LZO_CC_MWERKS || LZO_CC_PELLESC || LZO_CC_PGI || LZO_CC_SUNPROC)) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_ARCH_I386 && (LZO_CC_INTELC || LZO_CC_MSC)) # define LZO_SIZEOF___INT64 8 #elif ((LZO_OS_WIN32 || defined(_WIN32)) && (LZO_CC_MSC)) # define LZO_SIZEOF___INT64 8 #elif (LZO_ARCH_I386 && (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0520))) # define LZO_SIZEOF___INT64 8 #elif (LZO_ARCH_I386 && (LZO_CC_WATCOMC && (__WATCOMC__ >= 1100))) # define LZO_SIZEOF___INT64 8 #elif (LZO_CC_GHS && defined(__LLONG_BIT) && ((__LLONG_BIT-0) == 64)) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_CC_WATCOMC && defined(_INTEGRAL_MAX_BITS) && ((_INTEGRAL_MAX_BITS-0) == 64)) # define LZO_SIZEOF___INT64 8 #elif (LZO_OS_OS400 || defined(__OS400__)) && defined(__LLP64_IFC__) # define LZO_SIZEOF_LONG_LONG 8 #elif (defined(__vms) || defined(__VMS)) && ((__INITIAL_POINTER_SIZE-0) == 64) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_CC_SDCC) && (LZO_SIZEOF_INT == 2) #elif 1 && defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) # define LZO_SIZEOF_LONG_LONG 8 #endif #endif #endif #if defined(__cplusplus) && (LZO_CC_GNUC) # if (LZO_CC_GNUC < 0x020800ul) # undef LZO_SIZEOF_LONG_LONG # endif #endif #if (LZO_CFG_NO_LONG_LONG) # undef LZO_SIZEOF_LONG_LONG #elif defined(__NO_LONG_LONG) # undef LZO_SIZEOF_LONG_LONG #elif defined(_NO_LONGLONG) # undef LZO_SIZEOF_LONG_LONG #endif #if !defined(LZO_WORDSIZE) #if (LZO_ARCH_ALPHA) # define LZO_WORDSIZE 8 #elif (LZO_ARCH_AMD64) # define LZO_WORDSIZE 8 #elif (LZO_ARCH_ARM64) # define LZO_WORDSIZE 8 #elif (LZO_ARCH_AVR) # define LZO_WORDSIZE 1 #elif (LZO_ARCH_H8300) # if defined(__H8300H__) || defined(__H8300S__) || defined(__H8300SX__) # define LZO_WORDSIZE 4 # else # define LZO_WORDSIZE 2 # endif #elif (LZO_ARCH_I086) # define LZO_WORDSIZE 2 #elif (LZO_ARCH_IA64) # define LZO_WORDSIZE 8 #elif (LZO_ARCH_M16C) # define LZO_WORDSIZE 2 #elif (LZO_ARCH_SPU) # define LZO_WORDSIZE 4 #elif (LZO_ARCH_Z80) # define LZO_WORDSIZE 1 #elif (LZO_SIZEOF_LONG == 8) && ((defined(__mips__) && defined(__R5900__)) || defined(__MIPS_PSX2__)) # define LZO_WORDSIZE 8 #elif (LZO_OS_OS400 || defined(__OS400__)) # define LZO_WORDSIZE 8 #elif (defined(__vms) || defined(__VMS)) && (__INITIAL_POINTER_SIZE+0 == 64) # define LZO_WORDSIZE 8 #endif #endif #if !defined(LZO_SIZEOF_VOID_P) #if defined(__ILP32__) || defined(__ILP32) || defined(_ILP32) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(int) == 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long) == 4) # define LZO_SIZEOF_VOID_P 4 #elif defined(__ILP64__) || defined(__ILP64) || defined(_ILP64) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(int) == 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long) == 8) # define LZO_SIZEOF_VOID_P 8 #elif defined(__LLP64__) || defined(__LLP64) || defined(_LLP64) || defined(_WIN64) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long) == 4) # define LZO_SIZEOF_VOID_P 8 #elif defined(__LP64__) || defined(__LP64) || defined(_LP64) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long) == 8) # define LZO_SIZEOF_VOID_P 8 #elif (LZO_ARCH_AVR) # define LZO_SIZEOF_VOID_P 2 #elif (LZO_ARCH_C166 || LZO_ARCH_MCS51 || LZO_ARCH_MCS251 || LZO_ARCH_MSP430) # define LZO_SIZEOF_VOID_P 2 #elif (LZO_ARCH_H8300) # if defined(__H8300H__) || defined(__H8300S__) || defined(__H8300SX__) LZO_COMPILE_TIME_ASSERT_HEADER(LZO_WORDSIZE == 4) # if defined(__NORMAL_MODE__) # define LZO_SIZEOF_VOID_P 2 # else # define LZO_SIZEOF_VOID_P 4 # endif # else LZO_COMPILE_TIME_ASSERT_HEADER(LZO_WORDSIZE == 2) # define LZO_SIZEOF_VOID_P 2 # endif # if (LZO_CC_GNUC && (LZO_CC_GNUC < 0x040000ul)) && (LZO_SIZEOF_INT == 4) # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_INT # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_INT # endif #elif (LZO_ARCH_I086) # if (LZO_MM_TINY || LZO_MM_SMALL || LZO_MM_MEDIUM) # define LZO_SIZEOF_VOID_P 2 # elif (LZO_MM_COMPACT || LZO_MM_LARGE || LZO_MM_HUGE) # define LZO_SIZEOF_VOID_P 4 # else # error "invalid LZO_ARCH_I086 memory model" # endif #elif (LZO_ARCH_M16C) # if defined(__m32c_cpu__) || defined(__m32cm_cpu__) # define LZO_SIZEOF_VOID_P 4 # else # define LZO_SIZEOF_VOID_P 2 # endif #elif (LZO_ARCH_SPU) # define LZO_SIZEOF_VOID_P 4 #elif (LZO_ARCH_Z80) # define LZO_SIZEOF_VOID_P 2 #elif (LZO_SIZEOF_LONG == 8) && ((defined(__mips__) && defined(__R5900__)) || defined(__MIPS_PSX2__)) # define LZO_SIZEOF_VOID_P 4 #elif (LZO_OS_OS400 || defined(__OS400__)) # if defined(__LLP64_IFC__) # define LZO_SIZEOF_VOID_P 8 # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_LONG # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_LONG # else # define LZO_SIZEOF_VOID_P 16 # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_LONG # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_LONG # endif #elif (defined(__vms) || defined(__VMS)) && (__INITIAL_POINTER_SIZE+0 == 64) # define LZO_SIZEOF_VOID_P 8 # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_LONG # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_LONG #endif #endif #if !defined(LZO_SIZEOF_VOID_P) # define LZO_SIZEOF_VOID_P LZO_SIZEOF_LONG #endif LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_VOID_P == sizeof(void *)) #if !defined(LZO_SIZEOF_SIZE_T) #if (LZO_ARCH_I086 || LZO_ARCH_M16C) # define LZO_SIZEOF_SIZE_T 2 #endif #endif #if !defined(LZO_SIZEOF_SIZE_T) # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_VOID_P #endif #if defined(offsetof) LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_SIZE_T == sizeof(size_t)) #endif #if !defined(LZO_SIZEOF_PTRDIFF_T) #if (LZO_ARCH_I086) # if (LZO_MM_TINY || LZO_MM_SMALL || LZO_MM_MEDIUM || LZO_MM_HUGE) # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_VOID_P # elif (LZO_MM_COMPACT || LZO_MM_LARGE) # if (LZO_CC_BORLANDC || LZO_CC_TURBOC) # define LZO_SIZEOF_PTRDIFF_T 4 # else # define LZO_SIZEOF_PTRDIFF_T 2 # endif # else # error "invalid LZO_ARCH_I086 memory model" # endif #endif #endif #if !defined(LZO_SIZEOF_PTRDIFF_T) # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_SIZE_T #endif #if defined(offsetof) LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_PTRDIFF_T == sizeof(ptrdiff_t)) #endif #if !defined(LZO_WORDSIZE) # define LZO_WORDSIZE LZO_SIZEOF_VOID_P #endif #if (LZO_ABI_NEUTRAL_ENDIAN) # undef LZO_ABI_BIG_ENDIAN # undef LZO_ABI_LITTLE_ENDIAN #elif !(LZO_ABI_BIG_ENDIAN) && !(LZO_ABI_LITTLE_ENDIAN) #if (LZO_ARCH_ALPHA) && (LZO_ARCH_CRAY_MPP) # define LZO_ABI_BIG_ENDIAN 1 #elif (LZO_ARCH_IA64) && (LZO_OS_POSIX_LINUX || LZO_OS_WIN64) # define LZO_ABI_LITTLE_ENDIAN 1 #elif (LZO_ARCH_ALPHA || LZO_ARCH_AMD64 || LZO_ARCH_BLACKFIN || LZO_ARCH_CRIS || LZO_ARCH_I086 || LZO_ARCH_I386 || LZO_ARCH_MSP430 || LZO_ARCH_RISCV) # define LZO_ABI_LITTLE_ENDIAN 1 #elif (LZO_ARCH_AVR32 || LZO_ARCH_M68K || LZO_ARCH_S390 || LZO_ARCH_SPU) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && defined(__IAR_SYSTEMS_ICC__) && defined(__LITTLE_ENDIAN__) # if (__LITTLE_ENDIAN__ == 1) # define LZO_ABI_LITTLE_ENDIAN 1 # else # define LZO_ABI_BIG_ENDIAN 1 # endif #elif 1 && defined(__BIG_ENDIAN__) && !defined(__LITTLE_ENDIAN__) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && defined(__LITTLE_ENDIAN__) && !defined(__BIG_ENDIAN__) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM) && defined(__ARM_BIG_ENDIAN) && ((__ARM_BIG_ENDIAN)+0) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM) && defined(__ARMEB__) && !defined(__ARMEL__) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM) && defined(__ARMEL__) && !defined(__ARMEB__) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM) && defined(_MSC_VER) && defined(_WIN32) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM && LZO_CC_ARMCC_ARMCC) # if defined(__BIG_ENDIAN) && defined(__LITTLE_ENDIAN) # error "unexpected configuration - check your compiler defines" # elif defined(__BIG_ENDIAN) # define LZO_ABI_BIG_ENDIAN 1 # else # define LZO_ABI_LITTLE_ENDIAN 1 # endif # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM64) && defined(__ARM_BIG_ENDIAN) && ((__ARM_BIG_ENDIAN)+0) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM64) && defined(__AARCH64EB__) && !defined(__AARCH64EL__) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM64) && defined(__AARCH64EL__) && !defined(__AARCH64EB__) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM64) && defined(_MSC_VER) && defined(_WIN32) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_MIPS) && defined(__MIPSEB__) && !defined(__MIPSEL__) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_MIPS) && defined(__MIPSEL__) && !defined(__MIPSEB__) # define LZO_ABI_LITTLE_ENDIAN 1 #endif #endif #if (LZO_ABI_BIG_ENDIAN) && (LZO_ABI_LITTLE_ENDIAN) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ABI_BIG_ENDIAN) # define LZO_INFO_ABI_ENDIAN "be" #elif (LZO_ABI_LITTLE_ENDIAN) # define LZO_INFO_ABI_ENDIAN "le" #elif (LZO_ABI_NEUTRAL_ENDIAN) # define LZO_INFO_ABI_ENDIAN "neutral" #endif #if (LZO_SIZEOF_INT == 1 && LZO_SIZEOF_LONG == 2 && LZO_SIZEOF_VOID_P == 2) # define LZO_ABI_I8LP16 1 # define LZO_INFO_ABI_PM "i8lp16" #elif (LZO_SIZEOF_INT == 2 && LZO_SIZEOF_LONG == 2 && LZO_SIZEOF_VOID_P == 2) # define LZO_ABI_ILP16 1 # define LZO_INFO_ABI_PM "ilp16" #elif (LZO_SIZEOF_INT == 2 && LZO_SIZEOF_LONG == 4 && LZO_SIZEOF_VOID_P == 4) # define LZO_ABI_LP32 1 # define LZO_INFO_ABI_PM "lp32" #elif (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_LONG == 4 && LZO_SIZEOF_VOID_P == 4) # define LZO_ABI_ILP32 1 # define LZO_INFO_ABI_PM "ilp32" #elif (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_LONG == 4 && LZO_SIZEOF_VOID_P == 8 && LZO_SIZEOF_SIZE_T == 8) # define LZO_ABI_LLP64 1 # define LZO_INFO_ABI_PM "llp64" #elif (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_LONG == 8 && LZO_SIZEOF_VOID_P == 8) # define LZO_ABI_LP64 1 # define LZO_INFO_ABI_PM "lp64" #elif (LZO_SIZEOF_INT == 8 && LZO_SIZEOF_LONG == 8 && LZO_SIZEOF_VOID_P == 8) # define LZO_ABI_ILP64 1 # define LZO_INFO_ABI_PM "ilp64" #elif (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_LONG == 8 && LZO_SIZEOF_VOID_P == 4) # define LZO_ABI_IP32L64 1 # define LZO_INFO_ABI_PM "ip32l64" #endif #if (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_VOID_P == 4 && LZO_WORDSIZE == 8) # define LZO_ABI_IP32W64 1 # ifndef LZO_INFO_ABI_PM # define LZO_INFO_ABI_PM "ip32w64" # endif #endif #if 0 #elif !defined(__LZO_LIBC_OVERRIDE) #if (LZO_LIBC_NAKED) # define LZO_INFO_LIBC "naked" #elif (LZO_LIBC_FREESTANDING) # define LZO_INFO_LIBC "freestanding" #elif (LZO_LIBC_MOSTLY_FREESTANDING) # define LZO_INFO_LIBC "mfreestanding" #elif (LZO_LIBC_ISOC90) # define LZO_INFO_LIBC "isoc90" #elif (LZO_LIBC_ISOC99) # define LZO_INFO_LIBC "isoc99" #elif (LZO_CC_ARMCC_ARMCC) && defined(__ARMCLIB_VERSION) # define LZO_LIBC_ISOC90 1 # define LZO_INFO_LIBC "isoc90" #elif defined(__dietlibc__) # define LZO_LIBC_DIETLIBC 1 # define LZO_INFO_LIBC "dietlibc" #elif defined(_NEWLIB_VERSION) # define LZO_LIBC_NEWLIB 1 # define LZO_INFO_LIBC "newlib" #elif defined(__UCLIBC__) && defined(__UCLIBC_MAJOR__) && defined(__UCLIBC_MINOR__) # if defined(__UCLIBC_SUBLEVEL__) # define LZO_LIBC_UCLIBC (__UCLIBC_MAJOR__ * 0x10000L + (__UCLIBC_MINOR__-0) * 0x100 + (__UCLIBC_SUBLEVEL__-0)) # else # define LZO_LIBC_UCLIBC 0x00090bL # endif # define LZO_INFO_LIBC "uc" "libc" #elif defined(__GLIBC__) && defined(__GLIBC_MINOR__) # define LZO_LIBC_GLIBC (__GLIBC__ * 0x10000L + (__GLIBC_MINOR__-0) * 0x100) # define LZO_INFO_LIBC "glibc" #elif (LZO_CC_MWERKS) && defined(__MSL__) # define LZO_LIBC_MSL __MSL__ # define LZO_INFO_LIBC "msl" #elif 1 && defined(__IAR_SYSTEMS_ICC__) # define LZO_LIBC_ISOC90 1 # define LZO_INFO_LIBC "isoc90" #else # define LZO_LIBC_DEFAULT 1 # define LZO_INFO_LIBC "default" #endif #endif #if (LZO_ARCH_I386 && (LZO_OS_DOS32 || LZO_OS_WIN32) && (LZO_CC_DMC || LZO_CC_INTELC || LZO_CC_MSC || LZO_CC_PELLESC)) # define LZO_ASM_SYNTAX_MSC 1 #elif (LZO_OS_WIN64 && (LZO_CC_DMC || LZO_CC_INTELC || LZO_CC_MSC || LZO_CC_PELLESC)) #elif (LZO_ARCH_I386 && LZO_CC_GNUC && (LZO_CC_GNUC == 0x011f00ul)) #elif (LZO_ARCH_I386 && (LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_INTELC || LZO_CC_PATHSCALE)) # define LZO_ASM_SYNTAX_GNUC 1 #elif (LZO_ARCH_AMD64 && (LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_INTELC || LZO_CC_PATHSCALE)) # define LZO_ASM_SYNTAX_GNUC 1 #elif (LZO_CC_GNUC) # define LZO_ASM_SYNTAX_GNUC 1 #endif #if (LZO_ASM_SYNTAX_GNUC) #if (LZO_ARCH_I386 && LZO_CC_GNUC && (LZO_CC_GNUC < 0x020000ul)) # define __LZO_ASM_CLOBBER "ax" # define __LZO_ASM_CLOBBER_LIST_CC /*empty*/ # define __LZO_ASM_CLOBBER_LIST_CC_MEMORY /*empty*/ # define __LZO_ASM_CLOBBER_LIST_EMPTY /*empty*/ #elif (LZO_CC_INTELC && (__INTEL_COMPILER < 1000)) # define __LZO_ASM_CLOBBER "memory" # define __LZO_ASM_CLOBBER_LIST_CC /*empty*/ # define __LZO_ASM_CLOBBER_LIST_CC_MEMORY : "memory" # define __LZO_ASM_CLOBBER_LIST_EMPTY /*empty*/ #else # define __LZO_ASM_CLOBBER "cc", "memory" # define __LZO_ASM_CLOBBER_LIST_CC : "cc" # define __LZO_ASM_CLOBBER_LIST_CC_MEMORY : "cc", "memory" # define __LZO_ASM_CLOBBER_LIST_EMPTY /*empty*/ #endif #endif #if (LZO_ARCH_ALPHA) # define LZO_OPT_AVOID_UINT_INDEX 1 #elif (LZO_ARCH_AMD64) # define LZO_OPT_AVOID_INT_INDEX 1 # define LZO_OPT_AVOID_UINT_INDEX 1 # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif #elif (LZO_ARCH_ARM) # if defined(__ARM_FEATURE_UNALIGNED) # if ((__ARM_FEATURE_UNALIGNED)+0) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # endif # elif 1 && (LZO_ARCH_ARM_THUMB2) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # elif 1 && defined(__ARM_ARCH) && ((__ARM_ARCH)+0 >= 7) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # elif 1 && defined(__TARGET_ARCH_ARM) && ((__TARGET_ARCH_ARM)+0 >= 7) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # elif 1 && defined(__TARGET_ARCH_ARM) && ((__TARGET_ARCH_ARM)+0 >= 6) && (defined(__TARGET_PROFILE_A) || defined(__TARGET_PROFILE_R)) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # elif 1 && defined(_MSC_VER) && defined(_M_ARM) && ((_M_ARM)+0 >= 7) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # endif #elif (LZO_ARCH_ARM64) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif #elif (LZO_ARCH_CRIS) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif #elif (LZO_ARCH_I386) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif #elif (LZO_ARCH_IA64) # define LZO_OPT_AVOID_INT_INDEX 1 # define LZO_OPT_AVOID_UINT_INDEX 1 # define LZO_OPT_PREFER_POSTINC 1 #elif (LZO_ARCH_M68K) # define LZO_OPT_PREFER_POSTINC 1 # define LZO_OPT_PREFER_PREDEC 1 # if defined(__mc68020__) && !defined(__mcoldfire__) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # endif #elif (LZO_ARCH_MIPS) # define LZO_OPT_AVOID_UINT_INDEX 1 #elif (LZO_ARCH_POWERPC) # define LZO_OPT_PREFER_PREINC 1 # define LZO_OPT_PREFER_PREDEC 1 # if (LZO_ABI_BIG_ENDIAN) || (LZO_WORDSIZE == 8) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # if (LZO_WORDSIZE == 8) # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif # endif # endif #elif (LZO_ARCH_RISCV) # define LZO_OPT_AVOID_UINT_INDEX 1 # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # if (LZO_WORDSIZE == 8) # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif # endif #elif (LZO_ARCH_S390) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # if (LZO_WORDSIZE == 8) # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif # endif #elif (LZO_ARCH_SH) # define LZO_OPT_PREFER_POSTINC 1 # define LZO_OPT_PREFER_PREDEC 1 #endif #ifndef LZO_CFG_NO_INLINE_ASM #if (LZO_ABI_NEUTRAL_ENDIAN) || (LZO_ARCH_GENERIC) # define LZO_CFG_NO_INLINE_ASM 1 #elif (LZO_CC_LLVM) # define LZO_CFG_NO_INLINE_ASM 1 #endif #endif #if (LZO_CFG_NO_INLINE_ASM) # undef LZO_ASM_SYNTAX_MSC # undef LZO_ASM_SYNTAX_GNUC # undef __LZO_ASM_CLOBBER # undef __LZO_ASM_CLOBBER_LIST_CC # undef __LZO_ASM_CLOBBER_LIST_CC_MEMORY # undef __LZO_ASM_CLOBBER_LIST_EMPTY #endif #ifndef LZO_CFG_NO_UNALIGNED #if (LZO_ABI_NEUTRAL_ENDIAN) || (LZO_ARCH_GENERIC) # define LZO_CFG_NO_UNALIGNED 1 #endif #endif #if (LZO_CFG_NO_UNALIGNED) # undef LZO_OPT_UNALIGNED16 # undef LZO_OPT_UNALIGNED32 # undef LZO_OPT_UNALIGNED64 #endif #if defined(__LZO_INFOSTR_MM) #elif (LZO_MM_FLAT) && (defined(__LZO_INFOSTR_PM) || defined(LZO_INFO_ABI_PM)) # define __LZO_INFOSTR_MM "" #elif defined(LZO_INFO_MM) # define __LZO_INFOSTR_MM "." LZO_INFO_MM #else # define __LZO_INFOSTR_MM "" #endif #if defined(__LZO_INFOSTR_PM) #elif defined(LZO_INFO_ABI_PM) # define __LZO_INFOSTR_PM "." LZO_INFO_ABI_PM #else # define __LZO_INFOSTR_PM "" #endif #if defined(__LZO_INFOSTR_ENDIAN) #elif defined(LZO_INFO_ABI_ENDIAN) # define __LZO_INFOSTR_ENDIAN "." LZO_INFO_ABI_ENDIAN #else # define __LZO_INFOSTR_ENDIAN "" #endif #if defined(__LZO_INFOSTR_OSNAME) #elif defined(LZO_INFO_OS_CONSOLE) # define __LZO_INFOSTR_OSNAME LZO_INFO_OS "." LZO_INFO_OS_CONSOLE #elif defined(LZO_INFO_OS_POSIX) # define __LZO_INFOSTR_OSNAME LZO_INFO_OS "." LZO_INFO_OS_POSIX #else # define __LZO_INFOSTR_OSNAME LZO_INFO_OS #endif #if defined(__LZO_INFOSTR_LIBC) #elif defined(LZO_INFO_LIBC) # define __LZO_INFOSTR_LIBC "." LZO_INFO_LIBC #else # define __LZO_INFOSTR_LIBC "" #endif #if defined(__LZO_INFOSTR_CCVER) #elif defined(LZO_INFO_CCVER) # define __LZO_INFOSTR_CCVER " " LZO_INFO_CCVER #else # define __LZO_INFOSTR_CCVER "" #endif #define LZO_INFO_STRING \ LZO_INFO_ARCH __LZO_INFOSTR_MM __LZO_INFOSTR_PM __LZO_INFOSTR_ENDIAN \ " " __LZO_INFOSTR_OSNAME __LZO_INFOSTR_LIBC " " LZO_INFO_CC __LZO_INFOSTR_CCVER #if !(LZO_CFG_SKIP_LZO_TYPES) #if (!(LZO_SIZEOF_SHORT+0 > 0 && LZO_SIZEOF_INT+0 > 0 && LZO_SIZEOF_LONG+0 > 0)) # error "missing defines for sizes" #endif #if (!(LZO_SIZEOF_PTRDIFF_T+0 > 0 && LZO_SIZEOF_SIZE_T+0 > 0 && LZO_SIZEOF_VOID_P+0 > 0)) # error "missing defines for sizes" #endif #define LZO_TYPEOF_CHAR 1u #define LZO_TYPEOF_SHORT 2u #define LZO_TYPEOF_INT 3u #define LZO_TYPEOF_LONG 4u #define LZO_TYPEOF_LONG_LONG 5u #define LZO_TYPEOF___INT8 17u #define LZO_TYPEOF___INT16 18u #define LZO_TYPEOF___INT32 19u #define LZO_TYPEOF___INT64 20u #define LZO_TYPEOF___INT128 21u #define LZO_TYPEOF___INT256 22u #define LZO_TYPEOF___MODE_QI 33u #define LZO_TYPEOF___MODE_HI 34u #define LZO_TYPEOF___MODE_SI 35u #define LZO_TYPEOF___MODE_DI 36u #define LZO_TYPEOF___MODE_TI 37u #define LZO_TYPEOF_CHAR_P 129u #if !defined(lzo_llong_t) #if (LZO_SIZEOF_LONG_LONG+0 > 0) # if !(LZO_LANG_ASSEMBLER) __lzo_gnuc_extension__ typedef long long lzo_llong_t__; __lzo_gnuc_extension__ typedef unsigned long long lzo_ullong_t__; # endif # define lzo_llong_t lzo_llong_t__ # define lzo_ullong_t lzo_ullong_t__ #endif #endif #if !defined(lzo_int16e_t) #if (LZO_CFG_PREFER_TYPEOF_ACC_INT16E_T == LZO_TYPEOF_SHORT) && (LZO_SIZEOF_SHORT != 2) # undef LZO_CFG_PREFER_TYPEOF_ACC_INT16E_T #endif #if (LZO_SIZEOF_LONG == 2) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT16E_T == LZO_TYPEOF_SHORT) # define lzo_int16e_t long # define lzo_uint16e_t unsigned long # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF_LONG #elif (LZO_SIZEOF_INT == 2) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT16E_T == LZO_TYPEOF_SHORT) # define lzo_int16e_t int # define lzo_uint16e_t unsigned int # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_SHORT == 2) # define lzo_int16e_t short int # define lzo_uint16e_t unsigned short int # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF_SHORT #elif 1 && !(LZO_CFG_TYPE_NO_MODE_HI) && (LZO_CC_CLANG || (LZO_CC_GNUC >= 0x025f00ul) || LZO_CC_LLVM) # if !(LZO_LANG_ASSEMBLER) typedef int lzo_int16e_hi_t__ __attribute__((__mode__(__HI__))); typedef unsigned int lzo_uint16e_hi_t__ __attribute__((__mode__(__HI__))); # endif # define lzo_int16e_t lzo_int16e_hi_t__ # define lzo_uint16e_t lzo_uint16e_hi_t__ # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF___MODE_HI #elif (LZO_SIZEOF___INT16 == 2) # define lzo_int16e_t __int16 # define lzo_uint16e_t unsigned __int16 # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF___INT16 #else #endif #endif #if defined(lzo_int16e_t) # define LZO_SIZEOF_LZO_INT16E_T 2 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16e_t) == 2) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16e_t) == LZO_SIZEOF_LZO_INT16E_T) #endif #if !defined(lzo_int32e_t) #if (LZO_CFG_PREFER_TYPEOF_ACC_INT32E_T == LZO_TYPEOF_INT) && (LZO_SIZEOF_INT != 4) # undef LZO_CFG_PREFER_TYPEOF_ACC_INT32E_T #endif #if (LZO_SIZEOF_LONG == 4) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT32E_T == LZO_TYPEOF_INT) # define lzo_int32e_t long int # define lzo_uint32e_t unsigned long int # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF_LONG #elif (LZO_SIZEOF_INT == 4) # define lzo_int32e_t int # define lzo_uint32e_t unsigned int # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_SHORT == 4) # define lzo_int32e_t short int # define lzo_uint32e_t unsigned short int # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF_SHORT #elif (LZO_SIZEOF_LONG_LONG == 4) # define lzo_int32e_t lzo_llong_t # define lzo_uint32e_t lzo_ullong_t # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF_LONG_LONG #elif 1 && !(LZO_CFG_TYPE_NO_MODE_SI) && (LZO_CC_CLANG || (LZO_CC_GNUC >= 0x025f00ul) || LZO_CC_LLVM) && (__INT_MAX__+0 > 2147483647L) # if !(LZO_LANG_ASSEMBLER) typedef int lzo_int32e_si_t__ __attribute__((__mode__(__SI__))); typedef unsigned int lzo_uint32e_si_t__ __attribute__((__mode__(__SI__))); # endif # define lzo_int32e_t lzo_int32e_si_t__ # define lzo_uint32e_t lzo_uint32e_si_t__ # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF___MODE_SI #elif 1 && !(LZO_CFG_TYPE_NO_MODE_SI) && (LZO_CC_GNUC >= 0x025f00ul) && defined(__AVR__) && (__LONG_MAX__+0 == 32767L) # if !(LZO_LANG_ASSEMBLER) typedef int lzo_int32e_si_t__ __attribute__((__mode__(__SI__))); typedef unsigned int lzo_uint32e_si_t__ __attribute__((__mode__(__SI__))); # endif # define lzo_int32e_t lzo_int32e_si_t__ # define lzo_uint32e_t lzo_uint32e_si_t__ # define LZO_INT32_C(c) (c##LL) # define LZO_UINT32_C(c) (c##ULL) # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF___MODE_SI #elif (LZO_SIZEOF___INT32 == 4) # define lzo_int32e_t __int32 # define lzo_uint32e_t unsigned __int32 # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF___INT32 #else #endif #endif #if defined(lzo_int32e_t) # define LZO_SIZEOF_LZO_INT32E_T 4 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32e_t) == 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32e_t) == LZO_SIZEOF_LZO_INT32E_T) #endif #if !defined(lzo_int64e_t) #if (LZO_SIZEOF___INT64 == 8) # if (LZO_CC_BORLANDC) && !defined(LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T) # define LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T LZO_TYPEOF___INT64 # endif #endif #if (LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF_LONG_LONG) && (LZO_SIZEOF_LONG_LONG != 8) # undef LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T #endif #if (LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF___INT64) && (LZO_SIZEOF___INT64 != 8) # undef LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T #endif #if (LZO_SIZEOF_INT == 8) && (LZO_SIZEOF_INT < LZO_SIZEOF_LONG) # define lzo_int64e_t int # define lzo_uint64e_t unsigned int # define LZO_TYPEOF_LZO_INT64E_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_LONG == 8) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF_LONG_LONG) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF___INT64) # define lzo_int64e_t long int # define lzo_uint64e_t unsigned long int # define LZO_TYPEOF_LZO_INT64E_T LZO_TYPEOF_LONG #elif (LZO_SIZEOF_LONG_LONG == 8) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF___INT64) # define lzo_int64e_t lzo_llong_t # define lzo_uint64e_t lzo_ullong_t # define LZO_TYPEOF_LZO_INT64E_T LZO_TYPEOF_LONG_LONG # if (LZO_CC_BORLANDC) # define LZO_INT64_C(c) ((c) + 0ll) # define LZO_UINT64_C(c) ((c) + 0ull) # elif 0 # define LZO_INT64_C(c) (__lzo_gnuc_extension__ (c##LL)) # define LZO_UINT64_C(c) (__lzo_gnuc_extension__ (c##ULL)) # else # define LZO_INT64_C(c) (c##LL) # define LZO_UINT64_C(c) (c##ULL) # endif #elif (LZO_SIZEOF___INT64 == 8) # define lzo_int64e_t __int64 # define lzo_uint64e_t unsigned __int64 # define LZO_TYPEOF_LZO_INT64E_T LZO_TYPEOF___INT64 # if (LZO_CC_BORLANDC) # define LZO_INT64_C(c) ((c) + 0i64) # define LZO_UINT64_C(c) ((c) + 0ui64) # else # define LZO_INT64_C(c) (c##i64) # define LZO_UINT64_C(c) (c##ui64) # endif #else #endif #endif #if defined(lzo_int64e_t) # define LZO_SIZEOF_LZO_INT64E_T 8 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64e_t) == 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64e_t) == LZO_SIZEOF_LZO_INT64E_T) #endif #if !defined(lzo_int32l_t) #if defined(lzo_int32e_t) # define lzo_int32l_t lzo_int32e_t # define lzo_uint32l_t lzo_uint32e_t # define LZO_SIZEOF_LZO_INT32L_T LZO_SIZEOF_LZO_INT32E_T # define LZO_TYPEOF_LZO_INT32L_T LZO_TYPEOF_LZO_INT32E_T #elif (LZO_SIZEOF_INT >= 4) && (LZO_SIZEOF_INT < LZO_SIZEOF_LONG) # define lzo_int32l_t int # define lzo_uint32l_t unsigned int # define LZO_SIZEOF_LZO_INT32L_T LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_INT32L_T LZO_SIZEOF_INT #elif (LZO_SIZEOF_LONG >= 4) # define lzo_int32l_t long int # define lzo_uint32l_t unsigned long int # define LZO_SIZEOF_LZO_INT32L_T LZO_SIZEOF_LONG # define LZO_TYPEOF_LZO_INT32L_T LZO_SIZEOF_LONG #else # error "lzo_int32l_t" #endif #endif #if 1 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32l_t) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32l_t) == LZO_SIZEOF_LZO_INT32L_T) #endif #if !defined(lzo_int64l_t) #if defined(lzo_int64e_t) # define lzo_int64l_t lzo_int64e_t # define lzo_uint64l_t lzo_uint64e_t # define LZO_SIZEOF_LZO_INT64L_T LZO_SIZEOF_LZO_INT64E_T # define LZO_TYPEOF_LZO_INT64L_T LZO_TYPEOF_LZO_INT64E_T #else #endif #endif #if defined(lzo_int64l_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64l_t) >= 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64l_t) == LZO_SIZEOF_LZO_INT64L_T) #endif #if !defined(lzo_int32f_t) #if (LZO_SIZEOF_SIZE_T >= 8) # define lzo_int32f_t lzo_int64l_t # define lzo_uint32f_t lzo_uint64l_t # define LZO_SIZEOF_LZO_INT32F_T LZO_SIZEOF_LZO_INT64L_T # define LZO_TYPEOF_LZO_INT32F_T LZO_TYPEOF_LZO_INT64L_T #else # define lzo_int32f_t lzo_int32l_t # define lzo_uint32f_t lzo_uint32l_t # define LZO_SIZEOF_LZO_INT32F_T LZO_SIZEOF_LZO_INT32L_T # define LZO_TYPEOF_LZO_INT32F_T LZO_TYPEOF_LZO_INT32L_T #endif #endif #if 1 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32f_t) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32f_t) == LZO_SIZEOF_LZO_INT32F_T) #endif #if !defined(lzo_int64f_t) #if defined(lzo_int64l_t) # define lzo_int64f_t lzo_int64l_t # define lzo_uint64f_t lzo_uint64l_t # define LZO_SIZEOF_LZO_INT64F_T LZO_SIZEOF_LZO_INT64L_T # define LZO_TYPEOF_LZO_INT64F_T LZO_TYPEOF_LZO_INT64L_T #else #endif #endif #if defined(lzo_int64f_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64f_t) >= 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64f_t) == LZO_SIZEOF_LZO_INT64F_T) #endif #if !defined(lzo_intptr_t) #if 1 && (LZO_OS_OS400 && (LZO_SIZEOF_VOID_P == 16)) # define __LZO_INTPTR_T_IS_POINTER 1 # if !(LZO_LANG_ASSEMBLER) typedef char * lzo_intptr_t; typedef char * lzo_uintptr_t; # endif # define lzo_intptr_t lzo_intptr_t # define lzo_uintptr_t lzo_uintptr_t # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_VOID_P # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_CHAR_P #elif (LZO_CC_MSC && (_MSC_VER >= 1300) && (LZO_SIZEOF_VOID_P == 4) && (LZO_SIZEOF_INT == 4)) # if !(LZO_LANG_ASSEMBLER) typedef __w64 int lzo_intptr_t; typedef __w64 unsigned int lzo_uintptr_t; # endif # define lzo_intptr_t lzo_intptr_t # define lzo_uintptr_t lzo_uintptr_t # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_SHORT == LZO_SIZEOF_VOID_P) && (LZO_SIZEOF_INT > LZO_SIZEOF_VOID_P) # define lzo_intptr_t short # define lzo_uintptr_t unsigned short # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_SHORT # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_SHORT #elif (LZO_SIZEOF_INT >= LZO_SIZEOF_VOID_P) && (LZO_SIZEOF_INT < LZO_SIZEOF_LONG) # define lzo_intptr_t int # define lzo_uintptr_t unsigned int # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_LONG >= LZO_SIZEOF_VOID_P) # define lzo_intptr_t long # define lzo_uintptr_t unsigned long # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_LONG # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_LONG #elif (LZO_SIZEOF_LZO_INT64L_T >= LZO_SIZEOF_VOID_P) # define lzo_intptr_t lzo_int64l_t # define lzo_uintptr_t lzo_uint64l_t # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_LZO_INT64L_T # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_LZO_INT64L_T #else # error "lzo_intptr_t" #endif #endif #if 1 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_intptr_t) >= sizeof(void *)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_intptr_t) == sizeof(lzo_uintptr_t)) #endif #if !defined(lzo_word_t) #if defined(LZO_WORDSIZE) && (LZO_WORDSIZE+0 > 0) #if (LZO_WORDSIZE == LZO_SIZEOF_LZO_INTPTR_T) && !(__LZO_INTPTR_T_IS_POINTER) # define lzo_word_t lzo_uintptr_t # define lzo_sword_t lzo_intptr_t # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_LZO_INTPTR_T # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_LZO_INTPTR_T #elif (LZO_WORDSIZE == LZO_SIZEOF_LONG) # define lzo_word_t unsigned long # define lzo_sword_t long # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_LONG # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_LONG #elif (LZO_WORDSIZE == LZO_SIZEOF_INT) # define lzo_word_t unsigned int # define lzo_sword_t int # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_INT #elif (LZO_WORDSIZE == LZO_SIZEOF_SHORT) # define lzo_word_t unsigned short # define lzo_sword_t short # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_SHORT # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_SHORT #elif (LZO_WORDSIZE == 1) # define lzo_word_t unsigned char # define lzo_sword_t signed char # define LZO_SIZEOF_LZO_WORD_T 1 # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_CHAR #elif (LZO_WORDSIZE == LZO_SIZEOF_LZO_INT64L_T) # define lzo_word_t lzo_uint64l_t # define lzo_sword_t lzo_int64l_t # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_LZO_INT64L_T # define LZO_TYPEOF_LZO_WORD_T LZO_SIZEOF_LZO_INT64L_T #elif (LZO_ARCH_SPU) && (LZO_CC_GNUC) #if 0 # if !(LZO_LANG_ASSEMBLER) typedef unsigned lzo_word_t __attribute__((__mode__(__V16QI__))); typedef int lzo_sword_t __attribute__((__mode__(__V16QI__))); # endif # define lzo_word_t lzo_word_t # define lzo_sword_t lzo_sword_t # define LZO_SIZEOF_LZO_WORD_T 16 # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF___MODE_V16QI #endif #else # error "lzo_word_t" #endif #endif #endif #if 1 && defined(lzo_word_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_word_t) == LZO_WORDSIZE) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_sword_t) == LZO_WORDSIZE) #endif #if 1 #define lzo_int8_t signed char #define lzo_uint8_t unsigned char #define LZO_SIZEOF_LZO_INT8_T 1 #define LZO_TYPEOF_LZO_INT8_T LZO_TYPEOF_CHAR LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int8_t) == 1) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int8_t) == sizeof(lzo_uint8_t)) #endif #if defined(lzo_int16e_t) #define lzo_int16_t lzo_int16e_t #define lzo_uint16_t lzo_uint16e_t #define LZO_SIZEOF_LZO_INT16_T LZO_SIZEOF_LZO_INT16E_T #define LZO_TYPEOF_LZO_INT16_T LZO_TYPEOF_LZO_INT16E_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16_t) == 2) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16_t) == sizeof(lzo_uint16_t)) #endif #if defined(lzo_int32e_t) #define lzo_int32_t lzo_int32e_t #define lzo_uint32_t lzo_uint32e_t #define LZO_SIZEOF_LZO_INT32_T LZO_SIZEOF_LZO_INT32E_T #define LZO_TYPEOF_LZO_INT32_T LZO_TYPEOF_LZO_INT32E_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32_t) == 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32_t) == sizeof(lzo_uint32_t)) #endif #if defined(lzo_int64e_t) #define lzo_int64_t lzo_int64e_t #define lzo_uint64_t lzo_uint64e_t #define LZO_SIZEOF_LZO_INT64_T LZO_SIZEOF_LZO_INT64E_T #define LZO_TYPEOF_LZO_INT64_T LZO_TYPEOF_LZO_INT64E_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64_t) == 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64_t) == sizeof(lzo_uint64_t)) #endif #if 1 #define lzo_int_least32_t lzo_int32l_t #define lzo_uint_least32_t lzo_uint32l_t #define LZO_SIZEOF_LZO_INT_LEAST32_T LZO_SIZEOF_LZO_INT32L_T #define LZO_TYPEOF_LZO_INT_LEAST32_T LZO_TYPEOF_LZO_INT32L_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_least32_t) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_least32_t) == sizeof(lzo_uint_least32_t)) #endif #if defined(lzo_int64l_t) #define lzo_int_least64_t lzo_int64l_t #define lzo_uint_least64_t lzo_uint64l_t #define LZO_SIZEOF_LZO_INT_LEAST64_T LZO_SIZEOF_LZO_INT64L_T #define LZO_TYPEOF_LZO_INT_LEAST64_T LZO_TYPEOF_LZO_INT64L_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_least64_t) >= 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_least64_t) == sizeof(lzo_uint_least64_t)) #endif #if 1 #define lzo_int_fast32_t lzo_int32f_t #define lzo_uint_fast32_t lzo_uint32f_t #define LZO_SIZEOF_LZO_INT_FAST32_T LZO_SIZEOF_LZO_INT32F_T #define LZO_TYPEOF_LZO_INT_FAST32_T LZO_TYPEOF_LZO_INT32F_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_fast32_t) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_fast32_t) == sizeof(lzo_uint_fast32_t)) #endif #if defined(lzo_int64f_t) #define lzo_int_fast64_t lzo_int64f_t #define lzo_uint_fast64_t lzo_uint64f_t #define LZO_SIZEOF_LZO_INT_FAST64_T LZO_SIZEOF_LZO_INT64F_T #define LZO_TYPEOF_LZO_INT_FAST64_T LZO_TYPEOF_LZO_INT64F_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_fast64_t) >= 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_fast64_t) == sizeof(lzo_uint_fast64_t)) #endif #if !defined(LZO_INT16_C) # if (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_INT >= 2) # define LZO_INT16_C(c) ((c) + 0) # define LZO_UINT16_C(c) ((c) + 0U) # elif (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_LONG >= 2) # define LZO_INT16_C(c) ((c) + 0L) # define LZO_UINT16_C(c) ((c) + 0UL) # elif (LZO_SIZEOF_INT >= 2) # define LZO_INT16_C(c) (c) # define LZO_UINT16_C(c) (c##U) # elif (LZO_SIZEOF_LONG >= 2) # define LZO_INT16_C(c) (c##L) # define LZO_UINT16_C(c) (c##UL) # else # error "LZO_INT16_C" # endif #endif #if !defined(LZO_INT32_C) # if (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_INT >= 4) # define LZO_INT32_C(c) ((c) + 0) # define LZO_UINT32_C(c) ((c) + 0U) # elif (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_LONG >= 4) # define LZO_INT32_C(c) ((c) + 0L) # define LZO_UINT32_C(c) ((c) + 0UL) # elif (LZO_SIZEOF_INT >= 4) # define LZO_INT32_C(c) (c) # define LZO_UINT32_C(c) (c##U) # elif (LZO_SIZEOF_LONG >= 4) # define LZO_INT32_C(c) (c##L) # define LZO_UINT32_C(c) (c##UL) # elif (LZO_SIZEOF_LONG_LONG >= 4) # define LZO_INT32_C(c) (c##LL) # define LZO_UINT32_C(c) (c##ULL) # else # error "LZO_INT32_C" # endif #endif #if !defined(LZO_INT64_C) && defined(lzo_int64l_t) # if (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_INT >= 8) # define LZO_INT64_C(c) ((c) + 0) # define LZO_UINT64_C(c) ((c) + 0U) # elif (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_LONG >= 8) # define LZO_INT64_C(c) ((c) + 0L) # define LZO_UINT64_C(c) ((c) + 0UL) # elif (LZO_SIZEOF_INT >= 8) # define LZO_INT64_C(c) (c) # define LZO_UINT64_C(c) (c##U) # elif (LZO_SIZEOF_LONG >= 8) # define LZO_INT64_C(c) (c##L) # define LZO_UINT64_C(c) (c##UL) # else # error "LZO_INT64_C" # endif #endif #endif #endif /* already included */ /* vim:set ts=4 sw=4 et: */ nfdump-1.6.23/bin/minilzo.c000077500000000000000000006412521404501030700155000ustar00rootroot00000000000000/* minilzo.c -- mini subset of the LZO real-time data compression library This file is part of the LZO real-time data compression library. Copyright (C) 1996-2017 Markus Franz Xaver Johannes Oberhumer All Rights Reserved. The LZO library is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. The LZO library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with the LZO library; see the file COPYING. If not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Markus F.X.J. Oberhumer http://www.oberhumer.com/opensource/lzo/ */ /* * NOTE: * the full LZO package can be found at * http://www.oberhumer.com/opensource/lzo/ */ #define __LZO_IN_MINILZO 1 #if defined(LZO_CFG_FREESTANDING) # undef MINILZO_HAVE_CONFIG_H # define LZO_LIBC_FREESTANDING 1 # define LZO_OS_FREESTANDING 1 #endif #ifdef MINILZO_HAVE_CONFIG_H # include #endif #include #include #if defined(MINILZO_CFG_USE_INTERNAL_LZODEFS) #ifndef __LZODEFS_H_INCLUDED #define __LZODEFS_H_INCLUDED 1 #if defined(__CYGWIN32__) && !defined(__CYGWIN__) # define __CYGWIN__ __CYGWIN32__ #endif #if 1 && defined(__INTERIX) && defined(__GNUC__) && !defined(_ALL_SOURCE) # define _ALL_SOURCE 1 #endif #if defined(__mips__) && defined(__R5900__) # if !defined(__LONG_MAX__) # define __LONG_MAX__ 9223372036854775807L # endif #endif #if 0 #elif !defined(__LZO_LANG_OVERRIDE) #if (defined(__clang__) || defined(__GNUC__)) && defined(__ASSEMBLER__) # if (__ASSEMBLER__+0) <= 0 # error "__ASSEMBLER__" # else # define LZO_LANG_ASSEMBLER 1 # endif #elif defined(__cplusplus) # if (__cplusplus+0) <= 0 # error "__cplusplus" # elif (__cplusplus < 199711L) # define LZO_LANG_CXX 1 # elif defined(_MSC_VER) && defined(_MSVC_LANG) && (_MSVC_LANG+0 >= 201402L) && 1 # define LZO_LANG_CXX _MSVC_LANG # else # define LZO_LANG_CXX __cplusplus # endif # define LZO_LANG_CPLUSPLUS LZO_LANG_CXX #else # if defined(__STDC_VERSION__) && (__STDC_VERSION__+0 >= 199409L) # define LZO_LANG_C __STDC_VERSION__ # else # define LZO_LANG_C 1 # endif #endif #endif #if !defined(LZO_CFG_NO_DISABLE_WUNDEF) #if defined(__ARMCC_VERSION) # pragma diag_suppress 193 #elif defined(__clang__) && defined(__clang_minor__) # pragma clang diagnostic ignored "-Wundef" #elif defined(__INTEL_COMPILER) # pragma warning(disable: 193) #elif defined(__KEIL__) && defined(__C166__) # pragma warning disable = 322 #elif defined(__GNUC__) && defined(__GNUC_MINOR__) && !defined(__PATHSCALE__) # if ((__GNUC__-0) >= 5 || ((__GNUC__-0) == 4 && (__GNUC_MINOR__-0) >= 2)) # pragma GCC diagnostic ignored "-Wundef" # endif #elif defined(_MSC_VER) && !defined(__clang__) && !defined(__INTEL_COMPILER) && !defined(__MWERKS__) # if ((_MSC_VER-0) >= 1300) # pragma warning(disable: 4668) # endif #endif #endif #if 0 && defined(__POCC__) && defined(_WIN32) # if (__POCC__ >= 400) # pragma warn(disable: 2216) # endif #endif #if 0 && defined(__WATCOMC__) # if (__WATCOMC__ >= 1050) && (__WATCOMC__ < 1060) # pragma warning 203 9 # endif #endif #if defined(__BORLANDC__) && defined(__MSDOS__) && !defined(__FLAT__) # pragma option -h #endif #if !(LZO_CFG_NO_DISABLE_WCRTNONSTDC) #ifndef _CRT_NONSTDC_NO_DEPRECATE #define _CRT_NONSTDC_NO_DEPRECATE 1 #endif #ifndef _CRT_NONSTDC_NO_WARNINGS #define _CRT_NONSTDC_NO_WARNINGS 1 #endif #ifndef _CRT_SECURE_NO_DEPRECATE #define _CRT_SECURE_NO_DEPRECATE 1 #endif #ifndef _CRT_SECURE_NO_WARNINGS #define _CRT_SECURE_NO_WARNINGS 1 #endif #endif #if 0 #define LZO_0xffffUL 0xfffful #define LZO_0xffffffffUL 0xfffffffful #else #define LZO_0xffffUL 65535ul #define LZO_0xffffffffUL 4294967295ul #endif #define LZO_0xffffL LZO_0xffffUL #define LZO_0xffffffffL LZO_0xffffffffUL #if (LZO_0xffffL == LZO_0xffffffffL) # error "your preprocessor is broken 1" #endif #if (16ul * 16384ul != 262144ul) # error "your preprocessor is broken 2" #endif #if 0 #if (32767 >= 4294967295ul) # error "your preprocessor is broken 3" #endif #if (65535u >= 4294967295ul) # error "your preprocessor is broken 4" #endif #endif #if defined(__COUNTER__) # ifndef LZO_CFG_USE_COUNTER # define LZO_CFG_USE_COUNTER 1 # endif #else # undef LZO_CFG_USE_COUNTER #endif #if (UINT_MAX == LZO_0xffffL) #if defined(__ZTC__) && defined(__I86__) && !defined(__OS2__) # if !defined(MSDOS) # define MSDOS 1 # endif # if !defined(_MSDOS) # define _MSDOS 1 # endif #elif 0 && defined(__VERSION) && defined(MB_LEN_MAX) # if (__VERSION == 520) && (MB_LEN_MAX == 1) # if !defined(__AZTEC_C__) # define __AZTEC_C__ __VERSION # endif # if !defined(__DOS__) # define __DOS__ 1 # endif # endif #endif #endif #if (UINT_MAX == LZO_0xffffL) #if defined(_MSC_VER) && defined(M_I86HM) # define ptrdiff_t long # define _PTRDIFF_T_DEFINED 1 #endif #endif #if (UINT_MAX == LZO_0xffffL) # undef __LZO_RENAME_A # undef __LZO_RENAME_B # if defined(__AZTEC_C__) && defined(__DOS__) # define __LZO_RENAME_A 1 # elif defined(_MSC_VER) && defined(MSDOS) # if (_MSC_VER < 600) # define __LZO_RENAME_A 1 # elif (_MSC_VER < 700) # define __LZO_RENAME_B 1 # endif # elif defined(__TSC__) && defined(__OS2__) # define __LZO_RENAME_A 1 # elif defined(__MSDOS__) && defined(__TURBOC__) && (__TURBOC__ < 0x0410) # define __LZO_RENAME_A 1 # elif defined(__PACIFIC__) && defined(DOS) # if !defined(__far) # define __far far # endif # if !defined(__near) # define __near near # endif # endif # if defined(__LZO_RENAME_A) # if !defined(__cdecl) # define __cdecl cdecl # endif # if !defined(__far) # define __far far # endif # if !defined(__huge) # define __huge huge # endif # if !defined(__near) # define __near near # endif # if !defined(__pascal) # define __pascal pascal # endif # if !defined(__huge) # define __huge huge # endif # elif defined(__LZO_RENAME_B) # if !defined(__cdecl) # define __cdecl _cdecl # endif # if !defined(__far) # define __far _far # endif # if !defined(__huge) # define __huge _huge # endif # if !defined(__near) # define __near _near # endif # if !defined(__pascal) # define __pascal _pascal # endif # elif (defined(__PUREC__) || defined(__TURBOC__)) && defined(__TOS__) # if !defined(__cdecl) # define __cdecl cdecl # endif # if !defined(__pascal) # define __pascal pascal # endif # endif # undef __LZO_RENAME_A # undef __LZO_RENAME_B #endif #if (UINT_MAX == LZO_0xffffL) #if defined(__AZTEC_C__) && defined(__DOS__) # define LZO_BROKEN_CDECL_ALT_SYNTAX 1 #elif defined(_MSC_VER) && defined(MSDOS) # if (_MSC_VER < 600) # define LZO_BROKEN_INTEGRAL_CONSTANTS 1 # endif # if (_MSC_VER < 700) # define LZO_BROKEN_INTEGRAL_PROMOTION 1 # define LZO_BROKEN_SIZEOF 1 # endif #elif defined(__PACIFIC__) && defined(DOS) # define LZO_BROKEN_INTEGRAL_CONSTANTS 1 #elif defined(__TURBOC__) && defined(__MSDOS__) # if (__TURBOC__ < 0x0150) # define LZO_BROKEN_CDECL_ALT_SYNTAX 1 # define LZO_BROKEN_INTEGRAL_CONSTANTS 1 # define LZO_BROKEN_INTEGRAL_PROMOTION 1 # endif # if (__TURBOC__ < 0x0200) # define LZO_BROKEN_SIZEOF 1 # endif # if (__TURBOC__ < 0x0400) && defined(__cplusplus) # define LZO_BROKEN_CDECL_ALT_SYNTAX 1 # endif #elif (defined(__PUREC__) || defined(__TURBOC__)) && defined(__TOS__) # define LZO_BROKEN_CDECL_ALT_SYNTAX 1 # define LZO_BROKEN_SIZEOF 1 #endif #endif #if defined(__WATCOMC__) && (__WATCOMC__ < 900) # define LZO_BROKEN_INTEGRAL_CONSTANTS 1 #endif #if defined(_CRAY) && defined(_CRAY1) # define LZO_BROKEN_SIGNED_RIGHT_SHIFT 1 #endif #define LZO_PP_STRINGIZE(x) #x #define LZO_PP_MACRO_EXPAND(x) LZO_PP_STRINGIZE(x) #define LZO_PP_CONCAT0() /*empty*/ #define LZO_PP_CONCAT1(a) a #define LZO_PP_CONCAT2(a,b) a ## b #define LZO_PP_CONCAT3(a,b,c) a ## b ## c #define LZO_PP_CONCAT4(a,b,c,d) a ## b ## c ## d #define LZO_PP_CONCAT5(a,b,c,d,e) a ## b ## c ## d ## e #define LZO_PP_CONCAT6(a,b,c,d,e,f) a ## b ## c ## d ## e ## f #define LZO_PP_CONCAT7(a,b,c,d,e,f,g) a ## b ## c ## d ## e ## f ## g #define LZO_PP_ECONCAT0() LZO_PP_CONCAT0() #define LZO_PP_ECONCAT1(a) LZO_PP_CONCAT1(a) #define LZO_PP_ECONCAT2(a,b) LZO_PP_CONCAT2(a,b) #define LZO_PP_ECONCAT3(a,b,c) LZO_PP_CONCAT3(a,b,c) #define LZO_PP_ECONCAT4(a,b,c,d) LZO_PP_CONCAT4(a,b,c,d) #define LZO_PP_ECONCAT5(a,b,c,d,e) LZO_PP_CONCAT5(a,b,c,d,e) #define LZO_PP_ECONCAT6(a,b,c,d,e,f) LZO_PP_CONCAT6(a,b,c,d,e,f) #define LZO_PP_ECONCAT7(a,b,c,d,e,f,g) LZO_PP_CONCAT7(a,b,c,d,e,f,g) #define LZO_PP_EMPTY /*empty*/ #define LZO_PP_EMPTY0() /*empty*/ #define LZO_PP_EMPTY1(a) /*empty*/ #define LZO_PP_EMPTY2(a,b) /*empty*/ #define LZO_PP_EMPTY3(a,b,c) /*empty*/ #define LZO_PP_EMPTY4(a,b,c,d) /*empty*/ #define LZO_PP_EMPTY5(a,b,c,d,e) /*empty*/ #define LZO_PP_EMPTY6(a,b,c,d,e,f) /*empty*/ #define LZO_PP_EMPTY7(a,b,c,d,e,f,g) /*empty*/ #if 1 #define LZO_CPP_STRINGIZE(x) #x #define LZO_CPP_MACRO_EXPAND(x) LZO_CPP_STRINGIZE(x) #define LZO_CPP_CONCAT2(a,b) a ## b #define LZO_CPP_CONCAT3(a,b,c) a ## b ## c #define LZO_CPP_CONCAT4(a,b,c,d) a ## b ## c ## d #define LZO_CPP_CONCAT5(a,b,c,d,e) a ## b ## c ## d ## e #define LZO_CPP_CONCAT6(a,b,c,d,e,f) a ## b ## c ## d ## e ## f #define LZO_CPP_CONCAT7(a,b,c,d,e,f,g) a ## b ## c ## d ## e ## f ## g #define LZO_CPP_ECONCAT2(a,b) LZO_CPP_CONCAT2(a,b) #define LZO_CPP_ECONCAT3(a,b,c) LZO_CPP_CONCAT3(a,b,c) #define LZO_CPP_ECONCAT4(a,b,c,d) LZO_CPP_CONCAT4(a,b,c,d) #define LZO_CPP_ECONCAT5(a,b,c,d,e) LZO_CPP_CONCAT5(a,b,c,d,e) #define LZO_CPP_ECONCAT6(a,b,c,d,e,f) LZO_CPP_CONCAT6(a,b,c,d,e,f) #define LZO_CPP_ECONCAT7(a,b,c,d,e,f,g) LZO_CPP_CONCAT7(a,b,c,d,e,f,g) #endif #define __LZO_MASK_GEN(o,b) (((((o) << ((b)-((b)!=0))) - (o)) << 1) + (o)*((b)!=0)) #if 1 && defined(__cplusplus) # if !defined(__STDC_CONSTANT_MACROS) # define __STDC_CONSTANT_MACROS 1 # endif # if !defined(__STDC_LIMIT_MACROS) # define __STDC_LIMIT_MACROS 1 # endif #endif #if defined(__cplusplus) # define LZO_EXTERN_C extern "C" # define LZO_EXTERN_C_BEGIN extern "C" { # define LZO_EXTERN_C_END } #else # define LZO_EXTERN_C extern # define LZO_EXTERN_C_BEGIN /*empty*/ # define LZO_EXTERN_C_END /*empty*/ #endif #if !defined(__LZO_OS_OVERRIDE) #if (LZO_OS_FREESTANDING) # define LZO_INFO_OS "freestanding" #elif (LZO_OS_EMBEDDED) # define LZO_INFO_OS "embedded" #elif 1 && defined(__IAR_SYSTEMS_ICC__) # define LZO_OS_EMBEDDED 1 # define LZO_INFO_OS "embedded" #elif defined(__CYGWIN__) && defined(__GNUC__) # define LZO_OS_CYGWIN 1 # define LZO_INFO_OS "cygwin" #elif defined(__EMX__) && defined(__GNUC__) # define LZO_OS_EMX 1 # define LZO_INFO_OS "emx" #elif defined(__BEOS__) # define LZO_OS_BEOS 1 # define LZO_INFO_OS "beos" #elif defined(__Lynx__) # define LZO_OS_LYNXOS 1 # define LZO_INFO_OS "lynxos" #elif defined(__OS400__) # define LZO_OS_OS400 1 # define LZO_INFO_OS "os400" #elif defined(__QNX__) # define LZO_OS_QNX 1 # define LZO_INFO_OS "qnx" #elif defined(__BORLANDC__) && defined(__DPMI32__) && (__BORLANDC__ >= 0x0460) # define LZO_OS_DOS32 1 # define LZO_INFO_OS "dos32" #elif defined(__BORLANDC__) && defined(__DPMI16__) # define LZO_OS_DOS16 1 # define LZO_INFO_OS "dos16" #elif defined(__ZTC__) && defined(DOS386) # define LZO_OS_DOS32 1 # define LZO_INFO_OS "dos32" #elif defined(__OS2__) || defined(__OS2V2__) # if (UINT_MAX == LZO_0xffffL) # define LZO_OS_OS216 1 # define LZO_INFO_OS "os216" # elif (UINT_MAX == LZO_0xffffffffL) # define LZO_OS_OS2 1 # define LZO_INFO_OS "os2" # else # error "check your limits.h header" # endif #elif defined(__WIN64__) || defined(_WIN64) || defined(WIN64) # define LZO_OS_WIN64 1 # define LZO_INFO_OS "win64" #elif defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__WINDOWS_386__) # define LZO_OS_WIN32 1 # define LZO_INFO_OS "win32" #elif defined(__MWERKS__) && defined(__INTEL__) # define LZO_OS_WIN32 1 # define LZO_INFO_OS "win32" #elif defined(__WINDOWS__) || defined(_WINDOWS) || defined(_Windows) # if (UINT_MAX == LZO_0xffffL) # define LZO_OS_WIN16 1 # define LZO_INFO_OS "win16" # elif (UINT_MAX == LZO_0xffffffffL) # define LZO_OS_WIN32 1 # define LZO_INFO_OS "win32" # else # error "check your limits.h header" # endif #elif defined(__DOS__) || defined(__MSDOS__) || defined(_MSDOS) || defined(MSDOS) || (defined(__PACIFIC__) && defined(DOS)) # if (UINT_MAX == LZO_0xffffL) # define LZO_OS_DOS16 1 # define LZO_INFO_OS "dos16" # elif (UINT_MAX == LZO_0xffffffffL) # define LZO_OS_DOS32 1 # define LZO_INFO_OS "dos32" # else # error "check your limits.h header" # endif #elif defined(__WATCOMC__) # if defined(__NT__) && (UINT_MAX == LZO_0xffffL) # define LZO_OS_DOS16 1 # define LZO_INFO_OS "dos16" # elif defined(__NT__) && (__WATCOMC__ < 1100) # define LZO_OS_WIN32 1 # define LZO_INFO_OS "win32" # elif defined(__linux__) || defined(__LINUX__) # define LZO_OS_POSIX 1 # define LZO_INFO_OS "posix" # else # error "please specify a target using the -bt compiler option" # endif #elif defined(__palmos__) # define LZO_OS_PALMOS 1 # define LZO_INFO_OS "palmos" #elif defined(__TOS__) || defined(__atarist__) # define LZO_OS_TOS 1 # define LZO_INFO_OS "tos" #elif defined(macintosh) && !defined(__arm__) && !defined(__i386__) && !defined(__ppc__) && !defined(__x64_64__) # define LZO_OS_MACCLASSIC 1 # define LZO_INFO_OS "macclassic" #elif defined(__VMS) # define LZO_OS_VMS 1 # define LZO_INFO_OS "vms" #elif (defined(__mips__) && defined(__R5900__)) || defined(__MIPS_PSX2__) # define LZO_OS_CONSOLE 1 # define LZO_OS_CONSOLE_PS2 1 # define LZO_INFO_OS "console" # define LZO_INFO_OS_CONSOLE "ps2" #elif defined(__mips__) && defined(__psp__) # define LZO_OS_CONSOLE 1 # define LZO_OS_CONSOLE_PSP 1 # define LZO_INFO_OS "console" # define LZO_INFO_OS_CONSOLE "psp" #else # define LZO_OS_POSIX 1 # define LZO_INFO_OS "posix" #endif #if (LZO_OS_POSIX) # if defined(_AIX) || defined(__AIX__) || defined(__aix__) # define LZO_OS_POSIX_AIX 1 # define LZO_INFO_OS_POSIX "aix" # elif defined(__FreeBSD__) # define LZO_OS_POSIX_FREEBSD 1 # define LZO_INFO_OS_POSIX "freebsd" # elif defined(__hpux__) || defined(__hpux) # define LZO_OS_POSIX_HPUX 1 # define LZO_INFO_OS_POSIX "hpux" # elif defined(__INTERIX) # define LZO_OS_POSIX_INTERIX 1 # define LZO_INFO_OS_POSIX "interix" # elif defined(__IRIX__) || defined(__irix__) # define LZO_OS_POSIX_IRIX 1 # define LZO_INFO_OS_POSIX "irix" # elif defined(__linux__) || defined(__linux) || defined(__LINUX__) # define LZO_OS_POSIX_LINUX 1 # define LZO_INFO_OS_POSIX "linux" # elif defined(__APPLE__) && defined(__MACH__) # if ((__ENVIRONMENT_IPHONE_OS_VERSION_MIN_REQUIRED__-0) >= 20000) # define LZO_OS_POSIX_DARWIN 1040 # define LZO_INFO_OS_POSIX "darwin_iphone" # elif ((__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__-0) >= 1040) # define LZO_OS_POSIX_DARWIN __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ # define LZO_INFO_OS_POSIX "darwin" # else # define LZO_OS_POSIX_DARWIN 1 # define LZO_INFO_OS_POSIX "darwin" # endif # define LZO_OS_POSIX_MACOSX LZO_OS_POSIX_DARWIN # elif defined(__minix__) || defined(__minix) # define LZO_OS_POSIX_MINIX 1 # define LZO_INFO_OS_POSIX "minix" # elif defined(__NetBSD__) # define LZO_OS_POSIX_NETBSD 1 # define LZO_INFO_OS_POSIX "netbsd" # elif defined(__OpenBSD__) # define LZO_OS_POSIX_OPENBSD 1 # define LZO_INFO_OS_POSIX "openbsd" # elif defined(__osf__) # define LZO_OS_POSIX_OSF 1 # define LZO_INFO_OS_POSIX "osf" # elif defined(__solaris__) || defined(__sun) # if defined(__SVR4) || defined(__svr4__) # define LZO_OS_POSIX_SOLARIS 1 # define LZO_INFO_OS_POSIX "solaris" # else # define LZO_OS_POSIX_SUNOS 1 # define LZO_INFO_OS_POSIX "sunos" # endif # elif defined(__ultrix__) || defined(__ultrix) # define LZO_OS_POSIX_ULTRIX 1 # define LZO_INFO_OS_POSIX "ultrix" # elif defined(_UNICOS) # define LZO_OS_POSIX_UNICOS 1 # define LZO_INFO_OS_POSIX "unicos" # else # define LZO_OS_POSIX_UNKNOWN 1 # define LZO_INFO_OS_POSIX "unknown" # endif #endif #endif #if (LZO_OS_DOS16 || LZO_OS_OS216 || LZO_OS_WIN16) # if (UINT_MAX != LZO_0xffffL) # error "unexpected configuration - check your compiler defines" # endif # if (ULONG_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif #endif #if (LZO_OS_DOS32 || LZO_OS_OS2 || LZO_OS_WIN32 || LZO_OS_WIN64) # if (UINT_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif # if (ULONG_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif #endif #if defined(CIL) && defined(_GNUCC) && defined(__GNUC__) # define LZO_CC_CILLY 1 # define LZO_INFO_CC "Cilly" # if defined(__CILLY__) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__CILLY__) # else # define LZO_INFO_CCVER "unknown" # endif #elif 0 && defined(SDCC) && defined(__VERSION__) && !defined(__GNUC__) # define LZO_CC_SDCC 1 # define LZO_INFO_CC "sdcc" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(SDCC) #elif defined(__PATHSCALE__) && defined(__PATHCC_PATCHLEVEL__) # define LZO_CC_PATHSCALE (__PATHCC__ * 0x10000L + (__PATHCC_MINOR__-0) * 0x100 + (__PATHCC_PATCHLEVEL__-0)) # define LZO_INFO_CC "Pathscale C" # define LZO_INFO_CCVER __PATHSCALE__ # if defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # define LZO_CC_PATHSCALE_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # endif #elif defined(__INTEL_COMPILER) && ((__INTEL_COMPILER-0) > 0) # define LZO_CC_INTELC __INTEL_COMPILER # define LZO_INFO_CC "Intel C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__INTEL_COMPILER) # if defined(_MSC_VER) && ((_MSC_VER-0) > 0) # define LZO_CC_INTELC_MSC _MSC_VER # elif defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # define LZO_CC_INTELC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # endif #elif defined(__POCC__) && defined(_WIN32) # define LZO_CC_PELLESC 1 # define LZO_INFO_CC "Pelles C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__POCC__) #elif defined(__ARMCC_VERSION) && defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # if defined(__GNUC_PATCHLEVEL__) # define LZO_CC_ARMCC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # else # define LZO_CC_ARMCC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100) # endif # define LZO_CC_ARMCC __ARMCC_VERSION # define LZO_INFO_CC "ARM C Compiler" # define LZO_INFO_CCVER __VERSION__ #elif defined(__clang__) && defined(__c2__) && defined(__c2_version__) && defined(_MSC_VER) # define LZO_CC_CLANG (__clang_major__ * 0x10000L + (__clang_minor__-0) * 0x100 + (__clang_patchlevel__-0)) # define LZO_CC_CLANG_C2 _MSC_VER # define LZO_CC_CLANG_VENDOR_MICROSOFT 1 # define LZO_INFO_CC "clang/c2" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__c2_version__) #elif defined(__clang__) && defined(__llvm__) && defined(__VERSION__) # if defined(__clang_major__) && defined(__clang_minor__) && defined(__clang_patchlevel__) # define LZO_CC_CLANG (__clang_major__ * 0x10000L + (__clang_minor__-0) * 0x100 + (__clang_patchlevel__-0)) # else # define LZO_CC_CLANG 0x010000L # endif # if defined(_MSC_VER) && ((_MSC_VER-0) > 0) # define LZO_CC_CLANG_MSC _MSC_VER # elif defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # define LZO_CC_CLANG_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # endif # if defined(__APPLE_CC__) # define LZO_CC_CLANG_VENDOR_APPLE 1 # define LZO_INFO_CC "clang/apple" # else # define LZO_CC_CLANG_VENDOR_LLVM 1 # define LZO_INFO_CC "clang" # endif # if defined(__clang_version__) # define LZO_INFO_CCVER __clang_version__ # else # define LZO_INFO_CCVER __VERSION__ # endif #elif defined(__llvm__) && defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # if defined(__GNUC_PATCHLEVEL__) # define LZO_CC_LLVM_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # else # define LZO_CC_LLVM_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100) # endif # define LZO_CC_LLVM LZO_CC_LLVM_GNUC # define LZO_INFO_CC "llvm-gcc" # define LZO_INFO_CCVER __VERSION__ #elif defined(__ACK__) && defined(_ACK) # define LZO_CC_ACK 1 # define LZO_INFO_CC "Amsterdam Compiler Kit C" # define LZO_INFO_CCVER "unknown" #elif defined(__ARMCC_VERSION) && !defined(__GNUC__) # define LZO_CC_ARMCC __ARMCC_VERSION # define LZO_CC_ARMCC_ARMCC __ARMCC_VERSION # define LZO_INFO_CC "ARM C Compiler" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__ARMCC_VERSION) #elif defined(__AZTEC_C__) # define LZO_CC_AZTECC 1 # define LZO_INFO_CC "Aztec C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__AZTEC_C__) #elif defined(__CODEGEARC__) # define LZO_CC_CODEGEARC 1 # define LZO_INFO_CC "CodeGear C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__CODEGEARC__) #elif defined(__BORLANDC__) # define LZO_CC_BORLANDC 1 # define LZO_INFO_CC "Borland C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__BORLANDC__) #elif defined(_CRAYC) && defined(_RELEASE) # define LZO_CC_CRAYC 1 # define LZO_INFO_CC "Cray C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(_RELEASE) #elif defined(__DMC__) && defined(__SC__) # define LZO_CC_DMC 1 # define LZO_INFO_CC "Digital Mars C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__DMC__) #elif defined(__DECC) # define LZO_CC_DECC 1 # define LZO_INFO_CC "DEC C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__DECC) #elif (defined(__ghs) || defined(__ghs__)) && defined(__GHS_VERSION_NUMBER) && ((__GHS_VERSION_NUMBER-0) > 0) # define LZO_CC_GHS 1 # define LZO_INFO_CC "Green Hills C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__GHS_VERSION_NUMBER) # if defined(_MSC_VER) && ((_MSC_VER-0) > 0) # define LZO_CC_GHS_MSC _MSC_VER # elif defined(__GNUC__) && defined(__GNUC_MINOR__) && defined(__VERSION__) # define LZO_CC_GHS_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # endif #elif defined(__HIGHC__) # define LZO_CC_HIGHC 1 # define LZO_INFO_CC "MetaWare High C" # define LZO_INFO_CCVER "unknown" #elif defined(__HP_aCC) && ((__HP_aCC-0) > 0) # define LZO_CC_HPACC __HP_aCC # define LZO_INFO_CC "HP aCC" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__HP_aCC) #elif defined(__IAR_SYSTEMS_ICC__) # define LZO_CC_IARC 1 # define LZO_INFO_CC "IAR C" # if defined(__VER__) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__VER__) # else # define LZO_INFO_CCVER "unknown" # endif #elif defined(__IBMC__) && ((__IBMC__-0) > 0) # define LZO_CC_IBMC __IBMC__ # define LZO_INFO_CC "IBM C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__IBMC__) #elif defined(__IBMCPP__) && ((__IBMCPP__-0) > 0) # define LZO_CC_IBMC __IBMCPP__ # define LZO_INFO_CC "IBM C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__IBMCPP__) #elif defined(__KEIL__) && defined(__C166__) # define LZO_CC_KEILC 1 # define LZO_INFO_CC "Keil C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__C166__) #elif defined(__LCC__) && defined(_WIN32) && defined(__LCCOPTIMLEVEL) # define LZO_CC_LCCWIN32 1 # define LZO_INFO_CC "lcc-win32" # define LZO_INFO_CCVER "unknown" #elif defined(__LCC__) # define LZO_CC_LCC 1 # define LZO_INFO_CC "lcc" # if defined(__LCC_VERSION__) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__LCC_VERSION__) # else # define LZO_INFO_CCVER "unknown" # endif #elif defined(__MWERKS__) && ((__MWERKS__-0) > 0) # define LZO_CC_MWERKS __MWERKS__ # define LZO_INFO_CC "Metrowerks C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__MWERKS__) #elif (defined(__NDPC__) || defined(__NDPX__)) && defined(__i386) # define LZO_CC_NDPC 1 # define LZO_INFO_CC "Microway NDP C" # define LZO_INFO_CCVER "unknown" #elif defined(__PACIFIC__) # define LZO_CC_PACIFICC 1 # define LZO_INFO_CC "Pacific C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__PACIFIC__) #elif defined(__PGI) && defined(__PGIC__) && defined(__PGIC_MINOR__) # if defined(__PGIC_PATCHLEVEL__) # define LZO_CC_PGI (__PGIC__ * 0x10000L + (__PGIC_MINOR__-0) * 0x100 + (__PGIC_PATCHLEVEL__-0)) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__PGIC__) "." LZO_PP_MACRO_EXPAND(__PGIC_MINOR__) "." LZO_PP_MACRO_EXPAND(__PGIC_PATCHLEVEL__) # else # define LZO_CC_PGI (__PGIC__ * 0x10000L + (__PGIC_MINOR__-0) * 0x100) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__PGIC__) "." LZO_PP_MACRO_EXPAND(__PGIC_MINOR__) ".0" # endif # define LZO_INFO_CC "Portland Group PGI C" #elif defined(__PGI) && (defined(__linux__) || defined(__WIN32__)) # define LZO_CC_PGI 1 # define LZO_INFO_CC "Portland Group PGI C" # define LZO_INFO_CCVER "unknown" #elif defined(__PUREC__) && defined(__TOS__) # define LZO_CC_PUREC 1 # define LZO_INFO_CC "Pure C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__PUREC__) #elif defined(__SC__) && defined(__ZTC__) # define LZO_CC_SYMANTECC 1 # define LZO_INFO_CC "Symantec C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__SC__) #elif defined(__SUNPRO_C) # define LZO_INFO_CC "SunPro C" # if ((__SUNPRO_C-0) > 0) # define LZO_CC_SUNPROC __SUNPRO_C # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__SUNPRO_C) # else # define LZO_CC_SUNPROC 1 # define LZO_INFO_CCVER "unknown" # endif #elif defined(__SUNPRO_CC) # define LZO_INFO_CC "SunPro C" # if ((__SUNPRO_CC-0) > 0) # define LZO_CC_SUNPROC __SUNPRO_CC # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__SUNPRO_CC) # else # define LZO_CC_SUNPROC 1 # define LZO_INFO_CCVER "unknown" # endif #elif defined(__TINYC__) # define LZO_CC_TINYC 1 # define LZO_INFO_CC "Tiny C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__TINYC__) #elif defined(__TSC__) # define LZO_CC_TOPSPEEDC 1 # define LZO_INFO_CC "TopSpeed C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__TSC__) #elif defined(__WATCOMC__) # define LZO_CC_WATCOMC 1 # define LZO_INFO_CC "Watcom C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__WATCOMC__) #elif defined(__TURBOC__) # define LZO_CC_TURBOC 1 # define LZO_INFO_CC "Turbo C" # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__TURBOC__) #elif defined(__ZTC__) # define LZO_CC_ZORTECHC 1 # define LZO_INFO_CC "Zortech C" # if ((__ZTC__-0) == 0x310) # define LZO_INFO_CCVER "0x310" # else # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(__ZTC__) # endif #elif defined(__GNUC__) && defined(__VERSION__) # if defined(__GNUC_MINOR__) && defined(__GNUC_PATCHLEVEL__) # define LZO_CC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100 + (__GNUC_PATCHLEVEL__-0)) # elif defined(__GNUC_MINOR__) # define LZO_CC_GNUC (__GNUC__ * 0x10000L + (__GNUC_MINOR__-0) * 0x100) # else # define LZO_CC_GNUC (__GNUC__ * 0x10000L) # endif # define LZO_INFO_CC "gcc" # define LZO_INFO_CCVER __VERSION__ #elif defined(_MSC_VER) && ((_MSC_VER-0) > 0) # define LZO_CC_MSC _MSC_VER # define LZO_INFO_CC "Microsoft C" # if defined(_MSC_FULL_VER) # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(_MSC_VER) "." LZO_PP_MACRO_EXPAND(_MSC_FULL_VER) # else # define LZO_INFO_CCVER LZO_PP_MACRO_EXPAND(_MSC_VER) # endif #else # define LZO_CC_UNKNOWN 1 # define LZO_INFO_CC "unknown" # define LZO_INFO_CCVER "unknown" #endif #if (LZO_CC_GNUC) && defined(__OPEN64__) # if defined(__OPENCC__) && defined(__OPENCC_MINOR__) && defined(__OPENCC_PATCHLEVEL__) # define LZO_CC_OPEN64 (__OPENCC__ * 0x10000L + (__OPENCC_MINOR__-0) * 0x100 + (__OPENCC_PATCHLEVEL__-0)) # define LZO_CC_OPEN64_GNUC LZO_CC_GNUC # endif #endif #if (LZO_CC_GNUC) && defined(__PCC__) # if defined(__PCC__) && defined(__PCC_MINOR__) && defined(__PCC_MINORMINOR__) # define LZO_CC_PCC (__PCC__ * 0x10000L + (__PCC_MINOR__-0) * 0x100 + (__PCC_MINORMINOR__-0)) # define LZO_CC_PCC_GNUC LZO_CC_GNUC # endif #endif #if 0 && (LZO_CC_MSC && (_MSC_VER >= 1200)) && !defined(_MSC_FULL_VER) # error "LZO_CC_MSC: _MSC_FULL_VER is not defined" #endif #if !defined(__LZO_ARCH_OVERRIDE) && !(LZO_ARCH_GENERIC) && defined(_CRAY) # if (UINT_MAX > LZO_0xffffffffL) && defined(_CRAY) # if defined(_CRAYMPP) || defined(_CRAYT3D) || defined(_CRAYT3E) # define LZO_ARCH_CRAY_MPP 1 # elif defined(_CRAY1) # define LZO_ARCH_CRAY_PVP 1 # endif # endif #endif #if !defined(__LZO_ARCH_OVERRIDE) #if (LZO_ARCH_GENERIC) # define LZO_INFO_ARCH "generic" #elif (LZO_OS_DOS16 || LZO_OS_OS216 || LZO_OS_WIN16) # define LZO_ARCH_I086 1 # define LZO_INFO_ARCH "i086" #elif defined(__aarch64__) || defined(_M_ARM64) # define LZO_ARCH_ARM64 1 # define LZO_INFO_ARCH "arm64" #elif defined(__alpha__) || defined(__alpha) || defined(_M_ALPHA) # define LZO_ARCH_ALPHA 1 # define LZO_INFO_ARCH "alpha" #elif (LZO_ARCH_CRAY_MPP) && (defined(_CRAYT3D) || defined(_CRAYT3E)) # define LZO_ARCH_ALPHA 1 # define LZO_INFO_ARCH "alpha" #elif defined(__amd64__) || defined(__x86_64__) || defined(_M_AMD64) # define LZO_ARCH_AMD64 1 # define LZO_INFO_ARCH "amd64" #elif defined(__arm__) || defined(_M_ARM) # define LZO_ARCH_ARM 1 # define LZO_INFO_ARCH "arm" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICCARM__) # define LZO_ARCH_ARM 1 # define LZO_INFO_ARCH "arm" #elif (UINT_MAX <= LZO_0xffffL) && defined(__AVR__) # define LZO_ARCH_AVR 1 # define LZO_INFO_ARCH "avr" #elif defined(__avr32__) || defined(__AVR32__) # define LZO_ARCH_AVR32 1 # define LZO_INFO_ARCH "avr32" #elif defined(__bfin__) # define LZO_ARCH_BLACKFIN 1 # define LZO_INFO_ARCH "blackfin" #elif (UINT_MAX == LZO_0xffffL) && defined(__C166__) # define LZO_ARCH_C166 1 # define LZO_INFO_ARCH "c166" #elif defined(__cris__) # define LZO_ARCH_CRIS 1 # define LZO_INFO_ARCH "cris" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICCEZ80__) # define LZO_ARCH_EZ80 1 # define LZO_INFO_ARCH "ez80" #elif defined(__H8300__) || defined(__H8300H__) || defined(__H8300S__) || defined(__H8300SX__) # define LZO_ARCH_H8300 1 # define LZO_INFO_ARCH "h8300" #elif defined(__hppa__) || defined(__hppa) # define LZO_ARCH_HPPA 1 # define LZO_INFO_ARCH "hppa" #elif defined(__386__) || defined(__i386__) || defined(__i386) || defined(_M_IX86) || defined(_M_I386) # define LZO_ARCH_I386 1 # define LZO_ARCH_IA32 1 # define LZO_INFO_ARCH "i386" #elif (LZO_CC_ZORTECHC && defined(__I86__)) # define LZO_ARCH_I386 1 # define LZO_ARCH_IA32 1 # define LZO_INFO_ARCH "i386" #elif (LZO_OS_DOS32 && LZO_CC_HIGHC) && defined(_I386) # define LZO_ARCH_I386 1 # define LZO_ARCH_IA32 1 # define LZO_INFO_ARCH "i386" #elif defined(__ia64__) || defined(__ia64) || defined(_M_IA64) # define LZO_ARCH_IA64 1 # define LZO_INFO_ARCH "ia64" #elif (UINT_MAX == LZO_0xffffL) && defined(__m32c__) # define LZO_ARCH_M16C 1 # define LZO_INFO_ARCH "m16c" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICCM16C__) # define LZO_ARCH_M16C 1 # define LZO_INFO_ARCH "m16c" #elif defined(__m32r__) # define LZO_ARCH_M32R 1 # define LZO_INFO_ARCH "m32r" #elif (LZO_OS_TOS) || defined(__m68k__) || defined(__m68000__) || defined(__mc68000__) || defined(__mc68020__) || defined(_M_M68K) # define LZO_ARCH_M68K 1 # define LZO_INFO_ARCH "m68k" #elif (UINT_MAX == LZO_0xffffL) && defined(__C251__) # define LZO_ARCH_MCS251 1 # define LZO_INFO_ARCH "mcs251" #elif (UINT_MAX == LZO_0xffffL) && defined(__C51__) # define LZO_ARCH_MCS51 1 # define LZO_INFO_ARCH "mcs51" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICC8051__) # define LZO_ARCH_MCS51 1 # define LZO_INFO_ARCH "mcs51" #elif defined(__mips__) || defined(__mips) || defined(_MIPS_ARCH) || defined(_M_MRX000) # define LZO_ARCH_MIPS 1 # define LZO_INFO_ARCH "mips" #elif (UINT_MAX == LZO_0xffffL) && defined(__MSP430__) # define LZO_ARCH_MSP430 1 # define LZO_INFO_ARCH "msp430" #elif defined(__IAR_SYSTEMS_ICC__) && defined(__ICC430__) # define LZO_ARCH_MSP430 1 # define LZO_INFO_ARCH "msp430" #elif defined(__powerpc__) || defined(__powerpc) || defined(__ppc__) || defined(__PPC__) || defined(_M_PPC) || defined(_ARCH_PPC) || defined(_ARCH_PWR) # define LZO_ARCH_POWERPC 1 # define LZO_INFO_ARCH "powerpc" #elif defined(__powerpc64__) || defined(__powerpc64) || defined(__ppc64__) || defined(__PPC64__) # define LZO_ARCH_POWERPC 1 # define LZO_INFO_ARCH "powerpc" #elif defined(__powerpc64le__) || defined(__powerpc64le) || defined(__ppc64le__) || defined(__PPC64LE__) # define LZO_ARCH_POWERPC 1 # define LZO_INFO_ARCH "powerpc" #elif defined(__riscv) # define LZO_ARCH_RISCV 1 # define LZO_INFO_ARCH "riscv" #elif defined(__s390__) || defined(__s390) || defined(__s390x__) || defined(__s390x) # define LZO_ARCH_S390 1 # define LZO_INFO_ARCH "s390" #elif defined(__sh__) || defined(_M_SH) # define LZO_ARCH_SH 1 # define LZO_INFO_ARCH "sh" #elif defined(__sparc__) || defined(__sparc) || defined(__sparcv8) # define LZO_ARCH_SPARC 1 # define LZO_INFO_ARCH "sparc" #elif defined(__SPU__) # define LZO_ARCH_SPU 1 # define LZO_INFO_ARCH "spu" #elif (UINT_MAX == LZO_0xffffL) && defined(__z80) # define LZO_ARCH_Z80 1 # define LZO_INFO_ARCH "z80" #elif (LZO_ARCH_CRAY_PVP) # if defined(_CRAYSV1) # define LZO_ARCH_CRAY_SV1 1 # define LZO_INFO_ARCH "cray_sv1" # elif (_ADDR64) # define LZO_ARCH_CRAY_T90 1 # define LZO_INFO_ARCH "cray_t90" # elif (_ADDR32) # define LZO_ARCH_CRAY_YMP 1 # define LZO_INFO_ARCH "cray_ymp" # else # define LZO_ARCH_CRAY_XMP 1 # define LZO_INFO_ARCH "cray_xmp" # endif #else # define LZO_ARCH_UNKNOWN 1 # define LZO_INFO_ARCH "unknown" #endif #endif #if !defined(LZO_ARCH_ARM_THUMB2) #if (LZO_ARCH_ARM) # if defined(__thumb__) || defined(__thumb) || defined(_M_THUMB) # if defined(__thumb2__) # define LZO_ARCH_ARM_THUMB2 1 # elif 1 && defined(__TARGET_ARCH_THUMB) && ((__TARGET_ARCH_THUMB)+0 >= 4) # define LZO_ARCH_ARM_THUMB2 1 # elif 1 && defined(_MSC_VER) && defined(_M_THUMB) && ((_M_THUMB)+0 >= 7) # define LZO_ARCH_ARM_THUMB2 1 # endif # endif #endif #endif #if (LZO_ARCH_ARM_THUMB2) # undef LZO_INFO_ARCH # define LZO_INFO_ARCH "arm_thumb2" #endif #if 1 && (LZO_ARCH_UNKNOWN) && (LZO_OS_DOS32 || LZO_OS_OS2) # error "FIXME - missing define for CPU architecture" #endif #if 1 && (LZO_ARCH_UNKNOWN) && (LZO_OS_WIN32) # error "FIXME - missing LZO_OS_WIN32 define for CPU architecture" #endif #if 1 && (LZO_ARCH_UNKNOWN) && (LZO_OS_WIN64) # error "FIXME - missing LZO_OS_WIN64 define for CPU architecture" #endif #if (LZO_OS_OS216 || LZO_OS_WIN16) # define LZO_ARCH_I086PM 1 #elif 1 && (LZO_OS_DOS16 && defined(BLX286)) # define LZO_ARCH_I086PM 1 #elif 1 && (LZO_OS_DOS16 && defined(DOSX286)) # define LZO_ARCH_I086PM 1 #elif 1 && (LZO_OS_DOS16 && LZO_CC_BORLANDC && defined(__DPMI16__)) # define LZO_ARCH_I086PM 1 #endif #if (LZO_ARCH_AMD64 && !LZO_ARCH_X64) # define LZO_ARCH_X64 1 #elif (!LZO_ARCH_AMD64 && LZO_ARCH_X64) && defined(__LZO_ARCH_OVERRIDE) # define LZO_ARCH_AMD64 1 #endif #if (LZO_ARCH_ARM64 && !LZO_ARCH_AARCH64) # define LZO_ARCH_AARCH64 1 #elif (!LZO_ARCH_ARM64 && LZO_ARCH_AARCH64) && defined(__LZO_ARCH_OVERRIDE) # define LZO_ARCH_ARM64 1 #endif #if (LZO_ARCH_I386 && !LZO_ARCH_X86) # define LZO_ARCH_X86 1 #elif (!LZO_ARCH_I386 && LZO_ARCH_X86) && defined(__LZO_ARCH_OVERRIDE) # define LZO_ARCH_I386 1 #endif #if (LZO_ARCH_AMD64 && !LZO_ARCH_X64) || (!LZO_ARCH_AMD64 && LZO_ARCH_X64) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM64 && !LZO_ARCH_AARCH64) || (!LZO_ARCH_ARM64 && LZO_ARCH_AARCH64) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_I386 && !LZO_ARCH_X86) || (!LZO_ARCH_I386 && LZO_ARCH_X86) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM_THUMB1 && !LZO_ARCH_ARM) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM_THUMB2 && !LZO_ARCH_ARM) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM_THUMB1 && LZO_ARCH_ARM_THUMB2) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_I086PM && !LZO_ARCH_I086) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_I086) # if (UINT_MAX != LZO_0xffffL) # error "unexpected configuration - check your compiler defines" # endif # if (ULONG_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif #endif #if (LZO_ARCH_I386) # if (UINT_MAX != LZO_0xffffL) && defined(__i386_int16__) # error "unexpected configuration - check your compiler defines" # endif # if (UINT_MAX != LZO_0xffffffffL) && !defined(__i386_int16__) # error "unexpected configuration - check your compiler defines" # endif # if (ULONG_MAX != LZO_0xffffffffL) # error "unexpected configuration - check your compiler defines" # endif #endif #if (LZO_ARCH_AMD64 || LZO_ARCH_I386) # if !defined(LZO_TARGET_FEATURE_SSE2) # if defined(__SSE2__) # define LZO_TARGET_FEATURE_SSE2 1 # elif defined(_MSC_VER) && (defined(_M_IX86_FP) && ((_M_IX86_FP)+0 >= 2)) # define LZO_TARGET_FEATURE_SSE2 1 # elif (LZO_CC_INTELC_MSC || LZO_CC_MSC) && defined(_M_AMD64) # define LZO_TARGET_FEATURE_SSE2 1 # endif # endif # if !defined(LZO_TARGET_FEATURE_SSSE3) # if (LZO_TARGET_FEATURE_SSE2) # if defined(__SSSE3__) # define LZO_TARGET_FEATURE_SSSE3 1 # elif defined(_MSC_VER) && defined(__AVX__) # define LZO_TARGET_FEATURE_SSSE3 1 # endif # endif # endif # if !defined(LZO_TARGET_FEATURE_SSE4_2) # if (LZO_TARGET_FEATURE_SSSE3) # if defined(__SSE4_2__) # define LZO_TARGET_FEATURE_SSE4_2 1 # endif # endif # endif # if !defined(LZO_TARGET_FEATURE_AVX) # if (LZO_TARGET_FEATURE_SSSE3) # if defined(__AVX__) # define LZO_TARGET_FEATURE_AVX 1 # endif # endif # endif # if !defined(LZO_TARGET_FEATURE_AVX2) # if (LZO_TARGET_FEATURE_AVX) # if defined(__AVX2__) # define LZO_TARGET_FEATURE_AVX2 1 # endif # endif # endif #endif #if (LZO_TARGET_FEATURE_SSSE3 && !(LZO_TARGET_FEATURE_SSE2)) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_TARGET_FEATURE_SSE4_2 && !(LZO_TARGET_FEATURE_SSSE3)) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_TARGET_FEATURE_AVX && !(LZO_TARGET_FEATURE_SSSE3)) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_TARGET_FEATURE_AVX2 && !(LZO_TARGET_FEATURE_AVX)) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ARCH_ARM) # if !defined(LZO_TARGET_FEATURE_NEON) # if defined(__ARM_NEON) && ((__ARM_NEON)+0) # define LZO_TARGET_FEATURE_NEON 1 # elif 1 && defined(__ARM_NEON__) && ((__ARM_NEON__)+0) # define LZO_TARGET_FEATURE_NEON 1 # elif 1 && defined(__TARGET_FEATURE_NEON) && ((__TARGET_FEATURE_NEON)+0) # define LZO_TARGET_FEATURE_NEON 1 # endif # endif #elif (LZO_ARCH_ARM64) # if !defined(LZO_TARGET_FEATURE_NEON) # if 1 # define LZO_TARGET_FEATURE_NEON 1 # endif # endif #endif #if 0 #elif !defined(__LZO_MM_OVERRIDE) #if (LZO_ARCH_I086) #if (UINT_MAX != LZO_0xffffL) # error "unexpected configuration - check your compiler defines" #endif #if defined(__TINY__) || defined(M_I86TM) || defined(_M_I86TM) # define LZO_MM_TINY 1 #elif defined(__HUGE__) || defined(_HUGE_) || defined(M_I86HM) || defined(_M_I86HM) # define LZO_MM_HUGE 1 #elif defined(__SMALL__) || defined(M_I86SM) || defined(_M_I86SM) || defined(SMALL_MODEL) # define LZO_MM_SMALL 1 #elif defined(__MEDIUM__) || defined(M_I86MM) || defined(_M_I86MM) # define LZO_MM_MEDIUM 1 #elif defined(__COMPACT__) || defined(M_I86CM) || defined(_M_I86CM) # define LZO_MM_COMPACT 1 #elif defined(__LARGE__) || defined(M_I86LM) || defined(_M_I86LM) || defined(LARGE_MODEL) # define LZO_MM_LARGE 1 #elif (LZO_CC_AZTECC) # if defined(_LARGE_CODE) && defined(_LARGE_DATA) # define LZO_MM_LARGE 1 # elif defined(_LARGE_CODE) # define LZO_MM_MEDIUM 1 # elif defined(_LARGE_DATA) # define LZO_MM_COMPACT 1 # else # define LZO_MM_SMALL 1 # endif #elif (LZO_CC_ZORTECHC && defined(__VCM__)) # define LZO_MM_LARGE 1 #else # error "unknown LZO_ARCH_I086 memory model" #endif #if (LZO_OS_DOS16 || LZO_OS_OS216 || LZO_OS_WIN16) #define LZO_HAVE_MM_HUGE_PTR 1 #define LZO_HAVE_MM_HUGE_ARRAY 1 #if (LZO_MM_TINY) # undef LZO_HAVE_MM_HUGE_ARRAY #endif #if (LZO_CC_AZTECC || LZO_CC_PACIFICC || LZO_CC_ZORTECHC) # undef LZO_HAVE_MM_HUGE_PTR # undef LZO_HAVE_MM_HUGE_ARRAY #elif (LZO_CC_DMC || LZO_CC_SYMANTECC) # undef LZO_HAVE_MM_HUGE_ARRAY #elif (LZO_CC_MSC && defined(_QC)) # undef LZO_HAVE_MM_HUGE_ARRAY # if (_MSC_VER < 600) # undef LZO_HAVE_MM_HUGE_PTR # endif #elif (LZO_CC_TURBOC && (__TURBOC__ < 0x0295)) # undef LZO_HAVE_MM_HUGE_ARRAY #endif #if (LZO_ARCH_I086PM) && !(LZO_HAVE_MM_HUGE_PTR) # if (LZO_OS_DOS16) # error "unexpected configuration - check your compiler defines" # elif (LZO_CC_ZORTECHC) # else # error "unexpected configuration - check your compiler defines" # endif #endif #if defined(__cplusplus) extern "C" { #endif #if (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0200)) extern void __near __cdecl _AHSHIFT(void); # define LZO_MM_AHSHIFT ((unsigned) _AHSHIFT) #elif (LZO_CC_DMC || LZO_CC_SYMANTECC || LZO_CC_ZORTECHC) extern void __near __cdecl _AHSHIFT(void); # define LZO_MM_AHSHIFT ((unsigned) _AHSHIFT) #elif (LZO_CC_MSC || LZO_CC_TOPSPEEDC) extern void __near __cdecl _AHSHIFT(void); # define LZO_MM_AHSHIFT ((unsigned) _AHSHIFT) #elif (LZO_CC_TURBOC && (__TURBOC__ >= 0x0295)) extern void __near __cdecl _AHSHIFT(void); # define LZO_MM_AHSHIFT ((unsigned) _AHSHIFT) #elif ((LZO_CC_AZTECC || LZO_CC_PACIFICC || LZO_CC_TURBOC) && LZO_OS_DOS16) # define LZO_MM_AHSHIFT 12 #elif (LZO_CC_WATCOMC) extern unsigned char _HShift; # define LZO_MM_AHSHIFT ((unsigned) _HShift) #else # error "FIXME - implement LZO_MM_AHSHIFT" #endif #if defined(__cplusplus) } #endif #endif #elif (LZO_ARCH_C166) #if !defined(__MODEL__) # error "FIXME - LZO_ARCH_C166 __MODEL__" #elif ((__MODEL__) == 0) # define LZO_MM_SMALL 1 #elif ((__MODEL__) == 1) # define LZO_MM_SMALL 1 #elif ((__MODEL__) == 2) # define LZO_MM_LARGE 1 #elif ((__MODEL__) == 3) # define LZO_MM_TINY 1 #elif ((__MODEL__) == 4) # define LZO_MM_XTINY 1 #elif ((__MODEL__) == 5) # define LZO_MM_XSMALL 1 #else # error "FIXME - LZO_ARCH_C166 __MODEL__" #endif #elif (LZO_ARCH_MCS251) #if !defined(__MODEL__) # error "FIXME - LZO_ARCH_MCS251 __MODEL__" #elif ((__MODEL__) == 0) # define LZO_MM_SMALL 1 #elif ((__MODEL__) == 2) # define LZO_MM_LARGE 1 #elif ((__MODEL__) == 3) # define LZO_MM_TINY 1 #elif ((__MODEL__) == 4) # define LZO_MM_XTINY 1 #elif ((__MODEL__) == 5) # define LZO_MM_XSMALL 1 #else # error "FIXME - LZO_ARCH_MCS251 __MODEL__" #endif #elif (LZO_ARCH_MCS51) #if !defined(__MODEL__) # error "FIXME - LZO_ARCH_MCS51 __MODEL__" #elif ((__MODEL__) == 1) # define LZO_MM_SMALL 1 #elif ((__MODEL__) == 2) # define LZO_MM_LARGE 1 #elif ((__MODEL__) == 3) # define LZO_MM_TINY 1 #elif ((__MODEL__) == 4) # define LZO_MM_XTINY 1 #elif ((__MODEL__) == 5) # define LZO_MM_XSMALL 1 #else # error "FIXME - LZO_ARCH_MCS51 __MODEL__" #endif #elif (LZO_ARCH_CRAY_PVP) # define LZO_MM_PVP 1 #else # define LZO_MM_FLAT 1 #endif #if (LZO_MM_COMPACT) # define LZO_INFO_MM "compact" #elif (LZO_MM_FLAT) # define LZO_INFO_MM "flat" #elif (LZO_MM_HUGE) # define LZO_INFO_MM "huge" #elif (LZO_MM_LARGE) # define LZO_INFO_MM "large" #elif (LZO_MM_MEDIUM) # define LZO_INFO_MM "medium" #elif (LZO_MM_PVP) # define LZO_INFO_MM "pvp" #elif (LZO_MM_SMALL) # define LZO_INFO_MM "small" #elif (LZO_MM_TINY) # define LZO_INFO_MM "tiny" #else # error "unknown memory model" #endif #endif #if !defined(__lzo_gnuc_extension__) #if (LZO_CC_GNUC >= 0x020800ul) # define __lzo_gnuc_extension__ __extension__ #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_gnuc_extension__ __extension__ #elif (LZO_CC_IBMC >= 600) # define __lzo_gnuc_extension__ __extension__ #endif #endif #if !defined(__lzo_gnuc_extension__) # define __lzo_gnuc_extension__ /*empty*/ #endif #if !defined(lzo_has_builtin) #if (LZO_CC_CLANG) && defined(__has_builtin) # define lzo_has_builtin __has_builtin #endif #endif #if !defined(lzo_has_builtin) # define lzo_has_builtin(x) 0 #endif #if !defined(lzo_has_attribute) #if (LZO_CC_CLANG) && defined(__has_attribute) # define lzo_has_attribute __has_attribute #endif #endif #if !defined(lzo_has_attribute) # define lzo_has_attribute(x) 0 #endif #if !defined(lzo_has_declspec_attribute) #if (LZO_CC_CLANG) && defined(__has_declspec_attribute) # define lzo_has_declspec_attribute __has_declspec_attribute #endif #endif #if !defined(lzo_has_declspec_attribute) # define lzo_has_declspec_attribute(x) 0 #endif #if !defined(lzo_has_feature) #if (LZO_CC_CLANG) && defined(__has_feature) # define lzo_has_feature __has_feature #endif #endif #if !defined(lzo_has_feature) # define lzo_has_feature(x) 0 #endif #if !defined(lzo_has_extension) #if (LZO_CC_CLANG) && defined(__has_extension) # define lzo_has_extension __has_extension #elif (LZO_CC_CLANG) && defined(__has_feature) # define lzo_has_extension __has_feature #endif #endif #if !defined(lzo_has_extension) # define lzo_has_extension(x) 0 #endif #if !defined(LZO_CFG_USE_NEW_STYLE_CASTS) && defined(__cplusplus) && 0 # if (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020800ul)) # define LZO_CFG_USE_NEW_STYLE_CASTS 0 # elif (LZO_CC_INTELC && (__INTEL_COMPILER < 1200)) # define LZO_CFG_USE_NEW_STYLE_CASTS 0 # else # define LZO_CFG_USE_NEW_STYLE_CASTS 1 # endif #endif #if !defined(LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_CFG_USE_NEW_STYLE_CASTS 0 #endif #if !defined(__cplusplus) # if defined(LZO_CFG_USE_NEW_STYLE_CASTS) # undef LZO_CFG_USE_NEW_STYLE_CASTS # endif # define LZO_CFG_USE_NEW_STYLE_CASTS 0 #endif #if !defined(LZO_REINTERPRET_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_REINTERPRET_CAST(t,e) (reinterpret_cast (e)) # endif #endif #if !defined(LZO_REINTERPRET_CAST) # define LZO_REINTERPRET_CAST(t,e) ((t) (e)) #endif #if !defined(LZO_STATIC_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_STATIC_CAST(t,e) (static_cast (e)) # endif #endif #if !defined(LZO_STATIC_CAST) # define LZO_STATIC_CAST(t,e) ((t) (e)) #endif #if !defined(LZO_STATIC_CAST2) # define LZO_STATIC_CAST2(t1,t2,e) LZO_STATIC_CAST(t1, LZO_STATIC_CAST(t2, e)) #endif #if !defined(LZO_UNCONST_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_UNCONST_CAST(t,e) (const_cast (e)) # elif (LZO_HAVE_MM_HUGE_PTR) # define LZO_UNCONST_CAST(t,e) ((t) (e)) # elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNCONST_CAST(t,e) ((t) ((void *) ((lzo_uintptr_t) ((const void *) (e))))) # endif #endif #if !defined(LZO_UNCONST_CAST) # define LZO_UNCONST_CAST(t,e) ((t) ((void *) ((const void *) (e)))) #endif #if !defined(LZO_UNCONST_VOLATILE_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_UNCONST_VOLATILE_CAST(t,e) (const_cast (e)) # elif (LZO_HAVE_MM_HUGE_PTR) # define LZO_UNCONST_VOLATILE_CAST(t,e) ((t) (e)) # elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNCONST_VOLATILE_CAST(t,e) ((t) ((volatile void *) ((lzo_uintptr_t) ((volatile const void *) (e))))) # endif #endif #if !defined(LZO_UNCONST_VOLATILE_CAST) # define LZO_UNCONST_VOLATILE_CAST(t,e) ((t) ((volatile void *) ((volatile const void *) (e)))) #endif #if !defined(LZO_UNVOLATILE_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_UNVOLATILE_CAST(t,e) (const_cast (e)) # elif (LZO_HAVE_MM_HUGE_PTR) # define LZO_UNVOLATILE_CAST(t,e) ((t) (e)) # elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNVOLATILE_CAST(t,e) ((t) ((void *) ((lzo_uintptr_t) ((volatile void *) (e))))) # endif #endif #if !defined(LZO_UNVOLATILE_CAST) # define LZO_UNVOLATILE_CAST(t,e) ((t) ((void *) ((volatile void *) (e)))) #endif #if !defined(LZO_UNVOLATILE_CONST_CAST) # if (LZO_CFG_USE_NEW_STYLE_CASTS) # define LZO_UNVOLATILE_CONST_CAST(t,e) (const_cast (e)) # elif (LZO_HAVE_MM_HUGE_PTR) # define LZO_UNVOLATILE_CONST_CAST(t,e) ((t) (e)) # elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNVOLATILE_CONST_CAST(t,e) ((t) ((const void *) ((lzo_uintptr_t) ((volatile const void *) (e))))) # endif #endif #if !defined(LZO_UNVOLATILE_CONST_CAST) # define LZO_UNVOLATILE_CONST_CAST(t,e) ((t) ((const void *) ((volatile const void *) (e)))) #endif #if !defined(LZO_PCAST) # if (LZO_HAVE_MM_HUGE_PTR) # define LZO_PCAST(t,e) ((t) (e)) # endif #endif #if !defined(LZO_PCAST) # define LZO_PCAST(t,e) LZO_STATIC_CAST(t, LZO_STATIC_CAST(void *, e)) #endif #if !defined(LZO_CCAST) # if (LZO_HAVE_MM_HUGE_PTR) # define LZO_CCAST(t,e) ((t) (e)) # endif #endif #if !defined(LZO_CCAST) # define LZO_CCAST(t,e) LZO_STATIC_CAST(t, LZO_STATIC_CAST(const void *, e)) #endif #if !defined(LZO_ICONV) # define LZO_ICONV(t,e) LZO_STATIC_CAST(t, e) #endif #if !defined(LZO_ICAST) # define LZO_ICAST(t,e) LZO_STATIC_CAST(t, e) #endif #if !defined(LZO_ITRUNC) # define LZO_ITRUNC(t,e) LZO_STATIC_CAST(t, e) #endif #if !defined(__lzo_cte) # if (LZO_CC_MSC || LZO_CC_WATCOMC) # define __lzo_cte(e) ((void)0,(e)) # elif 1 # define __lzo_cte(e) ((void)0,(e)) # endif #endif #if !defined(__lzo_cte) # define __lzo_cte(e) (e) #endif #if !defined(LZO_BLOCK_BEGIN) # define LZO_BLOCK_BEGIN do { # define LZO_BLOCK_END } while __lzo_cte(0) #endif #if !defined(LZO_UNUSED) # if (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0600)) # define LZO_UNUSED(var) ((void) &var) # elif (LZO_CC_BORLANDC || LZO_CC_HIGHC || LZO_CC_NDPC || LZO_CC_PELLESC || LZO_CC_TURBOC) # define LZO_UNUSED(var) if (&var) ; else # elif (LZO_CC_CLANG && (LZO_CC_CLANG >= 0x030200ul)) # define LZO_UNUSED(var) ((void) &var) # elif (LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_UNUSED(var) ((void) var) # elif (LZO_CC_MSC && (_MSC_VER < 900)) # define LZO_UNUSED(var) if (&var) ; else # elif (LZO_CC_KEILC) # define LZO_UNUSED(var) {extern int lzo_unused__[1-2*!(sizeof(var)>0)]; (void)lzo_unused__;} # elif (LZO_CC_PACIFICC) # define LZO_UNUSED(var) ((void) sizeof(var)) # elif (LZO_CC_WATCOMC) && defined(__cplusplus) # define LZO_UNUSED(var) ((void) var) # else # define LZO_UNUSED(var) ((void) &var) # endif #endif #if !defined(LZO_UNUSED_RESULT) # define LZO_UNUSED_RESULT(var) LZO_UNUSED(var) #endif #if !defined(LZO_UNUSED_FUNC) # if (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0600)) # define LZO_UNUSED_FUNC(func) ((void) func) # elif (LZO_CC_BORLANDC || LZO_CC_NDPC || LZO_CC_TURBOC) # define LZO_UNUSED_FUNC(func) if (func) ; else # elif (LZO_CC_CLANG || LZO_CC_LLVM) # define LZO_UNUSED_FUNC(func) ((void) &func) # elif (LZO_CC_MSC && (_MSC_VER < 900)) # define LZO_UNUSED_FUNC(func) if (func) ; else # elif (LZO_CC_MSC) # define LZO_UNUSED_FUNC(func) ((void) &func) # elif (LZO_CC_KEILC || LZO_CC_PELLESC) # define LZO_UNUSED_FUNC(func) {extern int lzo_unused__[1-2*!(sizeof((int)func)>0)]; (void)lzo_unused__;} # else # define LZO_UNUSED_FUNC(func) ((void) func) # endif #endif #if !defined(LZO_UNUSED_LABEL) # if (LZO_CC_CLANG >= 0x020800ul) # define LZO_UNUSED_LABEL(l) (__lzo_gnuc_extension__ ((void) ((const void *) &&l))) # elif (LZO_CC_ARMCC || LZO_CC_CLANG || LZO_CC_INTELC || LZO_CC_WATCOMC) # define LZO_UNUSED_LABEL(l) if __lzo_cte(0) goto l # else # define LZO_UNUSED_LABEL(l) switch (0) case 1:goto l # endif #endif #if !defined(LZO_DEFINE_UNINITIALIZED_VAR) # if 0 # define LZO_DEFINE_UNINITIALIZED_VAR(type,var,init) type var # elif 0 && (LZO_CC_GNUC) # define LZO_DEFINE_UNINITIALIZED_VAR(type,var,init) type var = var # else # define LZO_DEFINE_UNINITIALIZED_VAR(type,var,init) type var = init # endif #endif #if !defined(__lzo_inline) #if (LZO_CC_TURBOC && (__TURBOC__ <= 0x0295)) #elif defined(__cplusplus) # define __lzo_inline inline #elif defined(__STDC_VERSION__) && (__STDC_VERSION__-0 >= 199901L) # define __lzo_inline inline #elif (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0550)) # define __lzo_inline __inline #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CILLY || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || LZO_CC_PGI) # define __lzo_inline __inline__ #elif (LZO_CC_DMC) # define __lzo_inline __inline #elif (LZO_CC_GHS) # define __lzo_inline __inline__ #elif (LZO_CC_IBMC >= 600) # define __lzo_inline __inline__ #elif (LZO_CC_INTELC) # define __lzo_inline __inline #elif (LZO_CC_MWERKS && (__MWERKS__ >= 0x2405)) # define __lzo_inline __inline #elif (LZO_CC_MSC && (_MSC_VER >= 900)) # define __lzo_inline __inline #elif (LZO_CC_SUNPROC >= 0x5100) # define __lzo_inline __inline__ #endif #endif #if defined(__lzo_inline) # ifndef __lzo_HAVE_inline # define __lzo_HAVE_inline 1 # endif #else # define __lzo_inline /*empty*/ #endif #if !defined(__lzo_forceinline) #if (LZO_CC_GNUC >= 0x030200ul) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_IBMC >= 700) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 450)) # define __lzo_forceinline __forceinline #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 800)) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_MSC && (_MSC_VER >= 1200)) # define __lzo_forceinline __forceinline #elif (LZO_CC_PGI >= 0x0d0a00ul) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #elif (LZO_CC_SUNPROC >= 0x5100) # define __lzo_forceinline __inline__ __attribute__((__always_inline__)) #endif #endif #if defined(__lzo_forceinline) # ifndef __lzo_HAVE_forceinline # define __lzo_HAVE_forceinline 1 # endif #else # define __lzo_forceinline __lzo_inline #endif #if !defined(__lzo_noinline) #if 1 && (LZO_ARCH_I386) && (LZO_CC_GNUC >= 0x040000ul) && (LZO_CC_GNUC < 0x040003ul) # define __lzo_noinline __attribute__((__noinline__,__used__)) #elif (LZO_CC_GNUC >= 0x030200ul) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_IBMC >= 700) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 600)) # define __lzo_noinline __declspec(noinline) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 800)) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_noinline __declspec(noinline) #elif (LZO_CC_MWERKS && (__MWERKS__ >= 0x3200) && (LZO_OS_WIN32 || LZO_OS_WIN64)) # if defined(__cplusplus) # else # define __lzo_noinline __declspec(noinline) # endif #elif (LZO_CC_PGI >= 0x0d0a00ul) # define __lzo_noinline __attribute__((__noinline__)) #elif (LZO_CC_SUNPROC >= 0x5100) # define __lzo_noinline __attribute__((__noinline__)) #endif #endif #if defined(__lzo_noinline) # ifndef __lzo_HAVE_noinline # define __lzo_HAVE_noinline 1 # endif #else # define __lzo_noinline /*empty*/ #endif #if (__lzo_HAVE_forceinline || __lzo_HAVE_noinline) && !(__lzo_HAVE_inline) # error "unexpected configuration - check your compiler defines" #endif #if !defined(__lzo_static_inline) #if (LZO_CC_IBMC) # define __lzo_static_inline __lzo_gnuc_extension__ static __lzo_inline #endif #endif #if !defined(__lzo_static_inline) # define __lzo_static_inline static __lzo_inline #endif #if !defined(__lzo_static_forceinline) #if (LZO_CC_IBMC) # define __lzo_static_forceinline __lzo_gnuc_extension__ static __lzo_forceinline #endif #endif #if !defined(__lzo_static_forceinline) # define __lzo_static_forceinline static __lzo_forceinline #endif #if !defined(__lzo_static_noinline) #if (LZO_CC_IBMC) # define __lzo_static_noinline __lzo_gnuc_extension__ static __lzo_noinline #endif #endif #if !defined(__lzo_static_noinline) # define __lzo_static_noinline static __lzo_noinline #endif #if !defined(__lzo_c99_extern_inline) #if defined(__GNUC_GNU_INLINE__) # define __lzo_c99_extern_inline __lzo_inline #elif defined(__GNUC_STDC_INLINE__) # define __lzo_c99_extern_inline extern __lzo_inline #elif defined(__STDC_VERSION__) && (__STDC_VERSION__-0 >= 199901L) # define __lzo_c99_extern_inline extern __lzo_inline #endif #if !defined(__lzo_c99_extern_inline) && (__lzo_HAVE_inline) # define __lzo_c99_extern_inline __lzo_inline #endif #endif #if defined(__lzo_c99_extern_inline) # ifndef __lzo_HAVE_c99_extern_inline # define __lzo_HAVE_c99_extern_inline 1 # endif #else # define __lzo_c99_extern_inline /*empty*/ #endif #if !defined(__lzo_may_alias) #if (LZO_CC_GNUC >= 0x030400ul) # define __lzo_may_alias __attribute__((__may_alias__)) #elif (LZO_CC_CLANG >= 0x020900ul) # define __lzo_may_alias __attribute__((__may_alias__)) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 1210)) && 0 # define __lzo_may_alias __attribute__((__may_alias__)) #elif (LZO_CC_PGI >= 0x0d0a00ul) && 0 # define __lzo_may_alias __attribute__((__may_alias__)) #endif #endif #if defined(__lzo_may_alias) # ifndef __lzo_HAVE_may_alias # define __lzo_HAVE_may_alias 1 # endif #else # define __lzo_may_alias /*empty*/ #endif #if !defined(__lzo_noreturn) #if (LZO_CC_GNUC >= 0x020700ul) # define __lzo_noreturn __attribute__((__noreturn__)) #elif (LZO_CC_IBMC >= 700) # define __lzo_noreturn __attribute__((__noreturn__)) #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 450)) # define __lzo_noreturn __declspec(noreturn) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 600)) # define __lzo_noreturn __attribute__((__noreturn__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_noreturn __attribute__((__noreturn__)) #elif (LZO_CC_MSC && (_MSC_VER >= 1200)) # define __lzo_noreturn __declspec(noreturn) #elif (LZO_CC_PGI >= 0x0d0a00ul) # define __lzo_noreturn __attribute__((__noreturn__)) #endif #endif #if defined(__lzo_noreturn) # ifndef __lzo_HAVE_noreturn # define __lzo_HAVE_noreturn 1 # endif #else # define __lzo_noreturn /*empty*/ #endif #if !defined(__lzo_nothrow) #if (LZO_CC_GNUC >= 0x030300ul) # define __lzo_nothrow __attribute__((__nothrow__)) #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 450)) && defined(__cplusplus) # define __lzo_nothrow __declspec(nothrow) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 900)) # define __lzo_nothrow __attribute__((__nothrow__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_nothrow __attribute__((__nothrow__)) #elif (LZO_CC_MSC && (_MSC_VER >= 1200)) && defined(__cplusplus) # define __lzo_nothrow __declspec(nothrow) #endif #endif #if defined(__lzo_nothrow) # ifndef __lzo_HAVE_nothrow # define __lzo_HAVE_nothrow 1 # endif #else # define __lzo_nothrow /*empty*/ #endif #if !defined(__lzo_restrict) #if (LZO_CC_GNUC >= 0x030400ul) # define __lzo_restrict __restrict__ #elif (LZO_CC_IBMC >= 800) && !defined(__cplusplus) # define __lzo_restrict __restrict__ #elif (LZO_CC_IBMC >= 1210) # define __lzo_restrict __restrict__ #elif (LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 600)) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 600)) # define __lzo_restrict __restrict__ #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM) # define __lzo_restrict __restrict__ #elif (LZO_CC_MSC && (_MSC_VER >= 1400)) # define __lzo_restrict __restrict #elif (LZO_CC_PGI >= 0x0d0a00ul) # define __lzo_restrict __restrict__ #endif #endif #if defined(__lzo_restrict) # ifndef __lzo_HAVE_restrict # define __lzo_HAVE_restrict 1 # endif #else # define __lzo_restrict /*empty*/ #endif #if !defined(__lzo_alignof) #if (LZO_CC_ARMCC || LZO_CC_CILLY || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || LZO_CC_PGI) # define __lzo_alignof(e) __alignof__(e) #elif (LZO_CC_GHS) && !defined(__cplusplus) # define __lzo_alignof(e) __alignof__(e) #elif (LZO_CC_IBMC >= 600) # define __lzo_alignof(e) (__lzo_gnuc_extension__ __alignof__(e)) #elif (LZO_CC_INTELC && (__INTEL_COMPILER >= 700)) # define __lzo_alignof(e) __alignof__(e) #elif (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_alignof(e) __alignof(e) #elif (LZO_CC_SUNPROC >= 0x5100) # define __lzo_alignof(e) __alignof__(e) #endif #endif #if defined(__lzo_alignof) # ifndef __lzo_HAVE_alignof # define __lzo_HAVE_alignof 1 # endif #endif #if !defined(__lzo_struct_packed) #if (LZO_CC_CLANG && (LZO_CC_CLANG < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020700ul)) #elif (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_PCC && (LZO_CC_PCC < 0x010100ul)) #elif (LZO_CC_SUNPROC && (LZO_CC_SUNPROC < 0x5110)) && !defined(__cplusplus) #elif (LZO_CC_GNUC >= 0x030400ul) && !(LZO_CC_PCC_GNUC) && (LZO_ARCH_AMD64 || LZO_ARCH_I386) # define __lzo_struct_packed(s) struct s { # define __lzo_struct_packed_end() } __attribute__((__gcc_struct__,__packed__)); # define __lzo_struct_packed_ma_end() } __lzo_may_alias __attribute__((__gcc_struct__,__packed__)); #elif (LZO_CC_ARMCC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || (LZO_CC_PGI >= 0x0d0a00ul) || (LZO_CC_SUNPROC >= 0x5100)) # define __lzo_struct_packed(s) struct s { # define __lzo_struct_packed_end() } __attribute__((__packed__)); # define __lzo_struct_packed_ma_end() } __lzo_may_alias __attribute__((__packed__)); #elif (LZO_CC_IBMC >= 700) # define __lzo_struct_packed(s) __lzo_gnuc_extension__ struct s { # define __lzo_struct_packed_end() } __attribute__((__packed__)); # define __lzo_struct_packed_ma_end() } __lzo_may_alias __attribute__((__packed__)); #elif (LZO_CC_INTELC_MSC) || (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_struct_packed(s) __pragma(pack(push,1)) struct s { # define __lzo_struct_packed_end() } __pragma(pack(pop)); #elif (LZO_CC_WATCOMC && (__WATCOMC__ >= 900)) # define __lzo_struct_packed(s) _Packed struct s { # define __lzo_struct_packed_end() }; #endif #endif #if defined(__lzo_struct_packed) && !defined(__lzo_struct_packed_ma) # define __lzo_struct_packed_ma(s) __lzo_struct_packed(s) #endif #if defined(__lzo_struct_packed_end) && !defined(__lzo_struct_packed_ma_end) # define __lzo_struct_packed_ma_end() __lzo_struct_packed_end() #endif #if !defined(__lzo_byte_struct) #if defined(__lzo_struct_packed) # define __lzo_byte_struct(s,n) __lzo_struct_packed(s) unsigned char a[n]; __lzo_struct_packed_end() # define __lzo_byte_struct_ma(s,n) __lzo_struct_packed_ma(s) unsigned char a[n]; __lzo_struct_packed_ma_end() #elif (LZO_CC_CILLY || LZO_CC_CLANG || LZO_CC_PGI || (LZO_CC_SUNPROC >= 0x5100)) # define __lzo_byte_struct(s,n) struct s { unsigned char a[n]; } __attribute__((__packed__)); # define __lzo_byte_struct_ma(s,n) struct s { unsigned char a[n]; } __lzo_may_alias __attribute__((__packed__)); #endif #endif #if defined(__lzo_byte_struct) && !defined(__lzo_byte_struct_ma) # define __lzo_byte_struct_ma(s,n) __lzo_byte_struct(s,n) #endif #if !defined(__lzo_struct_align16) && (__lzo_HAVE_alignof) #if (LZO_CC_GNUC && (LZO_CC_GNUC < 0x030000ul)) #elif (LZO_CC_CLANG && (LZO_CC_CLANG < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_CILLY || LZO_CC_PCC) #elif (LZO_CC_INTELC_MSC) || (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_struct_align16(s) struct __declspec(align(16)) s { # define __lzo_struct_align16_end() }; # define __lzo_struct_align32(s) struct __declspec(align(32)) s { # define __lzo_struct_align32_end() }; # define __lzo_struct_align64(s) struct __declspec(align(64)) s { # define __lzo_struct_align64_end() }; #elif (LZO_CC_ARMCC || LZO_CC_CLANG || LZO_CC_GNUC || (LZO_CC_IBMC >= 700) || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_struct_align16(s) struct s { # define __lzo_struct_align16_end() } __attribute__((__aligned__(16))); # define __lzo_struct_align32(s) struct s { # define __lzo_struct_align32_end() } __attribute__((__aligned__(32))); # define __lzo_struct_align64(s) struct s { # define __lzo_struct_align64_end() } __attribute__((__aligned__(64))); #endif #endif #if !defined(__lzo_union_um) #if (LZO_CC_CLANG && (LZO_CC_CLANG < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020700ul)) #elif (LZO_CC_GNUC && (LZO_CC_GNUC < 0x020800ul)) && defined(__cplusplus) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER < 810)) #elif (LZO_CC_PCC && (LZO_CC_PCC < 0x010100ul)) #elif (LZO_CC_SUNPROC && (LZO_CC_SUNPROC < 0x5110)) && !defined(__cplusplus) #elif (LZO_CC_ARMCC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || (LZO_CC_PGI >= 0x0d0a00ul) || (LZO_CC_SUNPROC >= 0x5100)) # define __lzo_union_am(s) union s { # define __lzo_union_am_end() } __lzo_may_alias; # define __lzo_union_um(s) union s { # define __lzo_union_um_end() } __lzo_may_alias __attribute__((__packed__)); #elif (LZO_CC_IBMC >= 700) # define __lzo_union_am(s) __lzo_gnuc_extension__ union s { # define __lzo_union_am_end() } __lzo_may_alias; # define __lzo_union_um(s) __lzo_gnuc_extension__ union s { # define __lzo_union_um_end() } __lzo_may_alias __attribute__((__packed__)); #elif (LZO_CC_INTELC_MSC) || (LZO_CC_MSC && (_MSC_VER >= 1300)) # define __lzo_union_um(s) __pragma(pack(push,1)) union s { # define __lzo_union_um_end() } __pragma(pack(pop)); #elif (LZO_CC_WATCOMC && (__WATCOMC__ >= 900)) # define __lzo_union_um(s) _Packed union s { # define __lzo_union_um_end() }; #endif #endif #if !defined(__lzo_union_am) # define __lzo_union_am(s) union s { # define __lzo_union_am_end() }; #endif #if !defined(__lzo_constructor) #if (LZO_CC_GNUC >= 0x030400ul) # define __lzo_constructor __attribute__((__constructor__,__used__)) #elif (LZO_CC_GNUC >= 0x020700ul) # define __lzo_constructor __attribute__((__constructor__)) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 800)) # define __lzo_constructor __attribute__((__constructor__,__used__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_constructor __attribute__((__constructor__)) #endif #endif #if defined(__lzo_constructor) # ifndef __lzo_HAVE_constructor # define __lzo_HAVE_constructor 1 # endif #endif #if !defined(__lzo_destructor) #if (LZO_CC_GNUC >= 0x030400ul) # define __lzo_destructor __attribute__((__destructor__,__used__)) #elif (LZO_CC_GNUC >= 0x020700ul) # define __lzo_destructor __attribute__((__destructor__)) #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 800)) # define __lzo_destructor __attribute__((__destructor__,__used__)) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_destructor __attribute__((__destructor__)) #endif #endif #if defined(__lzo_destructor) # ifndef __lzo_HAVE_destructor # define __lzo_HAVE_destructor 1 # endif #endif #if (__lzo_HAVE_destructor) && !(__lzo_HAVE_constructor) # error "unexpected configuration - check your compiler defines" #endif #if !defined(__lzo_likely) && !defined(__lzo_unlikely) #if (LZO_CC_GNUC >= 0x030200ul) # define __lzo_likely(e) (__builtin_expect(!!(e),1)) # define __lzo_unlikely(e) (__builtin_expect(!!(e),0)) #elif (LZO_CC_IBMC >= 1010) # define __lzo_likely(e) (__builtin_expect(!!(e),1)) # define __lzo_unlikely(e) (__builtin_expect(!!(e),0)) #elif (LZO_CC_INTELC && (__INTEL_COMPILER >= 800)) # define __lzo_likely(e) (__builtin_expect(!!(e),1)) # define __lzo_unlikely(e) (__builtin_expect(!!(e),0)) #elif (LZO_CC_CLANG && LZO_CC_CLANG_C2) #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define __lzo_likely(e) (__builtin_expect(!!(e),1)) # define __lzo_unlikely(e) (__builtin_expect(!!(e),0)) #endif #endif #if defined(__lzo_likely) # ifndef __lzo_HAVE_likely # define __lzo_HAVE_likely 1 # endif #else # define __lzo_likely(e) (e) #endif #if defined(__lzo_very_likely) # ifndef __lzo_HAVE_very_likely # define __lzo_HAVE_very_likely 1 # endif #else # define __lzo_very_likely(e) __lzo_likely(e) #endif #if defined(__lzo_unlikely) # ifndef __lzo_HAVE_unlikely # define __lzo_HAVE_unlikely 1 # endif #else # define __lzo_unlikely(e) (e) #endif #if defined(__lzo_very_unlikely) # ifndef __lzo_HAVE_very_unlikely # define __lzo_HAVE_very_unlikely 1 # endif #else # define __lzo_very_unlikely(e) __lzo_unlikely(e) #endif #if !defined(__lzo_loop_forever) # if (LZO_CC_IBMC) # define __lzo_loop_forever() LZO_BLOCK_BEGIN for (;;) { ; } LZO_BLOCK_END # else # define __lzo_loop_forever() do { ; } while __lzo_cte(1) # endif #endif #if !defined(__lzo_unreachable) #if (LZO_CC_CLANG && (LZO_CC_CLANG >= 0x020800ul)) && lzo_has_builtin(__builtin_unreachable) # define __lzo_unreachable() __builtin_unreachable(); #elif (LZO_CC_GNUC >= 0x040500ul) # define __lzo_unreachable() __builtin_unreachable(); #elif (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 1300)) && 1 # define __lzo_unreachable() __builtin_unreachable(); #endif #endif #if defined(__lzo_unreachable) # ifndef __lzo_HAVE_unreachable # define __lzo_HAVE_unreachable 1 # endif #else # if 0 # define __lzo_unreachable() ((void)0); # else # define __lzo_unreachable() __lzo_loop_forever(); # endif #endif #if !defined(lzo_unused_funcs_impl) # if 1 && (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || (LZO_CC_GNUC >= 0x020700ul) || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || LZO_CC_PGI) # define lzo_unused_funcs_impl(r,f) static r __attribute__((__unused__)) f # elif 1 && (LZO_CC_BORLANDC || LZO_CC_GNUC) # define lzo_unused_funcs_impl(r,f) static r f # else # define lzo_unused_funcs_impl(r,f) __lzo_static_forceinline r f # endif #endif #ifndef __LZO_CTA_NAME #if (LZO_CFG_USE_COUNTER) # define __LZO_CTA_NAME(a) LZO_PP_ECONCAT2(a,__COUNTER__) #else # define __LZO_CTA_NAME(a) LZO_PP_ECONCAT2(a,__LINE__) #endif #endif #if !defined(LZO_COMPILE_TIME_ASSERT_HEADER) # if (LZO_CC_AZTECC || LZO_CC_ZORTECHC) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN extern int __LZO_CTA_NAME(lzo_cta__)[1-!(e)]; LZO_EXTERN_C_END # elif (LZO_CC_DMC || LZO_CC_SYMANTECC) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN extern int __LZO_CTA_NAME(lzo_cta__)[1u-2*!(e)]; LZO_EXTERN_C_END # elif (LZO_CC_TURBOC && (__TURBOC__ == 0x0295)) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN extern int __LZO_CTA_NAME(lzo_cta__)[1-!(e)]; LZO_EXTERN_C_END # elif (LZO_CC_CLANG && (LZO_CC_CLANG < 0x020900ul)) && defined(__cplusplus) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN int __LZO_CTA_NAME(lzo_cta_f__)(int [1-2*!(e)]); LZO_EXTERN_C_END # elif (LZO_CC_GNUC) && defined(__CHECKER__) && defined(__SPARSE_CHECKER__) # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN enum {__LZO_CTA_NAME(lzo_cta_e__)=1/!!(e)} __attribute__((__unused__)); LZO_EXTERN_C_END # else # define LZO_COMPILE_TIME_ASSERT_HEADER(e) LZO_EXTERN_C_BEGIN extern int __LZO_CTA_NAME(lzo_cta__)[1-2*!(e)]; LZO_EXTERN_C_END # endif #endif #if !defined(LZO_COMPILE_TIME_ASSERT) # if (LZO_CC_AZTECC) # define LZO_COMPILE_TIME_ASSERT(e) {typedef int __LZO_CTA_NAME(lzo_cta_t__)[1-!(e)];} # elif (LZO_CC_CLANG && (LZO_CC_CLANG >= 0x030000ul)) # define LZO_COMPILE_TIME_ASSERT(e) {typedef int __LZO_CTA_NAME(lzo_cta_t__)[1-2*!(e)] __attribute__((__unused__));} # elif (LZO_CC_DMC || LZO_CC_PACIFICC || LZO_CC_SYMANTECC || LZO_CC_ZORTECHC) # define LZO_COMPILE_TIME_ASSERT(e) switch(0) case 1:case !(e):break; # elif (LZO_CC_GNUC) && defined(__CHECKER__) && defined(__SPARSE_CHECKER__) # define LZO_COMPILE_TIME_ASSERT(e) {(void) (0/!!(e));} # elif (LZO_CC_GNUC >= 0x040700ul) && (LZO_CFG_USE_COUNTER) && defined(__cplusplus) # define LZO_COMPILE_TIME_ASSERT(e) {enum {__LZO_CTA_NAME(lzo_cta_e__)=1/!!(e)} __attribute__((__unused__));} # elif (LZO_CC_GNUC >= 0x040700ul) # define LZO_COMPILE_TIME_ASSERT(e) {typedef int __LZO_CTA_NAME(lzo_cta_t__)[1-2*!(e)] __attribute__((__unused__));} # elif (LZO_CC_MSC && (_MSC_VER < 900)) # define LZO_COMPILE_TIME_ASSERT(e) switch(0) case 1:case !(e):break; # elif (LZO_CC_TURBOC && (__TURBOC__ == 0x0295)) # define LZO_COMPILE_TIME_ASSERT(e) switch(0) case 1:case !(e):break; # else # define LZO_COMPILE_TIME_ASSERT(e) {typedef int __LZO_CTA_NAME(lzo_cta_t__)[1-2*!(e)];} # endif #endif #if (LZO_LANG_ASSEMBLER) # undef LZO_COMPILE_TIME_ASSERT_HEADER # define LZO_COMPILE_TIME_ASSERT_HEADER(e) /*empty*/ #else LZO_COMPILE_TIME_ASSERT_HEADER(1 == 1) #if defined(__cplusplus) extern "C" { LZO_COMPILE_TIME_ASSERT_HEADER(2 == 2) } #endif LZO_COMPILE_TIME_ASSERT_HEADER(3 == 3) #endif #if (LZO_ARCH_I086 || LZO_ARCH_I386) && (LZO_OS_DOS16 || LZO_OS_DOS32 || LZO_OS_OS2 || LZO_OS_OS216 || LZO_OS_WIN16 || LZO_OS_WIN32 || LZO_OS_WIN64) # if (LZO_CC_GNUC || LZO_CC_HIGHC || LZO_CC_NDPC || LZO_CC_PACIFICC) # elif (LZO_CC_DMC || LZO_CC_SYMANTECC || LZO_CC_ZORTECHC) # define __lzo_cdecl __cdecl # define __lzo_cdecl_atexit /*empty*/ # define __lzo_cdecl_main __cdecl # if (LZO_OS_OS2 && (LZO_CC_DMC || LZO_CC_SYMANTECC)) # define __lzo_cdecl_qsort __pascal # elif (LZO_OS_OS2 && (LZO_CC_ZORTECHC)) # define __lzo_cdecl_qsort _stdcall # else # define __lzo_cdecl_qsort __cdecl # endif # elif (LZO_CC_WATCOMC) # define __lzo_cdecl __cdecl # else # define __lzo_cdecl __cdecl # define __lzo_cdecl_atexit __cdecl # define __lzo_cdecl_main __cdecl # define __lzo_cdecl_qsort __cdecl # endif # if (LZO_CC_GNUC || LZO_CC_HIGHC || LZO_CC_NDPC || LZO_CC_PACIFICC || LZO_CC_WATCOMC) # elif (LZO_OS_OS2 && (LZO_CC_DMC || LZO_CC_SYMANTECC)) # define __lzo_cdecl_sighandler __pascal # elif (LZO_OS_OS2 && (LZO_CC_ZORTECHC)) # define __lzo_cdecl_sighandler _stdcall # elif (LZO_CC_MSC && (_MSC_VER >= 1400)) && defined(_M_CEE_PURE) # define __lzo_cdecl_sighandler __clrcall # elif (LZO_CC_MSC && (_MSC_VER >= 600 && _MSC_VER < 700)) # if defined(_DLL) # define __lzo_cdecl_sighandler _far _cdecl _loadds # elif defined(_MT) # define __lzo_cdecl_sighandler _far _cdecl # else # define __lzo_cdecl_sighandler _cdecl # endif # else # define __lzo_cdecl_sighandler __cdecl # endif #elif (LZO_ARCH_I386) && (LZO_CC_WATCOMC) # define __lzo_cdecl __cdecl #elif (LZO_ARCH_M68K && LZO_OS_TOS && (LZO_CC_PUREC || LZO_CC_TURBOC)) # define __lzo_cdecl cdecl #endif #if !defined(__lzo_cdecl) # define __lzo_cdecl /*empty*/ #endif #if !defined(__lzo_cdecl_atexit) # define __lzo_cdecl_atexit /*empty*/ #endif #if !defined(__lzo_cdecl_main) # define __lzo_cdecl_main /*empty*/ #endif #if !defined(__lzo_cdecl_qsort) # define __lzo_cdecl_qsort /*empty*/ #endif #if !defined(__lzo_cdecl_sighandler) # define __lzo_cdecl_sighandler /*empty*/ #endif #if !defined(__lzo_cdecl_va) # define __lzo_cdecl_va __lzo_cdecl #endif #if !(LZO_CFG_NO_WINDOWS_H) #if !defined(LZO_HAVE_WINDOWS_H) #if (LZO_OS_CYGWIN || (LZO_OS_EMX && defined(__RSXNT__)) || LZO_OS_WIN32 || LZO_OS_WIN64) # if (LZO_CC_WATCOMC && (__WATCOMC__ < 1000)) # elif ((LZO_OS_WIN32 && defined(__PW32__)) && (LZO_CC_GNUC && (LZO_CC_GNUC < 0x030000ul))) # elif ((LZO_OS_CYGWIN || defined(__MINGW32__)) && (LZO_CC_GNUC && (LZO_CC_GNUC < 0x025f00ul))) # else # define LZO_HAVE_WINDOWS_H 1 # endif #endif #endif #endif #define LZO_SIZEOF_CHAR 1 #ifndef LZO_SIZEOF_SHORT #if defined(SIZEOF_SHORT) # define LZO_SIZEOF_SHORT (SIZEOF_SHORT) #elif defined(__SIZEOF_SHORT__) # define LZO_SIZEOF_SHORT (__SIZEOF_SHORT__) #endif #endif #ifndef LZO_SIZEOF_INT #if defined(SIZEOF_INT) # define LZO_SIZEOF_INT (SIZEOF_INT) #elif defined(__SIZEOF_INT__) # define LZO_SIZEOF_INT (__SIZEOF_INT__) #endif #endif #ifndef LZO_SIZEOF_LONG #if defined(SIZEOF_LONG) # define LZO_SIZEOF_LONG (SIZEOF_LONG) #elif defined(__SIZEOF_LONG__) # define LZO_SIZEOF_LONG (__SIZEOF_LONG__) #endif #endif #ifndef LZO_SIZEOF_LONG_LONG #if defined(SIZEOF_LONG_LONG) # define LZO_SIZEOF_LONG_LONG (SIZEOF_LONG_LONG) #elif defined(__SIZEOF_LONG_LONG__) # define LZO_SIZEOF_LONG_LONG (__SIZEOF_LONG_LONG__) #endif #endif #ifndef LZO_SIZEOF___INT16 #if defined(SIZEOF___INT16) # define LZO_SIZEOF___INT16 (SIZEOF___INT16) #endif #endif #ifndef LZO_SIZEOF___INT32 #if defined(SIZEOF___INT32) # define LZO_SIZEOF___INT32 (SIZEOF___INT32) #endif #endif #ifndef LZO_SIZEOF___INT64 #if defined(SIZEOF___INT64) # define LZO_SIZEOF___INT64 (SIZEOF___INT64) #endif #endif #ifndef LZO_SIZEOF_VOID_P #if defined(SIZEOF_VOID_P) # define LZO_SIZEOF_VOID_P (SIZEOF_VOID_P) #elif defined(__SIZEOF_POINTER__) # define LZO_SIZEOF_VOID_P (__SIZEOF_POINTER__) #endif #endif #ifndef LZO_SIZEOF_SIZE_T #if defined(SIZEOF_SIZE_T) # define LZO_SIZEOF_SIZE_T (SIZEOF_SIZE_T) #elif defined(__SIZEOF_SIZE_T__) # define LZO_SIZEOF_SIZE_T (__SIZEOF_SIZE_T__) #endif #endif #ifndef LZO_SIZEOF_PTRDIFF_T #if defined(SIZEOF_PTRDIFF_T) # define LZO_SIZEOF_PTRDIFF_T (SIZEOF_PTRDIFF_T) #elif defined(__SIZEOF_PTRDIFF_T__) # define LZO_SIZEOF_PTRDIFF_T (__SIZEOF_PTRDIFF_T__) #endif #endif #define __LZO_LSR(x,b) (((x)+0ul) >> (b)) #if !defined(LZO_SIZEOF_SHORT) # if (LZO_ARCH_CRAY_PVP) # define LZO_SIZEOF_SHORT 8 # elif (USHRT_MAX == LZO_0xffffL) # define LZO_SIZEOF_SHORT 2 # elif (__LZO_LSR(USHRT_MAX,7) == 1) # define LZO_SIZEOF_SHORT 1 # elif (__LZO_LSR(USHRT_MAX,15) == 1) # define LZO_SIZEOF_SHORT 2 # elif (__LZO_LSR(USHRT_MAX,31) == 1) # define LZO_SIZEOF_SHORT 4 # elif (__LZO_LSR(USHRT_MAX,63) == 1) # define LZO_SIZEOF_SHORT 8 # elif (__LZO_LSR(USHRT_MAX,127) == 1) # define LZO_SIZEOF_SHORT 16 # else # error "LZO_SIZEOF_SHORT" # endif #endif LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_SHORT == sizeof(short)) #if !defined(LZO_SIZEOF_INT) # if (LZO_ARCH_CRAY_PVP) # define LZO_SIZEOF_INT 8 # elif (UINT_MAX == LZO_0xffffL) # define LZO_SIZEOF_INT 2 # elif (UINT_MAX == LZO_0xffffffffL) # define LZO_SIZEOF_INT 4 # elif (__LZO_LSR(UINT_MAX,7) == 1) # define LZO_SIZEOF_INT 1 # elif (__LZO_LSR(UINT_MAX,15) == 1) # define LZO_SIZEOF_INT 2 # elif (__LZO_LSR(UINT_MAX,31) == 1) # define LZO_SIZEOF_INT 4 # elif (__LZO_LSR(UINT_MAX,63) == 1) # define LZO_SIZEOF_INT 8 # elif (__LZO_LSR(UINT_MAX,127) == 1) # define LZO_SIZEOF_INT 16 # else # error "LZO_SIZEOF_INT" # endif #endif LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_INT == sizeof(int)) #if !defined(LZO_SIZEOF_LONG) # if (ULONG_MAX == LZO_0xffffffffL) # define LZO_SIZEOF_LONG 4 # elif (__LZO_LSR(ULONG_MAX,7) == 1) # define LZO_SIZEOF_LONG 1 # elif (__LZO_LSR(ULONG_MAX,15) == 1) # define LZO_SIZEOF_LONG 2 # elif (__LZO_LSR(ULONG_MAX,31) == 1) # define LZO_SIZEOF_LONG 4 # elif (__LZO_LSR(ULONG_MAX,39) == 1) # define LZO_SIZEOF_LONG 5 # elif (__LZO_LSR(ULONG_MAX,63) == 1) # define LZO_SIZEOF_LONG 8 # elif (__LZO_LSR(ULONG_MAX,127) == 1) # define LZO_SIZEOF_LONG 16 # else # error "LZO_SIZEOF_LONG" # endif #endif LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_LONG == sizeof(long)) #if !defined(LZO_SIZEOF_LONG_LONG) && !defined(LZO_SIZEOF___INT64) #if (LZO_SIZEOF_LONG > 0 && LZO_SIZEOF_LONG < 8) # if defined(__LONG_MAX__) && defined(__LONG_LONG_MAX__) # if (LZO_CC_GNUC >= 0x030300ul) # if ((__LONG_MAX__-0) == (__LONG_LONG_MAX__-0)) # define LZO_SIZEOF_LONG_LONG LZO_SIZEOF_LONG # elif (__LZO_LSR(__LONG_LONG_MAX__,30) == 1) # define LZO_SIZEOF_LONG_LONG 4 # endif # endif # endif #endif #endif #if !defined(LZO_SIZEOF_LONG_LONG) && !defined(LZO_SIZEOF___INT64) #if (LZO_SIZEOF_LONG > 0 && LZO_SIZEOF_LONG < 8) #if (LZO_ARCH_I086 && LZO_CC_DMC) #elif (LZO_CC_CILLY) && defined(__GNUC__) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE) # define LZO_SIZEOF_LONG_LONG 8 #elif ((LZO_OS_WIN32 || LZO_OS_WIN64 || defined(_WIN32)) && LZO_CC_MSC && (_MSC_VER >= 1400)) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_OS_WIN64 || defined(_WIN64)) # define LZO_SIZEOF___INT64 8 #elif (LZO_ARCH_I386 && (LZO_CC_DMC)) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_ARCH_I386 && (LZO_CC_SYMANTECC && (__SC__ >= 0x700))) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_ARCH_I386 && (LZO_CC_INTELC && defined(__linux__))) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_ARCH_I386 && (LZO_CC_MWERKS || LZO_CC_PELLESC || LZO_CC_PGI || LZO_CC_SUNPROC)) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_ARCH_I386 && (LZO_CC_INTELC || LZO_CC_MSC)) # define LZO_SIZEOF___INT64 8 #elif ((LZO_OS_WIN32 || defined(_WIN32)) && (LZO_CC_MSC)) # define LZO_SIZEOF___INT64 8 #elif (LZO_ARCH_I386 && (LZO_CC_BORLANDC && (__BORLANDC__ >= 0x0520))) # define LZO_SIZEOF___INT64 8 #elif (LZO_ARCH_I386 && (LZO_CC_WATCOMC && (__WATCOMC__ >= 1100))) # define LZO_SIZEOF___INT64 8 #elif (LZO_CC_GHS && defined(__LLONG_BIT) && ((__LLONG_BIT-0) == 64)) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_CC_WATCOMC && defined(_INTEGRAL_MAX_BITS) && ((_INTEGRAL_MAX_BITS-0) == 64)) # define LZO_SIZEOF___INT64 8 #elif (LZO_OS_OS400 || defined(__OS400__)) && defined(__LLP64_IFC__) # define LZO_SIZEOF_LONG_LONG 8 #elif (defined(__vms) || defined(__VMS)) && ((__INITIAL_POINTER_SIZE-0) == 64) # define LZO_SIZEOF_LONG_LONG 8 #elif (LZO_CC_SDCC) && (LZO_SIZEOF_INT == 2) #elif 1 && defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) # define LZO_SIZEOF_LONG_LONG 8 #endif #endif #endif #if defined(__cplusplus) && (LZO_CC_GNUC) # if (LZO_CC_GNUC < 0x020800ul) # undef LZO_SIZEOF_LONG_LONG # endif #endif #if (LZO_CFG_NO_LONG_LONG) # undef LZO_SIZEOF_LONG_LONG #elif defined(__NO_LONG_LONG) # undef LZO_SIZEOF_LONG_LONG #elif defined(_NO_LONGLONG) # undef LZO_SIZEOF_LONG_LONG #endif #if !defined(LZO_WORDSIZE) #if (LZO_ARCH_ALPHA) # define LZO_WORDSIZE 8 #elif (LZO_ARCH_AMD64) # define LZO_WORDSIZE 8 #elif (LZO_ARCH_ARM64) # define LZO_WORDSIZE 8 #elif (LZO_ARCH_AVR) # define LZO_WORDSIZE 1 #elif (LZO_ARCH_H8300) # if defined(__H8300H__) || defined(__H8300S__) || defined(__H8300SX__) # define LZO_WORDSIZE 4 # else # define LZO_WORDSIZE 2 # endif #elif (LZO_ARCH_I086) # define LZO_WORDSIZE 2 #elif (LZO_ARCH_IA64) # define LZO_WORDSIZE 8 #elif (LZO_ARCH_M16C) # define LZO_WORDSIZE 2 #elif (LZO_ARCH_SPU) # define LZO_WORDSIZE 4 #elif (LZO_ARCH_Z80) # define LZO_WORDSIZE 1 #elif (LZO_SIZEOF_LONG == 8) && ((defined(__mips__) && defined(__R5900__)) || defined(__MIPS_PSX2__)) # define LZO_WORDSIZE 8 #elif (LZO_OS_OS400 || defined(__OS400__)) # define LZO_WORDSIZE 8 #elif (defined(__vms) || defined(__VMS)) && (__INITIAL_POINTER_SIZE+0 == 64) # define LZO_WORDSIZE 8 #endif #endif #if !defined(LZO_SIZEOF_VOID_P) #if defined(__ILP32__) || defined(__ILP32) || defined(_ILP32) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(int) == 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long) == 4) # define LZO_SIZEOF_VOID_P 4 #elif defined(__ILP64__) || defined(__ILP64) || defined(_ILP64) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(int) == 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long) == 8) # define LZO_SIZEOF_VOID_P 8 #elif defined(__LLP64__) || defined(__LLP64) || defined(_LLP64) || defined(_WIN64) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long) == 4) # define LZO_SIZEOF_VOID_P 8 #elif defined(__LP64__) || defined(__LP64) || defined(_LP64) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long) == 8) # define LZO_SIZEOF_VOID_P 8 #elif (LZO_ARCH_AVR) # define LZO_SIZEOF_VOID_P 2 #elif (LZO_ARCH_C166 || LZO_ARCH_MCS51 || LZO_ARCH_MCS251 || LZO_ARCH_MSP430) # define LZO_SIZEOF_VOID_P 2 #elif (LZO_ARCH_H8300) # if defined(__H8300H__) || defined(__H8300S__) || defined(__H8300SX__) LZO_COMPILE_TIME_ASSERT_HEADER(LZO_WORDSIZE == 4) # if defined(__NORMAL_MODE__) # define LZO_SIZEOF_VOID_P 2 # else # define LZO_SIZEOF_VOID_P 4 # endif # else LZO_COMPILE_TIME_ASSERT_HEADER(LZO_WORDSIZE == 2) # define LZO_SIZEOF_VOID_P 2 # endif # if (LZO_CC_GNUC && (LZO_CC_GNUC < 0x040000ul)) && (LZO_SIZEOF_INT == 4) # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_INT # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_INT # endif #elif (LZO_ARCH_I086) # if (LZO_MM_TINY || LZO_MM_SMALL || LZO_MM_MEDIUM) # define LZO_SIZEOF_VOID_P 2 # elif (LZO_MM_COMPACT || LZO_MM_LARGE || LZO_MM_HUGE) # define LZO_SIZEOF_VOID_P 4 # else # error "invalid LZO_ARCH_I086 memory model" # endif #elif (LZO_ARCH_M16C) # if defined(__m32c_cpu__) || defined(__m32cm_cpu__) # define LZO_SIZEOF_VOID_P 4 # else # define LZO_SIZEOF_VOID_P 2 # endif #elif (LZO_ARCH_SPU) # define LZO_SIZEOF_VOID_P 4 #elif (LZO_ARCH_Z80) # define LZO_SIZEOF_VOID_P 2 #elif (LZO_SIZEOF_LONG == 8) && ((defined(__mips__) && defined(__R5900__)) || defined(__MIPS_PSX2__)) # define LZO_SIZEOF_VOID_P 4 #elif (LZO_OS_OS400 || defined(__OS400__)) # if defined(__LLP64_IFC__) # define LZO_SIZEOF_VOID_P 8 # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_LONG # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_LONG # else # define LZO_SIZEOF_VOID_P 16 # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_LONG # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_LONG # endif #elif (defined(__vms) || defined(__VMS)) && (__INITIAL_POINTER_SIZE+0 == 64) # define LZO_SIZEOF_VOID_P 8 # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_LONG # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_LONG #endif #endif #if !defined(LZO_SIZEOF_VOID_P) # define LZO_SIZEOF_VOID_P LZO_SIZEOF_LONG #endif LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_VOID_P == sizeof(void *)) #if !defined(LZO_SIZEOF_SIZE_T) #if (LZO_ARCH_I086 || LZO_ARCH_M16C) # define LZO_SIZEOF_SIZE_T 2 #endif #endif #if !defined(LZO_SIZEOF_SIZE_T) # define LZO_SIZEOF_SIZE_T LZO_SIZEOF_VOID_P #endif #if defined(offsetof) LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_SIZE_T == sizeof(size_t)) #endif #if !defined(LZO_SIZEOF_PTRDIFF_T) #if (LZO_ARCH_I086) # if (LZO_MM_TINY || LZO_MM_SMALL || LZO_MM_MEDIUM || LZO_MM_HUGE) # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_VOID_P # elif (LZO_MM_COMPACT || LZO_MM_LARGE) # if (LZO_CC_BORLANDC || LZO_CC_TURBOC) # define LZO_SIZEOF_PTRDIFF_T 4 # else # define LZO_SIZEOF_PTRDIFF_T 2 # endif # else # error "invalid LZO_ARCH_I086 memory model" # endif #endif #endif #if !defined(LZO_SIZEOF_PTRDIFF_T) # define LZO_SIZEOF_PTRDIFF_T LZO_SIZEOF_SIZE_T #endif #if defined(offsetof) LZO_COMPILE_TIME_ASSERT_HEADER(LZO_SIZEOF_PTRDIFF_T == sizeof(ptrdiff_t)) #endif #if !defined(LZO_WORDSIZE) # define LZO_WORDSIZE LZO_SIZEOF_VOID_P #endif #if (LZO_ABI_NEUTRAL_ENDIAN) # undef LZO_ABI_BIG_ENDIAN # undef LZO_ABI_LITTLE_ENDIAN #elif !(LZO_ABI_BIG_ENDIAN) && !(LZO_ABI_LITTLE_ENDIAN) #if (LZO_ARCH_ALPHA) && (LZO_ARCH_CRAY_MPP) # define LZO_ABI_BIG_ENDIAN 1 #elif (LZO_ARCH_IA64) && (LZO_OS_POSIX_LINUX || LZO_OS_WIN64) # define LZO_ABI_LITTLE_ENDIAN 1 #elif (LZO_ARCH_ALPHA || LZO_ARCH_AMD64 || LZO_ARCH_BLACKFIN || LZO_ARCH_CRIS || LZO_ARCH_I086 || LZO_ARCH_I386 || LZO_ARCH_MSP430 || LZO_ARCH_RISCV) # define LZO_ABI_LITTLE_ENDIAN 1 #elif (LZO_ARCH_AVR32 || LZO_ARCH_M68K || LZO_ARCH_S390 || LZO_ARCH_SPU) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && defined(__IAR_SYSTEMS_ICC__) && defined(__LITTLE_ENDIAN__) # if (__LITTLE_ENDIAN__ == 1) # define LZO_ABI_LITTLE_ENDIAN 1 # else # define LZO_ABI_BIG_ENDIAN 1 # endif #elif 1 && defined(__BIG_ENDIAN__) && !defined(__LITTLE_ENDIAN__) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && defined(__LITTLE_ENDIAN__) && !defined(__BIG_ENDIAN__) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM) && defined(__ARM_BIG_ENDIAN) && ((__ARM_BIG_ENDIAN)+0) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM) && defined(__ARMEB__) && !defined(__ARMEL__) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM) && defined(__ARMEL__) && !defined(__ARMEB__) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM) && defined(_MSC_VER) && defined(_WIN32) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM && LZO_CC_ARMCC_ARMCC) # if defined(__BIG_ENDIAN) && defined(__LITTLE_ENDIAN) # error "unexpected configuration - check your compiler defines" # elif defined(__BIG_ENDIAN) # define LZO_ABI_BIG_ENDIAN 1 # else # define LZO_ABI_LITTLE_ENDIAN 1 # endif # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM64) && defined(__ARM_BIG_ENDIAN) && ((__ARM_BIG_ENDIAN)+0) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM64) && defined(__AARCH64EB__) && !defined(__AARCH64EL__) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM64) && defined(__AARCH64EL__) && !defined(__AARCH64EB__) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_ARM64) && defined(_MSC_VER) && defined(_WIN32) # define LZO_ABI_LITTLE_ENDIAN 1 #elif 1 && (LZO_ARCH_MIPS) && defined(__MIPSEB__) && !defined(__MIPSEL__) # define LZO_ABI_BIG_ENDIAN 1 #elif 1 && (LZO_ARCH_MIPS) && defined(__MIPSEL__) && !defined(__MIPSEB__) # define LZO_ABI_LITTLE_ENDIAN 1 #endif #endif #if (LZO_ABI_BIG_ENDIAN) && (LZO_ABI_LITTLE_ENDIAN) # error "unexpected configuration - check your compiler defines" #endif #if (LZO_ABI_BIG_ENDIAN) # define LZO_INFO_ABI_ENDIAN "be" #elif (LZO_ABI_LITTLE_ENDIAN) # define LZO_INFO_ABI_ENDIAN "le" #elif (LZO_ABI_NEUTRAL_ENDIAN) # define LZO_INFO_ABI_ENDIAN "neutral" #endif #if (LZO_SIZEOF_INT == 1 && LZO_SIZEOF_LONG == 2 && LZO_SIZEOF_VOID_P == 2) # define LZO_ABI_I8LP16 1 # define LZO_INFO_ABI_PM "i8lp16" #elif (LZO_SIZEOF_INT == 2 && LZO_SIZEOF_LONG == 2 && LZO_SIZEOF_VOID_P == 2) # define LZO_ABI_ILP16 1 # define LZO_INFO_ABI_PM "ilp16" #elif (LZO_SIZEOF_INT == 2 && LZO_SIZEOF_LONG == 4 && LZO_SIZEOF_VOID_P == 4) # define LZO_ABI_LP32 1 # define LZO_INFO_ABI_PM "lp32" #elif (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_LONG == 4 && LZO_SIZEOF_VOID_P == 4) # define LZO_ABI_ILP32 1 # define LZO_INFO_ABI_PM "ilp32" #elif (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_LONG == 4 && LZO_SIZEOF_VOID_P == 8 && LZO_SIZEOF_SIZE_T == 8) # define LZO_ABI_LLP64 1 # define LZO_INFO_ABI_PM "llp64" #elif (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_LONG == 8 && LZO_SIZEOF_VOID_P == 8) # define LZO_ABI_LP64 1 # define LZO_INFO_ABI_PM "lp64" #elif (LZO_SIZEOF_INT == 8 && LZO_SIZEOF_LONG == 8 && LZO_SIZEOF_VOID_P == 8) # define LZO_ABI_ILP64 1 # define LZO_INFO_ABI_PM "ilp64" #elif (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_LONG == 8 && LZO_SIZEOF_VOID_P == 4) # define LZO_ABI_IP32L64 1 # define LZO_INFO_ABI_PM "ip32l64" #endif #if (LZO_SIZEOF_INT == 4 && LZO_SIZEOF_VOID_P == 4 && LZO_WORDSIZE == 8) # define LZO_ABI_IP32W64 1 # ifndef LZO_INFO_ABI_PM # define LZO_INFO_ABI_PM "ip32w64" # endif #endif #if 0 #elif !defined(__LZO_LIBC_OVERRIDE) #if (LZO_LIBC_NAKED) # define LZO_INFO_LIBC "naked" #elif (LZO_LIBC_FREESTANDING) # define LZO_INFO_LIBC "freestanding" #elif (LZO_LIBC_MOSTLY_FREESTANDING) # define LZO_INFO_LIBC "mfreestanding" #elif (LZO_LIBC_ISOC90) # define LZO_INFO_LIBC "isoc90" #elif (LZO_LIBC_ISOC99) # define LZO_INFO_LIBC "isoc99" #elif (LZO_CC_ARMCC_ARMCC) && defined(__ARMCLIB_VERSION) # define LZO_LIBC_ISOC90 1 # define LZO_INFO_LIBC "isoc90" #elif defined(__dietlibc__) # define LZO_LIBC_DIETLIBC 1 # define LZO_INFO_LIBC "dietlibc" #elif defined(_NEWLIB_VERSION) # define LZO_LIBC_NEWLIB 1 # define LZO_INFO_LIBC "newlib" #elif defined(__UCLIBC__) && defined(__UCLIBC_MAJOR__) && defined(__UCLIBC_MINOR__) # if defined(__UCLIBC_SUBLEVEL__) # define LZO_LIBC_UCLIBC (__UCLIBC_MAJOR__ * 0x10000L + (__UCLIBC_MINOR__-0) * 0x100 + (__UCLIBC_SUBLEVEL__-0)) # else # define LZO_LIBC_UCLIBC 0x00090bL # endif # define LZO_INFO_LIBC "uc" "libc" #elif defined(__GLIBC__) && defined(__GLIBC_MINOR__) # define LZO_LIBC_GLIBC (__GLIBC__ * 0x10000L + (__GLIBC_MINOR__-0) * 0x100) # define LZO_INFO_LIBC "glibc" #elif (LZO_CC_MWERKS) && defined(__MSL__) # define LZO_LIBC_MSL __MSL__ # define LZO_INFO_LIBC "msl" #elif 1 && defined(__IAR_SYSTEMS_ICC__) # define LZO_LIBC_ISOC90 1 # define LZO_INFO_LIBC "isoc90" #else # define LZO_LIBC_DEFAULT 1 # define LZO_INFO_LIBC "default" #endif #endif #if (LZO_ARCH_I386 && (LZO_OS_DOS32 || LZO_OS_WIN32) && (LZO_CC_DMC || LZO_CC_INTELC || LZO_CC_MSC || LZO_CC_PELLESC)) # define LZO_ASM_SYNTAX_MSC 1 #elif (LZO_OS_WIN64 && (LZO_CC_DMC || LZO_CC_INTELC || LZO_CC_MSC || LZO_CC_PELLESC)) #elif (LZO_ARCH_I386 && LZO_CC_GNUC && (LZO_CC_GNUC == 0x011f00ul)) #elif (LZO_ARCH_I386 && (LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_INTELC || LZO_CC_PATHSCALE)) # define LZO_ASM_SYNTAX_GNUC 1 #elif (LZO_ARCH_AMD64 && (LZO_CC_CLANG || LZO_CC_GNUC || LZO_CC_INTELC || LZO_CC_PATHSCALE)) # define LZO_ASM_SYNTAX_GNUC 1 #elif (LZO_CC_GNUC) # define LZO_ASM_SYNTAX_GNUC 1 #endif #if (LZO_ASM_SYNTAX_GNUC) #if (LZO_ARCH_I386 && LZO_CC_GNUC && (LZO_CC_GNUC < 0x020000ul)) # define __LZO_ASM_CLOBBER "ax" # define __LZO_ASM_CLOBBER_LIST_CC /*empty*/ # define __LZO_ASM_CLOBBER_LIST_CC_MEMORY /*empty*/ # define __LZO_ASM_CLOBBER_LIST_EMPTY /*empty*/ #elif (LZO_CC_INTELC && (__INTEL_COMPILER < 1000)) # define __LZO_ASM_CLOBBER "memory" # define __LZO_ASM_CLOBBER_LIST_CC /*empty*/ # define __LZO_ASM_CLOBBER_LIST_CC_MEMORY : "memory" # define __LZO_ASM_CLOBBER_LIST_EMPTY /*empty*/ #else # define __LZO_ASM_CLOBBER "cc", "memory" # define __LZO_ASM_CLOBBER_LIST_CC : "cc" # define __LZO_ASM_CLOBBER_LIST_CC_MEMORY : "cc", "memory" # define __LZO_ASM_CLOBBER_LIST_EMPTY /*empty*/ #endif #endif #if (LZO_ARCH_ALPHA) # define LZO_OPT_AVOID_UINT_INDEX 1 #elif (LZO_ARCH_AMD64) # define LZO_OPT_AVOID_INT_INDEX 1 # define LZO_OPT_AVOID_UINT_INDEX 1 # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif #elif (LZO_ARCH_ARM) # if defined(__ARM_FEATURE_UNALIGNED) # if ((__ARM_FEATURE_UNALIGNED)+0) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # endif # elif 1 && (LZO_ARCH_ARM_THUMB2) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # elif 1 && defined(__ARM_ARCH) && ((__ARM_ARCH)+0 >= 7) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # elif 1 && defined(__TARGET_ARCH_ARM) && ((__TARGET_ARCH_ARM)+0 >= 7) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # elif 1 && defined(__TARGET_ARCH_ARM) && ((__TARGET_ARCH_ARM)+0 >= 6) && (defined(__TARGET_PROFILE_A) || defined(__TARGET_PROFILE_R)) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # elif 1 && defined(_MSC_VER) && defined(_M_ARM) && ((_M_ARM)+0 >= 7) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # endif #elif (LZO_ARCH_ARM64) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif #elif (LZO_ARCH_CRIS) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif #elif (LZO_ARCH_I386) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif #elif (LZO_ARCH_IA64) # define LZO_OPT_AVOID_INT_INDEX 1 # define LZO_OPT_AVOID_UINT_INDEX 1 # define LZO_OPT_PREFER_POSTINC 1 #elif (LZO_ARCH_M68K) # define LZO_OPT_PREFER_POSTINC 1 # define LZO_OPT_PREFER_PREDEC 1 # if defined(__mc68020__) && !defined(__mcoldfire__) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # endif #elif (LZO_ARCH_MIPS) # define LZO_OPT_AVOID_UINT_INDEX 1 #elif (LZO_ARCH_POWERPC) # define LZO_OPT_PREFER_PREINC 1 # define LZO_OPT_PREFER_PREDEC 1 # if (LZO_ABI_BIG_ENDIAN) || (LZO_WORDSIZE == 8) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # if (LZO_WORDSIZE == 8) # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif # endif # endif #elif (LZO_ARCH_RISCV) # define LZO_OPT_AVOID_UINT_INDEX 1 # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # if (LZO_WORDSIZE == 8) # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif # endif #elif (LZO_ARCH_S390) # ifndef LZO_OPT_UNALIGNED16 # define LZO_OPT_UNALIGNED16 1 # endif # ifndef LZO_OPT_UNALIGNED32 # define LZO_OPT_UNALIGNED32 1 # endif # if (LZO_WORDSIZE == 8) # ifndef LZO_OPT_UNALIGNED64 # define LZO_OPT_UNALIGNED64 1 # endif # endif #elif (LZO_ARCH_SH) # define LZO_OPT_PREFER_POSTINC 1 # define LZO_OPT_PREFER_PREDEC 1 #endif #ifndef LZO_CFG_NO_INLINE_ASM #if (LZO_ABI_NEUTRAL_ENDIAN) || (LZO_ARCH_GENERIC) # define LZO_CFG_NO_INLINE_ASM 1 #elif (LZO_CC_LLVM) # define LZO_CFG_NO_INLINE_ASM 1 #endif #endif #if (LZO_CFG_NO_INLINE_ASM) # undef LZO_ASM_SYNTAX_MSC # undef LZO_ASM_SYNTAX_GNUC # undef __LZO_ASM_CLOBBER # undef __LZO_ASM_CLOBBER_LIST_CC # undef __LZO_ASM_CLOBBER_LIST_CC_MEMORY # undef __LZO_ASM_CLOBBER_LIST_EMPTY #endif #ifndef LZO_CFG_NO_UNALIGNED #if (LZO_ABI_NEUTRAL_ENDIAN) || (LZO_ARCH_GENERIC) # define LZO_CFG_NO_UNALIGNED 1 #endif #endif #if (LZO_CFG_NO_UNALIGNED) # undef LZO_OPT_UNALIGNED16 # undef LZO_OPT_UNALIGNED32 # undef LZO_OPT_UNALIGNED64 #endif #if defined(__LZO_INFOSTR_MM) #elif (LZO_MM_FLAT) && (defined(__LZO_INFOSTR_PM) || defined(LZO_INFO_ABI_PM)) # define __LZO_INFOSTR_MM "" #elif defined(LZO_INFO_MM) # define __LZO_INFOSTR_MM "." LZO_INFO_MM #else # define __LZO_INFOSTR_MM "" #endif #if defined(__LZO_INFOSTR_PM) #elif defined(LZO_INFO_ABI_PM) # define __LZO_INFOSTR_PM "." LZO_INFO_ABI_PM #else # define __LZO_INFOSTR_PM "" #endif #if defined(__LZO_INFOSTR_ENDIAN) #elif defined(LZO_INFO_ABI_ENDIAN) # define __LZO_INFOSTR_ENDIAN "." LZO_INFO_ABI_ENDIAN #else # define __LZO_INFOSTR_ENDIAN "" #endif #if defined(__LZO_INFOSTR_OSNAME) #elif defined(LZO_INFO_OS_CONSOLE) # define __LZO_INFOSTR_OSNAME LZO_INFO_OS "." LZO_INFO_OS_CONSOLE #elif defined(LZO_INFO_OS_POSIX) # define __LZO_INFOSTR_OSNAME LZO_INFO_OS "." LZO_INFO_OS_POSIX #else # define __LZO_INFOSTR_OSNAME LZO_INFO_OS #endif #if defined(__LZO_INFOSTR_LIBC) #elif defined(LZO_INFO_LIBC) # define __LZO_INFOSTR_LIBC "." LZO_INFO_LIBC #else # define __LZO_INFOSTR_LIBC "" #endif #if defined(__LZO_INFOSTR_CCVER) #elif defined(LZO_INFO_CCVER) # define __LZO_INFOSTR_CCVER " " LZO_INFO_CCVER #else # define __LZO_INFOSTR_CCVER "" #endif #define LZO_INFO_STRING \ LZO_INFO_ARCH __LZO_INFOSTR_MM __LZO_INFOSTR_PM __LZO_INFOSTR_ENDIAN \ " " __LZO_INFOSTR_OSNAME __LZO_INFOSTR_LIBC " " LZO_INFO_CC __LZO_INFOSTR_CCVER #if !(LZO_CFG_SKIP_LZO_TYPES) #if (!(LZO_SIZEOF_SHORT+0 > 0 && LZO_SIZEOF_INT+0 > 0 && LZO_SIZEOF_LONG+0 > 0)) # error "missing defines for sizes" #endif #if (!(LZO_SIZEOF_PTRDIFF_T+0 > 0 && LZO_SIZEOF_SIZE_T+0 > 0 && LZO_SIZEOF_VOID_P+0 > 0)) # error "missing defines for sizes" #endif #define LZO_TYPEOF_CHAR 1u #define LZO_TYPEOF_SHORT 2u #define LZO_TYPEOF_INT 3u #define LZO_TYPEOF_LONG 4u #define LZO_TYPEOF_LONG_LONG 5u #define LZO_TYPEOF___INT8 17u #define LZO_TYPEOF___INT16 18u #define LZO_TYPEOF___INT32 19u #define LZO_TYPEOF___INT64 20u #define LZO_TYPEOF___INT128 21u #define LZO_TYPEOF___INT256 22u #define LZO_TYPEOF___MODE_QI 33u #define LZO_TYPEOF___MODE_HI 34u #define LZO_TYPEOF___MODE_SI 35u #define LZO_TYPEOF___MODE_DI 36u #define LZO_TYPEOF___MODE_TI 37u #define LZO_TYPEOF_CHAR_P 129u #if !defined(lzo_llong_t) #if (LZO_SIZEOF_LONG_LONG+0 > 0) # if !(LZO_LANG_ASSEMBLER) __lzo_gnuc_extension__ typedef long long lzo_llong_t__; __lzo_gnuc_extension__ typedef unsigned long long lzo_ullong_t__; # endif # define lzo_llong_t lzo_llong_t__ # define lzo_ullong_t lzo_ullong_t__ #endif #endif #if !defined(lzo_int16e_t) #if (LZO_CFG_PREFER_TYPEOF_ACC_INT16E_T == LZO_TYPEOF_SHORT) && (LZO_SIZEOF_SHORT != 2) # undef LZO_CFG_PREFER_TYPEOF_ACC_INT16E_T #endif #if (LZO_SIZEOF_LONG == 2) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT16E_T == LZO_TYPEOF_SHORT) # define lzo_int16e_t long # define lzo_uint16e_t unsigned long # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF_LONG #elif (LZO_SIZEOF_INT == 2) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT16E_T == LZO_TYPEOF_SHORT) # define lzo_int16e_t int # define lzo_uint16e_t unsigned int # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_SHORT == 2) # define lzo_int16e_t short int # define lzo_uint16e_t unsigned short int # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF_SHORT #elif 1 && !(LZO_CFG_TYPE_NO_MODE_HI) && (LZO_CC_CLANG || (LZO_CC_GNUC >= 0x025f00ul) || LZO_CC_LLVM) # if !(LZO_LANG_ASSEMBLER) typedef int lzo_int16e_hi_t__ __attribute__((__mode__(__HI__))); typedef unsigned int lzo_uint16e_hi_t__ __attribute__((__mode__(__HI__))); # endif # define lzo_int16e_t lzo_int16e_hi_t__ # define lzo_uint16e_t lzo_uint16e_hi_t__ # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF___MODE_HI #elif (LZO_SIZEOF___INT16 == 2) # define lzo_int16e_t __int16 # define lzo_uint16e_t unsigned __int16 # define LZO_TYPEOF_LZO_INT16E_T LZO_TYPEOF___INT16 #else #endif #endif #if defined(lzo_int16e_t) # define LZO_SIZEOF_LZO_INT16E_T 2 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16e_t) == 2) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16e_t) == LZO_SIZEOF_LZO_INT16E_T) #endif #if !defined(lzo_int32e_t) #if (LZO_CFG_PREFER_TYPEOF_ACC_INT32E_T == LZO_TYPEOF_INT) && (LZO_SIZEOF_INT != 4) # undef LZO_CFG_PREFER_TYPEOF_ACC_INT32E_T #endif #if (LZO_SIZEOF_LONG == 4) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT32E_T == LZO_TYPEOF_INT) # define lzo_int32e_t long int # define lzo_uint32e_t unsigned long int # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF_LONG #elif (LZO_SIZEOF_INT == 4) # define lzo_int32e_t int # define lzo_uint32e_t unsigned int # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_SHORT == 4) # define lzo_int32e_t short int # define lzo_uint32e_t unsigned short int # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF_SHORT #elif (LZO_SIZEOF_LONG_LONG == 4) # define lzo_int32e_t lzo_llong_t # define lzo_uint32e_t lzo_ullong_t # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF_LONG_LONG #elif 1 && !(LZO_CFG_TYPE_NO_MODE_SI) && (LZO_CC_CLANG || (LZO_CC_GNUC >= 0x025f00ul) || LZO_CC_LLVM) && (__INT_MAX__+0 > 2147483647L) # if !(LZO_LANG_ASSEMBLER) typedef int lzo_int32e_si_t__ __attribute__((__mode__(__SI__))); typedef unsigned int lzo_uint32e_si_t__ __attribute__((__mode__(__SI__))); # endif # define lzo_int32e_t lzo_int32e_si_t__ # define lzo_uint32e_t lzo_uint32e_si_t__ # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF___MODE_SI #elif 1 && !(LZO_CFG_TYPE_NO_MODE_SI) && (LZO_CC_GNUC >= 0x025f00ul) && defined(__AVR__) && (__LONG_MAX__+0 == 32767L) # if !(LZO_LANG_ASSEMBLER) typedef int lzo_int32e_si_t__ __attribute__((__mode__(__SI__))); typedef unsigned int lzo_uint32e_si_t__ __attribute__((__mode__(__SI__))); # endif # define lzo_int32e_t lzo_int32e_si_t__ # define lzo_uint32e_t lzo_uint32e_si_t__ # define LZO_INT32_C(c) (c##LL) # define LZO_UINT32_C(c) (c##ULL) # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF___MODE_SI #elif (LZO_SIZEOF___INT32 == 4) # define lzo_int32e_t __int32 # define lzo_uint32e_t unsigned __int32 # define LZO_TYPEOF_LZO_INT32E_T LZO_TYPEOF___INT32 #else #endif #endif #if defined(lzo_int32e_t) # define LZO_SIZEOF_LZO_INT32E_T 4 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32e_t) == 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32e_t) == LZO_SIZEOF_LZO_INT32E_T) #endif #if !defined(lzo_int64e_t) #if (LZO_SIZEOF___INT64 == 8) # if (LZO_CC_BORLANDC) && !defined(LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T) # define LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T LZO_TYPEOF___INT64 # endif #endif #if (LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF_LONG_LONG) && (LZO_SIZEOF_LONG_LONG != 8) # undef LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T #endif #if (LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF___INT64) && (LZO_SIZEOF___INT64 != 8) # undef LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T #endif #if (LZO_SIZEOF_INT == 8) && (LZO_SIZEOF_INT < LZO_SIZEOF_LONG) # define lzo_int64e_t int # define lzo_uint64e_t unsigned int # define LZO_TYPEOF_LZO_INT64E_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_LONG == 8) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF_LONG_LONG) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF___INT64) # define lzo_int64e_t long int # define lzo_uint64e_t unsigned long int # define LZO_TYPEOF_LZO_INT64E_T LZO_TYPEOF_LONG #elif (LZO_SIZEOF_LONG_LONG == 8) && !(LZO_CFG_PREFER_TYPEOF_ACC_INT64E_T == LZO_TYPEOF___INT64) # define lzo_int64e_t lzo_llong_t # define lzo_uint64e_t lzo_ullong_t # define LZO_TYPEOF_LZO_INT64E_T LZO_TYPEOF_LONG_LONG # if (LZO_CC_BORLANDC) # define LZO_INT64_C(c) ((c) + 0ll) # define LZO_UINT64_C(c) ((c) + 0ull) # elif 0 # define LZO_INT64_C(c) (__lzo_gnuc_extension__ (c##LL)) # define LZO_UINT64_C(c) (__lzo_gnuc_extension__ (c##ULL)) # else # define LZO_INT64_C(c) (c##LL) # define LZO_UINT64_C(c) (c##ULL) # endif #elif (LZO_SIZEOF___INT64 == 8) # define lzo_int64e_t __int64 # define lzo_uint64e_t unsigned __int64 # define LZO_TYPEOF_LZO_INT64E_T LZO_TYPEOF___INT64 # if (LZO_CC_BORLANDC) # define LZO_INT64_C(c) ((c) + 0i64) # define LZO_UINT64_C(c) ((c) + 0ui64) # else # define LZO_INT64_C(c) (c##i64) # define LZO_UINT64_C(c) (c##ui64) # endif #else #endif #endif #if defined(lzo_int64e_t) # define LZO_SIZEOF_LZO_INT64E_T 8 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64e_t) == 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64e_t) == LZO_SIZEOF_LZO_INT64E_T) #endif #if !defined(lzo_int32l_t) #if defined(lzo_int32e_t) # define lzo_int32l_t lzo_int32e_t # define lzo_uint32l_t lzo_uint32e_t # define LZO_SIZEOF_LZO_INT32L_T LZO_SIZEOF_LZO_INT32E_T # define LZO_TYPEOF_LZO_INT32L_T LZO_TYPEOF_LZO_INT32E_T #elif (LZO_SIZEOF_INT >= 4) && (LZO_SIZEOF_INT < LZO_SIZEOF_LONG) # define lzo_int32l_t int # define lzo_uint32l_t unsigned int # define LZO_SIZEOF_LZO_INT32L_T LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_INT32L_T LZO_SIZEOF_INT #elif (LZO_SIZEOF_LONG >= 4) # define lzo_int32l_t long int # define lzo_uint32l_t unsigned long int # define LZO_SIZEOF_LZO_INT32L_T LZO_SIZEOF_LONG # define LZO_TYPEOF_LZO_INT32L_T LZO_SIZEOF_LONG #else # error "lzo_int32l_t" #endif #endif #if 1 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32l_t) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32l_t) == LZO_SIZEOF_LZO_INT32L_T) #endif #if !defined(lzo_int64l_t) #if defined(lzo_int64e_t) # define lzo_int64l_t lzo_int64e_t # define lzo_uint64l_t lzo_uint64e_t # define LZO_SIZEOF_LZO_INT64L_T LZO_SIZEOF_LZO_INT64E_T # define LZO_TYPEOF_LZO_INT64L_T LZO_TYPEOF_LZO_INT64E_T #else #endif #endif #if defined(lzo_int64l_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64l_t) >= 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64l_t) == LZO_SIZEOF_LZO_INT64L_T) #endif #if !defined(lzo_int32f_t) #if (LZO_SIZEOF_SIZE_T >= 8) # define lzo_int32f_t lzo_int64l_t # define lzo_uint32f_t lzo_uint64l_t # define LZO_SIZEOF_LZO_INT32F_T LZO_SIZEOF_LZO_INT64L_T # define LZO_TYPEOF_LZO_INT32F_T LZO_TYPEOF_LZO_INT64L_T #else # define lzo_int32f_t lzo_int32l_t # define lzo_uint32f_t lzo_uint32l_t # define LZO_SIZEOF_LZO_INT32F_T LZO_SIZEOF_LZO_INT32L_T # define LZO_TYPEOF_LZO_INT32F_T LZO_TYPEOF_LZO_INT32L_T #endif #endif #if 1 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32f_t) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32f_t) == LZO_SIZEOF_LZO_INT32F_T) #endif #if !defined(lzo_int64f_t) #if defined(lzo_int64l_t) # define lzo_int64f_t lzo_int64l_t # define lzo_uint64f_t lzo_uint64l_t # define LZO_SIZEOF_LZO_INT64F_T LZO_SIZEOF_LZO_INT64L_T # define LZO_TYPEOF_LZO_INT64F_T LZO_TYPEOF_LZO_INT64L_T #else #endif #endif #if defined(lzo_int64f_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64f_t) >= 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64f_t) == LZO_SIZEOF_LZO_INT64F_T) #endif #if !defined(lzo_intptr_t) #if 1 && (LZO_OS_OS400 && (LZO_SIZEOF_VOID_P == 16)) # define __LZO_INTPTR_T_IS_POINTER 1 # if !(LZO_LANG_ASSEMBLER) typedef char * lzo_intptr_t; typedef char * lzo_uintptr_t; # endif # define lzo_intptr_t lzo_intptr_t # define lzo_uintptr_t lzo_uintptr_t # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_VOID_P # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_CHAR_P #elif (LZO_CC_MSC && (_MSC_VER >= 1300) && (LZO_SIZEOF_VOID_P == 4) && (LZO_SIZEOF_INT == 4)) # if !(LZO_LANG_ASSEMBLER) typedef __w64 int lzo_intptr_t; typedef __w64 unsigned int lzo_uintptr_t; # endif # define lzo_intptr_t lzo_intptr_t # define lzo_uintptr_t lzo_uintptr_t # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_SHORT == LZO_SIZEOF_VOID_P) && (LZO_SIZEOF_INT > LZO_SIZEOF_VOID_P) # define lzo_intptr_t short # define lzo_uintptr_t unsigned short # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_SHORT # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_SHORT #elif (LZO_SIZEOF_INT >= LZO_SIZEOF_VOID_P) && (LZO_SIZEOF_INT < LZO_SIZEOF_LONG) # define lzo_intptr_t int # define lzo_uintptr_t unsigned int # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_INT #elif (LZO_SIZEOF_LONG >= LZO_SIZEOF_VOID_P) # define lzo_intptr_t long # define lzo_uintptr_t unsigned long # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_LONG # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_LONG #elif (LZO_SIZEOF_LZO_INT64L_T >= LZO_SIZEOF_VOID_P) # define lzo_intptr_t lzo_int64l_t # define lzo_uintptr_t lzo_uint64l_t # define LZO_SIZEOF_LZO_INTPTR_T LZO_SIZEOF_LZO_INT64L_T # define LZO_TYPEOF_LZO_INTPTR_T LZO_TYPEOF_LZO_INT64L_T #else # error "lzo_intptr_t" #endif #endif #if 1 LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_intptr_t) >= sizeof(void *)) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_intptr_t) == sizeof(lzo_uintptr_t)) #endif #if !defined(lzo_word_t) #if defined(LZO_WORDSIZE) && (LZO_WORDSIZE+0 > 0) #if (LZO_WORDSIZE == LZO_SIZEOF_LZO_INTPTR_T) && !(__LZO_INTPTR_T_IS_POINTER) # define lzo_word_t lzo_uintptr_t # define lzo_sword_t lzo_intptr_t # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_LZO_INTPTR_T # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_LZO_INTPTR_T #elif (LZO_WORDSIZE == LZO_SIZEOF_LONG) # define lzo_word_t unsigned long # define lzo_sword_t long # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_LONG # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_LONG #elif (LZO_WORDSIZE == LZO_SIZEOF_INT) # define lzo_word_t unsigned int # define lzo_sword_t int # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_INT # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_INT #elif (LZO_WORDSIZE == LZO_SIZEOF_SHORT) # define lzo_word_t unsigned short # define lzo_sword_t short # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_SHORT # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_SHORT #elif (LZO_WORDSIZE == 1) # define lzo_word_t unsigned char # define lzo_sword_t signed char # define LZO_SIZEOF_LZO_WORD_T 1 # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF_CHAR #elif (LZO_WORDSIZE == LZO_SIZEOF_LZO_INT64L_T) # define lzo_word_t lzo_uint64l_t # define lzo_sword_t lzo_int64l_t # define LZO_SIZEOF_LZO_WORD_T LZO_SIZEOF_LZO_INT64L_T # define LZO_TYPEOF_LZO_WORD_T LZO_SIZEOF_LZO_INT64L_T #elif (LZO_ARCH_SPU) && (LZO_CC_GNUC) #if 0 # if !(LZO_LANG_ASSEMBLER) typedef unsigned lzo_word_t __attribute__((__mode__(__V16QI__))); typedef int lzo_sword_t __attribute__((__mode__(__V16QI__))); # endif # define lzo_word_t lzo_word_t # define lzo_sword_t lzo_sword_t # define LZO_SIZEOF_LZO_WORD_T 16 # define LZO_TYPEOF_LZO_WORD_T LZO_TYPEOF___MODE_V16QI #endif #else # error "lzo_word_t" #endif #endif #endif #if 1 && defined(lzo_word_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_word_t) == LZO_WORDSIZE) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_sword_t) == LZO_WORDSIZE) #endif #if 1 #define lzo_int8_t signed char #define lzo_uint8_t unsigned char #define LZO_SIZEOF_LZO_INT8_T 1 #define LZO_TYPEOF_LZO_INT8_T LZO_TYPEOF_CHAR LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int8_t) == 1) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int8_t) == sizeof(lzo_uint8_t)) #endif #if defined(lzo_int16e_t) #define lzo_int16_t lzo_int16e_t #define lzo_uint16_t lzo_uint16e_t #define LZO_SIZEOF_LZO_INT16_T LZO_SIZEOF_LZO_INT16E_T #define LZO_TYPEOF_LZO_INT16_T LZO_TYPEOF_LZO_INT16E_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16_t) == 2) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16_t) == sizeof(lzo_uint16_t)) #endif #if defined(lzo_int32e_t) #define lzo_int32_t lzo_int32e_t #define lzo_uint32_t lzo_uint32e_t #define LZO_SIZEOF_LZO_INT32_T LZO_SIZEOF_LZO_INT32E_T #define LZO_TYPEOF_LZO_INT32_T LZO_TYPEOF_LZO_INT32E_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32_t) == 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32_t) == sizeof(lzo_uint32_t)) #endif #if defined(lzo_int64e_t) #define lzo_int64_t lzo_int64e_t #define lzo_uint64_t lzo_uint64e_t #define LZO_SIZEOF_LZO_INT64_T LZO_SIZEOF_LZO_INT64E_T #define LZO_TYPEOF_LZO_INT64_T LZO_TYPEOF_LZO_INT64E_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64_t) == 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64_t) == sizeof(lzo_uint64_t)) #endif #if 1 #define lzo_int_least32_t lzo_int32l_t #define lzo_uint_least32_t lzo_uint32l_t #define LZO_SIZEOF_LZO_INT_LEAST32_T LZO_SIZEOF_LZO_INT32L_T #define LZO_TYPEOF_LZO_INT_LEAST32_T LZO_TYPEOF_LZO_INT32L_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_least32_t) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_least32_t) == sizeof(lzo_uint_least32_t)) #endif #if defined(lzo_int64l_t) #define lzo_int_least64_t lzo_int64l_t #define lzo_uint_least64_t lzo_uint64l_t #define LZO_SIZEOF_LZO_INT_LEAST64_T LZO_SIZEOF_LZO_INT64L_T #define LZO_TYPEOF_LZO_INT_LEAST64_T LZO_TYPEOF_LZO_INT64L_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_least64_t) >= 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_least64_t) == sizeof(lzo_uint_least64_t)) #endif #if 1 #define lzo_int_fast32_t lzo_int32f_t #define lzo_uint_fast32_t lzo_uint32f_t #define LZO_SIZEOF_LZO_INT_FAST32_T LZO_SIZEOF_LZO_INT32F_T #define LZO_TYPEOF_LZO_INT_FAST32_T LZO_TYPEOF_LZO_INT32F_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_fast32_t) >= 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_fast32_t) == sizeof(lzo_uint_fast32_t)) #endif #if defined(lzo_int64f_t) #define lzo_int_fast64_t lzo_int64f_t #define lzo_uint_fast64_t lzo_uint64f_t #define LZO_SIZEOF_LZO_INT_FAST64_T LZO_SIZEOF_LZO_INT64F_T #define LZO_TYPEOF_LZO_INT_FAST64_T LZO_TYPEOF_LZO_INT64F_T LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_fast64_t) >= 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int_fast64_t) == sizeof(lzo_uint_fast64_t)) #endif #if !defined(LZO_INT16_C) # if (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_INT >= 2) # define LZO_INT16_C(c) ((c) + 0) # define LZO_UINT16_C(c) ((c) + 0U) # elif (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_LONG >= 2) # define LZO_INT16_C(c) ((c) + 0L) # define LZO_UINT16_C(c) ((c) + 0UL) # elif (LZO_SIZEOF_INT >= 2) # define LZO_INT16_C(c) (c) # define LZO_UINT16_C(c) (c##U) # elif (LZO_SIZEOF_LONG >= 2) # define LZO_INT16_C(c) (c##L) # define LZO_UINT16_C(c) (c##UL) # else # error "LZO_INT16_C" # endif #endif #if !defined(LZO_INT32_C) # if (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_INT >= 4) # define LZO_INT32_C(c) ((c) + 0) # define LZO_UINT32_C(c) ((c) + 0U) # elif (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_LONG >= 4) # define LZO_INT32_C(c) ((c) + 0L) # define LZO_UINT32_C(c) ((c) + 0UL) # elif (LZO_SIZEOF_INT >= 4) # define LZO_INT32_C(c) (c) # define LZO_UINT32_C(c) (c##U) # elif (LZO_SIZEOF_LONG >= 4) # define LZO_INT32_C(c) (c##L) # define LZO_UINT32_C(c) (c##UL) # elif (LZO_SIZEOF_LONG_LONG >= 4) # define LZO_INT32_C(c) (c##LL) # define LZO_UINT32_C(c) (c##ULL) # else # error "LZO_INT32_C" # endif #endif #if !defined(LZO_INT64_C) && defined(lzo_int64l_t) # if (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_INT >= 8) # define LZO_INT64_C(c) ((c) + 0) # define LZO_UINT64_C(c) ((c) + 0U) # elif (LZO_BROKEN_INTEGRAL_CONSTANTS) && (LZO_SIZEOF_LONG >= 8) # define LZO_INT64_C(c) ((c) + 0L) # define LZO_UINT64_C(c) ((c) + 0UL) # elif (LZO_SIZEOF_INT >= 8) # define LZO_INT64_C(c) (c) # define LZO_UINT64_C(c) (c##U) # elif (LZO_SIZEOF_LONG >= 8) # define LZO_INT64_C(c) (c##L) # define LZO_UINT64_C(c) (c##UL) # else # error "LZO_INT64_C" # endif #endif #endif #endif #endif #undef LZO_HAVE_CONFIG_H #include "minilzo.h" #if !defined(MINILZO_VERSION) || (MINILZO_VERSION != 0x20a0) # error "version mismatch in miniLZO source files" #endif #ifdef MINILZO_HAVE_CONFIG_H # define LZO_HAVE_CONFIG_H 1 #endif #ifndef __LZO_CONF_H #define __LZO_CONF_H 1 #if !defined(__LZO_IN_MINILZO) #if defined(LZO_CFG_FREESTANDING) && (LZO_CFG_FREESTANDING) # define LZO_LIBC_FREESTANDING 1 # define LZO_OS_FREESTANDING 1 #endif #if defined(LZO_CFG_EXTRA_CONFIG_HEADER) # include LZO_CFG_EXTRA_CONFIG_HEADER #endif #if defined(__LZOCONF_H) || defined(__LZOCONF_H_INCLUDED) # error "include this file first" #endif #if defined(LZO_CFG_BUILD_DLL) && (LZO_CFG_BUILD_DLL+0) && !defined(__LZO_EXPORT1) && !defined(__LZO_EXPORT2) && 0 #ifndef __LZODEFS_H_INCLUDED #if defined(LZO_HAVE_CONFIG_H) # include #endif #include #include #include #endif #endif #include #if defined(LZO_CFG_EXTRA_CONFIG_HEADER2) # include LZO_CFG_EXTRA_CONFIG_HEADER2 #endif #endif #if !defined(__LZOCONF_H_INCLUDED) || (LZO_VERSION+0 != 0x20a0) # error "version mismatch" #endif #if (LZO_CC_MSC && (_MSC_VER >= 1000 && _MSC_VER < 1100)) # pragma warning(disable: 4702) #endif #if (LZO_CC_MSC && (_MSC_VER >= 1000)) # pragma warning(disable: 4127 4701) # pragma warning(disable: 4514 4710 4711) #endif #if (LZO_CC_MSC && (_MSC_VER >= 1300)) # pragma warning(disable: 4820) #endif #if (LZO_CC_MSC && (_MSC_VER >= 1800)) # pragma warning(disable: 4746) #endif #if (LZO_CC_INTELC && (__INTEL_COMPILER >= 900)) # pragma warning(disable: 1684) #endif #if (LZO_CC_SUNPROC) #if !defined(__cplusplus) # pragma error_messages(off,E_END_OF_LOOP_CODE_NOT_REACHED) # pragma error_messages(off,E_LOOP_NOT_ENTERED_AT_TOP) # pragma error_messages(off,E_STATEMENT_NOT_REACHED) #endif #endif #if !defined(__LZO_NOEXPORT1) # define __LZO_NOEXPORT1 /*empty*/ #endif #if !defined(__LZO_NOEXPORT2) # define __LZO_NOEXPORT2 /*empty*/ #endif #if 1 # define LZO_PUBLIC_DECL(r) LZO_EXTERN(r) #endif #if 1 # define LZO_PUBLIC_IMPL(r) LZO_PUBLIC(r) #endif #if !defined(LZO_LOCAL_DECL) # define LZO_LOCAL_DECL(r) __LZO_EXTERN_C LZO_LOCAL_IMPL(r) #endif #if !defined(LZO_LOCAL_IMPL) # define LZO_LOCAL_IMPL(r) __LZO_NOEXPORT1 r __LZO_NOEXPORT2 __LZO_CDECL #endif #if 1 # define LZO_STATIC_DECL(r) LZO_PRIVATE(r) #endif #if 1 # define LZO_STATIC_IMPL(r) LZO_PRIVATE(r) #endif #if defined(__LZO_IN_MINILZO) || (LZO_CFG_FREESTANDING) #elif 1 # include #else # define LZO_WANT_ACC_INCD_H 1 #endif #if defined(LZO_HAVE_CONFIG_H) # define LZO_CFG_NO_CONFIG_HEADER 1 #endif #if 1 && !defined(LZO_CFG_FREESTANDING) #if 1 && !defined(HAVE_STRING_H) #define HAVE_STRING_H 1 #endif #if 1 && !defined(HAVE_MEMCMP) #define HAVE_MEMCMP 1 #endif #if 1 && !defined(HAVE_MEMCPY) #define HAVE_MEMCPY 1 #endif #if 1 && !defined(HAVE_MEMMOVE) #define HAVE_MEMMOVE 1 #endif #if 1 && !defined(HAVE_MEMSET) #define HAVE_MEMSET 1 #endif #endif #if 1 && defined(HAVE_STRING_H) #include #endif #if 1 || defined(lzo_int8_t) || defined(lzo_uint8_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int8_t) == 1) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint8_t) == 1) #endif #if 1 || defined(lzo_int16_t) || defined(lzo_uint16_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int16_t) == 2) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint16_t) == 2) #endif #if 1 || defined(lzo_int32_t) || defined(lzo_uint32_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int32_t) == 4) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint32_t) == 4) #endif #if defined(lzo_int64_t) || defined(lzo_uint64_t) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_int64_t) == 8) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint64_t) == 8) #endif #if (LZO_CFG_FREESTANDING) # undef HAVE_MEMCMP # undef HAVE_MEMCPY # undef HAVE_MEMMOVE # undef HAVE_MEMSET #endif #if !(HAVE_MEMCMP) # undef memcmp # define memcmp(a,b,c) lzo_memcmp(a,b,c) #else # undef lzo_memcmp # define lzo_memcmp(a,b,c) memcmp(a,b,c) #endif #if !(HAVE_MEMCPY) # undef memcpy # define memcpy(a,b,c) lzo_memcpy(a,b,c) #else # undef lzo_memcpy # define lzo_memcpy(a,b,c) memcpy(a,b,c) #endif #if !(HAVE_MEMMOVE) # undef memmove # define memmove(a,b,c) lzo_memmove(a,b,c) #else # undef lzo_memmove # define lzo_memmove(a,b,c) memmove(a,b,c) #endif #if !(HAVE_MEMSET) # undef memset # define memset(a,b,c) lzo_memset(a,b,c) #else # undef lzo_memset # define lzo_memset(a,b,c) memset(a,b,c) #endif #undef NDEBUG #if (LZO_CFG_FREESTANDING) # undef LZO_DEBUG # define NDEBUG 1 # undef assert # define assert(e) ((void)0) #else # if !defined(LZO_DEBUG) # define NDEBUG 1 # endif # include #endif #if 0 && defined(__BOUNDS_CHECKING_ON) # include #else # define BOUNDS_CHECKING_OFF_DURING(stmt) stmt # define BOUNDS_CHECKING_OFF_IN_EXPR(expr) (expr) #endif #if (LZO_CFG_PGO) # undef __lzo_likely # undef __lzo_unlikely # define __lzo_likely(e) (e) # define __lzo_unlikely(e) (e) #endif #undef _ #undef __ #undef ___ #undef ____ #undef _p0 #undef _p1 #undef _p2 #undef _p3 #undef _p4 #undef _s0 #undef _s1 #undef _s2 #undef _s3 #undef _s4 #undef _ww #if 1 # define LZO_BYTE(x) ((unsigned char) (x)) #else # define LZO_BYTE(x) ((unsigned char) ((x) & 0xff)) #endif #define LZO_MAX(a,b) ((a) >= (b) ? (a) : (b)) #define LZO_MIN(a,b) ((a) <= (b) ? (a) : (b)) #define LZO_MAX3(a,b,c) ((a) >= (b) ? LZO_MAX(a,c) : LZO_MAX(b,c)) #define LZO_MIN3(a,b,c) ((a) <= (b) ? LZO_MIN(a,c) : LZO_MIN(b,c)) #define lzo_sizeof(type) ((lzo_uint) (sizeof(type))) #define LZO_HIGH(array) ((lzo_uint) (sizeof(array)/sizeof(*(array)))) #define LZO_SIZE(bits) (1u << (bits)) #define LZO_MASK(bits) (LZO_SIZE(bits) - 1) #define LZO_USIZE(bits) ((lzo_uint) 1 << (bits)) #define LZO_UMASK(bits) (LZO_USIZE(bits) - 1) #if !defined(DMUL) #if 0 # define DMUL(a,b) ((lzo_xint) ((lzo_uint32_t)(a) * (lzo_uint32_t)(b))) #else # define DMUL(a,b) ((lzo_xint) ((a) * (b))) #endif #endif #ifndef __LZO_FUNC_H #define __LZO_FUNC_H 1 #if !defined(LZO_BITOPS_USE_ASM_BITSCAN) && !defined(LZO_BITOPS_USE_GNUC_BITSCAN) && !defined(LZO_BITOPS_USE_MSC_BITSCAN) #if 1 && (LZO_ARCH_AMD64) && (LZO_CC_GNUC && (LZO_CC_GNUC < 0x040000ul)) && (LZO_ASM_SYNTAX_GNUC) #define LZO_BITOPS_USE_ASM_BITSCAN 1 #elif (LZO_CC_CLANG || (LZO_CC_GNUC >= 0x030400ul) || (LZO_CC_INTELC_GNUC && (__INTEL_COMPILER >= 1000)) || (LZO_CC_LLVM && (!defined(__llvm_tools_version__) || (__llvm_tools_version__+0 >= 0x010500ul)))) #define LZO_BITOPS_USE_GNUC_BITSCAN 1 #elif (LZO_OS_WIN32 || LZO_OS_WIN64) && ((LZO_CC_INTELC_MSC && (__INTEL_COMPILER >= 1010)) || (LZO_CC_MSC && (_MSC_VER >= 1400))) #define LZO_BITOPS_USE_MSC_BITSCAN 1 #if (LZO_CC_MSC) && (LZO_ARCH_AMD64 || LZO_ARCH_I386) #include #endif #if (LZO_CC_MSC) && (LZO_ARCH_AMD64 || LZO_ARCH_I386) #pragma intrinsic(_BitScanReverse) #pragma intrinsic(_BitScanForward) #endif #if (LZO_CC_MSC) && (LZO_ARCH_AMD64) #pragma intrinsic(_BitScanReverse64) #pragma intrinsic(_BitScanForward64) #endif #endif #endif __lzo_static_forceinline unsigned lzo_bitops_ctlz32_func(lzo_uint32_t v) { #if (LZO_BITOPS_USE_MSC_BITSCAN) && (LZO_ARCH_AMD64 || LZO_ARCH_I386) unsigned long r; (void) _BitScanReverse(&r, v); return (unsigned) r ^ 31; #define lzo_bitops_ctlz32(v) lzo_bitops_ctlz32_func(v) #elif (LZO_BITOPS_USE_ASM_BITSCAN) && (LZO_ARCH_AMD64 || LZO_ARCH_I386) && (LZO_ASM_SYNTAX_GNUC) lzo_uint32_t r; __asm__("bsr %1,%0" : "=r" (r) : "rm" (v) __LZO_ASM_CLOBBER_LIST_CC); return (unsigned) r ^ 31; #define lzo_bitops_ctlz32(v) lzo_bitops_ctlz32_func(v) #elif (LZO_BITOPS_USE_GNUC_BITSCAN) && (LZO_SIZEOF_INT == 4) unsigned r; r = (unsigned) __builtin_clz(v); return r; #define lzo_bitops_ctlz32(v) ((unsigned) __builtin_clz(v)) #elif (LZO_BITOPS_USE_GNUC_BITSCAN) && (LZO_SIZEOF_LONG == 8) && (LZO_WORDSIZE >= 8) unsigned r; r = (unsigned) __builtin_clzl(v); return r ^ 32; #define lzo_bitops_ctlz32(v) (((unsigned) __builtin_clzl(v)) ^ 32) #else LZO_UNUSED(v); return 0; #endif } #if defined(lzo_uint64_t) __lzo_static_forceinline unsigned lzo_bitops_ctlz64_func(lzo_uint64_t v) { #if (LZO_BITOPS_USE_MSC_BITSCAN) && (LZO_ARCH_AMD64) unsigned long r; (void) _BitScanReverse64(&r, v); return (unsigned) r ^ 63; #define lzo_bitops_ctlz64(v) lzo_bitops_ctlz64_func(v) #elif (LZO_BITOPS_USE_ASM_BITSCAN) && (LZO_ARCH_AMD64) && (LZO_ASM_SYNTAX_GNUC) lzo_uint64_t r; __asm__("bsr %1,%0" : "=r" (r) : "rm" (v) __LZO_ASM_CLOBBER_LIST_CC); return (unsigned) r ^ 63; #define lzo_bitops_ctlz64(v) lzo_bitops_ctlz64_func(v) #elif (LZO_BITOPS_USE_GNUC_BITSCAN) && (LZO_SIZEOF_LONG == 8) && (LZO_WORDSIZE >= 8) unsigned r; r = (unsigned) __builtin_clzl(v); return r; #define lzo_bitops_ctlz64(v) ((unsigned) __builtin_clzl(v)) #elif (LZO_BITOPS_USE_GNUC_BITSCAN) && (LZO_SIZEOF_LONG_LONG == 8) && (LZO_WORDSIZE >= 8) unsigned r; r = (unsigned) __builtin_clzll(v); return r; #define lzo_bitops_ctlz64(v) ((unsigned) __builtin_clzll(v)) #else LZO_UNUSED(v); return 0; #endif } #endif __lzo_static_forceinline unsigned lzo_bitops_cttz32_func(lzo_uint32_t v) { #if (LZO_BITOPS_USE_MSC_BITSCAN) && (LZO_ARCH_AMD64 || LZO_ARCH_I386) unsigned long r; (void) _BitScanForward(&r, v); return (unsigned) r; #define lzo_bitops_cttz32(v) lzo_bitops_cttz32_func(v) #elif (LZO_BITOPS_USE_ASM_BITSCAN) && (LZO_ARCH_AMD64 || LZO_ARCH_I386) && (LZO_ASM_SYNTAX_GNUC) lzo_uint32_t r; __asm__("bsf %1,%0" : "=r" (r) : "rm" (v) __LZO_ASM_CLOBBER_LIST_CC); return (unsigned) r; #define lzo_bitops_cttz32(v) lzo_bitops_cttz32_func(v) #elif (LZO_BITOPS_USE_GNUC_BITSCAN) && (LZO_SIZEOF_INT >= 4) unsigned r; r = (unsigned) __builtin_ctz(v); return r; #define lzo_bitops_cttz32(v) ((unsigned) __builtin_ctz(v)) #else LZO_UNUSED(v); return 0; #endif } #if defined(lzo_uint64_t) __lzo_static_forceinline unsigned lzo_bitops_cttz64_func(lzo_uint64_t v) { #if (LZO_BITOPS_USE_MSC_BITSCAN) && (LZO_ARCH_AMD64) unsigned long r; (void) _BitScanForward64(&r, v); return (unsigned) r; #define lzo_bitops_cttz64(v) lzo_bitops_cttz64_func(v) #elif (LZO_BITOPS_USE_ASM_BITSCAN) && (LZO_ARCH_AMD64) && (LZO_ASM_SYNTAX_GNUC) lzo_uint64_t r; __asm__("bsf %1,%0" : "=r" (r) : "rm" (v) __LZO_ASM_CLOBBER_LIST_CC); return (unsigned) r; #define lzo_bitops_cttz64(v) lzo_bitops_cttz64_func(v) #elif (LZO_BITOPS_USE_GNUC_BITSCAN) && (LZO_SIZEOF_LONG >= 8) && (LZO_WORDSIZE >= 8) unsigned r; r = (unsigned) __builtin_ctzl(v); return r; #define lzo_bitops_cttz64(v) ((unsigned) __builtin_ctzl(v)) #elif (LZO_BITOPS_USE_GNUC_BITSCAN) && (LZO_SIZEOF_LONG_LONG >= 8) && (LZO_WORDSIZE >= 8) unsigned r; r = (unsigned) __builtin_ctzll(v); return r; #define lzo_bitops_cttz64(v) ((unsigned) __builtin_ctzll(v)) #else LZO_UNUSED(v); return 0; #endif } #endif lzo_unused_funcs_impl(void, lzo_bitops_unused_funcs)(void) { LZO_UNUSED_FUNC(lzo_bitops_unused_funcs); LZO_UNUSED_FUNC(lzo_bitops_ctlz32_func); LZO_UNUSED_FUNC(lzo_bitops_cttz32_func); #if defined(lzo_uint64_t) LZO_UNUSED_FUNC(lzo_bitops_ctlz64_func); LZO_UNUSED_FUNC(lzo_bitops_cttz64_func); #endif } #if defined(__lzo_alignof) && !(LZO_CFG_NO_UNALIGNED) #if !defined(lzo_memops_tcheck__) && 0 #define lzo_memops_tcheck__(t,a,b) ((void)0, sizeof(t) == (a) && __lzo_alignof(t) == (b)) #endif #endif #ifndef lzo_memops_TU0p #define lzo_memops_TU0p void __LZO_MMODEL * #endif #ifndef lzo_memops_TU1p #define lzo_memops_TU1p unsigned char __LZO_MMODEL * #endif #ifndef lzo_memops_TU2p #if (LZO_OPT_UNALIGNED16) typedef lzo_uint16_t __lzo_may_alias lzo_memops_TU2; #define lzo_memops_TU2p volatile lzo_memops_TU2 * #elif defined(__lzo_byte_struct) __lzo_byte_struct(lzo_memops_TU2_struct,2) typedef struct lzo_memops_TU2_struct lzo_memops_TU2; #else struct lzo_memops_TU2_struct { unsigned char a[2]; } __lzo_may_alias; typedef struct lzo_memops_TU2_struct lzo_memops_TU2; #endif #ifndef lzo_memops_TU2p #define lzo_memops_TU2p lzo_memops_TU2 * #endif #endif #ifndef lzo_memops_TU4p #if (LZO_OPT_UNALIGNED32) typedef lzo_uint32_t __lzo_may_alias lzo_memops_TU4; #define lzo_memops_TU4p volatile lzo_memops_TU4 __LZO_MMODEL * #elif defined(__lzo_byte_struct) __lzo_byte_struct(lzo_memops_TU4_struct,4) typedef struct lzo_memops_TU4_struct lzo_memops_TU4; #else struct lzo_memops_TU4_struct { unsigned char a[4]; } __lzo_may_alias; typedef struct lzo_memops_TU4_struct lzo_memops_TU4; #endif #ifndef lzo_memops_TU4p #define lzo_memops_TU4p lzo_memops_TU4 __LZO_MMODEL * #endif #endif #ifndef lzo_memops_TU8p #if (LZO_OPT_UNALIGNED64) typedef lzo_uint64_t __lzo_may_alias lzo_memops_TU8; #define lzo_memops_TU8p volatile lzo_memops_TU8 __LZO_MMODEL * #elif defined(__lzo_byte_struct) __lzo_byte_struct(lzo_memops_TU8_struct,8) typedef struct lzo_memops_TU8_struct lzo_memops_TU8; #else struct lzo_memops_TU8_struct { unsigned char a[8]; } __lzo_may_alias; typedef struct lzo_memops_TU8_struct lzo_memops_TU8; #endif #ifndef lzo_memops_TU8p #define lzo_memops_TU8p lzo_memops_TU8 __LZO_MMODEL * #endif #endif #ifndef lzo_memops_set_TU1p #define lzo_memops_set_TU1p volatile lzo_memops_TU1p #endif #ifndef lzo_memops_move_TU1p #define lzo_memops_move_TU1p lzo_memops_TU1p #endif #define LZO_MEMOPS_SET1(dd,cc) \ LZO_BLOCK_BEGIN \ lzo_memops_set_TU1p d__1 = (lzo_memops_set_TU1p) (lzo_memops_TU0p) (dd); \ d__1[0] = LZO_BYTE(cc); \ LZO_BLOCK_END #define LZO_MEMOPS_SET2(dd,cc) \ LZO_BLOCK_BEGIN \ lzo_memops_set_TU1p d__2 = (lzo_memops_set_TU1p) (lzo_memops_TU0p) (dd); \ d__2[0] = LZO_BYTE(cc); d__2[1] = LZO_BYTE(cc); \ LZO_BLOCK_END #define LZO_MEMOPS_SET3(dd,cc) \ LZO_BLOCK_BEGIN \ lzo_memops_set_TU1p d__3 = (lzo_memops_set_TU1p) (lzo_memops_TU0p) (dd); \ d__3[0] = LZO_BYTE(cc); d__3[1] = LZO_BYTE(cc); d__3[2] = LZO_BYTE(cc); \ LZO_BLOCK_END #define LZO_MEMOPS_SET4(dd,cc) \ LZO_BLOCK_BEGIN \ lzo_memops_set_TU1p d__4 = (lzo_memops_set_TU1p) (lzo_memops_TU0p) (dd); \ d__4[0] = LZO_BYTE(cc); d__4[1] = LZO_BYTE(cc); d__4[2] = LZO_BYTE(cc); d__4[3] = LZO_BYTE(cc); \ LZO_BLOCK_END #define LZO_MEMOPS_MOVE1(dd,ss) \ LZO_BLOCK_BEGIN \ lzo_memops_move_TU1p d__1 = (lzo_memops_move_TU1p) (lzo_memops_TU0p) (dd); \ const lzo_memops_move_TU1p s__1 = (const lzo_memops_move_TU1p) (const lzo_memops_TU0p) (ss); \ d__1[0] = s__1[0]; \ LZO_BLOCK_END #define LZO_MEMOPS_MOVE2(dd,ss) \ LZO_BLOCK_BEGIN \ lzo_memops_move_TU1p d__2 = (lzo_memops_move_TU1p) (lzo_memops_TU0p) (dd); \ const lzo_memops_move_TU1p s__2 = (const lzo_memops_move_TU1p) (const lzo_memops_TU0p) (ss); \ d__2[0] = s__2[0]; d__2[1] = s__2[1]; \ LZO_BLOCK_END #define LZO_MEMOPS_MOVE3(dd,ss) \ LZO_BLOCK_BEGIN \ lzo_memops_move_TU1p d__3 = (lzo_memops_move_TU1p) (lzo_memops_TU0p) (dd); \ const lzo_memops_move_TU1p s__3 = (const lzo_memops_move_TU1p) (const lzo_memops_TU0p) (ss); \ d__3[0] = s__3[0]; d__3[1] = s__3[1]; d__3[2] = s__3[2]; \ LZO_BLOCK_END #define LZO_MEMOPS_MOVE4(dd,ss) \ LZO_BLOCK_BEGIN \ lzo_memops_move_TU1p d__4 = (lzo_memops_move_TU1p) (lzo_memops_TU0p) (dd); \ const lzo_memops_move_TU1p s__4 = (const lzo_memops_move_TU1p) (const lzo_memops_TU0p) (ss); \ d__4[0] = s__4[0]; d__4[1] = s__4[1]; d__4[2] = s__4[2]; d__4[3] = s__4[3]; \ LZO_BLOCK_END #define LZO_MEMOPS_MOVE8(dd,ss) \ LZO_BLOCK_BEGIN \ lzo_memops_move_TU1p d__8 = (lzo_memops_move_TU1p) (lzo_memops_TU0p) (dd); \ const lzo_memops_move_TU1p s__8 = (const lzo_memops_move_TU1p) (const lzo_memops_TU0p) (ss); \ d__8[0] = s__8[0]; d__8[1] = s__8[1]; d__8[2] = s__8[2]; d__8[3] = s__8[3]; \ d__8[4] = s__8[4]; d__8[5] = s__8[5]; d__8[6] = s__8[6]; d__8[7] = s__8[7]; \ LZO_BLOCK_END LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(*(lzo_memops_TU1p)0)==1) #define LZO_MEMOPS_COPY1(dd,ss) LZO_MEMOPS_MOVE1(dd,ss) #if (LZO_OPT_UNALIGNED16) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(*(lzo_memops_TU2p)0)==2) #define LZO_MEMOPS_COPY2(dd,ss) \ * (lzo_memops_TU2p) (lzo_memops_TU0p) (dd) = * (const lzo_memops_TU2p) (const lzo_memops_TU0p) (ss) #elif defined(lzo_memops_tcheck__) #define LZO_MEMOPS_COPY2(dd,ss) \ LZO_BLOCK_BEGIN if (lzo_memops_tcheck__(lzo_memops_TU2,2,1)) { \ * (lzo_memops_TU2p) (lzo_memops_TU0p) (dd) = * (const lzo_memops_TU2p) (const lzo_memops_TU0p) (ss); \ } else { LZO_MEMOPS_MOVE2(dd,ss); } LZO_BLOCK_END #else #define LZO_MEMOPS_COPY2(dd,ss) LZO_MEMOPS_MOVE2(dd,ss) #endif #if (LZO_OPT_UNALIGNED32) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(*(lzo_memops_TU4p)0)==4) #define LZO_MEMOPS_COPY4(dd,ss) \ * (lzo_memops_TU4p) (lzo_memops_TU0p) (dd) = * (const lzo_memops_TU4p) (const lzo_memops_TU0p) (ss) #elif defined(lzo_memops_tcheck__) #define LZO_MEMOPS_COPY4(dd,ss) \ LZO_BLOCK_BEGIN if (lzo_memops_tcheck__(lzo_memops_TU4,4,1)) { \ * (lzo_memops_TU4p) (lzo_memops_TU0p) (dd) = * (const lzo_memops_TU4p) (const lzo_memops_TU0p) (ss); \ } else { LZO_MEMOPS_MOVE4(dd,ss); } LZO_BLOCK_END #else #define LZO_MEMOPS_COPY4(dd,ss) LZO_MEMOPS_MOVE4(dd,ss) #endif #if (LZO_WORDSIZE != 8) #define LZO_MEMOPS_COPY8(dd,ss) \ LZO_BLOCK_BEGIN LZO_MEMOPS_COPY4(dd,ss); LZO_MEMOPS_COPY4((lzo_memops_TU1p)(lzo_memops_TU0p)(dd)+4,(const lzo_memops_TU1p)(const lzo_memops_TU0p)(ss)+4); LZO_BLOCK_END #else #if (LZO_OPT_UNALIGNED64) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(*(lzo_memops_TU8p)0)==8) #define LZO_MEMOPS_COPY8(dd,ss) \ * (lzo_memops_TU8p) (lzo_memops_TU0p) (dd) = * (const lzo_memops_TU8p) (const lzo_memops_TU0p) (ss) #elif (LZO_OPT_UNALIGNED32) #define LZO_MEMOPS_COPY8(dd,ss) \ LZO_BLOCK_BEGIN LZO_MEMOPS_COPY4(dd,ss); LZO_MEMOPS_COPY4((lzo_memops_TU1p)(lzo_memops_TU0p)(dd)+4,(const lzo_memops_TU1p)(const lzo_memops_TU0p)(ss)+4); LZO_BLOCK_END #elif defined(lzo_memops_tcheck__) #define LZO_MEMOPS_COPY8(dd,ss) \ LZO_BLOCK_BEGIN if (lzo_memops_tcheck__(lzo_memops_TU8,8,1)) { \ * (lzo_memops_TU8p) (lzo_memops_TU0p) (dd) = * (const lzo_memops_TU8p) (const lzo_memops_TU0p) (ss); \ } else { LZO_MEMOPS_MOVE8(dd,ss); } LZO_BLOCK_END #else #define LZO_MEMOPS_COPY8(dd,ss) LZO_MEMOPS_MOVE8(dd,ss) #endif #endif #define LZO_MEMOPS_COPYN(dd,ss,nn) \ LZO_BLOCK_BEGIN \ lzo_memops_TU1p d__n = (lzo_memops_TU1p) (lzo_memops_TU0p) (dd); \ const lzo_memops_TU1p s__n = (const lzo_memops_TU1p) (const lzo_memops_TU0p) (ss); \ lzo_uint n__n = (nn); \ while ((void)0, n__n >= 8) { LZO_MEMOPS_COPY8(d__n, s__n); d__n += 8; s__n += 8; n__n -= 8; } \ if ((void)0, n__n >= 4) { LZO_MEMOPS_COPY4(d__n, s__n); d__n += 4; s__n += 4; n__n -= 4; } \ if ((void)0, n__n > 0) do { *d__n++ = *s__n++; } while (--n__n > 0); \ LZO_BLOCK_END __lzo_static_forceinline lzo_uint16_t lzo_memops_get_le16(const lzo_voidp ss) { lzo_uint16_t v; #if (LZO_ABI_LITTLE_ENDIAN) LZO_MEMOPS_COPY2(&v, ss); #elif (LZO_OPT_UNALIGNED16 && LZO_ARCH_POWERPC && LZO_ABI_BIG_ENDIAN) && (LZO_ASM_SYNTAX_GNUC) const lzo_memops_TU2p s = (const lzo_memops_TU2p) ss; unsigned long vv; __asm__("lhbrx %0,0,%1" : "=r" (vv) : "r" (s), "m" (*s)); v = (lzo_uint16_t) vv; #else const lzo_memops_TU1p s = (const lzo_memops_TU1p) ss; v = (lzo_uint16_t) (((lzo_uint16_t)s[0]) | ((lzo_uint16_t)s[1] << 8)); #endif return v; } #if (LZO_OPT_UNALIGNED16) && (LZO_ABI_LITTLE_ENDIAN) #define LZO_MEMOPS_GET_LE16(ss) (* (const lzo_memops_TU2p) (const lzo_memops_TU0p) (ss)) #else #define LZO_MEMOPS_GET_LE16(ss) lzo_memops_get_le16(ss) #endif __lzo_static_forceinline lzo_uint32_t lzo_memops_get_le32(const lzo_voidp ss) { lzo_uint32_t v; #if (LZO_ABI_LITTLE_ENDIAN) LZO_MEMOPS_COPY4(&v, ss); #elif (LZO_OPT_UNALIGNED32 && LZO_ARCH_POWERPC && LZO_ABI_BIG_ENDIAN) && (LZO_ASM_SYNTAX_GNUC) const lzo_memops_TU4p s = (const lzo_memops_TU4p) ss; unsigned long vv; __asm__("lwbrx %0,0,%1" : "=r" (vv) : "r" (s), "m" (*s)); v = (lzo_uint32_t) vv; #else const lzo_memops_TU1p s = (const lzo_memops_TU1p) ss; v = (lzo_uint32_t) (((lzo_uint32_t)s[0]) | ((lzo_uint32_t)s[1] << 8) | ((lzo_uint32_t)s[2] << 16) | ((lzo_uint32_t)s[3] << 24)); #endif return v; } #if (LZO_OPT_UNALIGNED32) && (LZO_ABI_LITTLE_ENDIAN) #define LZO_MEMOPS_GET_LE32(ss) (* (const lzo_memops_TU4p) (const lzo_memops_TU0p) (ss)) #else #define LZO_MEMOPS_GET_LE32(ss) lzo_memops_get_le32(ss) #endif #if (LZO_OPT_UNALIGNED64) && (LZO_ABI_LITTLE_ENDIAN) #define LZO_MEMOPS_GET_LE64(ss) (* (const lzo_memops_TU8p) (const lzo_memops_TU0p) (ss)) #endif __lzo_static_forceinline lzo_uint16_t lzo_memops_get_ne16(const lzo_voidp ss) { lzo_uint16_t v; LZO_MEMOPS_COPY2(&v, ss); return v; } #if (LZO_OPT_UNALIGNED16) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(*(lzo_memops_TU2p)0)==2) #define LZO_MEMOPS_GET_NE16(ss) (* (const lzo_memops_TU2p) (const lzo_memops_TU0p) (ss)) #else #define LZO_MEMOPS_GET_NE16(ss) lzo_memops_get_ne16(ss) #endif __lzo_static_forceinline lzo_uint32_t lzo_memops_get_ne32(const lzo_voidp ss) { lzo_uint32_t v; LZO_MEMOPS_COPY4(&v, ss); return v; } #if (LZO_OPT_UNALIGNED32) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(*(lzo_memops_TU4p)0)==4) #define LZO_MEMOPS_GET_NE32(ss) (* (const lzo_memops_TU4p) (const lzo_memops_TU0p) (ss)) #else #define LZO_MEMOPS_GET_NE32(ss) lzo_memops_get_ne32(ss) #endif #if (LZO_OPT_UNALIGNED64) LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(*(lzo_memops_TU8p)0)==8) #define LZO_MEMOPS_GET_NE64(ss) (* (const lzo_memops_TU8p) (const lzo_memops_TU0p) (ss)) #endif __lzo_static_forceinline void lzo_memops_put_le16(lzo_voidp dd, lzo_uint16_t vv) { #if (LZO_ABI_LITTLE_ENDIAN) LZO_MEMOPS_COPY2(dd, &vv); #elif (LZO_OPT_UNALIGNED16 && LZO_ARCH_POWERPC && LZO_ABI_BIG_ENDIAN) && (LZO_ASM_SYNTAX_GNUC) lzo_memops_TU2p d = (lzo_memops_TU2p) dd; unsigned long v = vv; __asm__("sthbrx %2,0,%1" : "=m" (*d) : "r" (d), "r" (v)); #else lzo_memops_TU1p d = (lzo_memops_TU1p) dd; d[0] = LZO_BYTE((vv ) & 0xff); d[1] = LZO_BYTE((vv >> 8) & 0xff); #endif } #if (LZO_OPT_UNALIGNED16) && (LZO_ABI_LITTLE_ENDIAN) #define LZO_MEMOPS_PUT_LE16(dd,vv) (* (lzo_memops_TU2p) (lzo_memops_TU0p) (dd) = (vv)) #else #define LZO_MEMOPS_PUT_LE16(dd,vv) lzo_memops_put_le16(dd,vv) #endif __lzo_static_forceinline void lzo_memops_put_le32(lzo_voidp dd, lzo_uint32_t vv) { #if (LZO_ABI_LITTLE_ENDIAN) LZO_MEMOPS_COPY4(dd, &vv); #elif (LZO_OPT_UNALIGNED32 && LZO_ARCH_POWERPC && LZO_ABI_BIG_ENDIAN) && (LZO_ASM_SYNTAX_GNUC) lzo_memops_TU4p d = (lzo_memops_TU4p) dd; unsigned long v = vv; __asm__("stwbrx %2,0,%1" : "=m" (*d) : "r" (d), "r" (v)); #else lzo_memops_TU1p d = (lzo_memops_TU1p) dd; d[0] = LZO_BYTE((vv ) & 0xff); d[1] = LZO_BYTE((vv >> 8) & 0xff); d[2] = LZO_BYTE((vv >> 16) & 0xff); d[3] = LZO_BYTE((vv >> 24) & 0xff); #endif } #if (LZO_OPT_UNALIGNED32) && (LZO_ABI_LITTLE_ENDIAN) #define LZO_MEMOPS_PUT_LE32(dd,vv) (* (lzo_memops_TU4p) (lzo_memops_TU0p) (dd) = (vv)) #else #define LZO_MEMOPS_PUT_LE32(dd,vv) lzo_memops_put_le32(dd,vv) #endif __lzo_static_forceinline void lzo_memops_put_ne16(lzo_voidp dd, lzo_uint16_t vv) { LZO_MEMOPS_COPY2(dd, &vv); } #if (LZO_OPT_UNALIGNED16) #define LZO_MEMOPS_PUT_NE16(dd,vv) (* (lzo_memops_TU2p) (lzo_memops_TU0p) (dd) = (vv)) #else #define LZO_MEMOPS_PUT_NE16(dd,vv) lzo_memops_put_ne16(dd,vv) #endif __lzo_static_forceinline void lzo_memops_put_ne32(lzo_voidp dd, lzo_uint32_t vv) { LZO_MEMOPS_COPY4(dd, &vv); } #if (LZO_OPT_UNALIGNED32) #define LZO_MEMOPS_PUT_NE32(dd,vv) (* (lzo_memops_TU4p) (lzo_memops_TU0p) (dd) = (vv)) #else #define LZO_MEMOPS_PUT_NE32(dd,vv) lzo_memops_put_ne32(dd,vv) #endif lzo_unused_funcs_impl(void, lzo_memops_unused_funcs)(void) { LZO_UNUSED_FUNC(lzo_memops_unused_funcs); LZO_UNUSED_FUNC(lzo_memops_get_le16); LZO_UNUSED_FUNC(lzo_memops_get_le32); LZO_UNUSED_FUNC(lzo_memops_get_ne16); LZO_UNUSED_FUNC(lzo_memops_get_ne32); LZO_UNUSED_FUNC(lzo_memops_put_le16); LZO_UNUSED_FUNC(lzo_memops_put_le32); LZO_UNUSED_FUNC(lzo_memops_put_ne16); LZO_UNUSED_FUNC(lzo_memops_put_ne32); } #endif #ifndef UA_SET1 #define UA_SET1 LZO_MEMOPS_SET1 #endif #ifndef UA_SET2 #define UA_SET2 LZO_MEMOPS_SET2 #endif #ifndef UA_SET3 #define UA_SET3 LZO_MEMOPS_SET3 #endif #ifndef UA_SET4 #define UA_SET4 LZO_MEMOPS_SET4 #endif #ifndef UA_MOVE1 #define UA_MOVE1 LZO_MEMOPS_MOVE1 #endif #ifndef UA_MOVE2 #define UA_MOVE2 LZO_MEMOPS_MOVE2 #endif #ifndef UA_MOVE3 #define UA_MOVE3 LZO_MEMOPS_MOVE3 #endif #ifndef UA_MOVE4 #define UA_MOVE4 LZO_MEMOPS_MOVE4 #endif #ifndef UA_MOVE8 #define UA_MOVE8 LZO_MEMOPS_MOVE8 #endif #ifndef UA_COPY1 #define UA_COPY1 LZO_MEMOPS_COPY1 #endif #ifndef UA_COPY2 #define UA_COPY2 LZO_MEMOPS_COPY2 #endif #ifndef UA_COPY3 #define UA_COPY3 LZO_MEMOPS_COPY3 #endif #ifndef UA_COPY4 #define UA_COPY4 LZO_MEMOPS_COPY4 #endif #ifndef UA_COPY8 #define UA_COPY8 LZO_MEMOPS_COPY8 #endif #ifndef UA_COPYN #define UA_COPYN LZO_MEMOPS_COPYN #endif #ifndef UA_COPYN_X #define UA_COPYN_X LZO_MEMOPS_COPYN #endif #ifndef UA_GET_LE16 #define UA_GET_LE16 LZO_MEMOPS_GET_LE16 #endif #ifndef UA_GET_LE32 #define UA_GET_LE32 LZO_MEMOPS_GET_LE32 #endif #ifdef LZO_MEMOPS_GET_LE64 #ifndef UA_GET_LE64 #define UA_GET_LE64 LZO_MEMOPS_GET_LE64 #endif #endif #ifndef UA_GET_NE16 #define UA_GET_NE16 LZO_MEMOPS_GET_NE16 #endif #ifndef UA_GET_NE32 #define UA_GET_NE32 LZO_MEMOPS_GET_NE32 #endif #ifdef LZO_MEMOPS_GET_NE64 #ifndef UA_GET_NE64 #define UA_GET_NE64 LZO_MEMOPS_GET_NE64 #endif #endif #ifndef UA_PUT_LE16 #define UA_PUT_LE16 LZO_MEMOPS_PUT_LE16 #endif #ifndef UA_PUT_LE32 #define UA_PUT_LE32 LZO_MEMOPS_PUT_LE32 #endif #ifndef UA_PUT_NE16 #define UA_PUT_NE16 LZO_MEMOPS_PUT_NE16 #endif #ifndef UA_PUT_NE32 #define UA_PUT_NE32 LZO_MEMOPS_PUT_NE32 #endif #define MEMCPY8_DS(dest,src,len) \ lzo_memcpy(dest,src,len); dest += len; src += len #define BZERO8_PTR(s,l,n) \ lzo_memset((lzo_voidp)(s),0,(lzo_uint)(l)*(n)) #define MEMCPY_DS(dest,src,len) \ do *dest++ = *src++; while (--len > 0) LZO_EXTERN(const lzo_bytep) lzo_copyright(void); #ifndef __LZO_PTR_H #define __LZO_PTR_H 1 #ifdef __cplusplus extern "C" { #endif #if (LZO_ARCH_I086) #error "LZO_ARCH_I086 is unsupported" #elif (LZO_MM_PVP) #error "LZO_MM_PVP is unsupported" #else #define PTR(a) ((lzo_uintptr_t) (a)) #define PTR_LINEAR(a) PTR(a) #define PTR_ALIGNED_4(a) ((PTR_LINEAR(a) & 3) == 0) #define PTR_ALIGNED_8(a) ((PTR_LINEAR(a) & 7) == 0) #define PTR_ALIGNED2_4(a,b) (((PTR_LINEAR(a) | PTR_LINEAR(b)) & 3) == 0) #define PTR_ALIGNED2_8(a,b) (((PTR_LINEAR(a) | PTR_LINEAR(b)) & 7) == 0) #endif #define PTR_LT(a,b) (PTR(a) < PTR(b)) #define PTR_GE(a,b) (PTR(a) >= PTR(b)) #define PTR_DIFF(a,b) (PTR(a) - PTR(b)) #define pd(a,b) ((lzo_uint) ((a)-(b))) LZO_EXTERN(lzo_uintptr_t) __lzo_ptr_linear(const lzo_voidp ptr); typedef union { char a_char; unsigned char a_uchar; short a_short; unsigned short a_ushort; int a_int; unsigned int a_uint; long a_long; unsigned long a_ulong; lzo_int a_lzo_int; lzo_uint a_lzo_uint; lzo_xint a_lzo_xint; lzo_int16_t a_lzo_int16_t; lzo_uint16_t a_lzo_uint16_t; lzo_int32_t a_lzo_int32_t; lzo_uint32_t a_lzo_uint32_t; #if defined(lzo_uint64_t) lzo_int64_t a_lzo_int64_t; lzo_uint64_t a_lzo_uint64_t; #endif size_t a_size_t; ptrdiff_t a_ptrdiff_t; lzo_uintptr_t a_lzo_uintptr_t; void * a_void_p; char * a_char_p; unsigned char * a_uchar_p; const void * a_c_void_p; const char * a_c_char_p; const unsigned char * a_c_uchar_p; lzo_voidp a_lzo_voidp; lzo_bytep a_lzo_bytep; const lzo_voidp a_c_lzo_voidp; const lzo_bytep a_c_lzo_bytep; } lzo_full_align_t; #ifdef __cplusplus } #endif #endif #ifndef LZO_DETERMINISTIC #define LZO_DETERMINISTIC 1 #endif #ifndef LZO_DICT_USE_PTR #define LZO_DICT_USE_PTR 1 #endif #if (LZO_DICT_USE_PTR) # define lzo_dict_t const lzo_bytep # define lzo_dict_p lzo_dict_t * #else # define lzo_dict_t lzo_uint # define lzo_dict_p lzo_dict_t * #endif #endif #if !defined(MINILZO_CFG_SKIP_LZO_PTR) LZO_PUBLIC(lzo_uintptr_t) __lzo_ptr_linear(const lzo_voidp ptr) { lzo_uintptr_t p; #if (LZO_ARCH_I086) #error "LZO_ARCH_I086 is unsupported" #elif (LZO_MM_PVP) #error "LZO_MM_PVP is unsupported" #else p = (lzo_uintptr_t) PTR_LINEAR(ptr); #endif return p; } LZO_PUBLIC(unsigned) __lzo_align_gap(const lzo_voidp ptr, lzo_uint size) { #if (__LZO_UINTPTR_T_IS_POINTER) #error "__LZO_UINTPTR_T_IS_POINTER is unsupported" #else lzo_uintptr_t p, n; if (size < 2) return 0; p = __lzo_ptr_linear(ptr); #if 0 n = (((p + size - 1) / size) * size) - p; #else if ((size & (size - 1)) != 0) return 0; n = size; n = ((p + n - 1) & ~(n - 1)) - p; #endif #endif assert((long)n >= 0); assert(n <= size); return (unsigned)n; } #endif #if !defined(MINILZO_CFG_SKIP_LZO_UTIL) /* If you use the LZO library in a product, I would appreciate that you * keep this copyright string in the executable of your product. */ static const char lzo_copyright_[] = #if !defined(__LZO_IN_MINLZO) LZO_VERSION_STRING; #else "\r\n\n" "LZO data compression library.\n" "$Copyright: LZO Copyright (C) 1996-2017 Markus Franz Xaver Johannes Oberhumer\n" "\n" "http://www.oberhumer.com $\n\n" "$Id: LZO version: v" LZO_VERSION_STRING ", " LZO_VERSION_DATE " $\n" "$Info: " LZO_INFO_STRING " $\n"; #endif static const char lzo_version_string_[] = LZO_VERSION_STRING; static const char lzo_version_date_[] = LZO_VERSION_DATE; LZO_PUBLIC(const lzo_bytep) lzo_copyright(void) { return (const lzo_bytep) lzo_copyright_; } LZO_PUBLIC(unsigned) lzo_version(void) { return LZO_VERSION; } LZO_PUBLIC(const char *) lzo_version_string(void) { return lzo_version_string_; } LZO_PUBLIC(const char *) lzo_version_date(void) { return lzo_version_date_; } LZO_PUBLIC(const lzo_charp) _lzo_version_string(void) { return lzo_version_string_; } LZO_PUBLIC(const lzo_charp) _lzo_version_date(void) { return lzo_version_date_; } #define LZO_BASE 65521u #define LZO_NMAX 5552 #define LZO_DO1(buf,i) s1 += buf[i]; s2 += s1 #define LZO_DO2(buf,i) LZO_DO1(buf,i); LZO_DO1(buf,i+1) #define LZO_DO4(buf,i) LZO_DO2(buf,i); LZO_DO2(buf,i+2) #define LZO_DO8(buf,i) LZO_DO4(buf,i); LZO_DO4(buf,i+4) #define LZO_DO16(buf,i) LZO_DO8(buf,i); LZO_DO8(buf,i+8) LZO_PUBLIC(lzo_uint32_t) lzo_adler32(lzo_uint32_t adler, const lzo_bytep buf, lzo_uint len) { lzo_uint32_t s1 = adler & 0xffff; lzo_uint32_t s2 = (adler >> 16) & 0xffff; unsigned k; if (buf == NULL) return 1; while (len > 0) { k = len < LZO_NMAX ? (unsigned) len : LZO_NMAX; len -= k; if (k >= 16) do { LZO_DO16(buf,0); buf += 16; k -= 16; } while (k >= 16); if (k != 0) do { s1 += *buf++; s2 += s1; } while (--k > 0); s1 %= LZO_BASE; s2 %= LZO_BASE; } return (s2 << 16) | s1; } #undef LZO_DO1 #undef LZO_DO2 #undef LZO_DO4 #undef LZO_DO8 #undef LZO_DO16 #endif #if !defined(MINILZO_CFG_SKIP_LZO_STRING) #undef lzo_memcmp #undef lzo_memcpy #undef lzo_memmove #undef lzo_memset #if !defined(__LZO_MMODEL_HUGE) # undef LZO_HAVE_MM_HUGE_PTR #endif #define lzo_hsize_t lzo_uint #define lzo_hvoid_p lzo_voidp #define lzo_hbyte_p lzo_bytep #define LZOLIB_PUBLIC(r,f) LZO_PUBLIC(r) f #define lzo_hmemcmp lzo_memcmp #define lzo_hmemcpy lzo_memcpy #define lzo_hmemmove lzo_memmove #define lzo_hmemset lzo_memset #define __LZOLIB_HMEMCPY_CH_INCLUDED 1 #if !defined(LZOLIB_PUBLIC) # define LZOLIB_PUBLIC(r,f) r __LZOLIB_FUNCNAME(f) #endif LZOLIB_PUBLIC(int, lzo_hmemcmp) (const lzo_hvoid_p s1, const lzo_hvoid_p s2, lzo_hsize_t len) { #if (LZO_HAVE_MM_HUGE_PTR) || !(HAVE_MEMCMP) const lzo_hbyte_p p1 = LZO_STATIC_CAST(const lzo_hbyte_p, s1); const lzo_hbyte_p p2 = LZO_STATIC_CAST(const lzo_hbyte_p, s2); if __lzo_likely(len > 0) do { int d = *p1 - *p2; if (d != 0) return d; p1++; p2++; } while __lzo_likely(--len > 0); return 0; #else return memcmp(s1, s2, len); #endif } LZOLIB_PUBLIC(lzo_hvoid_p, lzo_hmemcpy) (lzo_hvoid_p dest, const lzo_hvoid_p src, lzo_hsize_t len) { #if (LZO_HAVE_MM_HUGE_PTR) || !(HAVE_MEMCPY) lzo_hbyte_p p1 = LZO_STATIC_CAST(lzo_hbyte_p, dest); const lzo_hbyte_p p2 = LZO_STATIC_CAST(const lzo_hbyte_p, src); if (!(len > 0) || p1 == p2) return dest; do *p1++ = *p2++; while __lzo_likely(--len > 0); return dest; #else return memcpy(dest, src, len); #endif } LZOLIB_PUBLIC(lzo_hvoid_p, lzo_hmemmove) (lzo_hvoid_p dest, const lzo_hvoid_p src, lzo_hsize_t len) { #if (LZO_HAVE_MM_HUGE_PTR) || !(HAVE_MEMMOVE) lzo_hbyte_p p1 = LZO_STATIC_CAST(lzo_hbyte_p, dest); const lzo_hbyte_p p2 = LZO_STATIC_CAST(const lzo_hbyte_p, src); if (!(len > 0) || p1 == p2) return dest; if (p1 < p2) { do *p1++ = *p2++; while __lzo_likely(--len > 0); } else { p1 += len; p2 += len; do *--p1 = *--p2; while __lzo_likely(--len > 0); } return dest; #else return memmove(dest, src, len); #endif } LZOLIB_PUBLIC(lzo_hvoid_p, lzo_hmemset) (lzo_hvoid_p s, int cc, lzo_hsize_t len) { #if (LZO_HAVE_MM_HUGE_PTR) || !(HAVE_MEMSET) lzo_hbyte_p p = LZO_STATIC_CAST(lzo_hbyte_p, s); unsigned char c = LZO_ITRUNC(unsigned char, cc); if __lzo_likely(len > 0) do *p++ = c; while __lzo_likely(--len > 0); return s; #else return memset(s, cc, len); #endif } #undef LZOLIB_PUBLIC #endif #if !defined(MINILZO_CFG_SKIP_LZO_INIT) #if !defined(__LZO_IN_MINILZO) #define LZO_WANT_ACC_CHK_CH 1 #undef LZOCHK_ASSERT LZOCHK_ASSERT((LZO_UINT32_C(1) << (int)(8*sizeof(LZO_UINT32_C(1))-1)) > 0) LZOCHK_ASSERT_IS_SIGNED_T(lzo_int) LZOCHK_ASSERT_IS_UNSIGNED_T(lzo_uint) #if !(__LZO_UINTPTR_T_IS_POINTER) LZOCHK_ASSERT_IS_UNSIGNED_T(lzo_uintptr_t) #endif LZOCHK_ASSERT(sizeof(lzo_uintptr_t) >= sizeof(lzo_voidp)) LZOCHK_ASSERT_IS_UNSIGNED_T(lzo_xint) #endif #undef LZOCHK_ASSERT union lzo_config_check_union { lzo_uint a[2]; unsigned char b[2*LZO_MAX(8,sizeof(lzo_uint))]; #if defined(lzo_uint64_t) lzo_uint64_t c[2]; #endif }; #if 0 #define u2p(ptr,off) ((lzo_voidp) (((lzo_bytep)(lzo_voidp)(ptr)) + (off))) #else static __lzo_noinline lzo_voidp u2p(lzo_voidp ptr, lzo_uint off) { return (lzo_voidp) ((lzo_bytep) ptr + off); } #endif LZO_PUBLIC(int) _lzo_config_check(void) { #if (LZO_CC_CLANG && (LZO_CC_CLANG >= 0x030100ul && LZO_CC_CLANG < 0x030300ul)) # if 0 volatile # endif #endif union lzo_config_check_union u; lzo_voidp p; unsigned r = 1; u.a[0] = u.a[1] = 0; p = u2p(&u, 0); r &= ((* (lzo_bytep) p) == 0); #if !(LZO_CFG_NO_CONFIG_CHECK) #if (LZO_ABI_BIG_ENDIAN) u.a[0] = u.a[1] = 0; u.b[sizeof(lzo_uint) - 1] = 128; p = u2p(&u, 0); r &= ((* (lzo_uintp) p) == 128); #endif #if (LZO_ABI_LITTLE_ENDIAN) u.a[0] = u.a[1] = 0; u.b[0] = 128; p = u2p(&u, 0); r &= ((* (lzo_uintp) p) == 128); #endif u.a[0] = u.a[1] = 0; u.b[0] = 1; u.b[3] = 2; p = u2p(&u, 1); r &= UA_GET_NE16(p) == 0; r &= UA_GET_LE16(p) == 0; u.b[1] = 128; r &= UA_GET_LE16(p) == 128; u.b[2] = 129; r &= UA_GET_LE16(p) == LZO_UINT16_C(0x8180); #if (LZO_ABI_BIG_ENDIAN) r &= UA_GET_NE16(p) == LZO_UINT16_C(0x8081); #endif #if (LZO_ABI_LITTLE_ENDIAN) r &= UA_GET_NE16(p) == LZO_UINT16_C(0x8180); #endif u.a[0] = u.a[1] = 0; u.b[0] = 3; u.b[5] = 4; p = u2p(&u, 1); r &= UA_GET_NE32(p) == 0; r &= UA_GET_LE32(p) == 0; u.b[1] = 128; r &= UA_GET_LE32(p) == 128; u.b[2] = 129; u.b[3] = 130; u.b[4] = 131; r &= UA_GET_LE32(p) == LZO_UINT32_C(0x83828180); #if (LZO_ABI_BIG_ENDIAN) r &= UA_GET_NE32(p) == LZO_UINT32_C(0x80818283); #endif #if (LZO_ABI_LITTLE_ENDIAN) r &= UA_GET_NE32(p) == LZO_UINT32_C(0x83828180); #endif #if defined(UA_GET_NE64) u.c[0] = u.c[1] = 0; u.b[0] = 5; u.b[9] = 6; p = u2p(&u, 1); u.c[0] = u.c[1] = 0; r &= UA_GET_NE64(p) == 0; #if defined(UA_GET_LE64) r &= UA_GET_LE64(p) == 0; u.b[1] = 128; r &= UA_GET_LE64(p) == 128; #endif #endif #if defined(lzo_bitops_ctlz32) { unsigned i = 0; lzo_uint32_t v; for (v = 1; v != 0 && r == 1; v <<= 1, i++) { r &= lzo_bitops_ctlz32(v) == 31 - i; r &= lzo_bitops_ctlz32_func(v) == 31 - i; }} #endif #if defined(lzo_bitops_ctlz64) { unsigned i = 0; lzo_uint64_t v; for (v = 1; v != 0 && r == 1; v <<= 1, i++) { r &= lzo_bitops_ctlz64(v) == 63 - i; r &= lzo_bitops_ctlz64_func(v) == 63 - i; }} #endif #if defined(lzo_bitops_cttz32) { unsigned i = 0; lzo_uint32_t v; for (v = 1; v != 0 && r == 1; v <<= 1, i++) { r &= lzo_bitops_cttz32(v) == i; r &= lzo_bitops_cttz32_func(v) == i; }} #endif #if defined(lzo_bitops_cttz64) { unsigned i = 0; lzo_uint64_t v; for (v = 1; v != 0 && r == 1; v <<= 1, i++) { r &= lzo_bitops_cttz64(v) == i; r &= lzo_bitops_cttz64_func(v) == i; }} #endif #endif LZO_UNUSED_FUNC(lzo_bitops_unused_funcs); return r == 1 ? LZO_E_OK : LZO_E_ERROR; } LZO_PUBLIC(int) __lzo_init_v2(unsigned v, int s1, int s2, int s3, int s4, int s5, int s6, int s7, int s8, int s9) { int r; #if defined(__LZO_IN_MINILZO) #elif (LZO_CC_MSC && ((_MSC_VER) < 700)) #else #define LZO_WANT_ACC_CHK_CH 1 #undef LZOCHK_ASSERT #define LZOCHK_ASSERT(expr) LZO_COMPILE_TIME_ASSERT(expr) #endif #undef LZOCHK_ASSERT if (v == 0) return LZO_E_ERROR; r = (s1 == -1 || s1 == (int) sizeof(short)) && (s2 == -1 || s2 == (int) sizeof(int)) && (s3 == -1 || s3 == (int) sizeof(long)) && (s4 == -1 || s4 == (int) sizeof(lzo_uint32_t)) && (s5 == -1 || s5 == (int) sizeof(lzo_uint)) && (s6 == -1 || s6 == (int) lzo_sizeof_dict_t) && (s7 == -1 || s7 == (int) sizeof(char *)) && (s8 == -1 || s8 == (int) sizeof(lzo_voidp)) && (s9 == -1 || s9 == (int) sizeof(lzo_callback_t)); if (!r) return LZO_E_ERROR; r = _lzo_config_check(); if (r != LZO_E_OK) return r; return r; } #if !defined(__LZO_IN_MINILZO) #if (LZO_OS_WIN16 && LZO_CC_WATCOMC) && defined(__SW_BD) #if 0 BOOL FAR PASCAL LibMain ( HANDLE hInstance, WORD wDataSegment, WORD wHeapSize, LPSTR lpszCmdLine ) #else int __far __pascal LibMain ( int a, short b, short c, long d ) #endif { LZO_UNUSED(a); LZO_UNUSED(b); LZO_UNUSED(c); LZO_UNUSED(d); return 1; } #endif #endif #endif #define LZO1X 1 #define LZO_EOF_CODE 1 #define M2_MAX_OFFSET 0x0800 #if !defined(MINILZO_CFG_SKIP_LZO1X_1_COMPRESS) #if 1 && defined(UA_GET_LE32) #undef LZO_DICT_USE_PTR #define LZO_DICT_USE_PTR 0 #undef lzo_dict_t #define lzo_dict_t lzo_uint16_t #endif #define LZO_NEED_DICT_H 1 #ifndef D_BITS #define D_BITS 14 #endif #define D_INDEX1(d,p) d = DM(DMUL(0x21,DX3(p,5,5,6)) >> 5) #define D_INDEX2(d,p) d = (d & (D_MASK & 0x7ff)) ^ (D_HIGH | 0x1f) #if 1 #define DINDEX(dv,p) DM(((DMUL(0x1824429d,dv)) >> (32-D_BITS))) #else #define DINDEX(dv,p) DM((dv) + ((dv) >> (32-D_BITS))) #endif #ifndef __LZO_CONFIG1X_H #define __LZO_CONFIG1X_H 1 #if !defined(LZO1X) && !defined(LZO1Y) && !defined(LZO1Z) # define LZO1X 1 #endif #if !defined(__LZO_IN_MINILZO) #include #endif #ifndef LZO_EOF_CODE #define LZO_EOF_CODE 1 #endif #undef LZO_DETERMINISTIC #define M1_MAX_OFFSET 0x0400 #ifndef M2_MAX_OFFSET #define M2_MAX_OFFSET 0x0800 #endif #define M3_MAX_OFFSET 0x4000 #define M4_MAX_OFFSET 0xbfff #define MX_MAX_OFFSET (M1_MAX_OFFSET + M2_MAX_OFFSET) #define M1_MIN_LEN 2 #define M1_MAX_LEN 2 #define M2_MIN_LEN 3 #ifndef M2_MAX_LEN #define M2_MAX_LEN 8 #endif #define M3_MIN_LEN 3 #define M3_MAX_LEN 33 #define M4_MIN_LEN 3 #define M4_MAX_LEN 9 #define M1_MARKER 0 #define M2_MARKER 64 #define M3_MARKER 32 #define M4_MARKER 16 #ifndef MIN_LOOKAHEAD #define MIN_LOOKAHEAD (M2_MAX_LEN + 1) #endif #if defined(LZO_NEED_DICT_H) #ifndef LZO_HASH #define LZO_HASH LZO_HASH_LZO_INCREMENTAL_B #endif #define DL_MIN_LEN M2_MIN_LEN #ifndef __LZO_DICT_H #define __LZO_DICT_H 1 #ifdef __cplusplus extern "C" { #endif #if !defined(D_BITS) && defined(DBITS) # define D_BITS DBITS #endif #if !defined(D_BITS) # error "D_BITS is not defined" #endif #if (D_BITS < 16) # define D_SIZE LZO_SIZE(D_BITS) # define D_MASK LZO_MASK(D_BITS) #else # define D_SIZE LZO_USIZE(D_BITS) # define D_MASK LZO_UMASK(D_BITS) #endif #define D_HIGH ((D_MASK >> 1) + 1) #if !defined(DD_BITS) # define DD_BITS 0 #endif #define DD_SIZE LZO_SIZE(DD_BITS) #define DD_MASK LZO_MASK(DD_BITS) #if !defined(DL_BITS) # define DL_BITS (D_BITS - DD_BITS) #endif #if (DL_BITS < 16) # define DL_SIZE LZO_SIZE(DL_BITS) # define DL_MASK LZO_MASK(DL_BITS) #else # define DL_SIZE LZO_USIZE(DL_BITS) # define DL_MASK LZO_UMASK(DL_BITS) #endif #if (D_BITS != DL_BITS + DD_BITS) # error "D_BITS does not match" #endif #if (D_BITS < 6 || D_BITS > 18) # error "invalid D_BITS" #endif #if (DL_BITS < 6 || DL_BITS > 20) # error "invalid DL_BITS" #endif #if (DD_BITS < 0 || DD_BITS > 6) # error "invalid DD_BITS" #endif #if !defined(DL_MIN_LEN) # define DL_MIN_LEN 3 #endif #if !defined(DL_SHIFT) # define DL_SHIFT ((DL_BITS + (DL_MIN_LEN - 1)) / DL_MIN_LEN) #endif #define LZO_HASH_GZIP 1 #define LZO_HASH_GZIP_INCREMENTAL 2 #define LZO_HASH_LZO_INCREMENTAL_A 3 #define LZO_HASH_LZO_INCREMENTAL_B 4 #if !defined(LZO_HASH) # error "choose a hashing strategy" #endif #undef DM #undef DX #if (DL_MIN_LEN == 3) # define _DV2_A(p,shift1,shift2) \ (((( (lzo_xint)((p)[0]) << shift1) ^ (p)[1]) << shift2) ^ (p)[2]) # define _DV2_B(p,shift1,shift2) \ (((( (lzo_xint)((p)[2]) << shift1) ^ (p)[1]) << shift2) ^ (p)[0]) # define _DV3_B(p,shift1,shift2,shift3) \ ((_DV2_B((p)+1,shift1,shift2) << (shift3)) ^ (p)[0]) #elif (DL_MIN_LEN == 2) # define _DV2_A(p,shift1,shift2) \ (( (lzo_xint)(p[0]) << shift1) ^ p[1]) # define _DV2_B(p,shift1,shift2) \ (( (lzo_xint)(p[1]) << shift1) ^ p[2]) #else # error "invalid DL_MIN_LEN" #endif #define _DV_A(p,shift) _DV2_A(p,shift,shift) #define _DV_B(p,shift) _DV2_B(p,shift,shift) #define DA2(p,s1,s2) \ (((((lzo_xint)((p)[2]) << (s2)) + (p)[1]) << (s1)) + (p)[0]) #define DS2(p,s1,s2) \ (((((lzo_xint)((p)[2]) << (s2)) - (p)[1]) << (s1)) - (p)[0]) #define DX2(p,s1,s2) \ (((((lzo_xint)((p)[2]) << (s2)) ^ (p)[1]) << (s1)) ^ (p)[0]) #define DA3(p,s1,s2,s3) ((DA2((p)+1,s2,s3) << (s1)) + (p)[0]) #define DS3(p,s1,s2,s3) ((DS2((p)+1,s2,s3) << (s1)) - (p)[0]) #define DX3(p,s1,s2,s3) ((DX2((p)+1,s2,s3) << (s1)) ^ (p)[0]) #define DMS(v,s) ((lzo_uint) (((v) & (D_MASK >> (s))) << (s))) #define DM(v) DMS(v,0) #if (LZO_HASH == LZO_HASH_GZIP) # define _DINDEX(dv,p) (_DV_A((p),DL_SHIFT)) #elif (LZO_HASH == LZO_HASH_GZIP_INCREMENTAL) # define __LZO_HASH_INCREMENTAL 1 # define DVAL_FIRST(dv,p) dv = _DV_A((p),DL_SHIFT) # define DVAL_NEXT(dv,p) dv = (((dv) << DL_SHIFT) ^ p[2]) # define _DINDEX(dv,p) (dv) # define DVAL_LOOKAHEAD DL_MIN_LEN #elif (LZO_HASH == LZO_HASH_LZO_INCREMENTAL_A) # define __LZO_HASH_INCREMENTAL 1 # define DVAL_FIRST(dv,p) dv = _DV_A((p),5) # define DVAL_NEXT(dv,p) \ dv ^= (lzo_xint)(p[-1]) << (2*5); dv = (((dv) << 5) ^ p[2]) # define _DINDEX(dv,p) ((DMUL(0x9f5f,dv)) >> 5) # define DVAL_LOOKAHEAD DL_MIN_LEN #elif (LZO_HASH == LZO_HASH_LZO_INCREMENTAL_B) # define __LZO_HASH_INCREMENTAL 1 # define DVAL_FIRST(dv,p) dv = _DV_B((p),5) # define DVAL_NEXT(dv,p) \ dv ^= p[-1]; dv = (((dv) >> 5) ^ ((lzo_xint)(p[2]) << (2*5))) # define _DINDEX(dv,p) ((DMUL(0x9f5f,dv)) >> 5) # define DVAL_LOOKAHEAD DL_MIN_LEN #else # error "choose a hashing strategy" #endif #ifndef DINDEX #define DINDEX(dv,p) ((lzo_uint)((_DINDEX(dv,p)) & DL_MASK) << DD_BITS) #endif #if !defined(DINDEX1) && defined(D_INDEX1) #define DINDEX1 D_INDEX1 #endif #if !defined(DINDEX2) && defined(D_INDEX2) #define DINDEX2 D_INDEX2 #endif #if !defined(__LZO_HASH_INCREMENTAL) # define DVAL_FIRST(dv,p) ((void) 0) # define DVAL_NEXT(dv,p) ((void) 0) # define DVAL_LOOKAHEAD 0 #endif #if !defined(DVAL_ASSERT) #if defined(__LZO_HASH_INCREMENTAL) && !defined(NDEBUG) #if 1 && (LZO_CC_ARMCC_GNUC || LZO_CC_CLANG || (LZO_CC_GNUC >= 0x020700ul) || LZO_CC_INTELC_GNUC || LZO_CC_LLVM || LZO_CC_PATHSCALE || LZO_CC_PGI) static void __attribute__((__unused__)) #else static void #endif DVAL_ASSERT(lzo_xint dv, const lzo_bytep p) { lzo_xint df; DVAL_FIRST(df,(p)); assert(DINDEX(dv,p) == DINDEX(df,p)); } #else # define DVAL_ASSERT(dv,p) ((void) 0) #endif #endif #if (LZO_DICT_USE_PTR) # define DENTRY(p,in) (p) # define GINDEX(m_pos,m_off,dict,dindex,in) m_pos = dict[dindex] #else # define DENTRY(p,in) ((lzo_dict_t) pd(p, in)) # define GINDEX(m_pos,m_off,dict,dindex,in) m_off = dict[dindex] #endif #if (DD_BITS == 0) # define UPDATE_D(dict,drun,dv,p,in) dict[ DINDEX(dv,p) ] = DENTRY(p,in) # define UPDATE_I(dict,drun,index,p,in) dict[index] = DENTRY(p,in) # define UPDATE_P(ptr,drun,p,in) (ptr)[0] = DENTRY(p,in) #else # define UPDATE_D(dict,drun,dv,p,in) \ dict[ DINDEX(dv,p) + drun++ ] = DENTRY(p,in); drun &= DD_MASK # define UPDATE_I(dict,drun,index,p,in) \ dict[ (index) + drun++ ] = DENTRY(p,in); drun &= DD_MASK # define UPDATE_P(ptr,drun,p,in) \ (ptr) [ drun++ ] = DENTRY(p,in); drun &= DD_MASK #endif #if (LZO_DICT_USE_PTR) #define LZO_CHECK_MPOS_DET(m_pos,m_off,in,ip,max_offset) \ (m_pos == NULL || (m_off = pd(ip, m_pos)) > max_offset) #define LZO_CHECK_MPOS_NON_DET(m_pos,m_off,in,ip,max_offset) \ (BOUNDS_CHECKING_OFF_IN_EXPR(( \ m_pos = ip - (lzo_uint) PTR_DIFF(ip,m_pos), \ PTR_LT(m_pos,in) || \ (m_off = (lzo_uint) PTR_DIFF(ip,m_pos)) == 0 || \ m_off > max_offset ))) #else #define LZO_CHECK_MPOS_DET(m_pos,m_off,in,ip,max_offset) \ (m_off == 0 || \ ((m_off = pd(ip, in) - m_off) > max_offset) || \ (m_pos = (ip) - (m_off), 0) ) #define LZO_CHECK_MPOS_NON_DET(m_pos,m_off,in,ip,max_offset) \ (pd(ip, in) <= m_off || \ ((m_off = pd(ip, in) - m_off) > max_offset) || \ (m_pos = (ip) - (m_off), 0) ) #endif #if (LZO_DETERMINISTIC) # define LZO_CHECK_MPOS LZO_CHECK_MPOS_DET #else # define LZO_CHECK_MPOS LZO_CHECK_MPOS_NON_DET #endif #ifdef __cplusplus } #endif #endif #endif #endif #define LZO_DETERMINISTIC !(LZO_DICT_USE_PTR) #ifndef DO_COMPRESS #define DO_COMPRESS lzo1x_1_compress #endif #if 1 && defined(DO_COMPRESS) && !defined(do_compress) # define do_compress LZO_PP_ECONCAT2(DO_COMPRESS,_core) #endif static __lzo_noinline lzo_uint do_compress ( const lzo_bytep in , lzo_uint in_len, lzo_bytep out, lzo_uintp out_len, lzo_uint ti, lzo_voidp wrkmem) { const lzo_bytep ip; lzo_bytep op; const lzo_bytep const in_end = in + in_len; const lzo_bytep const ip_end = in + in_len - 20; const lzo_bytep ii; lzo_dict_p const dict = (lzo_dict_p) wrkmem; op = out; ip = in; ii = ip; ip += ti < 4 ? 4 - ti : 0; for (;;) { const lzo_bytep m_pos; #if !(LZO_DETERMINISTIC) LZO_DEFINE_UNINITIALIZED_VAR(lzo_uint, m_off, 0); lzo_uint m_len; lzo_uint dindex; next: if __lzo_unlikely(ip >= ip_end) break; DINDEX1(dindex,ip); GINDEX(m_pos,m_off,dict,dindex,in); if (LZO_CHECK_MPOS_NON_DET(m_pos,m_off,in,ip,M4_MAX_OFFSET)) goto literal; #if 1 if (m_off <= M2_MAX_OFFSET || m_pos[3] == ip[3]) goto try_match; DINDEX2(dindex,ip); #endif GINDEX(m_pos,m_off,dict,dindex,in); if (LZO_CHECK_MPOS_NON_DET(m_pos,m_off,in,ip,M4_MAX_OFFSET)) goto literal; if (m_off <= M2_MAX_OFFSET || m_pos[3] == ip[3]) goto try_match; goto literal; try_match: #if (LZO_OPT_UNALIGNED32) if (UA_GET_NE32(m_pos) != UA_GET_NE32(ip)) #else if (m_pos[0] != ip[0] || m_pos[1] != ip[1] || m_pos[2] != ip[2] || m_pos[3] != ip[3]) #endif { literal: UPDATE_I(dict,0,dindex,ip,in); ip += 1 + ((ip - ii) >> 5); continue; } UPDATE_I(dict,0,dindex,ip,in); #else lzo_uint m_off; lzo_uint m_len; { lzo_uint32_t dv; lzo_uint dindex; literal: ip += 1 + ((ip - ii) >> 5); next: if __lzo_unlikely(ip >= ip_end) break; dv = UA_GET_LE32(ip); dindex = DINDEX(dv,ip); GINDEX(m_off,m_pos,in+dict,dindex,in); UPDATE_I(dict,0,dindex,ip,in); if __lzo_unlikely(dv != UA_GET_LE32(m_pos)) goto literal; } #endif ii -= ti; ti = 0; { lzo_uint t = pd(ip,ii); if (t != 0) { if (t <= 3) { op[-2] = LZO_BYTE(op[-2] | t); #if (LZO_OPT_UNALIGNED32) UA_COPY4(op, ii); op += t; #else { do *op++ = *ii++; while (--t > 0); } #endif } #if (LZO_OPT_UNALIGNED32) || (LZO_OPT_UNALIGNED64) else if (t <= 16) { *op++ = LZO_BYTE(t - 3); UA_COPY8(op, ii); UA_COPY8(op+8, ii+8); op += t; } #endif else { if (t <= 18) *op++ = LZO_BYTE(t - 3); else { lzo_uint tt = t - 18; *op++ = 0; while __lzo_unlikely(tt > 255) { tt -= 255; UA_SET1(op, 0); op++; } assert(tt > 0); *op++ = LZO_BYTE(tt); } #if (LZO_OPT_UNALIGNED32) || (LZO_OPT_UNALIGNED64) do { UA_COPY8(op, ii); UA_COPY8(op+8, ii+8); op += 16; ii += 16; t -= 16; } while (t >= 16); if (t > 0) #endif { do *op++ = *ii++; while (--t > 0); } } } } m_len = 4; { #if (LZO_OPT_UNALIGNED64) lzo_uint64_t v; v = UA_GET_NE64(ip + m_len) ^ UA_GET_NE64(m_pos + m_len); if __lzo_unlikely(v == 0) { do { m_len += 8; v = UA_GET_NE64(ip + m_len) ^ UA_GET_NE64(m_pos + m_len); if __lzo_unlikely(ip + m_len >= ip_end) goto m_len_done; } while (v == 0); } #if (LZO_ABI_BIG_ENDIAN) && defined(lzo_bitops_ctlz64) m_len += lzo_bitops_ctlz64(v) / CHAR_BIT; #elif (LZO_ABI_BIG_ENDIAN) if ((v >> (64 - CHAR_BIT)) == 0) do { v <<= CHAR_BIT; m_len += 1; } while ((v >> (64 - CHAR_BIT)) == 0); #elif (LZO_ABI_LITTLE_ENDIAN) && defined(lzo_bitops_cttz64) m_len += lzo_bitops_cttz64(v) / CHAR_BIT; #elif (LZO_ABI_LITTLE_ENDIAN) if ((v & UCHAR_MAX) == 0) do { v >>= CHAR_BIT; m_len += 1; } while ((v & UCHAR_MAX) == 0); #else if (ip[m_len] == m_pos[m_len]) do { m_len += 1; } while (ip[m_len] == m_pos[m_len]); #endif #elif (LZO_OPT_UNALIGNED32) lzo_uint32_t v; v = UA_GET_NE32(ip + m_len) ^ UA_GET_NE32(m_pos + m_len); if __lzo_unlikely(v == 0) { do { m_len += 4; v = UA_GET_NE32(ip + m_len) ^ UA_GET_NE32(m_pos + m_len); if (v != 0) break; m_len += 4; v = UA_GET_NE32(ip + m_len) ^ UA_GET_NE32(m_pos + m_len); if __lzo_unlikely(ip + m_len >= ip_end) goto m_len_done; } while (v == 0); } #if (LZO_ABI_BIG_ENDIAN) && defined(lzo_bitops_ctlz32) m_len += lzo_bitops_ctlz32(v) / CHAR_BIT; #elif (LZO_ABI_BIG_ENDIAN) if ((v >> (32 - CHAR_BIT)) == 0) do { v <<= CHAR_BIT; m_len += 1; } while ((v >> (32 - CHAR_BIT)) == 0); #elif (LZO_ABI_LITTLE_ENDIAN) && defined(lzo_bitops_cttz32) m_len += lzo_bitops_cttz32(v) / CHAR_BIT; #elif (LZO_ABI_LITTLE_ENDIAN) if ((v & UCHAR_MAX) == 0) do { v >>= CHAR_BIT; m_len += 1; } while ((v & UCHAR_MAX) == 0); #else if (ip[m_len] == m_pos[m_len]) do { m_len += 1; } while (ip[m_len] == m_pos[m_len]); #endif #else if __lzo_unlikely(ip[m_len] == m_pos[m_len]) { do { m_len += 1; if (ip[m_len] != m_pos[m_len]) break; m_len += 1; if (ip[m_len] != m_pos[m_len]) break; m_len += 1; if (ip[m_len] != m_pos[m_len]) break; m_len += 1; if (ip[m_len] != m_pos[m_len]) break; m_len += 1; if (ip[m_len] != m_pos[m_len]) break; m_len += 1; if (ip[m_len] != m_pos[m_len]) break; m_len += 1; if (ip[m_len] != m_pos[m_len]) break; m_len += 1; if __lzo_unlikely(ip + m_len >= ip_end) goto m_len_done; } while (ip[m_len] == m_pos[m_len]); } #endif } m_len_done: m_off = pd(ip,m_pos); ip += m_len; ii = ip; if (m_len <= M2_MAX_LEN && m_off <= M2_MAX_OFFSET) { m_off -= 1; #if defined(LZO1X) *op++ = LZO_BYTE(((m_len - 1) << 5) | ((m_off & 7) << 2)); *op++ = LZO_BYTE(m_off >> 3); #elif defined(LZO1Y) *op++ = LZO_BYTE(((m_len + 1) << 4) | ((m_off & 3) << 2)); *op++ = LZO_BYTE(m_off >> 2); #endif } else if (m_off <= M3_MAX_OFFSET) { m_off -= 1; if (m_len <= M3_MAX_LEN) *op++ = LZO_BYTE(M3_MARKER | (m_len - 2)); else { m_len -= M3_MAX_LEN; *op++ = M3_MARKER | 0; while __lzo_unlikely(m_len > 255) { m_len -= 255; UA_SET1(op, 0); op++; } *op++ = LZO_BYTE(m_len); } *op++ = LZO_BYTE(m_off << 2); *op++ = LZO_BYTE(m_off >> 6); } else { m_off -= 0x4000; if (m_len <= M4_MAX_LEN) *op++ = LZO_BYTE(M4_MARKER | ((m_off >> 11) & 8) | (m_len - 2)); else { m_len -= M4_MAX_LEN; *op++ = LZO_BYTE(M4_MARKER | ((m_off >> 11) & 8)); while __lzo_unlikely(m_len > 255) { m_len -= 255; UA_SET1(op, 0); op++; } *op++ = LZO_BYTE(m_len); } *op++ = LZO_BYTE(m_off << 2); *op++ = LZO_BYTE(m_off >> 6); } goto next; } *out_len = pd(op, out); return pd(in_end,ii-ti); } LZO_PUBLIC(int) DO_COMPRESS ( const lzo_bytep in , lzo_uint in_len, lzo_bytep out, lzo_uintp out_len, lzo_voidp wrkmem ) { const lzo_bytep ip = in; lzo_bytep op = out; lzo_uint l = in_len; lzo_uint t = 0; while (l > 20) { lzo_uint ll = l; lzo_uintptr_t ll_end; #if 0 || (LZO_DETERMINISTIC) ll = LZO_MIN(ll, 49152); #endif ll_end = (lzo_uintptr_t)ip + ll; if ((ll_end + ((t + ll) >> 5)) <= ll_end || (const lzo_bytep)(ll_end + ((t + ll) >> 5)) <= ip + ll) break; #if (LZO_DETERMINISTIC) lzo_memset(wrkmem, 0, ((lzo_uint)1 << D_BITS) * sizeof(lzo_dict_t)); #endif t = do_compress(ip,ll,op,out_len,t,wrkmem); ip += ll; op += *out_len; l -= ll; } t += l; if (t > 0) { const lzo_bytep ii = in + in_len - t; if (op == out && t <= 238) *op++ = LZO_BYTE(17 + t); else if (t <= 3) op[-2] = LZO_BYTE(op[-2] | t); else if (t <= 18) *op++ = LZO_BYTE(t - 3); else { lzo_uint tt = t - 18; *op++ = 0; while (tt > 255) { tt -= 255; UA_SET1(op, 0); op++; } assert(tt > 0); *op++ = LZO_BYTE(tt); } UA_COPYN(op, ii, t); op += t; } *op++ = M4_MARKER | 1; *op++ = 0; *op++ = 0; *out_len = pd(op, out); return LZO_E_OK; } #endif #undef do_compress #undef DO_COMPRESS #undef LZO_HASH #undef LZO_TEST_OVERRUN #undef DO_DECOMPRESS #define DO_DECOMPRESS lzo1x_decompress #if !defined(MINILZO_CFG_SKIP_LZO1X_DECOMPRESS) #if defined(LZO_TEST_OVERRUN) # if !defined(LZO_TEST_OVERRUN_INPUT) # define LZO_TEST_OVERRUN_INPUT 2 # endif # if !defined(LZO_TEST_OVERRUN_OUTPUT) # define LZO_TEST_OVERRUN_OUTPUT 2 # endif # if !defined(LZO_TEST_OVERRUN_LOOKBEHIND) # define LZO_TEST_OVERRUN_LOOKBEHIND 1 # endif #endif #undef TEST_IP #undef TEST_OP #undef TEST_IP_AND_TEST_OP #undef TEST_LB #undef TEST_LBO #undef NEED_IP #undef NEED_OP #undef TEST_IV #undef TEST_OV #undef HAVE_TEST_IP #undef HAVE_TEST_OP #undef HAVE_NEED_IP #undef HAVE_NEED_OP #undef HAVE_ANY_IP #undef HAVE_ANY_OP #if defined(LZO_TEST_OVERRUN_INPUT) # if (LZO_TEST_OVERRUN_INPUT >= 1) # define TEST_IP (ip < ip_end) # endif # if (LZO_TEST_OVERRUN_INPUT >= 2) # define NEED_IP(x) \ if ((lzo_uint)(ip_end - ip) < (lzo_uint)(x)) goto input_overrun # define TEST_IV(x) if ((x) > (lzo_uint)0 - (511)) goto input_overrun # endif #endif #if defined(LZO_TEST_OVERRUN_OUTPUT) # if (LZO_TEST_OVERRUN_OUTPUT >= 1) # define TEST_OP (op <= op_end) # endif # if (LZO_TEST_OVERRUN_OUTPUT >= 2) # undef TEST_OP # define NEED_OP(x) \ if ((lzo_uint)(op_end - op) < (lzo_uint)(x)) goto output_overrun # define TEST_OV(x) if ((x) > (lzo_uint)0 - (511)) goto output_overrun # endif #endif #if defined(LZO_TEST_OVERRUN_LOOKBEHIND) # define TEST_LB(m_pos) if (PTR_LT(m_pos,out) || PTR_GE(m_pos,op)) goto lookbehind_overrun # define TEST_LBO(m_pos,o) if (PTR_LT(m_pos,out) || PTR_GE(m_pos,op-(o))) goto lookbehind_overrun #else # define TEST_LB(m_pos) ((void) 0) # define TEST_LBO(m_pos,o) ((void) 0) #endif #if !defined(LZO_EOF_CODE) && !defined(TEST_IP) # define TEST_IP (ip < ip_end) #endif #if defined(TEST_IP) # define HAVE_TEST_IP 1 #else # define TEST_IP 1 #endif #if defined(TEST_OP) # define HAVE_TEST_OP 1 #else # define TEST_OP 1 #endif #if defined(HAVE_TEST_IP) && defined(HAVE_TEST_OP) # define TEST_IP_AND_TEST_OP (TEST_IP && TEST_OP) #elif defined(HAVE_TEST_IP) # define TEST_IP_AND_TEST_OP TEST_IP #elif defined(HAVE_TEST_OP) # define TEST_IP_AND_TEST_OP TEST_OP #else # define TEST_IP_AND_TEST_OP 1 #endif #if defined(NEED_IP) # define HAVE_NEED_IP 1 #else # define NEED_IP(x) ((void) 0) # define TEST_IV(x) ((void) 0) #endif #if defined(NEED_OP) # define HAVE_NEED_OP 1 #else # define NEED_OP(x) ((void) 0) # define TEST_OV(x) ((void) 0) #endif #if defined(HAVE_TEST_IP) || defined(HAVE_NEED_IP) # define HAVE_ANY_IP 1 #endif #if defined(HAVE_TEST_OP) || defined(HAVE_NEED_OP) # define HAVE_ANY_OP 1 #endif #if defined(DO_DECOMPRESS) LZO_PUBLIC(int) DO_DECOMPRESS ( const lzo_bytep in , lzo_uint in_len, lzo_bytep out, lzo_uintp out_len, lzo_voidp wrkmem ) #endif { lzo_bytep op; const lzo_bytep ip; lzo_uint t; #if defined(COPY_DICT) lzo_uint m_off; const lzo_bytep dict_end; #else const lzo_bytep m_pos; #endif const lzo_bytep const ip_end = in + in_len; #if defined(HAVE_ANY_OP) lzo_bytep const op_end = out + *out_len; #endif #if defined(LZO1Z) lzo_uint last_m_off = 0; #endif LZO_UNUSED(wrkmem); #if defined(COPY_DICT) if (dict) { if (dict_len > M4_MAX_OFFSET) { dict += dict_len - M4_MAX_OFFSET; dict_len = M4_MAX_OFFSET; } dict_end = dict + dict_len; } else { dict_len = 0; dict_end = NULL; } #endif *out_len = 0; op = out; ip = in; NEED_IP(1); if (*ip > 17) { t = *ip++ - 17; if (t < 4) goto match_next; assert(t > 0); NEED_OP(t); NEED_IP(t+3); do *op++ = *ip++; while (--t > 0); goto first_literal_run; } for (;;) { NEED_IP(3); t = *ip++; if (t >= 16) goto match; if (t == 0) { while (*ip == 0) { t += 255; ip++; TEST_IV(t); NEED_IP(1); } t += 15 + *ip++; } assert(t > 0); NEED_OP(t+3); NEED_IP(t+6); #if (LZO_OPT_UNALIGNED64) && (LZO_OPT_UNALIGNED32) t += 3; if (t >= 8) do { UA_COPY8(op,ip); op += 8; ip += 8; t -= 8; } while (t >= 8); if (t >= 4) { UA_COPY4(op,ip); op += 4; ip += 4; t -= 4; } if (t > 0) { *op++ = *ip++; if (t > 1) { *op++ = *ip++; if (t > 2) { *op++ = *ip++; } } } #elif (LZO_OPT_UNALIGNED32) || (LZO_ALIGNED_OK_4) #if !(LZO_OPT_UNALIGNED32) if (PTR_ALIGNED2_4(op,ip)) { #endif UA_COPY4(op,ip); op += 4; ip += 4; if (--t > 0) { if (t >= 4) { do { UA_COPY4(op,ip); op += 4; ip += 4; t -= 4; } while (t >= 4); if (t > 0) do *op++ = *ip++; while (--t > 0); } else do *op++ = *ip++; while (--t > 0); } #if !(LZO_OPT_UNALIGNED32) } else #endif #endif #if !(LZO_OPT_UNALIGNED32) { *op++ = *ip++; *op++ = *ip++; *op++ = *ip++; do *op++ = *ip++; while (--t > 0); } #endif first_literal_run: t = *ip++; if (t >= 16) goto match; #if defined(COPY_DICT) #if defined(LZO1Z) m_off = (1 + M2_MAX_OFFSET) + (t << 6) + (*ip++ >> 2); last_m_off = m_off; #else m_off = (1 + M2_MAX_OFFSET) + (t >> 2) + (*ip++ << 2); #endif NEED_OP(3); t = 3; COPY_DICT(t,m_off) #else #if defined(LZO1Z) t = (1 + M2_MAX_OFFSET) + (t << 6) + (*ip++ >> 2); m_pos = op - t; last_m_off = t; #else m_pos = op - (1 + M2_MAX_OFFSET); m_pos -= t >> 2; m_pos -= *ip++ << 2; #endif TEST_LB(m_pos); NEED_OP(3); *op++ = *m_pos++; *op++ = *m_pos++; *op++ = *m_pos; #endif goto match_done; for (;;) { match: if (t >= 64) { #if defined(COPY_DICT) #if defined(LZO1X) m_off = 1 + ((t >> 2) & 7) + (*ip++ << 3); t = (t >> 5) - 1; #elif defined(LZO1Y) m_off = 1 + ((t >> 2) & 3) + (*ip++ << 2); t = (t >> 4) - 3; #elif defined(LZO1Z) m_off = t & 0x1f; if (m_off >= 0x1c) m_off = last_m_off; else { m_off = 1 + (m_off << 6) + (*ip++ >> 2); last_m_off = m_off; } t = (t >> 5) - 1; #endif #else #if defined(LZO1X) m_pos = op - 1; m_pos -= (t >> 2) & 7; m_pos -= *ip++ << 3; t = (t >> 5) - 1; #elif defined(LZO1Y) m_pos = op - 1; m_pos -= (t >> 2) & 3; m_pos -= *ip++ << 2; t = (t >> 4) - 3; #elif defined(LZO1Z) { lzo_uint off = t & 0x1f; m_pos = op; if (off >= 0x1c) { assert(last_m_off > 0); m_pos -= last_m_off; } else { off = 1 + (off << 6) + (*ip++ >> 2); m_pos -= off; last_m_off = off; } } t = (t >> 5) - 1; #endif TEST_LB(m_pos); assert(t > 0); NEED_OP(t+3-1); goto copy_match; #endif } else if (t >= 32) { t &= 31; if (t == 0) { while (*ip == 0) { t += 255; ip++; TEST_OV(t); NEED_IP(1); } t += 31 + *ip++; NEED_IP(2); } #if defined(COPY_DICT) #if defined(LZO1Z) m_off = 1 + (ip[0] << 6) + (ip[1] >> 2); last_m_off = m_off; #else m_off = 1 + (ip[0] >> 2) + (ip[1] << 6); #endif #else #if defined(LZO1Z) { lzo_uint off = 1 + (ip[0] << 6) + (ip[1] >> 2); m_pos = op - off; last_m_off = off; } #elif (LZO_OPT_UNALIGNED16) && (LZO_ABI_LITTLE_ENDIAN) m_pos = op - 1; m_pos -= UA_GET_LE16(ip) >> 2; #else m_pos = op - 1; m_pos -= (ip[0] >> 2) + (ip[1] << 6); #endif #endif ip += 2; } else if (t >= 16) { #if defined(COPY_DICT) m_off = (t & 8) << 11; #else m_pos = op; m_pos -= (t & 8) << 11; #endif t &= 7; if (t == 0) { while (*ip == 0) { t += 255; ip++; TEST_OV(t); NEED_IP(1); } t += 7 + *ip++; NEED_IP(2); } #if defined(COPY_DICT) #if defined(LZO1Z) m_off += (ip[0] << 6) + (ip[1] >> 2); #else m_off += (ip[0] >> 2) + (ip[1] << 6); #endif ip += 2; if (m_off == 0) goto eof_found; m_off += 0x4000; #if defined(LZO1Z) last_m_off = m_off; #endif #else #if defined(LZO1Z) m_pos -= (ip[0] << 6) + (ip[1] >> 2); #elif (LZO_OPT_UNALIGNED16) && (LZO_ABI_LITTLE_ENDIAN) m_pos -= UA_GET_LE16(ip) >> 2; #else m_pos -= (ip[0] >> 2) + (ip[1] << 6); #endif ip += 2; if (m_pos == op) goto eof_found; m_pos -= 0x4000; #if defined(LZO1Z) last_m_off = pd((const lzo_bytep)op, m_pos); #endif #endif } else { #if defined(COPY_DICT) #if defined(LZO1Z) m_off = 1 + (t << 6) + (*ip++ >> 2); last_m_off = m_off; #else m_off = 1 + (t >> 2) + (*ip++ << 2); #endif NEED_OP(2); t = 2; COPY_DICT(t,m_off) #else #if defined(LZO1Z) t = 1 + (t << 6) + (*ip++ >> 2); m_pos = op - t; last_m_off = t; #else m_pos = op - 1; m_pos -= t >> 2; m_pos -= *ip++ << 2; #endif TEST_LB(m_pos); NEED_OP(2); *op++ = *m_pos++; *op++ = *m_pos; #endif goto match_done; } #if defined(COPY_DICT) NEED_OP(t+3-1); t += 3-1; COPY_DICT(t,m_off) #else TEST_LB(m_pos); assert(t > 0); NEED_OP(t+3-1); #if (LZO_OPT_UNALIGNED64) && (LZO_OPT_UNALIGNED32) if (op - m_pos >= 8) { t += (3 - 1); if (t >= 8) do { UA_COPY8(op,m_pos); op += 8; m_pos += 8; t -= 8; } while (t >= 8); if (t >= 4) { UA_COPY4(op,m_pos); op += 4; m_pos += 4; t -= 4; } if (t > 0) { *op++ = m_pos[0]; if (t > 1) { *op++ = m_pos[1]; if (t > 2) { *op++ = m_pos[2]; } } } } else #elif (LZO_OPT_UNALIGNED32) || (LZO_ALIGNED_OK_4) #if !(LZO_OPT_UNALIGNED32) if (t >= 2 * 4 - (3 - 1) && PTR_ALIGNED2_4(op,m_pos)) { assert((op - m_pos) >= 4); #else if (t >= 2 * 4 - (3 - 1) && (op - m_pos) >= 4) { #endif UA_COPY4(op,m_pos); op += 4; m_pos += 4; t -= 4 - (3 - 1); do { UA_COPY4(op,m_pos); op += 4; m_pos += 4; t -= 4; } while (t >= 4); if (t > 0) do *op++ = *m_pos++; while (--t > 0); } else #endif { copy_match: *op++ = *m_pos++; *op++ = *m_pos++; do *op++ = *m_pos++; while (--t > 0); } #endif match_done: #if defined(LZO1Z) t = ip[-1] & 3; #else t = ip[-2] & 3; #endif if (t == 0) break; match_next: assert(t > 0); assert(t < 4); NEED_OP(t); NEED_IP(t+3); #if 0 do *op++ = *ip++; while (--t > 0); #else *op++ = *ip++; if (t > 1) { *op++ = *ip++; if (t > 2) { *op++ = *ip++; } } #endif t = *ip++; } } eof_found: *out_len = pd(op, out); return (ip == ip_end ? LZO_E_OK : (ip < ip_end ? LZO_E_INPUT_NOT_CONSUMED : LZO_E_INPUT_OVERRUN)); #if defined(HAVE_NEED_IP) input_overrun: *out_len = pd(op, out); return LZO_E_INPUT_OVERRUN; #endif #if defined(HAVE_NEED_OP) output_overrun: *out_len = pd(op, out); return LZO_E_OUTPUT_OVERRUN; #endif #if defined(LZO_TEST_OVERRUN_LOOKBEHIND) lookbehind_overrun: *out_len = pd(op, out); return LZO_E_LOOKBEHIND_OVERRUN; #endif } #endif #define LZO_TEST_OVERRUN 1 #undef DO_DECOMPRESS #define DO_DECOMPRESS lzo1x_decompress_safe #if !defined(MINILZO_CFG_SKIP_LZO1X_DECOMPRESS_SAFE) #if defined(LZO_TEST_OVERRUN) # if !defined(LZO_TEST_OVERRUN_INPUT) # define LZO_TEST_OVERRUN_INPUT 2 # endif # if !defined(LZO_TEST_OVERRUN_OUTPUT) # define LZO_TEST_OVERRUN_OUTPUT 2 # endif # if !defined(LZO_TEST_OVERRUN_LOOKBEHIND) # define LZO_TEST_OVERRUN_LOOKBEHIND 1 # endif #endif #undef TEST_IP #undef TEST_OP #undef TEST_IP_AND_TEST_OP #undef TEST_LB #undef TEST_LBO #undef NEED_IP #undef NEED_OP #undef TEST_IV #undef TEST_OV #undef HAVE_TEST_IP #undef HAVE_TEST_OP #undef HAVE_NEED_IP #undef HAVE_NEED_OP #undef HAVE_ANY_IP #undef HAVE_ANY_OP #if defined(LZO_TEST_OVERRUN_INPUT) # if (LZO_TEST_OVERRUN_INPUT >= 1) # define TEST_IP (ip < ip_end) # endif # if (LZO_TEST_OVERRUN_INPUT >= 2) # define NEED_IP(x) \ if ((lzo_uint)(ip_end - ip) < (lzo_uint)(x)) goto input_overrun # define TEST_IV(x) if ((x) > (lzo_uint)0 - (511)) goto input_overrun # endif #endif #if defined(LZO_TEST_OVERRUN_OUTPUT) # if (LZO_TEST_OVERRUN_OUTPUT >= 1) # define TEST_OP (op <= op_end) # endif # if (LZO_TEST_OVERRUN_OUTPUT >= 2) # undef TEST_OP # define NEED_OP(x) \ if ((lzo_uint)(op_end - op) < (lzo_uint)(x)) goto output_overrun # define TEST_OV(x) if ((x) > (lzo_uint)0 - (511)) goto output_overrun # endif #endif #if defined(LZO_TEST_OVERRUN_LOOKBEHIND) # define TEST_LB(m_pos) if (PTR_LT(m_pos,out) || PTR_GE(m_pos,op)) goto lookbehind_overrun # define TEST_LBO(m_pos,o) if (PTR_LT(m_pos,out) || PTR_GE(m_pos,op-(o))) goto lookbehind_overrun #else # define TEST_LB(m_pos) ((void) 0) # define TEST_LBO(m_pos,o) ((void) 0) #endif #if !defined(LZO_EOF_CODE) && !defined(TEST_IP) # define TEST_IP (ip < ip_end) #endif #if defined(TEST_IP) # define HAVE_TEST_IP 1 #else # define TEST_IP 1 #endif #if defined(TEST_OP) # define HAVE_TEST_OP 1 #else # define TEST_OP 1 #endif #if defined(HAVE_TEST_IP) && defined(HAVE_TEST_OP) # define TEST_IP_AND_TEST_OP (TEST_IP && TEST_OP) #elif defined(HAVE_TEST_IP) # define TEST_IP_AND_TEST_OP TEST_IP #elif defined(HAVE_TEST_OP) # define TEST_IP_AND_TEST_OP TEST_OP #else # define TEST_IP_AND_TEST_OP 1 #endif #if defined(NEED_IP) # define HAVE_NEED_IP 1 #else # define NEED_IP(x) ((void) 0) # define TEST_IV(x) ((void) 0) #endif #if defined(NEED_OP) # define HAVE_NEED_OP 1 #else # define NEED_OP(x) ((void) 0) # define TEST_OV(x) ((void) 0) #endif #if defined(HAVE_TEST_IP) || defined(HAVE_NEED_IP) # define HAVE_ANY_IP 1 #endif #if defined(HAVE_TEST_OP) || defined(HAVE_NEED_OP) # define HAVE_ANY_OP 1 #endif #if defined(DO_DECOMPRESS) LZO_PUBLIC(int) DO_DECOMPRESS ( const lzo_bytep in , lzo_uint in_len, lzo_bytep out, lzo_uintp out_len, lzo_voidp wrkmem ) #endif { lzo_bytep op; const lzo_bytep ip; lzo_uint t; #if defined(COPY_DICT) lzo_uint m_off; const lzo_bytep dict_end; #else const lzo_bytep m_pos; #endif const lzo_bytep const ip_end = in + in_len; #if defined(HAVE_ANY_OP) lzo_bytep const op_end = out + *out_len; #endif #if defined(LZO1Z) lzo_uint last_m_off = 0; #endif LZO_UNUSED(wrkmem); #if defined(COPY_DICT) if (dict) { if (dict_len > M4_MAX_OFFSET) { dict += dict_len - M4_MAX_OFFSET; dict_len = M4_MAX_OFFSET; } dict_end = dict + dict_len; } else { dict_len = 0; dict_end = NULL; } #endif *out_len = 0; op = out; ip = in; NEED_IP(1); if (*ip > 17) { t = *ip++ - 17; if (t < 4) goto match_next; assert(t > 0); NEED_OP(t); NEED_IP(t+3); do *op++ = *ip++; while (--t > 0); goto first_literal_run; } for (;;) { NEED_IP(3); t = *ip++; if (t >= 16) goto match; if (t == 0) { while (*ip == 0) { t += 255; ip++; TEST_IV(t); NEED_IP(1); } t += 15 + *ip++; } assert(t > 0); NEED_OP(t+3); NEED_IP(t+6); #if (LZO_OPT_UNALIGNED64) && (LZO_OPT_UNALIGNED32) t += 3; if (t >= 8) do { UA_COPY8(op,ip); op += 8; ip += 8; t -= 8; } while (t >= 8); if (t >= 4) { UA_COPY4(op,ip); op += 4; ip += 4; t -= 4; } if (t > 0) { *op++ = *ip++; if (t > 1) { *op++ = *ip++; if (t > 2) { *op++ = *ip++; } } } #elif (LZO_OPT_UNALIGNED32) || (LZO_ALIGNED_OK_4) #if !(LZO_OPT_UNALIGNED32) if (PTR_ALIGNED2_4(op,ip)) { #endif UA_COPY4(op,ip); op += 4; ip += 4; if (--t > 0) { if (t >= 4) { do { UA_COPY4(op,ip); op += 4; ip += 4; t -= 4; } while (t >= 4); if (t > 0) do *op++ = *ip++; while (--t > 0); } else do *op++ = *ip++; while (--t > 0); } #if !(LZO_OPT_UNALIGNED32) } else #endif #endif #if !(LZO_OPT_UNALIGNED32) { *op++ = *ip++; *op++ = *ip++; *op++ = *ip++; do *op++ = *ip++; while (--t > 0); } #endif first_literal_run: t = *ip++; if (t >= 16) goto match; #if defined(COPY_DICT) #if defined(LZO1Z) m_off = (1 + M2_MAX_OFFSET) + (t << 6) + (*ip++ >> 2); last_m_off = m_off; #else m_off = (1 + M2_MAX_OFFSET) + (t >> 2) + (*ip++ << 2); #endif NEED_OP(3); t = 3; COPY_DICT(t,m_off) #else #if defined(LZO1Z) t = (1 + M2_MAX_OFFSET) + (t << 6) + (*ip++ >> 2); m_pos = op - t; last_m_off = t; #else m_pos = op - (1 + M2_MAX_OFFSET); m_pos -= t >> 2; m_pos -= *ip++ << 2; #endif TEST_LB(m_pos); NEED_OP(3); *op++ = *m_pos++; *op++ = *m_pos++; *op++ = *m_pos; #endif goto match_done; for (;;) { match: if (t >= 64) { #if defined(COPY_DICT) #if defined(LZO1X) m_off = 1 + ((t >> 2) & 7) + (*ip++ << 3); t = (t >> 5) - 1; #elif defined(LZO1Y) m_off = 1 + ((t >> 2) & 3) + (*ip++ << 2); t = (t >> 4) - 3; #elif defined(LZO1Z) m_off = t & 0x1f; if (m_off >= 0x1c) m_off = last_m_off; else { m_off = 1 + (m_off << 6) + (*ip++ >> 2); last_m_off = m_off; } t = (t >> 5) - 1; #endif #else #if defined(LZO1X) m_pos = op - 1; m_pos -= (t >> 2) & 7; m_pos -= *ip++ << 3; t = (t >> 5) - 1; #elif defined(LZO1Y) m_pos = op - 1; m_pos -= (t >> 2) & 3; m_pos -= *ip++ << 2; t = (t >> 4) - 3; #elif defined(LZO1Z) { lzo_uint off = t & 0x1f; m_pos = op; if (off >= 0x1c) { assert(last_m_off > 0); m_pos -= last_m_off; } else { off = 1 + (off << 6) + (*ip++ >> 2); m_pos -= off; last_m_off = off; } } t = (t >> 5) - 1; #endif TEST_LB(m_pos); assert(t > 0); NEED_OP(t+3-1); goto copy_match; #endif } else if (t >= 32) { t &= 31; if (t == 0) { while (*ip == 0) { t += 255; ip++; TEST_OV(t); NEED_IP(1); } t += 31 + *ip++; NEED_IP(2); } #if defined(COPY_DICT) #if defined(LZO1Z) m_off = 1 + (ip[0] << 6) + (ip[1] >> 2); last_m_off = m_off; #else m_off = 1 + (ip[0] >> 2) + (ip[1] << 6); #endif #else #if defined(LZO1Z) { lzo_uint off = 1 + (ip[0] << 6) + (ip[1] >> 2); m_pos = op - off; last_m_off = off; } #elif (LZO_OPT_UNALIGNED16) && (LZO_ABI_LITTLE_ENDIAN) m_pos = op - 1; m_pos -= UA_GET_LE16(ip) >> 2; #else m_pos = op - 1; m_pos -= (ip[0] >> 2) + (ip[1] << 6); #endif #endif ip += 2; } else if (t >= 16) { #if defined(COPY_DICT) m_off = (t & 8) << 11; #else m_pos = op; m_pos -= (t & 8) << 11; #endif t &= 7; if (t == 0) { while (*ip == 0) { t += 255; ip++; TEST_OV(t); NEED_IP(1); } t += 7 + *ip++; NEED_IP(2); } #if defined(COPY_DICT) #if defined(LZO1Z) m_off += (ip[0] << 6) + (ip[1] >> 2); #else m_off += (ip[0] >> 2) + (ip[1] << 6); #endif ip += 2; if (m_off == 0) goto eof_found; m_off += 0x4000; #if defined(LZO1Z) last_m_off = m_off; #endif #else #if defined(LZO1Z) m_pos -= (ip[0] << 6) + (ip[1] >> 2); #elif (LZO_OPT_UNALIGNED16) && (LZO_ABI_LITTLE_ENDIAN) m_pos -= UA_GET_LE16(ip) >> 2; #else m_pos -= (ip[0] >> 2) + (ip[1] << 6); #endif ip += 2; if (m_pos == op) goto eof_found; m_pos -= 0x4000; #if defined(LZO1Z) last_m_off = pd((const lzo_bytep)op, m_pos); #endif #endif } else { #if defined(COPY_DICT) #if defined(LZO1Z) m_off = 1 + (t << 6) + (*ip++ >> 2); last_m_off = m_off; #else m_off = 1 + (t >> 2) + (*ip++ << 2); #endif NEED_OP(2); t = 2; COPY_DICT(t,m_off) #else #if defined(LZO1Z) t = 1 + (t << 6) + (*ip++ >> 2); m_pos = op - t; last_m_off = t; #else m_pos = op - 1; m_pos -= t >> 2; m_pos -= *ip++ << 2; #endif TEST_LB(m_pos); NEED_OP(2); *op++ = *m_pos++; *op++ = *m_pos; #endif goto match_done; } #if defined(COPY_DICT) NEED_OP(t+3-1); t += 3-1; COPY_DICT(t,m_off) #else TEST_LB(m_pos); assert(t > 0); NEED_OP(t+3-1); #if (LZO_OPT_UNALIGNED64) && (LZO_OPT_UNALIGNED32) if (op - m_pos >= 8) { t += (3 - 1); if (t >= 8) do { UA_COPY8(op,m_pos); op += 8; m_pos += 8; t -= 8; } while (t >= 8); if (t >= 4) { UA_COPY4(op,m_pos); op += 4; m_pos += 4; t -= 4; } if (t > 0) { *op++ = m_pos[0]; if (t > 1) { *op++ = m_pos[1]; if (t > 2) { *op++ = m_pos[2]; } } } } else #elif (LZO_OPT_UNALIGNED32) || (LZO_ALIGNED_OK_4) #if !(LZO_OPT_UNALIGNED32) if (t >= 2 * 4 - (3 - 1) && PTR_ALIGNED2_4(op,m_pos)) { assert((op - m_pos) >= 4); #else if (t >= 2 * 4 - (3 - 1) && (op - m_pos) >= 4) { #endif UA_COPY4(op,m_pos); op += 4; m_pos += 4; t -= 4 - (3 - 1); do { UA_COPY4(op,m_pos); op += 4; m_pos += 4; t -= 4; } while (t >= 4); if (t > 0) do *op++ = *m_pos++; while (--t > 0); } else #endif { copy_match: *op++ = *m_pos++; *op++ = *m_pos++; do *op++ = *m_pos++; while (--t > 0); } #endif match_done: #if defined(LZO1Z) t = ip[-1] & 3; #else t = ip[-2] & 3; #endif if (t == 0) break; match_next: assert(t > 0); assert(t < 4); NEED_OP(t); NEED_IP(t+3); #if 0 do *op++ = *ip++; while (--t > 0); #else *op++ = *ip++; if (t > 1) { *op++ = *ip++; if (t > 2) { *op++ = *ip++; } } #endif t = *ip++; } } eof_found: *out_len = pd(op, out); return (ip == ip_end ? LZO_E_OK : (ip < ip_end ? LZO_E_INPUT_NOT_CONSUMED : LZO_E_INPUT_OVERRUN)); #if defined(HAVE_NEED_IP) input_overrun: *out_len = pd(op, out); return LZO_E_INPUT_OVERRUN; #endif #if defined(HAVE_NEED_OP) output_overrun: *out_len = pd(op, out); return LZO_E_OUTPUT_OVERRUN; #endif #if defined(LZO_TEST_OVERRUN_LOOKBEHIND) lookbehind_overrun: *out_len = pd(op, out); return LZO_E_LOOKBEHIND_OVERRUN; #endif } #endif /***** End of minilzo.c *****/ nfdump-1.6.23/bin/minilzo.h000077500000000000000000000060211404501030700154720ustar00rootroot00000000000000/* minilzo.h -- mini subset of the LZO real-time data compression library This file is part of the LZO real-time data compression library. Copyright (C) 1996-2017 Markus Franz Xaver Johannes Oberhumer All Rights Reserved. The LZO library is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. The LZO library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with the LZO library; see the file COPYING. If not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. Markus F.X.J. Oberhumer http://www.oberhumer.com/opensource/lzo/ */ /* * NOTE: * the full LZO package can be found at * http://www.oberhumer.com/opensource/lzo/ */ #ifndef __MINILZO_H_INCLUDED #define __MINILZO_H_INCLUDED 1 #define MINILZO_VERSION 0x20a0 /* 2.10 */ #if defined(__LZOCONF_H_INCLUDED) # error "you cannot use both LZO and miniLZO" #endif /* internal Autoconf configuration file - only used when building miniLZO */ #ifdef MINILZO_HAVE_CONFIG_H # include #endif #include #include #ifndef __LZODEFS_H_INCLUDED #include "lzodefs.h" #endif #undef LZO_HAVE_CONFIG_H #include "lzoconf.h" #if !defined(LZO_VERSION) || (LZO_VERSION != MINILZO_VERSION) # error "version mismatch in header files" #endif #ifdef __cplusplus extern "C" { #endif /*********************************************************************** // ************************************************************************/ /* Memory required for the wrkmem parameter. * When the required size is 0, you can also pass a NULL pointer. */ #define LZO1X_MEM_COMPRESS LZO1X_1_MEM_COMPRESS #define LZO1X_1_MEM_COMPRESS ((lzo_uint32_t) (16384L * lzo_sizeof_dict_t)) #define LZO1X_MEM_DECOMPRESS (0) /* compression */ LZO_EXTERN(int) lzo1x_1_compress ( const lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp dst_len, lzo_voidp wrkmem ); /* decompression */ LZO_EXTERN(int) lzo1x_decompress ( const lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp dst_len, lzo_voidp wrkmem /* NOT USED */ ); /* safe decompression with overrun testing */ LZO_EXTERN(int) lzo1x_decompress_safe ( const lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp dst_len, lzo_voidp wrkmem /* NOT USED */ ); #ifdef __cplusplus } /* extern "C" */ #endif #endif /* already included */ /* vim:set ts=4 sw=4 et: */ nfdump-1.6.23/bin/netflow_pcap.c000066400000000000000000000313131404501030700164640ustar00rootroot00000000000000/* * Copyright (c) 2013-2020, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "nfnet.h" #include "output_raw.h" #include "bookkeeper.h" #include "collector.h" #include "exporter.h" #include "flowtree.h" #include "netflow_pcap.h" extern int verbose; extern extension_descriptor_t extension_descriptor[]; /* module limited globals */ static extension_info_t pcap_extension_info; // common for all pcap records static extension_map_t *pcap_extension_map; static uint32_t pcap_output_record_size_v4; static uint32_t pcap_output_record_size_v6; typedef struct pcap_v4_block_s { uint32_t srcaddr; uint32_t dstaddr; uint32_t dPkts; uint32_t dOctets; uint32_t data[1]; // link to next record } __attribute__((__packed__ )) pcap_v4_block_t; #define PCAP_V4_BLOCK_DATA_SIZE (sizeof(pcap_v4_block_t) - sizeof(uint32_t)) typedef struct pcap_v6_block_s { uint64_t srcaddr[2]; uint64_t dstaddr[2]; uint32_t dPkts; uint32_t dOctets; uint32_t data[1]; // link to next record } __attribute__((__packed__ )) pcap_v6_block_t; #define PCAP_V6_BLOCK_DATA_SIZE (sizeof(pcap_v6_block_t) - sizeof(uint32_t)) // All required extension to save full pcap records static uint16_t pcap_full_map[] = { EX_LATENCY, 0 }; #include "nffile_inline.c" int Init_pcap2nf(void) { int i, id, map_index; int extension_size; uint16_t map_size; // prepare pcap extension map pcap_extension_info.map = NULL; extension_size = 0; map_size = 0; i=0; while ( (id = pcap_full_map[i]) != 0 ) { if ( extension_descriptor[id].enabled ) { extension_size += extension_descriptor[id].size; map_size += sizeof(uint16_t); } i++; } // extension_size contains the sum of all optional extensions // caculate the record size pcap_output_record_size_v4 = COMMON_RECORD_DATA_SIZE + PCAP_V4_BLOCK_DATA_SIZE + extension_size; pcap_output_record_size_v6 = COMMON_RECORD_DATA_SIZE + PCAP_V6_BLOCK_DATA_SIZE + extension_size; // now the full extension map size map_size += sizeof(extension_map_t); // align 32 bits if ( ( map_size & 0x3 ) != 0 ) map_size += 2; // Create a generic pcap extension map pcap_extension_info.map = (extension_map_t *)malloc((size_t)map_size); if ( !pcap_extension_info.map ) { LogError("Process_pcap: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return 0; } pcap_extension_info.map->type = ExtensionMapType; pcap_extension_info.map->size = map_size; pcap_extension_info.map->map_id = INIT_ID; pcap_extension_info.map->extension_size = extension_size; // see netflow_pcap.h for extension map description map_index = 0; i=0; while ( (id = pcap_full_map[i]) != 0 ) { if ( extension_descriptor[id].enabled ) pcap_extension_info.map->ex_id[map_index++] = id; i++; } pcap_extension_info.map->ex_id[map_index] = 0; pcap_extension_map = NULL; return 1; } // End of Init_pcap2nf int StorePcapFlow(FlowSource_t *fs, struct FlowNode *Node) { common_record_t *common_record; uint32_t packets, bytes, pcap_output_record_size; uint64_t start_time, end_time; int j, id; char *string; void *data_ptr; if ( !pcap_extension_map ) { pcap_extension_map = (extension_map_t *)malloc(pcap_extension_info.map->size); if ( !pcap_extension_map ) { LogError("Process_pcap: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return 0; } memcpy((void *)pcap_extension_map, (void *)pcap_extension_info.map, pcap_extension_info.map->size); if ( !AddExtensionMap(fs, pcap_extension_map) ) { LogError("Process_pcap: Fatal: AddExtensionMap() failed in %s line %d\n", __FILE__, __LINE__); return 0; } } if ( Node->version == AF_INET6 ) { pcap_output_record_size = pcap_output_record_size_v6; dbg_printf("Store Flow v6 node: size: %u\n", pcap_output_record_size); } else if ( Node->version == AF_INET ) { pcap_output_record_size = pcap_output_record_size_v4; dbg_printf("Store Flow v4 node: size: %u\n", pcap_output_record_size); } else { LogError("Process_pcap: Unexpected version in %s line %d: %u\n", __FILE__, __LINE__, Node->version); return 0; } // output buffer size check for all expected records if ( !CheckBufferSpace(fs->nffile, pcap_output_record_size) ) { // fishy! - should never happen. maybe disk full? LogError("Process_pcap: output buffer size error. Abort pcap record processing"); return 0; } // map output record to memory buffer common_record = (common_record_t *)fs->nffile->buff_ptr; // header data common_record->flags = 0; common_record->nfversion = 0x41; common_record->type = CommonRecordType; common_record->exporter_sysid = 0; common_record->ext_map = pcap_extension_map->map_id; common_record->size = pcap_output_record_size; // pcap common fields common_record->srcport = Node->src_port; common_record->dstport = Node->dst_port; common_record->tcp_flags = Node->flags; common_record->prot = Node->proto; common_record->tos = 0; common_record->fwd_status = 0; if ( Node->version == AF_INET6 ) { SetFlag(common_record->flags, FLAG_IPV6_ADDR); pcap_v6_block_t *pcap_v6_block = (pcap_v6_block_t *)common_record->data; pcap_v6_block->srcaddr[0] = Node->src_addr.v6[0]; pcap_v6_block->srcaddr[1] = Node->src_addr.v6[1]; pcap_v6_block->dstaddr[0] = Node->dst_addr.v6[0]; pcap_v6_block->dstaddr[1] = Node->dst_addr.v6[1]; pcap_v6_block->dPkts = packets = Node->packets; pcap_v6_block->dOctets = bytes = Node->bytes; data_ptr = (void *)pcap_v6_block->data; } else { pcap_v4_block_t *pcap_v4_block = (pcap_v4_block_t *)common_record->data; pcap_v4_block->srcaddr = Node->src_addr.v4; pcap_v4_block->dstaddr = Node->dst_addr.v4; pcap_v4_block->dPkts = packets = Node->packets; pcap_v4_block->dOctets = bytes = Node->bytes; data_ptr = (void *)pcap_v4_block->data; } // process optional extensions j = 0; while ( (id = pcap_extension_map->ex_id[j]) != 0 ) { switch (id) { case EX_IO_SNMP_2: { // 2 byte input/output interface index tpl_ext_4_t *tpl = (tpl_ext_4_t *)data_ptr; tpl->input = 0; tpl->output = 0; data_ptr = (void *)tpl->data; } break; case EX_LATENCY: { // latecy extension tpl_ext_latency_t *tpl = (tpl_ext_latency_t *)data_ptr; tpl->client_nw_delay_usec = Node->latency.client; tpl->server_nw_delay_usec = Node->latency.server; tpl->appl_latency_usec = Node->latency.application; data_ptr = (void *)tpl->data; } break; default: // this should never happen, as pcap has no other extensions LogError("Process_pcap: Unexpected extension %i for pcap record. Skip extension", id); } j++; } common_record->first = Node->t_first.tv_sec; common_record->msec_first = Node->t_first.tv_usec / 1000; common_record->last = Node->t_last.tv_sec; common_record->msec_last = Node->t_last.tv_usec / 1000; start_time = (1000LL * (uint64_t)common_record->first) + (uint64_t)common_record->msec_first; end_time = (1000LL * (uint64_t)common_record->last) + (uint64_t)common_record->msec_last; // update first_seen, last_seen if ( start_time < fs->first_seen ) fs->first_seen = start_time; if ( end_time > fs->last_seen ) fs->last_seen = end_time; // Update stats switch (common_record->prot) { case IPPROTO_ICMP: case IPPROTO_ICMPV6: fs->nffile->stat_record->numflows_icmp++; fs->nffile->stat_record->numpackets_icmp += packets; fs->nffile->stat_record->numbytes_icmp += bytes; // fix odd CISCO behaviour for ICMP port/type in src port if ( common_record->srcport != 0 ) { uint8_t *s1, *s2; s1 = (uint8_t *)&(common_record->srcport); s2 = (uint8_t *)&(common_record->dstport); s2[0] = s1[1]; s2[1] = s1[0]; common_record->srcport = 0; } break; case IPPROTO_TCP: fs->nffile->stat_record->numflows_tcp++; fs->nffile->stat_record->numpackets_tcp += packets; fs->nffile->stat_record->numbytes_tcp += bytes; break; case IPPROTO_UDP: fs->nffile->stat_record->numflows_udp++; fs->nffile->stat_record->numpackets_udp += packets; fs->nffile->stat_record->numbytes_udp += bytes; break; default: fs->nffile->stat_record->numflows_other++; fs->nffile->stat_record->numpackets_other += packets; fs->nffile->stat_record->numbytes_other += bytes; } fs->nffile->stat_record->numflows++; fs->nffile->stat_record->numpackets += packets; fs->nffile->stat_record->numbytes += bytes; if ( verbose ) { master_record_t master_record; ExpandRecord_v2((common_record_t *)common_record, &pcap_extension_info, NULL, &master_record); flow_record_to_raw(&master_record, &string, 0); printf("%s\n", string); } // update file record size ( -> output buffer size ) fs->nffile->block_header->NumRecords += 1; fs->nffile->block_header->size += pcap_output_record_size; fs->nffile->buff_ptr = data_ptr; return 1; } /* End of StorePcapFlow */ // Server latency = t(SYN Server) - t(SYN CLient) void SetServer_latency(struct FlowNode *node) { struct FlowNode *Client_node; uint64_t latency; Client_node = node->rev_node; if ( !Client_node ) return; latency = ((uint64_t)node->t_first.tv_sec * (uint64_t)1000000 + (uint64_t)node->t_first.tv_usec) - ((uint64_t)Client_node->t_first.tv_sec * (uint64_t)1000000 + (uint64_t)Client_node->t_first.tv_usec); node->latency.server = latency; Client_node->latency.server = latency; // set flag, to calc client latency with nex packet from client Client_node->latency.flag = 1; dbg_printf("Server latency: %llu\n", (long long unsigned)latency); } // End of SetServerClient_latency // Client latency = t(ACK CLient) - t(SYN Server) void SetClient_latency(struct FlowNode *node, struct timeval *t_packet) { struct FlowNode *Server_node; uint64_t latency; Server_node = node->rev_node; if ( !Server_node ) return; latency = ((uint64_t)t_packet->tv_sec * (uint64_t)1000000 + (uint64_t)t_packet->tv_usec) - ((uint64_t)Server_node->t_first.tv_sec * (uint64_t)1000000 + (uint64_t)Server_node->t_first.tv_usec); node->latency.client = latency; Server_node->latency.client = latency; // reset flag node->latency.flag = 0; // set flag, to calc application latency with nex packet from server Server_node->latency.flag = 2; Server_node->latency.t_request = *t_packet; dbg_printf("Client latency: %llu\n", (long long unsigned)latency); } // End of SetClient_latency // Application latency = t(ACK Server) - t(ACK CLient) void SetApplication_latency(struct FlowNode *node, struct timeval *t_packet) { struct FlowNode *Client_node; uint64_t latency; Client_node = node->rev_node; if ( !Client_node ) return; latency = ((uint64_t)t_packet->tv_sec * (uint64_t)1000000 + (uint64_t)t_packet->tv_usec) - ((uint64_t)node->latency.t_request.tv_sec * (uint64_t)1000000 + (uint64_t)node->latency.t_request.tv_usec); node->latency.application = latency; Client_node->latency.application = latency; // reset flag node->latency.flag = 0; dbg_printf("Application latency: %llu\n", (long long unsigned)latency); } // End of SetApplication_latency nfdump-1.6.23/bin/netflow_pcap.h000066400000000000000000000041241404501030700164710ustar00rootroot00000000000000/* * Copyright (c) 2017, Peter Haag * Copyright (c) 2016, Peter Haag * Copyright (c) 2014, Peter Haag * Copyright (c) 2013, Peter Haag * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _NETFLOW_PCAP_H #define _NETFLOW_PCAP_H 1 #include #include "collector.h" #include "flowtree.h" int Init_pcap2nf(void); int StorePcapFlow(FlowSource_t *fs, struct FlowNode *Node); void SetServer_latency(struct FlowNode *node); void SetClient_latency(struct FlowNode *node, struct timeval *t_packet); void SetApplication_latency(struct FlowNode *node, struct timeval *t_packet); #endif nfdump-1.6.23/bin/netflow_v1.c000066400000000000000000000413711404501030700160740ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifdef HAVE_CONFIG_H #include "config.h" #endif #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "nfnet.h" #include "output_raw.h" #include "bookkeeper.h" #include "collector.h" #include "exporter.h" #include "netflow_v1.h" extern extension_descriptor_t extension_descriptor[]; /* module limited globals */ static int verbose; static extension_info_t v1_extension_info; // common for all v1 records static uint16_t v1_output_record_size; // All required extension to save full v1 records static uint16_t v1_full_map[] = { EX_IO_SNMP_2, EX_NEXT_HOP_v4, EX_ROUTER_IP_v4, EX_RECEIVED, 0 }; typedef struct v1_block_s { uint32_t srcaddr; uint32_t dstaddr; uint32_t dPkts; uint32_t dOctets; uint32_t data[1]; // link to next record } v1_block_t; #define V1_BLOCK_DATA_SIZE (sizeof(v1_block_t) - sizeof(uint32_t)) typedef struct exporter_v1_s { // identical to exporter_t struct exporter_v1_s *next; // exporter information exporter_info_record_t info; uint64_t packets; // number of packets sent by this exporter uint64_t flows; // number of flow records sent by this exporter uint32_t sequence_failure; // number of sequence failues uint32_t padding_errors; // number of sequence failues sampler_t *sampler; // End of exporter_t // extension map extension_map_t *extension_map; } exporter_v1_t; static inline exporter_v1_t *GetExporter(FlowSource_t *fs, netflow_v1_header_t *header); /* functions */ #include "nffile_inline.c" int Init_v1(int v) { int i, id, map_index; int extension_size; uint16_t map_size; verbose = v; // prepare v1 extension map v1_extension_info.map = NULL; v1_extension_info.next = NULL; v1_extension_info.offset_cache = NULL; v1_extension_info.ref_count = 0; extension_size = 0; // default map - 0 extensions map_size = sizeof(extension_map_t); i=0; dbg_printf("v1 map: map size start: %u\n", map_size); while ( (id = v1_full_map[i]) != 0 ) { if ( extension_descriptor[id].enabled ) { extension_size += extension_descriptor[id].size; map_size += sizeof(uint16_t); dbg_printf("v1 map: enabled extension %u\n", id); } i++; } dbg_printf("v1 map: map size so far: %u\n", map_size); // extension_size contains the sum of all optional extensions // caculate the record size v1_output_record_size = COMMON_RECORD_DATA_SIZE + V1_BLOCK_DATA_SIZE + extension_size; // align 32 bits if ( ( map_size & 0x3 ) != 0 ) map_size += 2; // Create a netflow v1 extension map v1_extension_info.map = (extension_map_t *)malloc((size_t)map_size); if ( !v1_extension_info.map ) { LogError("Process_v1: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return 0; } v1_extension_info.map->type = ExtensionMapType; v1_extension_info.map->size = map_size; v1_extension_info.map->map_id = INIT_ID; v1_extension_info.map->extension_size = extension_size; // see netflow_v1.h for extension map description map_index = 0; i=0; while ( (id = v1_full_map[i]) != 0 ) { if ( extension_descriptor[id].enabled ) v1_extension_info.map->ex_id[map_index++] = id; i++; } v1_extension_info.map->ex_id[map_index] = 0; return 1; } // End of Init_v1 /* * functions used for receiving netflow v1 records */ static inline exporter_v1_t *GetExporter(FlowSource_t *fs, netflow_v1_header_t *header) { exporter_v1_t **e = (exporter_v1_t **)&(fs->exporter_data); uint16_t version = ntohs(header->version); #define IP_STRING_LEN 40 char ipstr[IP_STRING_LEN]; // search the appropriate exporter engine while ( *e ) { if ( (*e)->info.version == version && (*e)->info.ip.V6[0] == fs->ip.V6[0] && (*e)->info.ip.V6[1] == fs->ip.V6[1]) return *e; e = &((*e)->next); } // nothing found *e = (exporter_v1_t *)malloc(sizeof(exporter_v1_t)); if ( !(*e)) { LogError("Process_v1: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return NULL; } memset((void *)(*e), 0, sizeof(exporter_v1_t)); (*e)->info.header.type = ExporterInfoRecordType; (*e)->info.header.size = sizeof(exporter_info_record_t); (*e)->info.version = version; (*e)->info.id = 0; (*e)->info.ip = fs->ip; (*e)->info.sa_family = fs->sa_family; (*e)->next = NULL; (*e)->packets = 0; (*e)->flows = 0; (*e)->sequence_failure = 0; (*e)->padding_errors = 0; (*e)->sampler = NULL; // copy the v1 extension map (*e)->extension_map = (extension_map_t *)malloc(v1_extension_info.map->size); if ( !(*e)->extension_map ) { LogError("Process_v1: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); free(*e); *e = NULL; return NULL; } memcpy((void *)(*e)->extension_map, (void *)v1_extension_info.map, v1_extension_info.map->size); if ( !AddExtensionMap(fs, (*e)->extension_map) ) { // bad - we must free this map and fail - otherwise data can not be read any more free((*e)->extension_map); free(*e); *e = NULL; return NULL; } (*e)->info.sysid = 0; FlushInfoExporter(fs, &((*e)->info)); if ( fs->sa_family == AF_INET ) { uint32_t _ip = htonl(fs->ip.V4); inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr)); } else if ( fs->sa_family == AF_INET6 ) { uint64_t _ip[2]; _ip[0] = htonll(fs->ip.V6[0]); _ip[1] = htonll(fs->ip.V6[1]); inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr)); } else { strncpy(ipstr, "", IP_STRING_LEN); } dbg_printf("New Exporter: v1 SysID: %u, Extension ID: %i, IP: %s, \n", (*e)->info.sysid, (*e)->extension_map->map_id, ipstr); LogError("Process_v1: SysID: %u, New exporter: IP: %s\n", (*e)->info.sysid, ipstr); return (*e); } // End of GetExporter void Process_v1(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs) { netflow_v1_header_t *v1_header; netflow_v1_record_t *v1_record; exporter_v1_t *exporter; extension_map_t *extension_map; common_record_t *common_record; uint64_t start_time, end_time, boot_time; uint32_t First, Last; uint16_t count; uint8_t flags; int i, done, flow_record_length; ssize_t size_left; char *string; // map v1 data structure to input buffer v1_header = (netflow_v1_header_t *)in_buff; exporter = GetExporter(fs, v1_header); if ( !exporter ) { LogError("Process_v1: Exporter NULL: Abort v1 record processing"); return; } flags = 0; exporter->packets++; extension_map = exporter->extension_map; flow_record_length = NETFLOW_V1_RECORD_LENGTH; // this many data to process size_left = in_buff_cnt; common_record = fs->nffile->buff_ptr; done = 0; while ( !done ) { v1_block_t *v1_block; /* Process header */ // count check count = ntohs(v1_header->count); if ( count > NETFLOW_V1_MAX_RECORDS ) { LogError("Process_v1: Unexpected record count in header: %i. Abort v1 record processing", count); fs->nffile->buff_ptr = (void *)common_record; return; } // input buffer size check for all expected records if ( size_left < ( NETFLOW_V1_HEADER_LENGTH + count * flow_record_length) ) { LogError("Process_v1: Not enough data to process v1 record. Abort v1 record processing"); fs->nffile->buff_ptr = (void *)common_record; return; } // output buffer size check for all expected records if ( !CheckBufferSpace(fs->nffile, count * v1_output_record_size) ) { // fishy! - should never happen. maybe disk full? LogError("Process_v1: output buffer size error. Abort v1 record processing"); return; } // map output record to memory buffer common_record = (common_record_t *)fs->nffile->buff_ptr; v1_block = (v1_block_t *)common_record->data; v1_header->SysUptime = ntohl(v1_header->SysUptime); v1_header->unix_secs = ntohl(v1_header->unix_secs); v1_header->unix_nsecs = ntohl(v1_header->unix_nsecs); /* calculate boot time in msec */ boot_time = ((uint64_t)(v1_header->unix_secs)*1000 + ((uint64_t)(v1_header->unix_nsecs) / 1000000) ) - (uint64_t)(v1_header->SysUptime); // process all records v1_record = (netflow_v1_record_t *)((pointer_addr_t)v1_header + NETFLOW_V1_HEADER_LENGTH); /* loop over each records associated with this header */ for (i = 0; i < count; i++) { pointer_addr_t bsize; void *data_ptr; uint8_t *s1, *s2; int j, id; // header data common_record->flags = flags; common_record->type = CommonRecordType; common_record->exporter_sysid = exporter->info.sysid; common_record->ext_map = extension_map->map_id; common_record->size = v1_output_record_size; // v1 common fields common_record->srcport = ntohs(v1_record->srcport); common_record->dstport = ntohs(v1_record->dstport); common_record->tcp_flags = v1_record->tcp_flags; common_record->prot = v1_record->prot; common_record->tos = v1_record->tos; common_record->fwd_status = 0; common_record->nfversion = 1; common_record->biFlowDir = 0; common_record->flowEndReason = 0; // v1 typed data as fixed struct v1_block v1_block->srcaddr = ntohl(v1_record->srcaddr); v1_block->dstaddr = ntohl(v1_record->dstaddr); v1_block->dPkts = ntohl(v1_record->dPkts); v1_block->dOctets = ntohl(v1_record->dOctets); // process optional extensions data_ptr = (void *)v1_block->data; j = 0; while ( (id = extension_map->ex_id[j]) != 0 ) { switch (id) { case EX_IO_SNMP_2: { // 2 byte input/output interface index tpl_ext_4_t *tpl = (tpl_ext_4_t *)data_ptr; tpl->input = ntohs(v1_record->input); tpl->output = ntohs(v1_record->output); data_ptr = (void *)tpl->data; } break; case EX_NEXT_HOP_v4: { // IPv4 next hop tpl_ext_9_t *tpl = (tpl_ext_9_t *)data_ptr; tpl->nexthop = ntohl(v1_record->nexthop); data_ptr = (void *)tpl->data; } break; case EX_ROUTER_IP_v4: { // IPv4 router address tpl_ext_23_t *tpl = (tpl_ext_23_t *)data_ptr; tpl->router_ip = fs->ip.V4; data_ptr = (void *)tpl->data; ClearFlag(common_record->flags, FLAG_IPV6_EXP); } break; case EX_RECEIVED: { tpl_ext_27_t *tpl = (tpl_ext_27_t *)data_ptr; tpl->received = (uint64_t)((uint64_t)fs->received.tv_sec * 1000LL) + (uint64_t)((uint64_t)fs->received.tv_usec / 1000LL); data_ptr = (void *)tpl->data; } break; default: // this should never happen, as v1 has no other extensions LogError("Process_v1: Unexpected extension %i for v1 record. Skip extension", id); } j++; } // Time issues First = ntohl(v1_record->First); Last = ntohl(v1_record->Last); if ( First > Last ) { /* First in msec, in case of msec overflow, between start and end */ start_time = boot_time - 0x100000000LL + (uint64_t)First; } else { start_time = boot_time + (uint64_t)First; } /* end time in msecs */ end_time = (uint64_t)Last + boot_time; // if overflow happened after flow ended but before got exported if ( Last > v1_header->SysUptime ) { start_time -= 0x100000000LL; end_time -= 0x100000000LL; } common_record->first = start_time/1000; common_record->msec_first = start_time - common_record->first*1000; common_record->last = end_time/1000; common_record->msec_last = end_time - common_record->last*1000; // update first_seen, last_seen if ( start_time < fs->first_seen ) fs->first_seen = start_time; if ( end_time > fs->last_seen ) fs->last_seen = end_time; // Update stats switch (common_record->prot) { case IPPROTO_ICMP: fs->nffile->stat_record->numflows_icmp++; fs->nffile->stat_record->numpackets_icmp += v1_block->dPkts; fs->nffile->stat_record->numbytes_icmp += v1_block->dOctets; // fix odd CISCO behaviour for ICMP port/type in src port if ( common_record->srcport != 0 ) { s1 = (uint8_t *)&(common_record->srcport); s2 = (uint8_t *)&(common_record->dstport); s2[0] = s1[1]; s2[1] = s1[0]; common_record->srcport = 0; } break; case IPPROTO_TCP: fs->nffile->stat_record->numflows_tcp++; fs->nffile->stat_record->numpackets_tcp += v1_block->dPkts; fs->nffile->stat_record->numbytes_tcp += v1_block->dOctets; break; case IPPROTO_UDP: fs->nffile->stat_record->numflows_udp++; fs->nffile->stat_record->numpackets_udp += v1_block->dPkts; fs->nffile->stat_record->numbytes_udp += v1_block->dOctets; break; default: fs->nffile->stat_record->numflows_other++; fs->nffile->stat_record->numpackets_other += v1_block->dPkts; fs->nffile->stat_record->numbytes_other += v1_block->dOctets; } exporter->flows++; fs->nffile->stat_record->numflows++; fs->nffile->stat_record->numpackets += v1_block->dPkts; fs->nffile->stat_record->numbytes += v1_block->dOctets; if ( verbose ) { master_record_t master_record; memset((void *)&master_record, 0, sizeof(master_record_t)); ExpandRecord_v2((common_record_t *)common_record, &v1_extension_info, &(exporter->info), &master_record); flow_record_to_raw(&master_record, &string, 0); printf("%s\n", string); } // advance to next input flow record v1_record = (netflow_v1_record_t *)((pointer_addr_t)v1_record + flow_record_length); if ( ((pointer_addr_t)data_ptr - (pointer_addr_t)common_record) != v1_output_record_size ) { printf("Panic size check: ptr diff: %llu, record size: %u\n", (unsigned long long)((pointer_addr_t)data_ptr - (pointer_addr_t)common_record), v1_output_record_size ); abort(); } // advance to next output record common_record = (common_record_t *)data_ptr; v1_block = (v1_block_t *)common_record->data; // buffer size sanity check - should never happen, but check it anyway bsize = (pointer_addr_t)common_record - (pointer_addr_t)fs->nffile->block_header - sizeof(data_block_header_t); if ( bsize > BUFFSIZE ) { LogError("### Software error ###: %s line %d", __FILE__, __LINE__); LogError("Process_v1: Output buffer overflow! Flush buffer and skip records."); LogError("Buffer size: size: %u, bsize: %llu > %u", fs->nffile->block_header->size, (unsigned long long)bsize, BUFFSIZE); // reset buffer fs->nffile->block_header->size = 0; fs->nffile->block_header->NumRecords = 0; fs->nffile->buff_ptr = (void *)((pointer_addr_t)fs->nffile->block_header + sizeof(data_block_header_t) ); return; } } // End of foreach v1 record // update file record size ( -> output buffer size ) fs->nffile->block_header->NumRecords += count; fs->nffile->block_header->size += count * v1_output_record_size; fs->nffile->buff_ptr = (void *)common_record; // still to go for this many input bytes size_left -= NETFLOW_V1_HEADER_LENGTH + count * flow_record_length; // next header v1_header = (netflow_v1_header_t *)v1_record; // should never be < 0 done = size_left <= 0; } // End of while !done return; } /* End of Process_v1 */ nfdump-1.6.23/bin/netflow_v1.h000066400000000000000000000055761404501030700161100ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _NETFLOW_V1_H #define _NETFLOW_V1_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include "collector.h" #define NETFLOW_V1_HEADER_LENGTH 16 #define NETFLOW_V1_RECORD_LENGTH 48 #define NETFLOW_V1_MAX_RECORDS 24 /* v1 structures */ typedef struct netflow_v1_header { uint16_t version; uint16_t count; uint32_t SysUptime; uint32_t unix_secs; uint32_t unix_nsecs; } netflow_v1_header_t; typedef struct netflow_v1_record { uint32_t srcaddr; uint32_t dstaddr; uint32_t nexthop; uint16_t input; uint16_t output; uint32_t dPkts; uint32_t dOctets; uint32_t First; uint32_t Last; uint16_t srcport; uint16_t dstport; uint16_t pad1; uint8_t prot; uint8_t tos; uint8_t tcp_flags; uint8_t pad2[7]; } netflow_v1_record_t; /* prototypes */ int Init_v1(int v); void Process_v1(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs); /* * Extension map for v1 * * Required extensions: * * 4 byte byte counter * | 4byte packet counter * | | IPv4 * | | | * xxxx x0 0 0 * * Optional extensions: * * 4 : 2 byte input/output interface id * 9 : IPv4 next hop */ #endif //_NETFLOW_V1_H nfdump-1.6.23/bin/netflow_v5_v7.c000066400000000000000000000650451404501030700165200ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifdef HAVE_CONFIG_H #include "config.h" #endif #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "nfnet.h" #include "output_raw.h" #include "bookkeeper.h" #include "collector.h" #include "exporter.h" #include "netflow_v5_v7.h" extern extension_descriptor_t extension_descriptor[]; /* module limited globals */ static int verbose; static uint32_t default_sampling; static uint32_t overwrite_sampling; static extension_info_t v5_extension_info; // common for all v5 records static uint16_t v5_output_record_size, v5_output_record_base_size; // All required extension to save full v5 records static uint16_t v5_full_mapp[] = { EX_IO_SNMP_2, EX_AS_2, EX_MULIPLE, EX_NEXT_HOP_v4, EX_ROUTER_IP_v4, EX_ROUTER_ID, EX_RECEIVED, 0 }; // to simplify, assume blocks with 64 bit counters to check for enough buffer space // regardless if 32 or 64 bit packet/byte counters #define V5_BLOCK_DATA_SIZE (sizeof(ipv4_block_t) - sizeof(uint32_t) + 2 * sizeof(uint64_t)) typedef struct exporter_v5_s { // identical to exporter_t struct exporter_v5_s *next; // exporter information exporter_info_record_t info; uint64_t packets; // number of packets sent by this exporter uint64_t flows; // number of flow records sent by this exporter uint32_t sequence_failure; // number of sequence failues uint32_t padding_errors; // number of sequence failues // sampler sampler_t *sampler; // end of exporter_t // sequence vars int64_t last_sequence; int64_t sequence, distance; int64_t last_count; int first; // extension map extension_map_t *extension_map; } exporter_v5_t; // for sending netflow v5 static netflow_v5_header_t *v5_output_header; static netflow_v5_record_t *v5_output_record; static exporter_v5_t output_engine; static inline exporter_v5_t *GetExporter(FlowSource_t *fs, netflow_v5_header_t *header); static inline int CheckBufferSpace(nffile_t *nffile, size_t required); /* functions */ #include "nffile_inline.c" int Init_v5_v7_input(int v, uint32_t sampling, uint32_t overwrite) { int i, id, map_index; int extension_size; uint16_t map_size; assert(sizeof(netflow_v5_header_t) == NETFLOW_V5_HEADER_LENGTH); assert(sizeof(netflow_v5_record_t) == NETFLOW_V5_RECORD_LENGTH); verbose = v; default_sampling = sampling; overwrite_sampling = overwrite; extension_size = 0; // prepare v5 extension map v5_extension_info.map = NULL; v5_extension_info.next = NULL; v5_extension_info.offset_cache = NULL; v5_extension_info.ref_count = 0; // default map - 0 extensions map_size = sizeof(extension_map_t); i=0; while ( (id = v5_full_mapp[i]) != 0 ) { if ( extension_descriptor[id].enabled ) { extension_size += extension_descriptor[id].size; map_size += sizeof(uint16_t); } i++; } // extension_size contains the sum of all optional extensions // caculate the record size without counters! v5_output_record_base_size = COMMON_RECORD_DATA_SIZE + 8 + extension_size; // + 8 for 2 x IPv4 addr // align 32 bits if ( ( map_size & 0x3 ) != 0 ) map_size += 2; // Create a v5 extension map v5_extension_info.map = (extension_map_t *)malloc((size_t)map_size); if ( !v5_extension_info.map ) { LogError("Process_v5: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return 0; } v5_extension_info.map->type = ExtensionMapType; v5_extension_info.map->size = map_size; v5_extension_info.map->map_id = INIT_ID; v5_extension_info.map->extension_size = extension_size; // see netflow_v5_v7.h for extension map description map_index = 0; i=0; while ( (id = v5_full_mapp[i]) != 0 ) { if ( extension_descriptor[id].enabled ) v5_extension_info.map->ex_id[map_index++] = id; i++; } v5_extension_info.map->ex_id[map_index] = 0; return 1; } // End of Init_v5_input /* * functions used for receiving netflow v5 records */ static inline exporter_v5_t *GetExporter(FlowSource_t *fs, netflow_v5_header_t *header) { exporter_v5_t **e = (exporter_v5_t **)&(fs->exporter_data); sampler_t *sampler; uint16_t engine_tag = ntohs(header->engine_tag); uint16_t version = ntohs(header->version); #define IP_STRING_LEN 40 char ipstr[IP_STRING_LEN]; // search the appropriate exporter engine while ( *e ) { if ( (*e)->info.version == version && (*e)->info.id == engine_tag && (*e)->info.ip.V6[0] == fs->ip.V6[0] && (*e)->info.ip.V6[1] == fs->ip.V6[1]) return *e; e = &((*e)->next); } // nothing found *e = (exporter_v5_t *)malloc(sizeof(exporter_v5_t)); if ( !(*e)) { LogError("Process_v5: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return NULL; } memset((void *)(*e), 0, sizeof(exporter_v5_t)); (*e)->next = NULL; (*e)->info.header.type = ExporterInfoRecordType; (*e)->info.header.size = sizeof(exporter_info_record_t); (*e)->info.version = version; (*e)->info.id = engine_tag; (*e)->info.ip = fs->ip; (*e)->info.sa_family = fs->sa_family; (*e)->sequence_failure = 0; (*e)->padding_errors = 0; (*e)->packets = 0; (*e)->flows = 0; (*e)->first = 1; sampler = (sampler_t *)malloc(sizeof(sampler_t)); if ( !sampler ) { LogError("Process_v5: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); return NULL; } (*e)->sampler = sampler; sampler->info.header.type = SamplerInfoRecordype; sampler->info.header.size = sizeof(sampler_info_record_t); sampler->info.id = -1; sampler->info.mode = (0xC000 & ntohs(header->sampling_interval)) >> 14; sampler->info.interval = 0x3fff & ntohs(header->sampling_interval); sampler->next = NULL; // default is global default_sampling ( user defined or unsampled => 1 ) if ( sampler->info.interval == 0 ) sampler->info.interval = default_sampling; // copy the v5 extension map (*e)->extension_map = (extension_map_t *)malloc(v5_extension_info.map->size); if ( !(*e)->extension_map ) { LogError("Process_v5: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno)); free(*e); *e = NULL; return NULL; } memcpy((void *)(*e)->extension_map, (void *)v5_extension_info.map, v5_extension_info.map->size); if ( !AddExtensionMap(fs, (*e)->extension_map) ) { // bad - we must free this map and fail - otherwise data can not be read any more free((*e)->extension_map); free(*e); *e = NULL; return NULL; } (*e)->info.sysid = 0; FlushInfoExporter(fs, &((*e)->info)); sampler->info.exporter_sysid = (*e)->info.sysid; FlushInfoSampler(fs, &(sampler->info)); if ( fs->sa_family == AF_INET ) { uint32_t _ip = htonl(fs->ip.V4); inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr)); } else if ( fs->sa_family == AF_INET6 ) { uint64_t _ip[2]; _ip[0] = htonll(fs->ip.V6[0]); _ip[1] = htonll(fs->ip.V6[1]); inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr)); } else { strncpy(ipstr, "", IP_STRING_LEN); } dbg_printf("New Exporter: v5 SysID: %u, Extension ID: %i, IP: %s, Sampling Mode: %i, Sampling Interval: %u\n", (*e)->info.sysid, (*e)->extension_map->map_id, ipstr, sampler->info.mode ,sampler->info.interval); LogInfo("Process_v5: New exporter: SysID: %u, engine id %u, type %u, IP: %s, Sampling Mode: %i, Sampling Interval: %u\n", (*e)->info.sysid, ( engine_tag & 0xFF ),( (engine_tag >> 8) & 0xFF ), ipstr, sampler->info.mode ,sampler->info.interval ); if ( overwrite_sampling > 0 ) { sampler->info.interval = overwrite_sampling; LogInfo("Process_v5: Hard overwrite sampling rate: %u\n", sampler->info.interval); } return (*e); } // End of GetExporter void Process_v5_v7(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs) { netflow_v5_header_t *v5_header; netflow_v5_record_t *v5_record; exporter_v5_t *exporter; extension_map_t *extension_map; common_record_t *common_record; uint64_t start_time, end_time, boot_time; uint32_t First, Last; uint16_t count; uint8_t flags; int i, done, version, flow_record_length; ssize_t size_left; char *string; /* * v7 is treated as v5. It differes only in the record length, for what we process. */ // map v5 data structure to input buffer v5_header = (netflow_v5_header_t *)in_buff; exporter = GetExporter(fs, v5_header); if ( !exporter ) { LogError("Process_v5: Exporter NULL: Abort v5/v7 record processing"); return; } exporter->packets++; // calculate record size depending on counter size // sigh .. one day I should fix switch to 64bits if ( exporter->sampler->info.interval == 1 ) { flags = 0; v5_output_record_size = v5_output_record_base_size + 8; // 2 x 4 byte counters } else { flags = 0; SetFlag(flags, FLAG_SAMPLED); SetFlag(flags, FLAG_PKG_64); SetFlag(flags, FLAG_BYTES_64); v5_output_record_size = v5_output_record_base_size + 16; // 2 x 8 byte counters } extension_map = exporter->extension_map; version = ntohs(v5_header->version); flow_record_length = version == 5 ? NETFLOW_V5_RECORD_LENGTH : NETFLOW_V7_RECORD_LENGTH; // this many data to process size_left = in_buff_cnt; common_record = fs->nffile->buff_ptr; done = 0; while ( !done ) { ipv4_block_t *ipv4_block; /* Process header */ // count check count = ntohs(v5_header->count); if ( count > NETFLOW_V5_MAX_RECORDS ) { LogError("Process_v5: Unexpected record count in header: %i. Abort v5/v7 record processing", count); fs->nffile->buff_ptr = (void *)common_record; return; } // input buffer size check for all expected records if ( size_left < ( NETFLOW_V5_HEADER_LENGTH + count * flow_record_length) ) { LogError("Process_v5: Not enough data to process v5 record. Abort v5/v7 record processing"); fs->nffile->buff_ptr = (void *)common_record; return; } // output buffer size check for all expected records if ( !CheckBufferSpace(fs->nffile, count * v5_output_record_size) ) { // fishy! - should never happen. maybe disk full? LogError("Process_v5: output buffer size error. Abort v5/v7 record processing"); return; } // map output record to memory buffer common_record = (common_record_t *)fs->nffile->buff_ptr; ipv4_block = (ipv4_block_t *)common_record->data; // sequence check if ( exporter->first ) { exporter->last_sequence = ntohl(v5_header->flow_sequence); exporter->sequence = exporter->last_sequence; exporter->first = 0; } else { exporter->last_sequence = exporter->sequence; exporter->sequence = ntohl(v5_header->flow_sequence); exporter->distance = exporter->sequence - exporter->last_sequence; // handle overflow if (exporter->distance < 0) { exporter->distance = 0xffffffff + exporter->distance +1; } if (exporter->distance != exporter->last_count) { #define delta(a,b) ( (a)>(b) ? (a)-(b) : (b)-(a) ) fs->nffile->stat_record->sequence_failure++; exporter->sequence_failure++; /* LogError("Flow v%d sequence last:%llu now:%llu mismatch. Missing: dist:%lu flows", version, exporter->last_sequence, exporter->sequence, exporter->distance); */ } } exporter->last_count = count; v5_header->SysUptime = ntohl(v5_header->SysUptime); v5_header->unix_secs = ntohl(v5_header->unix_secs); v5_header->unix_nsecs = ntohl(v5_header->unix_nsecs); /* calculate boot time in msec */ boot_time = ((uint64_t)(v5_header->unix_secs)*1000 + ((uint64_t)(v5_header->unix_nsecs) / 1000000) ) - (uint64_t)(v5_header->SysUptime); // process all records v5_record = (netflow_v5_record_t *)((pointer_addr_t)v5_header + NETFLOW_V5_HEADER_LENGTH); /* loop over each records associated with this header */ for (i = 0; i < count; i++) { pointer_addr_t bsize; uint64_t packets, bytes; void *data_ptr; uint8_t *s1, *s2; int j, id; // header data common_record->flags = flags; common_record->type = CommonRecordType; common_record->exporter_sysid = exporter->info.sysid;; common_record->ext_map = extension_map->map_id; common_record->size = v5_output_record_size; // v5 common fields common_record->srcport = ntohs(v5_record->srcport); common_record->dstport = ntohs(v5_record->dstport); common_record->tcp_flags = v5_record->tcp_flags; common_record->prot = v5_record->prot; common_record->tos = v5_record->tos; common_record->fwd_status = 0; common_record->nfversion = 5; common_record->biFlowDir = 0; common_record->flowEndReason = 0; // v5 typed data as fixed struct v5_block ipv4_block->srcaddr = ntohl(v5_record->srcaddr); ipv4_block->dstaddr = ntohl(v5_record->dstaddr); if ( exporter->sampler->info.interval == 1 ) { value32_t *v = (value32_t *)ipv4_block->data; packets = (uint64_t)ntohl(v5_record->dPkts); bytes = (uint64_t)ntohl(v5_record->dOctets); v->val = packets; v = (value32_t *)v->data; v->val = bytes; data_ptr = (void *)v->data; } else { value64_t *v = (value64_t *)ipv4_block->data; uint32_t *ptr = (uint32_t *)&packets; packets = (uint64_t)ntohl(v5_record->dPkts) * (uint64_t)exporter->sampler->info.interval; bytes = (uint64_t)ntohl(v5_record->dOctets) * (uint64_t)exporter->sampler->info.interval; // pack packets in 32bit chunks v->val.val32[0] = ptr[0]; v->val.val32[1] = ptr[1]; // pack bytes in 32bit chunks v = (value64_t *)v->data; ptr = (uint32_t *)&bytes; v->val.val32[0] = ptr[0]; v->val.val32[1] = ptr[1]; data_ptr = (void *)v->data; } // process optional extensions j = 0; while ( (id = extension_map->ex_id[j]) != 0 ) { switch (id) { case EX_IO_SNMP_2: { // 2 byte input/output interface index tpl_ext_4_t *tpl = (tpl_ext_4_t *)data_ptr; tpl->input = ntohs(v5_record->input); tpl->output = ntohs(v5_record->output); data_ptr = (void *)tpl->data; } break; case EX_AS_2: { // 2 byte src/dst AS number tpl_ext_6_t *tpl = (tpl_ext_6_t *)data_ptr; tpl->src_as = ntohs(v5_record->src_as); tpl->dst_as = ntohs(v5_record->dst_as); data_ptr = (void *)tpl->data; } break; case EX_MULIPLE: { // dst tos, direction, src/dst mask tpl_ext_8_t *tpl = (tpl_ext_8_t *)data_ptr; tpl->dst_tos = 0; tpl->dir = 0; tpl->src_mask = v5_record->src_mask; tpl->dst_mask = v5_record->dst_mask; data_ptr = (void *)tpl->data; } break; case EX_NEXT_HOP_v4: { // IPv4 next hop tpl_ext_9_t *tpl = (tpl_ext_9_t *)data_ptr; tpl->nexthop = ntohl(v5_record->nexthop); data_ptr = (void *)tpl->data; } break; case EX_ROUTER_IP_v4: { // IPv4 router address tpl_ext_23_t *tpl = (tpl_ext_23_t *)data_ptr; tpl->router_ip = fs->ip.V4; data_ptr = (void *)tpl->data; ClearFlag(common_record->flags, FLAG_IPV6_EXP); } break; case EX_ROUTER_ID: { // engine type, engine ID tpl_ext_25_t *tpl = (tpl_ext_25_t *)data_ptr; uint16_t engine_tag = ntohs(v5_header->engine_tag); tpl->engine_type = (engine_tag >> 8) & 0xFF; tpl->engine_id = (engine_tag & 0xFF); data_ptr = (void *)tpl->data; } break; case EX_RECEIVED: { tpl_ext_27_t *tpl = (tpl_ext_27_t *)data_ptr; tpl->received = (uint64_t)((uint64_t)fs->received.tv_sec * 1000LL) + (uint64_t)((uint64_t)fs->received.tv_usec / 1000LL); data_ptr = (void *)tpl->data; } break; default: // this should never happen, as v5 has no other extensions LogError("Process_v5: Unexpected extension %i for v5 record. Skip extension", id); } j++; } // Time issues First = ntohl(v5_record->First); Last = ntohl(v5_record->Last); #ifdef FIXTIMEBUG /* * Some users report, that they see flows, which have duration time of about 40days * which is almost the overflow value. Investigating this, it cannot be an overflow * and the difference is always 15160 or 15176 msec too little for a classical * overflow. Therefore assume this must be an exporter bug */ if ( First > Last && ( (First - Last) < 20000) ) { uint32_t _t; LogError("Process_v5: Unexpected time swap: First 0x%llx smaller than boot time: 0x%llx", start_time, boot_time); _t= First; First = Last; Last = _t; } #endif if ( First > Last ) { /* First in msec, in case of msec overflow, between start and end */ start_time = boot_time - 0x100000000LL + (uint64_t)First; } else { start_time = boot_time + (uint64_t)First; } /* end time in msecs */ end_time = (uint64_t)Last + boot_time; // if overflow happened after flow ended but before got exported // the additional check > 100000 is required due to a CISCO IOS bug // CSCei12353 - thanks to Bojan if ( Last > v5_header->SysUptime && (( Last - v5_header->SysUptime) > 100000)) { start_time -= 0x100000000LL; end_time -= 0x100000000LL; } common_record->first = start_time/1000; common_record->msec_first = start_time - common_record->first*1000; common_record->last = end_time/1000; common_record->msec_last = end_time - common_record->last*1000; // update first_seen, last_seen if ( start_time < fs->first_seen ) fs->first_seen = start_time; if ( end_time > fs->last_seen ) fs->last_seen = end_time; // Update stats switch (common_record->prot) { case IPPROTO_ICMP: fs->nffile->stat_record->numflows_icmp++; fs->nffile->stat_record->numpackets_icmp += packets; fs->nffile->stat_record->numbytes_icmp += bytes; // fix odd CISCO behaviour for ICMP port/type in src port if ( common_record->srcport != 0 ) { s1 = (uint8_t *)&(common_record->srcport); s2 = (uint8_t *)&(common_record->dstport); s2[0] = s1[1]; s2[1] = s1[0]; common_record->srcport = 0; } break; case IPPROTO_TCP: fs->nffile->stat_record->numflows_tcp++; fs->nffile->stat_record->numpackets_tcp += packets; fs->nffile->stat_record->numbytes_tcp += bytes; break; case IPPROTO_UDP: fs->nffile->stat_record->numflows_udp++; fs->nffile->stat_record->numpackets_udp += packets; fs->nffile->stat_record->numbytes_udp += bytes; break; default: fs->nffile->stat_record->numflows_other++; fs->nffile->stat_record->numpackets_other += packets; fs->nffile->stat_record->numbytes_other += bytes; } exporter->flows++; fs->nffile->stat_record->numflows++; fs->nffile->stat_record->numpackets += packets; fs->nffile->stat_record->numbytes += bytes; if ( verbose ) { master_record_t master_record; memset((void *)&master_record, 0, sizeof(master_record_t)); ExpandRecord_v2((common_record_t *)common_record, &v5_extension_info, &(exporter->info), &master_record); flow_record_to_raw(&master_record, &string, 0); printf("%s\n", string); } // advance to next input flow record v5_record = (netflow_v5_record_t *)((pointer_addr_t)v5_record + flow_record_length); if ( ((pointer_addr_t)data_ptr - (pointer_addr_t)common_record) != v5_output_record_size ) { printf("Panic size check: ptr diff: %llu, record size: %u\n", (unsigned long long)((pointer_addr_t)data_ptr - (pointer_addr_t)common_record), v5_output_record_size ); abort(); } // advance to next output record common_record = (common_record_t *)data_ptr; ipv4_block = (ipv4_block_t *)common_record->data; // buffer size sanity check - should never happen, but check it anyway bsize = (pointer_addr_t)common_record - (pointer_addr_t)fs->nffile->block_header - sizeof(data_block_header_t); if ( bsize >= BUFFSIZE ) { LogError("### Software error ###: %s line %d", __FILE__, __LINE__); LogError("Process_v5: Output buffer overflow! Flush buffer and skip records."); LogError("Buffer size: size: %u, bsize: %llu > %u", fs->nffile->block_header->size, (unsigned long long)bsize, BUFFSIZE); // reset buffer fs->nffile->block_header->size = 0; fs->nffile->block_header->NumRecords = 0; fs->nffile->buff_ptr = (void *)((pointer_addr_t)fs->nffile->block_header + sizeof(data_block_header_t) ); return; } } // End of foreach v5 record // update file record size ( -> output buffer size ) fs->nffile->block_header->NumRecords += count; fs->nffile->block_header->size += count * v5_output_record_size; fs->nffile->buff_ptr = (void *)common_record; // still to go for this many input bytes size_left -= NETFLOW_V5_HEADER_LENGTH + count * flow_record_length; // next header v5_header = (netflow_v5_header_t *)v5_record; // should never be < 0 done = size_left <= 0; } // End of while !done return; } /* End of Process_v5 */ /* * functions used for sending netflow v5 records */ void Init_v5_v7_output(send_peer_t *peer) { assert(sizeof(netflow_v5_header_t) == NETFLOW_V5_HEADER_LENGTH); assert(sizeof(netflow_v5_record_t) == NETFLOW_V5_RECORD_LENGTH); v5_output_header = (netflow_v5_header_t *)peer->send_buffer; v5_output_header->version = htons(5); v5_output_header->SysUptime = 0; v5_output_header->unix_secs = 0; v5_output_header->unix_nsecs = 0; v5_output_header->count = 0; output_engine.first = 1; output_engine.sequence = 0; output_engine.last_sequence = 0; output_engine.last_count = 0; output_engine.sequence_failure = 0; v5_output_record = (netflow_v5_record_t *)((pointer_addr_t)v5_output_header + (pointer_addr_t)sizeof(netflow_v5_header_t)); } // End of Init_v5_v7_output int Add_v5_output_record(master_record_t *master_record, send_peer_t *peer) { static uint64_t boot_time; // in msec static int cnt; extension_map_t *extension_map = master_record->map_ref; uint32_t i, id, t1, t2; // Skip IPv6 records if ( (master_record->flags & FLAG_IPV6_ADDR ) != 0 ) return 0; if ( output_engine.first ) { // first time a record is added // boot time is set one day back - assuming that the start time of every flow does not start ealier boot_time = (uint64_t)(master_record->first - 86400)*1000LL; cnt = 0; output_engine.first = 0; } if ( cnt == 0 ) { v5_output_record = (netflow_v5_record_t *)((pointer_addr_t)peer->send_buffer + NETFLOW_V5_HEADER_LENGTH); peer->buff_ptr = (void *)v5_output_record; memset(peer->buff_ptr, 0, NETFLOW_V5_MAX_RECORDS * NETFLOW_V5_RECORD_LENGTH); output_engine.sequence += output_engine.last_count; v5_output_header->flow_sequence = htonl(output_engine.sequence); uint32_t unix_secs = master_record->last + 3600; v5_output_header->unix_secs = htonl(unix_secs); v5_output_header->SysUptime = htonl((uint32_t)(unix_secs * 1000 - boot_time)); } t1 = (uint32_t)(1000LL * (uint64_t)master_record->first + (uint64_t)master_record->msec_first - boot_time); t2 = (uint32_t)(1000LL * (uint64_t)master_record->last + (uint64_t)master_record->msec_last - boot_time); v5_output_record->First = htonl(t1); v5_output_record->Last = htonl(t2); v5_output_record->srcaddr = htonl(master_record->V4.srcaddr); v5_output_record->dstaddr = htonl(master_record->V4.dstaddr); v5_output_record->srcport = htons(master_record->srcport); v5_output_record->dstport = htons(master_record->dstport); v5_output_record->tcp_flags = master_record->tcp_flags; v5_output_record->prot = master_record->prot; v5_output_record->tos = master_record->tos; // the 64bit counters are cut down to 32 bits for v5 v5_output_record->dPkts = htonl((uint32_t)master_record->dPkts); v5_output_record->dOctets = htonl((uint32_t)master_record->dOctets); v5_output_record->input = htons(master_record->input); v5_output_record->output = htons(master_record->output); v5_output_record->src_as = htons(master_record->srcas); v5_output_record->dst_as = htons(master_record->dstas); i = 0; while ( (id = extension_map->ex_id[i]) != 0 ) { switch (id) { case EX_IO_SNMP_2: v5_output_record->input = htons(master_record->input); v5_output_record->output = htons(master_record->output); break; case EX_AS_2: v5_output_record->src_as = htons(master_record->srcas); v5_output_record->dst_as = htons(master_record->dstas); break; case EX_MULIPLE: v5_output_record->src_mask = master_record->src_mask; v5_output_record->dst_mask = master_record->dst_mask; break; case EX_NEXT_HOP_v4: v5_output_record->nexthop = htonl(master_record->ip_nexthop.V4); break; // default: Other extensions can not be sent with v5 } i++; } cnt++; v5_output_header->count = htons(cnt); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + NETFLOW_V5_RECORD_LENGTH); v5_output_record++; if ( cnt == NETFLOW_V5_MAX_RECORDS ) { peer->flush = 1; output_engine.last_count = cnt; cnt = 0; } return 0; } // End of Add_v5_output_record nfdump-1.6.23/bin/netflow_v5_v7.h000066400000000000000000000102011404501030700165050ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #ifndef _NETFLOW_V5_V7_H #define _NETFLOW_V5_V7_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include "collector.h" #include "nffile.h" #include "nfnet.h" #define NETFLOW_V5_HEADER_LENGTH 24 #define NETFLOW_V5_RECORD_LENGTH 48 #define NETFLOW_V5_MAX_RECORDS 30 #define NETFLOW_V7_HEADER_LENGTH 24 #define NETFLOW_V7_RECORD_LENGTH 52 #define NETFLOW_V7_MAX_RECORDS 28 /* v5 structures */ typedef struct netflow_v5_header { uint16_t version; uint16_t count; uint32_t SysUptime; uint32_t unix_secs; uint32_t unix_nsecs; uint32_t flow_sequence; uint16_t engine_tag; uint16_t sampling_interval; } netflow_v5_header_t; typedef struct netflow_v5_record { uint32_t srcaddr; uint32_t dstaddr; uint32_t nexthop; uint16_t input; uint16_t output; uint32_t dPkts; uint32_t dOctets; uint32_t First; uint32_t Last; uint16_t srcport; uint16_t dstport; uint8_t pad1; uint8_t tcp_flags; uint8_t prot; uint8_t tos; uint16_t src_as; uint16_t dst_as; uint8_t src_mask; uint8_t dst_mask; uint16_t pad2; } netflow_v5_record_t; /* v7 structures */ typedef struct netflow_v7_header { uint16_t version; uint16_t count; uint32_t SysUptime; uint32_t unix_secs; uint32_t unix_nsecs; uint32_t flow_sequence; uint32_t reserved; } netflow_v7_header_t; typedef struct netflow_v7_record { uint32_t srcaddr; uint32_t dstaddr; uint32_t nexthop; uint16_t input; uint16_t output; uint32_t dPkts; uint32_t dOctets; uint32_t First; uint32_t Last; uint16_t srcport; uint16_t dstport; uint8_t flags; uint8_t tcp_flags; uint8_t prot; uint8_t tos; uint16_t src_as; uint16_t dst_as; uint8_t src_mask; uint8_t dst_mask; uint16_t pad; uint32_t router_sc; } netflow_v7_record_t; /* prototypes */ int Init_v5_v7_input(int v, uint32_t sampling, uint32_t overwrite); void Process_v5_v7(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs); void Init_v5_v7_output(send_peer_t *peer); int Add_v5_output_record(master_record_t *master_record, send_peer_t *peer); /* * Extension map for v5/v7 * * Required extensions: * * 4 byte byte counter * | 4byte packet counter * | | IPv4 * | | | * xxxx x0 0 0 * * Optional extensions: * * 4 : 2 byte input/output interface id * 6 : 2 byte src/dst as * 8 : srcmask/dst mask dst tos = 0, dir = 0 * 9 : IPv4 next hop */ #endif //_NETFLOW_V5_V7_H nfdump-1.6.23/bin/netflow_v9.c000066400000000000000000003410311404501030700161000ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "nfnet.h" #include "output_raw.h" #include "bookkeeper.h" #include "collector.h" #include "exporter.h" #include "netflow_v9.h" #define GET_FLOWSET_ID(p) (Get_val16(p)) #define GET_FLOWSET_LENGTH(p) (Get_val16((void *)((p) + 2))) #define GET_TEMPLATE_ID(p) (Get_val16(p)) #define GET_TEMPLATE_COUNT(p) (Get_val16((void *)((p) + 2))) #define GET_OPTION_TEMPLATE_ID(p) (Get_val16(p)) #define GET_OPTION_TEMPLATE_OPTION_SCOPE_LENGTH(p) (Get_val16((void *)((p) + 2))) #define GET_OPTION_TEMPLATE_OPTION_LENGTH(p) (Get_val16((void *)((p) + 4))) #define CHECK_OPTION_DATA(avail, tag) ((tag.offset + tag.length) <= avail) extern extension_descriptor_t extension_descriptor[]; extern uint32_t Max_num_extensions; static int verbose; static uint32_t default_sampling; static uint32_t overwrite_sampling; typedef struct sequence_map_s { /* sequence definition: just move a certain number of bytes -> moveXX set a certain number of output bytes to zero -> zeroXX process input data into appropriate output -> AnyName */ #define nop 0 #define move8 1 #define move16 2 #define move32 3 #define move40 4 #define move48 5 #define move56 6 #define move64 7 #define move64_32 8 #define move96 9 #define move128 10 #define move32_sampling 11 #define move64_sampling 12 #define move_mac 13 #define move_mpls 14 #define move_ulatency 15 #define move_slatency 16 #define move_user_20 17 #define move_user_65 18 #define TimeMsec 19 #define PushTimeMsec 20 #define saveICMP 21 #define zero8 22 #define zero16 23 #define zero32 24 #define zero64 25 #define zero96 26 #define zero128 27 uint32_t id; // sequence ID as defined above uint16_t input_offset; // copy/process data at this input offset uint16_t output_offset; // copy final data to this output offset void *stack; // optionally copy data onto this stack } sequence_map_t; typedef struct input_translation_s { struct input_translation_s *next; uint32_t flags; time_t updated; uint32_t id; uint32_t input_record_size; uint32_t output_record_size; // tmp vars needed while processing the data record uint32_t ICMP_offset; // offset of ICMP type/code in data stream uint64_t flow_start; // start time in msec uint64_t flow_end; // end time in msec uint64_t EventTimeMsec; // Event time in msec for NSEL/NEL uint64_t packets; // total packets - sampling corrected uint64_t bytes; // total bytes - sampling corrected uint64_t out_packets; // total out packets - sampling corrected uint64_t out_bytes; // total out bytes - sampling corrected uint32_t sampler_offset; uint32_t sampler_size; uint32_t engine_offset; uint32_t received_offset; uint32_t router_ip_offset; // extension map infos uint32_t extension_map_changed; // map changed while refreshing extension_info_t extension_info; // the nfcap extension map, reflecting this template // sequence map information uint32_t number_of_sequences; // number of sequences for the translate sequence_map_t *sequence; // sequence map } input_translation_t; typedef struct exporter_domain_s { // identical to exporter_t struct exporter_domain_s *next; // exporter information exporter_info_record_t info; uint64_t packets; // number of packets sent by this exporter uint64_t flows; // number of flow records sent by this exporter uint32_t sequence_failure; // number of sequence failues uint32_t padding_errors; // number of padding errors // sampler sampler_t *sampler; samplerOption_t *samplerOption; // sampler options table info // exporter parameters uint64_t boot_time; // sequence int64_t last_sequence; int64_t sequence; int first; // sampling information: // each flow source may have several sampler applied // tags #48, #49, #50 // each sampler is assinged a sampler struct // global sampling information #34 #35 // stored in a sampler with id = -1; // translation table input_translation_t *input_translation_table; input_translation_t *current_table; } exporterDomain_t; /* module limited globals */ static struct v9_element_map_s { uint16_t id; // v9 element id char *name; // name string uint16_t length; // type of this element ( input length ) uint16_t out_length; // type of this element ( output length ) uint32_t sequence; // sequencer ID uint32_t zero_sequence; // uint16_t extension; // maps into nfdump extension ID } v9_element_map[] = { {0, 0, 0}, // packets and bytes are always stored in 64bits { NF9_IN_BYTES, "bytes", _4bytes, _8bytes, move32_sampling, zero64, COMMON_BLOCK }, { NF9_IN_BYTES, "bytes", _8bytes, _8bytes, move64_sampling, zero64, COMMON_BLOCK }, { NF9_IN_PACKETS, "packets", _4bytes, _8bytes, move32_sampling, zero64, COMMON_BLOCK }, { NF9_IN_PACKETS, "packets", _8bytes, _8bytes, move64_sampling, zero64, COMMON_BLOCK }, { NF_F_INITIATORPACKETS, "packets", _8bytes, _8bytes, move64_sampling, zero64, COMMON_BLOCK }, { NF9_FLOWS_AGGR, "flows", _4bytes, _4bytes, move32, zero32, EX_AGGR_FLOWS_4 }, { NF9_FLOWS_AGGR, "flows", _8bytes, _8bytes, move64, zero64, EX_AGGR_FLOWS_8 }, { NF9_IN_PROTOCOL, "proto", _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { NF9_SRC_TOS, "tos", _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { NF9_TCP_FLAGS, "flags", _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { NF9_L4_SRC_PORT, "src port", _2bytes, _2bytes, move16, zero16, COMMON_BLOCK }, { NF9_IPV4_SRC_ADDR, "V4 src addr", _4bytes, _4bytes, move32, zero32, COMMON_BLOCK }, { NF9_SRC_MASK, "V4 src mask", _1byte, _1byte, move8, zero8, EX_MULIPLE }, { NF9_INPUT_SNMP, "input SNMP", _2bytes, _2bytes, move16, zero16, EX_IO_SNMP_2 }, { NF9_INPUT_SNMP, "input SNMP", _4bytes, _4bytes, move32, zero32, EX_IO_SNMP_4 }, { NF9_L4_DST_PORT, "dst port", _2bytes, _2bytes, move16, zero16, COMMON_BLOCK }, { NF9_IPV4_DST_ADDR, "V4 dst addr", _4bytes, _4bytes, move32, zero32, COMMON_BLOCK }, { NF9_DST_MASK, "V4 dst mask", _1byte, _1byte, move8, zero8, EX_MULIPLE }, { NF9_OUTPUT_SNMP, "output SNMP", _2bytes, _2bytes, move16, zero16, EX_IO_SNMP_2 }, { NF9_OUTPUT_SNMP, "output SNMP", _4bytes, _4bytes, move32, zero32, EX_IO_SNMP_4 }, { NF9_V4_NEXT_HOP, "V4 next hop IP", _4bytes, _4bytes, move32, zero32, EX_NEXT_HOP_v4 }, { NF9_SRC_AS, "src AS", _2bytes, _2bytes, move16, zero16, EX_AS_2 }, { NF9_SRC_AS, "src AS", _4bytes, _4bytes, move32, zero32, EX_AS_4 }, { NF9_DST_AS, "dst AS", _2bytes, _2bytes, move16, zero16, EX_AS_2 }, { NF9_DST_AS, "dst AS", _4bytes, _4bytes, move32, zero32, EX_AS_4 }, { NF9_BGP_V4_NEXT_HOP, "V4 BGP next hop", _4bytes, _4bytes, move32, zero32, EX_NEXT_HOP_BGP_v4 }, { NF9_FIRST_SWITCHED, "time sec create", _4bytes, _4bytes, move32, zero32, COMMON_BLOCK }, { NF9_FIRST_SWITCHED, "time sec create", _8bytes, _4bytes, move64_32, zero32, COMMON_BLOCK }, { NF9_LAST_SWITCHED, "time sec end", _4bytes, _4bytes, move32, zero32, COMMON_BLOCK }, { NF9_LAST_SWITCHED, "time sec end", _8bytes, _4bytes, move64_32, zero32, COMMON_BLOCK }, { NF_F_FLOW_CREATE_TIME_MSEC, "time msec start",_8bytes, _8bytes, TimeMsec, nop, COMMON_BLOCK }, { NF_F_FLOW_END_TIME_MSEC, "time msec end", _8bytes, _8bytes, TimeMsec, nop, COMMON_BLOCK }, { NF9_OUT_BYTES, "out bytes", _4bytes, _8bytes, move32_sampling, zero64, EX_OUT_BYTES_8 }, { NF9_OUT_BYTES, "out bytes", _8bytes, _8bytes, move64_sampling, zero64, EX_OUT_BYTES_8 }, { NF9_OUT_PKTS, "out packets", _4bytes, _8bytes, move32_sampling, zero64, EX_OUT_PKG_8 }, { NF9_OUT_PKTS, "out packets", _8bytes, _8bytes, move64_sampling, zero64, EX_OUT_PKG_8 }, { NF_F_RESPONDERPACKETS, "out packets", _8bytes, _8bytes, move64_sampling, zero64, EX_OUT_PKG_8 }, { NF9_IPV6_SRC_ADDR, "V6 src addr", _16bytes, _16bytes, move128, zero128, COMMON_BLOCK }, { NF9_IPV6_DST_ADDR, "V6 dst addr", _16bytes, _16bytes, move128, zero128, COMMON_BLOCK }, { NF9_IPV6_SRC_MASK, "V6 src mask", _1byte, _1byte, move8, zero8, EX_MULIPLE }, { NF9_IPV6_DST_MASK, "V6 dst mask", _1byte, _1byte, move8, zero8, EX_MULIPLE }, /* XXX fix */ { NF9_IPV6_FLOW_LABEL, "V6 flow label", _4bytes, _4bytes, nop, nop, COMMON_BLOCK }, { NF9_ICMP_TYPE, "ICMP type", _2bytes, _2bytes, nop, nop, COMMON_BLOCK }, // sampling { NF9_SAMPLING_INTERVAL, "sampling interval", _4bytes, _4bytes, move32, zero32, COMMON_BLOCK }, { NF9_SAMPLING_ALGORITHM, "sampling algorithm", _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { NF9_ENGINE_TYPE, "engine type", _1byte, _1byte, move8, zero8, EX_ROUTER_ID }, { NF9_ENGINE_ID, "engine ID", _1byte, _1byte, move8, zero8, EX_ROUTER_ID }, // sampling { NF9_FLOW_SAMPLER_ID, "sampler ID", _1byte, _1byte, nop, nop, COMMON_BLOCK }, { NF9_FLOW_SAMPLER_ID, "sampler ID", _2bytes, _2bytes, nop, nop, COMMON_BLOCK }, { NF9_FLOW_SAMPLER_ID, "sampler ID", _4bytes, _4bytes, nop, nop, COMMON_BLOCK }, { FLOW_SAMPLER_MODE, "sampler mode", _1byte, _1byte, nop, nop, COMMON_BLOCK }, { NF9_FLOW_SAMPLER_RANDOM_INTERVAL, "sampler rand interval", _4bytes, _4bytes, nop, nop, COMMON_BLOCK }, { NF9_DST_TOS, "dst tos", _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { NF9_IN_SRC_MAC, "in src mac", _6bytes, _8bytes, move_mac, zero64, EX_MAC_1}, { NF9_OUT_DST_MAC, "out dst mac", _6bytes, _8bytes, move_mac, zero64, EX_MAC_1}, { NF9_SRC_VLAN, "src vlan", _2bytes, _2bytes, move16, zero16, EX_VLAN}, { NF9_DST_VLAN, "dst vlan", _2bytes, _2bytes, move16, zero16, EX_VLAN}, { NF9_dot1qVlanId, "src vlan", _2bytes, _2bytes, move16, zero16, EX_VLAN}, { NF9_postDot1qVlanId, "dst vlan", _2bytes, _2bytes, move16, zero16, EX_VLAN}, { NF9_DIRECTION, "direction", _1byte, _1byte, move8, zero8, EX_MULIPLE }, { NF9_V6_NEXT_HOP, "V6 next hop IP", _16bytes, _16bytes, move128, zero128, EX_NEXT_HOP_v6 }, { NF9_BPG_V6_NEXT_HOP, "V6 BGP next hop", _16bytes, _16bytes, move128, zero128, EX_NEXT_HOP_BGP_v6 }, // mpls { NF9_MPLS_LABEL_1, "mpls label 1", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_2, "mpls label 2", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_3, "mpls label 3", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_4, "mpls label 4", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_5, "mpls label 5", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_6, "mpls label 6", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_7, "mpls label 7", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_8, "mpls label 8", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_9, "mpls label 9", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_MPLS_LABEL_10, "mpls label 10", _3bytes, _4bytes, move_mpls, zero32, EX_MPLS}, { NF9_IN_DST_MAC, "in dst mac", _6bytes, _8bytes, move_mac, zero64, EX_MAC_2}, { NF9_OUT_SRC_MAC, "out src mac", _6bytes, _8bytes, move_mac, zero64, EX_MAC_2}, { NF9_FORWARDING_STATUS, "fwd status", _1byte, _1byte, move8, zero8, COMMON_BLOCK }, { NF9_BGP_ADJ_NEXT_AS, "BGP next AS", _4bytes, _4bytes, move32, zero32, EX_BGPADJ }, { NF9_BGP_ADJ_PREV_AS, "BGP prev AS", _4bytes, _4bytes, move32, zero32, EX_BGPADJ }, // NSEL ASA extension // NSEL common { NF_F_EVENT_TIME_MSEC, "ASA event time", _8bytes, _8bytes, PushTimeMsec, zero64, EX_NSEL_COMMON }, { NF_F_CONN_ID, "ASA conn ID", _4bytes, _4bytes, move32, zero32, EX_NSEL_COMMON }, { NF_F_FW_EVENT_84, "ASA 8.4 event", _1byte, _1byte, move8, zero8, EX_NSEL_COMMON }, { NF_F_FW_EVENT, "ASA event", _1byte, _1byte, move8, zero8, EX_NSEL_COMMON }, { NF_F_FW_EXT_EVENT, "ASA ext event", _2bytes, _2bytes, move16, zero16, EX_NSEL_COMMON }, { NF_F_ICMP_TYPE, "FNF ICMP type", _1byte, _1byte, move8, zero8, EX_NSEL_COMMON }, { NF_F_ICMP_CODE, "FNF ICMP code", _1byte, _1byte, move8, zero8, EX_NSEL_COMMON }, { NF_F_ICMP_TYPE_IPV6, "ASA ICMP type V6", _1byte, _1byte, move8, zero8, EX_NSEL_COMMON }, { NF_F_ICMP_CODE_IPV6, "ASA ICMP code V6", _1byte, _1byte, move8, zero8, EX_NSEL_COMMON }, // zone base firewall { NF_FW_CTS_SRC_SGT, "Source security group tag", _2bytes, _2bytes, move16, zero16, EX_NSEL_COMMON }, // XlATE extensions { NF_F_XLATE_SRC_ADDR_IPV4, "ASA V4 xsrc addr", _4bytes, _4bytes, move32, zero32, EX_NSEL_XLATE_IP_v4 }, { NF_F_XLATE_DST_ADDR_IPV4, "ASA V4 xdst addr", _4bytes, _4bytes, move32, zero32, EX_NSEL_XLATE_IP_v4 }, { NF_F_XLATE_SRC_ADDR_IPV6, "ASA V6 xsrc addr", _16bytes, _16bytes, move128, zero128, EX_NSEL_XLATE_IP_v6 }, { NF_F_XLATE_DST_ADDR_IPV6, "ASA V6 xdst addr", _16bytes, _16bytes, move128, zero128, EX_NSEL_XLATE_IP_v6 }, { NF_F_XLATE_SRC_PORT, "ASA xsrc port", _2bytes, _2bytes, move16, zero16, EX_NSEL_XLATE_PORTS }, { NF_F_XLATE_DST_PORT, "ASA xdst port", _2bytes, _2bytes, move16, zero16, EX_NSEL_XLATE_PORTS }, // ASA 8.4 mapping { NF_F_XLATE_SRC_ADDR_84, "ASA V4 xsrc addr", _4bytes, _4bytes, move32, zero32, EX_NSEL_XLATE_IP_v4 }, { NF_F_XLATE_DST_ADDR_84, "ASA V4 xdst addr", _4bytes, _4bytes, move32, zero32, EX_NSEL_XLATE_IP_v4 }, { NF_F_XLATE_SRC_PORT_84, "ASA 8.4 xsrc port", _2bytes, _2bytes, move16, zero16, EX_NSEL_XLATE_PORTS }, { NF_F_XLATE_DST_PORT_84, "ASA 8.4 xdst port", _2bytes, _2bytes, move16, zero16, EX_NSEL_XLATE_PORTS }, // ACL extension { NF_F_INGRESS_ACL_ID, "ASA ingress ACL", _12bytes, _12bytes, move96, zero96, EX_NSEL_ACL }, { NF_F_EGRESS_ACL_ID, "ASA egress ACL", _12bytes, _12bytes, move96, zero96, EX_NSEL_ACL }, // byte count { NF_F_FLOW_BYTES, "ASA bytes", _4bytes, _8bytes, move32_sampling, zero64, EX_NSEL_COMMON }, { NF_F_FLOW_BYTES, "ASA bytes", _8bytes, _8bytes, move64_sampling, zero64, EX_NSEL_COMMON }, { NF_F_FWD_FLOW_DELTA_BYTES, "ASA fwd bytes", _4bytes, _8bytes, move32_sampling, zero64, EX_NSEL_COMMON }, { NF_F_FWD_FLOW_DELTA_BYTES, "ASA fwd bytes", _8bytes, _8bytes, move64_sampling, zero64, EX_NSEL_COMMON }, { NF_F_REV_FLOW_DELTA_BYTES, "ASA rew bytes", _4bytes, _4bytes, move32_sampling, zero32, EX_OUT_BYTES_4 }, { NF_F_REV_FLOW_DELTA_BYTES, "ASA rew bytes", _8bytes, _8bytes, move64_sampling, zero64, EX_OUT_BYTES_8 }, // NSEL user names { NF_F_USERNAME, "ASA user name 20", _20bytes, _24bytes, move_user_20, zero32, EX_NSEL_USER }, { NF_F_USERNAME, "ASA user name 65", _65bytes, _72bytes, move_user_65, zero32, EX_NSEL_USER_MAX }, // NEL CISCO ASR 1000 series NAT logging // NEL COMMON extension { NF_N_NAT_EVENT, "NAT event", _1byte, _1byte, move8, zero8, EX_NEL_COMMON }, { NF_N_EGRESS_VRFID, "NAT egress VRFID", _4bytes, _4bytes, move32, zero32, EX_NEL_COMMON }, { NF_N_INGRESS_VRFID, "NAT ingress VRFID", _4bytes, _4bytes, move32, zero32, EX_NEL_COMMON }, // NAT Port block allocation { NF_F_XLATE_PORT_BLOCK_START, "NAT port block start", _2bytes, _2bytes, move16, zero16, EX_PORT_BLOCK_ALLOC }, { NF_F_XLATE_PORT_BLOCK_END, "NAT port block end", _2bytes, _2bytes, move16, zero16, EX_PORT_BLOCK_ALLOC }, { NF_F_XLATE_PORT_BLOCK_STEP, "NAT port step size", _2bytes, _2bytes, move16, zero16, EX_PORT_BLOCK_ALLOC }, { NF_F_XLATE_PORT_BLOCK_SIZE, "NAT port block size", _2bytes, _2bytes, move16, zero16, EX_PORT_BLOCK_ALLOC }, // nprobe latency extension { NF9_NPROBE_CLIENT_NW_DELAY_USEC, "NPROBE client lat usec", _4bytes, _8bytes, move_ulatency, zero64, EX_LATENCY }, { NF9_NPROBE_SERVER_NW_DELAY_USEC, "NPROBE server lat usec", _4bytes, _8bytes, move_ulatency, zero64, EX_LATENCY }, { NF9_NPROBE_APPL_LATENCY_USEC, "NPROBE appl lat usec", _4bytes, _8bytes, move_ulatency, zero64, EX_LATENCY }, { NF9_NPROBE_CLIENT_NW_DELAY_SEC, "NPROBE client lat sec", _4bytes, _8bytes, move_slatency, nop, EX_LATENCY }, { NF9_NPROBE_SERVER_NW_DELAY_SEC, "NPROBE server lat sec", _4bytes, _8bytes, move_slatency, nop, EX_LATENCY }, { NF9_NPROBE_APPL_LATENCY_SEC, "NPROBE appl lat sec", _4bytes, _8bytes, move_slatency, nop, EX_LATENCY }, {0, "NULL", 0, 0} }; /* * tmp cache while processing template records * array index = extension id, * value = 1 -> extension exists, 0 -> extension does not exists */ static struct cache_s { struct element_param_s { uint16_t index; uint16_t found; uint16_t offset; uint16_t length; } *lookup_info; // 65535 element 16byte to map potentially // all possible elements uint32_t max_v9_elements; uint32_t *common_extensions; } cache; typedef struct output_templates_s { struct output_templates_s *next; uint32_t flags; extension_map_t *extension_map; // extension map; time_t time_sent; uint32_t record_length; // length of the data record resulting from this template uint32_t flowset_length; // length of the flowset record template_flowset_t *template_flowset; } output_template_t; #define MAX_LIFETIME 60 static output_template_t *output_templates; static uint64_t boot_time; // in msec static uint16_t template_id; static uint32_t Max_num_v9_tags; static uint32_t processed_records; /* local function prototypes */ static int HasOptionTable(exporterDomain_t *exporter, uint16_t tableID ); static void InsertSamplerOption(exporterDomain_t *exporter, samplerOption_t *samplerOption); static void InsertSampler( FlowSource_t *fs, exporterDomain_t *exporter, int32_t id, uint16_t mode, uint32_t interval); static inline void Process_v9_templates(exporterDomain_t *exporter, void *template_flowset, FlowSource_t *fs); static inline void Process_v9_option_templates(exporterDomain_t *exporter, void *option_template_flowset, FlowSource_t *fs); static inline void Process_v9_data(exporterDomain_t *exporter, void *data_flowset, FlowSource_t *fs, input_translation_t *table ); static inline void Process_v9_option_data(exporterDomain_t *exporter, void *data_flowset, FlowSource_t *fs); static inline exporterDomain_t *GetExporter(FlowSource_t *fs, uint32_t exporter_id); static inline input_translation_t *GetTranslationTable(exporterDomain_t *exporter, uint16_t id); static input_translation_t *setup_translation_table (exporterDomain_t *exporter, uint16_t id, uint16_t input_record_size); static input_translation_t *add_translation_table(exporterDomain_t *exporter, uint16_t id); static output_template_t *GetOutputTemplate(uint32_t flags, extension_map_t *extension_map); static void Append_Record(send_peer_t *peer, master_record_t *master_record); static uint16_t Get_val16(void *p); static uint32_t Get_val32(void *p); static uint64_t Get_val64(void *p); // for sending netflow v9 static netflow_v9_header_t *v9_output_header; #include "inline.c" #include "nffile_inline.c" int Init_v9(int v, uint32_t sampling, uint32_t overwrite) { int i; verbose = v; default_sampling = sampling; overwrite_sampling = overwrite; output_templates = NULL; cache.lookup_info = (struct element_param_s *)calloc(65536, sizeof(struct element_param_s)); cache.common_extensions = (uint32_t *)malloc((Max_num_extensions+1)*sizeof(uint32_t)); if ( !cache.common_extensions || !cache.lookup_info ) { LogError( "Process_v9: Panic! malloc(): %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return 0; } // init the helper element table for (i=1; v9_element_map[i].id != 0; i++ ) { uint32_t Type = v9_element_map[i].id; // multiple same type - save first index only // iterate through same Types afterwards if ( cache.lookup_info[Type].index == 0 ) cache.lookup_info[Type].index = i; } cache.max_v9_elements = i; dbg_printf("Init v9: Max number of v9 tags: %u\n", cache.max_v9_elements); return 1; } // End of Init_v9 static int HasOptionTable(exporterDomain_t *exporter, uint16_t tableID ) { samplerOption_t *s; s = exporter->samplerOption; while ( s && s->tableID != tableID ) s = s->next; dbg_printf("Has option table: %s\n", s == NULL ? "not found" : "found"); return s != NULL; } // End of HasOptionTable static inline exporterDomain_t *GetExporter(FlowSource_t *fs, uint32_t exporter_id) { #define IP_STRING_LEN 40 char ipstr[IP_STRING_LEN]; exporterDomain_t **e = (exporterDomain_t **)&(fs->exporter_data); while ( *e ) { if ( (*e)->info.id == exporter_id && (*e)->info.version == 9 && (*e)->info.ip.V6[0] == fs->ip.V6[0] && (*e)->info.ip.V6[1] == fs->ip.V6[1]) return *e; e = &((*e)->next); } if ( fs->sa_family == AF_INET ) { uint32_t _ip = htonl(fs->ip.V4); inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr)); } else if ( fs->sa_family == AF_INET6 ) { uint64_t _ip[2]; _ip[0] = htonll(fs->ip.V6[0]); _ip[1] = htonll(fs->ip.V6[1]); inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr)); } else { strncpy(ipstr, "", IP_STRING_LEN); } // nothing found *e = (exporterDomain_t *)malloc(sizeof(exporterDomain_t)); if ( !(*e)) { LogError( "Process_v9: Panic! malloc() %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return NULL; } memset((void *)(*e), 0, sizeof(exporterDomain_t)); (*e)->info.header.type = ExporterInfoRecordType; (*e)->info.header.size = sizeof(exporter_info_record_t); (*e)->info.version = 9; (*e)->info.id = exporter_id; (*e)->info.ip = fs->ip; (*e)->info.sa_family = fs->sa_family; (*e)->info.sysid = 0; (*e)->first = 1; (*e)->sequence_failure = 0; (*e)->padding_errors = 0; (*e)->sampler = NULL; (*e)->next = NULL; FlushInfoExporter(fs, &((*e)->info)); dbg_printf("Process_v9: New exporter: SysID: %u, Domain: %u, IP: %s\n", (*e)->info.sysid, exporter_id, ipstr); LogInfo("Process_v9: New exporter: SysID: %u, Domain: %u, IP: %s\n", (*e)->info.sysid, exporter_id, ipstr); return (*e); } // End of GetExporter static inline uint32_t MapElement(uint16_t Type, uint16_t Length, uint16_t Offset) { int index; index = cache.lookup_info[Type].index; if ( index ) { while ( index && v9_element_map[index].id == Type ) { if ( Length == v9_element_map[index].length ) { cache.lookup_info[Type].found = 1; cache.lookup_info[Type].offset = Offset; cache.lookup_info[Type].length = Length; cache.lookup_info[Type].index = index; dbg_printf("found extension %u for type: %u(%s), at index: %i, input length: %u output length: %u Extension: %u, Offset: %u\n", v9_element_map[index].extension, v9_element_map[index].id, v9_element_map[index].name, index, v9_element_map[index].length, v9_element_map[index].out_length, v9_element_map[index].extension, Offset); return v9_element_map[index].extension; } index++; } #ifdef DEVEL index--; printf("=> known type: %u(%s), at index: %i, length: %u not supported\n", Type, v9_element_map[index].name, index, Length); #endif } dbg_printf("Skip unknown element type: %u, Length: %u\n", Type, Length); return 0; } // End of MapElement static inline input_translation_t *GetTranslationTable(exporterDomain_t *exporter, uint16_t id) { input_translation_t *table; if ( exporter->current_table && ( exporter->current_table->id == id ) ) return exporter->current_table; table = exporter->input_translation_table; while ( table ) { if ( table->id == id ) { exporter->current_table = table; return table; } table = table->next; } dbg_printf("[%u/%u] Get translation table %u: %s\n", exporter->info.id, exporter->info.sysid, id, table == NULL ? "not found" : "found"); exporter->current_table = table; return table; } // End of GetTranslationTable static input_translation_t *add_translation_table(exporterDomain_t *exporter, uint16_t id) { input_translation_t **table; table = &(exporter->input_translation_table); while ( *table ) { table = &((*table)->next); } // Allocate enough space for all potential v9 tags, which we support in v9_element_map // so template refreshing may change the table size without danger of overflowing *table = calloc(1, sizeof(input_translation_t)); if ( !(*table) ) { LogError( "Process_v9: Panic! calloc() %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return NULL; } (*table)->sequence = calloc(cache.max_v9_elements, sizeof(sequence_map_t)); if ( !(*table)->sequence ) { LogError( "Process_v9: Panic! malloc() %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return NULL; } (*table)->id = id; (*table)->next = NULL; dbg_printf("[%u] Get new translation table %u\n", exporter->info.id, id); return *table; } // End of add_translation_table static inline void PushSequence(input_translation_t *table, uint16_t Type, uint32_t *offset, void *stack, int pair_offset) { uint32_t i = table->number_of_sequences; uint32_t index = cache.lookup_info[Type].index; if ( table->number_of_sequences >= cache.max_v9_elements ) { LogError( "Process_v9: Software bug! Sequence table full. at %s line %d", __FILE__, __LINE__); dbg_printf("Software bug! Sequence table full. at %s line %d", __FILE__, __LINE__); return; } if ( cache.lookup_info[Type].found ) { table->sequence[i].id = v9_element_map[index].sequence; table->sequence[i].input_offset = cache.lookup_info[Type].offset; table->sequence[i].output_offset = *offset; table->sequence[i].stack = stack; dbg_printf("Fill "); } else { // in case only on half of an extension is sent from the collector, make // sure the right zero sequence is taken, where two length types exists index += pair_offset; table->sequence[i].id = v9_element_map[index].zero_sequence; table->sequence[i].input_offset = 0; table->sequence[i].output_offset = *offset; table->sequence[i].stack = NULL; dbg_printf("Zero "); } dbg_printf("Push: sequence: %u, Type: %u, length: %u, out length: %u, id: %u, in offset: %u, out offset: %u\n", i, Type, v9_element_map[index].length, v9_element_map[index].out_length, table->sequence[i].id, table->sequence[i].input_offset, table->sequence[i].output_offset); table->number_of_sequences++; (*offset) += v9_element_map[index].out_length; } // End of PushSequence static input_translation_t *setup_translation_table (exporterDomain_t *exporter, uint16_t id, uint16_t input_record_size) { input_translation_t *table; extension_map_t *extension_map; uint32_t i, ipv6, offset, next_extension; size_t size_required; ipv6 = 0; table = GetTranslationTable(exporter, id); if ( !table ) { LogInfo( "Process_v9: [%u] Add template %u", exporter->info.id, id); dbg_printf("[%u] Add template %u\n", exporter->info.id, id); table = add_translation_table(exporter, id); if ( !table ) { return NULL; } // Add an extension map // The number of extensions for this template is currently unknown // Allocate enough space for all configured extensions - some may be unused later // make sure memory is 4byte alligned size_required = Max_num_extensions * sizeof(uint16_t) + sizeof(extension_map_t); size_required = (size_required + 3) &~(size_t)3; extension_map = malloc(size_required); if ( !extension_map ) { LogError( "Process_v9: Panic! malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return NULL; } extension_map->type = ExtensionMapType; // Set size to an empty table - will be updated later extension_map->size = sizeof(extension_map_t); extension_map->map_id = INIT_ID; // packed record size still unknown at this point - will be added later extension_map->extension_size = 0; table->extension_info.map = extension_map; table->extension_map_changed = 1; #ifdef DEVEL if ( !GetTranslationTable(exporter, id) ) { printf("*** ERROR failed to crosscheck translation table\n"); } else { printf("table lookup ok!\n"); } #endif } else { extension_map = table->extension_info.map; // reset size/extension size - it's refreshed automatically extension_map->size = sizeof(extension_map_t); extension_map->extension_size = 0; dbg_printf("[%u] Refresh template %u\n", exporter->info.id, id); // very noisy for some exporters dbg_printf("[%u] Refresh template %u\n", exporter->info.id, id); } // clear current table memset((void *)table->sequence, 0, cache.max_v9_elements * sizeof(sequence_map_t)); table->number_of_sequences = 0; table->updated = time(NULL); table->flags = 0; table->flow_start = 0; table->flow_end = 0; table->EventTimeMsec = 0; table->ICMP_offset = 0; table->sampler_offset = 0; table->sampler_size = 0; table->engine_offset = 0; table->received_offset = 0; table->router_ip_offset = 0; dbg_printf("[%u] Fill translation table %u\n", exporter->info.id, id); // fill table table->id = id; /* * common data block: The common record is expected in the output stream. If not available * in the template, fill values with 0 */ // All required extensions offset = BYTE_OFFSET_first; if ( cache.lookup_info[NF_F_FLOW_CREATE_TIME_MSEC].found ) { uint32_t _tmp = 0; PushSequence( table, NF_F_FLOW_CREATE_TIME_MSEC, &_tmp, &table->flow_start, 0); dbg_printf("Push NF_F_FLOW_CREATE_TIME_MSEC\n"); } if ( cache.lookup_info[NF_F_FLOW_END_TIME_MSEC].found ) { uint32_t _tmp = 0; PushSequence( table, NF_F_FLOW_END_TIME_MSEC, &_tmp, &table->flow_end, 0); dbg_printf("Push NF_F_FLOW_END_TIME_MSEC\n"); } PushSequence( table, NF9_FIRST_SWITCHED, &offset, NULL, 0); offset = BYTE_OFFSET_first + 4; PushSequence( table, NF9_LAST_SWITCHED, &offset, NULL, 0); offset = BYTE_OFFSET_first + 8; PushSequence( table, NF9_FORWARDING_STATUS, &offset, NULL, 0); PushSequence( table, NF9_TCP_FLAGS, &offset, NULL, 0); PushSequence( table, NF9_IN_PROTOCOL, &offset, NULL, 0); PushSequence( table, NF9_SRC_TOS, &offset, NULL, 0); PushSequence( table, NF9_L4_SRC_PORT, &offset, NULL, 0); PushSequence( table, NF9_L4_DST_PORT, &offset, NULL, 0); // skip exporter_sysid and reserved offset += 4; /* IP address record * This record is expected in the output stream. If not available * in the template, assume empty v4 address. */ if ( cache.lookup_info[NF9_IPV4_SRC_ADDR].found ) { // IPv4 addresses PushSequence( table, NF9_IPV4_SRC_ADDR, &offset, NULL, 0); PushSequence( table, NF9_IPV4_DST_ADDR, &offset, NULL, 0); } else if ( cache.lookup_info[NF9_IPV6_SRC_ADDR].found ) { // IPv6 addresses PushSequence( table, NF9_IPV6_SRC_ADDR, &offset, NULL, 0); PushSequence( table, NF9_IPV6_DST_ADDR, &offset, NULL, 0); // mark IPv6 SetFlag(table->flags, FLAG_IPV6_ADDR); ipv6 = 1; } else { // should not happen, assume empty IPv4 addresses PushSequence( table, NF9_IPV4_SRC_ADDR, &offset, NULL, 0); PushSequence( table, NF9_IPV4_DST_ADDR, &offset, NULL, 0); } /* packet counter * This record is expected in the output stream. If not available * in the template, assume empty 4 bytes value */ if ( cache.lookup_info[NF_F_INITIATORPACKETS].found ) { PushSequence( table, NF_F_INITIATORPACKETS, &offset, &table->packets, 0); dbg_printf("Push NF_F_INITIATORPACKETS\n"); } else { PushSequence( table, NF9_IN_PACKETS, &offset, &table->packets, 0); } // fix: always have 64bit counters due to possible sampling SetFlag(table->flags, FLAG_PKG_64); if ( cache.lookup_info[NF_F_FLOW_BYTES].found ) { // NSEL ASA bytes PushSequence( table, NF_F_FLOW_BYTES, &offset, &table->bytes, 0); } else if ( cache.lookup_info[NF_F_FWD_FLOW_DELTA_BYTES].found ) { // NSEL ASA 8.4 bytes PushSequence( table, NF_F_FWD_FLOW_DELTA_BYTES, &offset, &table->bytes, 0); } else { PushSequence( table, NF9_IN_BYTES, &offset, &table->bytes, 0); } // fix: always have 64bit counters due to possible sampling SetFlag(table->flags, FLAG_BYTES_64); #if defined NSEL || defined NEL if ( cache.lookup_info[NF_F_FW_EVENT].found || cache.lookup_info[NF_F_FW_EVENT_84].found || cache.lookup_info[NF_N_NAT_EVENT].found) { SetFlag(table->flags, FLAG_EVENT); } #endif // Optional extensions next_extension = 0; for (i=4; i <= Max_num_extensions; i++ ) { uint32_t map_index = i; if ( cache.common_extensions[i] == 0 ) continue; switch(i) { case EX_IO_SNMP_2: PushSequence( table, NF9_INPUT_SNMP, &offset, NULL, 0); PushSequence( table, NF9_OUTPUT_SNMP, &offset, NULL, 0); break; case EX_IO_SNMP_4: PushSequence( table, NF9_INPUT_SNMP, &offset, NULL, 1); PushSequence( table, NF9_OUTPUT_SNMP, &offset, NULL, 1); break; case EX_AS_2: PushSequence( table, NF9_SRC_AS, &offset, NULL, 0); PushSequence( table, NF9_DST_AS, &offset, NULL, 0); break; case EX_AS_4: PushSequence( table, NF9_SRC_AS, &offset, NULL, 1); PushSequence( table, NF9_DST_AS, &offset, NULL, 1); break; case EX_MULIPLE: PushSequence( table, NF9_DST_TOS, &offset, NULL, 0); PushSequence( table, NF9_DIRECTION, &offset, NULL, 0); if ( ipv6 ) { // IPv6 PushSequence( table, NF9_IPV6_SRC_MASK, &offset, NULL, 0); PushSequence( table, NF9_IPV6_DST_MASK, &offset, NULL, 0); } else { // IPv4 PushSequence( table, NF9_SRC_MASK, &offset, NULL, 0); PushSequence( table, NF9_DST_MASK, &offset, NULL, 0); } break; case EX_NEXT_HOP_v4: PushSequence( table, NF9_V4_NEXT_HOP, &offset, NULL, 0); break; case EX_NEXT_HOP_v6: PushSequence( table, NF9_V6_NEXT_HOP, &offset, NULL, 0); SetFlag(table->flags, FLAG_IPV6_NH); break; case EX_NEXT_HOP_BGP_v4: PushSequence( table, NF9_BGP_V4_NEXT_HOP, &offset, NULL, 0); break; case EX_NEXT_HOP_BGP_v6: PushSequence( table, NF9_BPG_V6_NEXT_HOP, &offset, NULL, 0); SetFlag(table->flags, FLAG_IPV6_NHB); break; case EX_VLAN: if ( cache.lookup_info[NF9_dot1qVlanId].found ) { PushSequence( table, NF9_dot1qVlanId, &offset, NULL, 0); PushSequence( table, NF9_postDot1qVlanId, &offset, NULL, 0); } else { PushSequence( table, NF9_SRC_VLAN, &offset, NULL, 0); PushSequence( table, NF9_DST_VLAN, &offset, NULL, 0); } break; case EX_OUT_PKG_4: PushSequence( table, NF9_OUT_PKTS, &offset, &table->out_packets, 0); break; case EX_OUT_PKG_8: if ( cache.lookup_info[NF_F_RESPONDERPACKETS].found ) { PushSequence( table, NF_F_RESPONDERPACKETS, &offset, &table->out_packets, 0); dbg_printf("Push NF_F_RESPONDERPACKETS\n"); } else { PushSequence( table, NF9_OUT_PKTS, &offset, &table->out_packets, 0); } break; case EX_OUT_BYTES_4: if ( cache.lookup_info[NF_F_REV_FLOW_DELTA_BYTES].found ) { PushSequence( table, NF_F_REV_FLOW_DELTA_BYTES, &offset, &table->out_bytes, 0); } else { PushSequence( table, NF9_OUT_BYTES, &offset, &table->out_bytes, 0); } break; case EX_OUT_BYTES_8: if ( cache.lookup_info[NF_F_REV_FLOW_DELTA_BYTES].found ) { PushSequence( table, NF_F_REV_FLOW_DELTA_BYTES, &offset, &table->out_bytes, 0); } else { PushSequence( table, NF9_OUT_BYTES, &offset, &table->out_bytes, 0); } break; case EX_AGGR_FLOWS_4: PushSequence( table, NF9_FLOWS_AGGR, &offset, NULL, 0); break; case EX_AGGR_FLOWS_8: PushSequence( table, NF9_FLOWS_AGGR, &offset, NULL, 0); break; case EX_MAC_1: PushSequence( table, NF9_IN_SRC_MAC, &offset, NULL, 0); PushSequence( table, NF9_OUT_DST_MAC, &offset, NULL, 0); break; case EX_MAC_2: PushSequence( table, NF9_IN_DST_MAC, &offset, NULL, 0); PushSequence( table, NF9_OUT_SRC_MAC, &offset, NULL, 0); break; case EX_MPLS: PushSequence( table, NF9_MPLS_LABEL_1, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_2, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_3, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_4, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_5, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_6, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_7, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_8, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_9, &offset, NULL, 0); PushSequence( table, NF9_MPLS_LABEL_10, &offset, NULL, 0); break; case EX_ROUTER_IP_v4: case EX_ROUTER_IP_v6: if ( exporter->info.sa_family == PF_INET6 ) { table->router_ip_offset = offset; dbg_printf("Router IPv6: Offset: %u, olen: %u\n", offset, 16 ); // not an entry for the translateion table. // but reserve space in the output record for IPv6 offset += 16; SetFlag(table->flags, FLAG_IPV6_EXP); map_index = EX_ROUTER_IP_v6; } else { table->router_ip_offset = offset; dbg_printf("Router IPv4: Offset: %u, olen: %u\n", offset, 4 ); // not an entry for the translateion table. // but reserve space in the output record for IPv4 offset += 4; ClearFlag(table->flags, FLAG_IPV6_EXP); map_index = EX_ROUTER_IP_v4; } break; case EX_ROUTER_ID: table->engine_offset = offset; dbg_printf("Engine offset: %u\n", offset); offset += 2; dbg_printf("Skip 2 unused bytes. Next offset: %u\n", offset); PushSequence( table, NF9_ENGINE_TYPE, &offset, NULL, 0); PushSequence( table, NF9_ENGINE_ID, &offset, NULL, 0); // unused fill element for 32bit alignment break; case EX_RECEIVED: table->received_offset = offset; dbg_printf("Received offset: %u\n", offset); offset += 8; break; case EX_LATENCY: { // it's bit of a hack, but .. sigh .. uint32_t i = table->number_of_sequences; // Insert a zero64 as subsequent sequences add values table->sequence[i].id = zero64; table->sequence[i].input_offset = 0; table->sequence[i].output_offset = offset; table->sequence[i].stack = NULL; table->number_of_sequences++; dbg_printf("Zero latency at offset: %u\n", offset); PushSequence( table, NF9_NPROBE_CLIENT_NW_DELAY_SEC, &offset, NULL, 0); offset -= 8; PushSequence( table, NF9_NPROBE_CLIENT_NW_DELAY_USEC, &offset, NULL, 0); table->sequence[i].id = zero64; table->sequence[i].input_offset = 0; table->sequence[i].output_offset = offset; table->sequence[i].stack = NULL; table->number_of_sequences++; dbg_printf("Zero latency at offset: %u\n", offset); PushSequence( table, NF9_NPROBE_SERVER_NW_DELAY_SEC, &offset, NULL, 0); offset -= 8; PushSequence( table, NF9_NPROBE_SERVER_NW_DELAY_USEC, &offset, NULL, 0); table->sequence[i].id = zero64; table->sequence[i].input_offset = 0; table->sequence[i].output_offset = offset; table->sequence[i].stack = NULL; table->number_of_sequences++; dbg_printf("Zero latency at offset: %u\n", offset); PushSequence( table, NF9_NPROBE_APPL_LATENCY_SEC, &offset, NULL, 0); offset -= 8; PushSequence( table, NF9_NPROBE_APPL_LATENCY_USEC, &offset, NULL, 0); } break; case EX_BGPADJ: PushSequence( table, NF9_BGP_ADJ_NEXT_AS, &offset, NULL, 0); PushSequence( table, NF9_BGP_ADJ_PREV_AS, &offset, NULL, 0); break; case EX_NSEL_COMMON: PushSequence( table, NF_F_EVENT_TIME_MSEC, &offset, &table->EventTimeMsec, 0); PushSequence( table, NF_F_CONN_ID, &offset, NULL, 0); if ( ipv6 ) { #ifdef WORDS_BIGENDIAN PushSequence( table, NF_F_ICMP_TYPE_IPV6, &offset, NULL, 0); PushSequence( table, NF_F_ICMP_CODE_IPV6, &offset, NULL, 0); #else PushSequence( table, NF_F_ICMP_CODE_IPV6, &offset, NULL, 0); PushSequence( table, NF_F_ICMP_TYPE_IPV6, &offset, NULL, 0); #endif } else { #ifdef WORDS_BIGENDIAN PushSequence( table, NF_F_ICMP_TYPE, &offset, NULL, 0); PushSequence( table, NF_F_ICMP_CODE, &offset, NULL, 0); #else PushSequence( table, NF_F_ICMP_CODE, &offset, NULL, 0); PushSequence( table, NF_F_ICMP_TYPE, &offset, NULL, 0); #endif } cache.lookup_info[NF_F_FW_EVENT_84].found ? PushSequence( table, NF_F_FW_EVENT_84, &offset, NULL, 0) : PushSequence( table, NF_F_FW_EVENT, &offset, NULL, 0); offset += 1; PushSequence( table, NF_F_FW_EXT_EVENT, &offset, NULL, 0); PushSequence( table, NF_FW_CTS_SRC_SGT, &offset, NULL, 0); break; case EX_NSEL_XLATE_PORTS: if ( cache.lookup_info[NF_F_XLATE_SRC_ADDR_84].found ) { PushSequence( table, NF_F_XLATE_SRC_PORT_84, &offset, NULL, 0); PushSequence( table, NF_F_XLATE_DST_PORT_84, &offset, NULL, 0); } else { PushSequence( table, NF_F_XLATE_SRC_PORT, &offset, NULL, 0); PushSequence( table, NF_F_XLATE_DST_PORT, &offset, NULL, 0); } break; case EX_NSEL_XLATE_IP_v4: if ( cache.lookup_info[NF_F_XLATE_SRC_ADDR_84].found ) { PushSequence( table, NF_F_XLATE_SRC_ADDR_84, &offset, NULL, 0); PushSequence( table, NF_F_XLATE_DST_ADDR_84, &offset, NULL, 0); } else { PushSequence( table, NF_F_XLATE_SRC_ADDR_IPV4, &offset, NULL, 0); PushSequence( table, NF_F_XLATE_DST_ADDR_IPV4, &offset, NULL, 0); } break; case EX_NSEL_XLATE_IP_v6: PushSequence( table, NF_F_XLATE_SRC_ADDR_IPV6, &offset, NULL, 0); PushSequence( table, NF_F_XLATE_DST_ADDR_IPV6, &offset, NULL, 0); break; case EX_NSEL_ACL: PushSequence( table, NF_F_INGRESS_ACL_ID, &offset, NULL, 0); PushSequence( table, NF_F_EGRESS_ACL_ID, &offset, NULL, 0); break; case EX_NSEL_USER: case EX_NSEL_USER_MAX: PushSequence( table, NF_F_USERNAME, &offset, NULL, 0); break; case EX_NEL_COMMON: PushSequence( table, NF_N_NAT_EVENT, &offset, NULL, 0); offset += 3; PushSequence( table, NF_N_EGRESS_VRFID, &offset, NULL, 0); PushSequence( table, NF_N_INGRESS_VRFID, &offset, NULL, 0); break; case EX_PORT_BLOCK_ALLOC: PushSequence( table, NF_F_XLATE_PORT_BLOCK_START, &offset, NULL, 0); PushSequence( table, NF_F_XLATE_PORT_BLOCK_END, &offset, NULL, 0); PushSequence( table, NF_F_XLATE_PORT_BLOCK_STEP, &offset, NULL, 0); PushSequence( table, NF_F_XLATE_PORT_BLOCK_SIZE, &offset, NULL, 0); break; case EX_NEL_GLOBAL_IP_v4: // XXX no longer used break; } extension_map->size += sizeof(uint16_t); extension_map->extension_size += extension_descriptor[map_index].size; // found extension in map_index must be the same as in map - otherwise map is dirty if ( extension_map->ex_id[next_extension] != map_index ) { // dirty map - needs to be refreshed in output stream extension_map->ex_id[next_extension] = map_index; table->extension_map_changed = 1; } next_extension++; } extension_map->ex_id[next_extension++] = 0; // make sure map is aligned if ( extension_map->size & 0x3 ) { extension_map->ex_id[next_extension] = 0; extension_map->size = ( extension_map->size + 3 ) &~ 0x3; } table->output_record_size = offset; table->input_record_size = input_record_size; /* ICMP hack for v9 */ // for netflow historical reason, ICMP type/code goes into dst port field // remember offset, for decoding if ( cache.lookup_info[NF9_ICMP_TYPE].found && cache.lookup_info[NF9_ICMP_TYPE].length == 2 ) { table->ICMP_offset = cache.lookup_info[NF9_ICMP_TYPE].offset; } /* Sampler ID */ if ( cache.lookup_info[NF9_FLOW_SAMPLER_ID].found ) { uint32_t length = cache.lookup_info[NF9_FLOW_SAMPLER_ID].length; switch (length) { case 1: case 2: case 4: table->sampler_offset = cache.lookup_info[NF9_FLOW_SAMPLER_ID].offset; table->sampler_size = length; dbg_printf("%d byte Sampling ID included at offset %u\n", length, table->sampler_offset); break; default: LogError( "Process_v9: Unexpected SAMPLER ID field length: %d", cache.lookup_info[NF9_FLOW_SAMPLER_ID].length); dbg_printf("Unexpected SAMPLER ID field length: %d", cache.lookup_info[NF9_FLOW_SAMPLER_ID].length); } } else { dbg_printf("No Sampling ID found\n"); } #ifdef DEVEL if ( table->extension_map_changed ) { printf("Extension Map id=%u changed!\n", extension_map->map_id); } else { printf("[%u] template %u unchanged\n", exporter->info.id, id); } printf("Process_v9: Check extension map: id: %d, size: %u, extension_size: %u\n", extension_map->map_id, extension_map->size, extension_map->extension_size); { int i; for (i=0; inumber_of_sequences; i++ ) { printf("Sequence %i: id: %u, in offset: %u, out offset: %u, stack: %llu\n", i, table->sequence[i].id, table->sequence[i].input_offset, table->sequence[i].output_offset, (unsigned long long)table->sequence[i].stack); } printf("Flags: 0x%x\n", table->flags); printf("Input record size: %u, output record size: %u\n", table->input_record_size, table->output_record_size); } PrintExtensionMap(extension_map); #endif return table; } // End of setup_translation_table static void InsertSamplerOption(exporterDomain_t *exporter, samplerOption_t *samplerOption) { samplerOption_t *s, *parent; parent = NULL; s = exporter->samplerOption; while (s) { if ( s->tableID == samplerOption->tableID ) { // table already known to us - update data dbg_printf("Found existing sampling info in template %i\n", samplerOption->tableID); break; } parent = s; s = s->next; } if ( s != NULL ) { // existing entry // replace existing table dbg_printf("Replace existing sampler table ID %i\n", samplerOption->tableID); if ( parent ) { parent->next = samplerOption; } else { exporter->samplerOption = samplerOption; } samplerOption->next = s->next; free(s); s = NULL; } else { // new entry dbg_printf("New sampling table ID %i\n", samplerOption->tableID); // push new sampling table samplerOption->next = exporter->samplerOption; exporter->samplerOption = samplerOption; } dbg_printf("Update/Insert sampler table id: %u flags: 0x%x - sampler ID: %u/%u, mode: %u/%u, interval: %u/%u\n", samplerOption->tableID, samplerOption->flags, samplerOption->id.offset, samplerOption->id.length, samplerOption->mode.offset, samplerOption->mode.length, samplerOption->interval.offset, samplerOption->interval.length); } // End of InsertSamplerOption static inline void Process_v9_templates(exporterDomain_t *exporter, void *template_flowset, FlowSource_t *fs) { void *template; input_translation_t *translation_table; uint16_t id, count, Offset; uint32_t size_left, size_required, num_extensions, num_v9tags; int i; size_left = GET_FLOWSET_LENGTH(template_flowset) - 4; // -4 for flowset header -> id and length template = template_flowset + 4; // the template description begins at offset 4 // process all templates in flowset, as long as any bytes are left size_required = 0; Offset = 0; while (size_left) { void *p; template = template + size_required; // clear helper tables memset((void *)cache.common_extensions, 0, (Max_num_extensions+1)*sizeof(uint32_t)); memset((void *)cache.lookup_info, 0, 65536 * sizeof(struct element_param_s)); for (i=1; v9_element_map[i].id != 0; i++ ) { uint32_t Type = v9_element_map[i].id; if ( v9_element_map[i].id == v9_element_map[i-1].id ) continue; cache.lookup_info[Type].index = i; // other elements cleared by memset } id = GET_TEMPLATE_ID(template); count = GET_TEMPLATE_COUNT(template); size_required = 4 + 4 * count; // id + count = 4 bytes, and 2 x 2 bytes for each entry dbg_printf("\n[%u] Template ID: %u\n", exporter->info.id, id); dbg_printf("template size: %u buffersize: %u\n", size_required, size_left); if ( size_left < size_required ) { LogError( "Process_v9: [%u] buffer size error: expected %u available %u", exporter->info.id, size_required, size_left); size_left = 0; continue; } Offset = 0; num_extensions = 0; // number of extensions num_v9tags = 0; // number of optional v9 tags p = template + 4; // type/length pairs start at template offset 4 for(i=0; i Extension: %u\n", fs->sa_family == PF_INET6 ? "ipv6" : "ipv4", EX_ROUTER_IP_v4); } // as the router IP address extension is not part announced in a template, we need to deal with it here if ( extension_descriptor[EX_ROUTER_ID].enabled ) { if ( cache.common_extensions[EX_ROUTER_ID] == 0 ) { cache.common_extensions[EX_ROUTER_ID] = 1; num_extensions++; } dbg_printf("Force add router ID (engine type/ID), Extension: %u\n", EX_ROUTER_ID); } // as the received time is not announced in a template, we need to deal with it here if ( extension_descriptor[EX_RECEIVED].enabled ) { if ( cache.common_extensions[EX_RECEIVED] == 0 ) { cache.common_extensions[EX_RECEIVED] = 1; num_extensions++; } dbg_printf("Force add packet received time, Extension: %u\n", EX_RECEIVED); } dbg_printf("Parsed %u v9 tags, total %u extensions\n", num_v9tags, num_extensions); #ifdef DEVEL { int i; for (i=0; i<=Max_num_extensions; i++ ) { if ( cache.common_extensions[i] ) { printf("Enabled extension: %2i: %s\n", i, extension_descriptor[i].description); } } } #endif translation_table = setup_translation_table(exporter, id, Offset); if (translation_table->extension_map_changed ) { translation_table->extension_map_changed = 0; // refresh he map in the ouput buffer dbg_printf("Translation Table changed! Add extension map ID: %i\n", translation_table->extension_info.map->map_id); AddExtensionMap(fs, translation_table->extension_info.map); dbg_printf("Translation Table added! map ID: %i\n", translation_table->extension_info.map->map_id); } size_left -= size_required; processed_records++; dbg_printf("\n"); } // End of while size_left } // End of Process_v9_templates static inline void Process_v9_option_templates(exporterDomain_t *exporter, void *option_template_flowset, FlowSource_t *fs) { uint8_t *option_template, *p; uint32_t size_left, nr_scopes, nr_options; uint16_t tableID, scope_length, option_length; samplerOption_t *samplerOption; size_left = GET_FLOWSET_LENGTH(option_template_flowset) - 4; // -4 for flowset header -> id and length option_template = option_template_flowset + 4; tableID = GET_OPTION_TEMPLATE_ID(option_template); scope_length = GET_OPTION_TEMPLATE_OPTION_SCOPE_LENGTH(option_template); option_length = GET_OPTION_TEMPLATE_OPTION_LENGTH(option_template); if ( scope_length & 0x3 ) { LogError( "Process_v9: [%u] scope length error: length %u not multiple of 4", exporter->info.id, scope_length); return; } if ( option_length & 0x3 ) { LogError( "Process_v9: [%u] option length error: length %u not multiple of 4", exporter->info.id, option_length); return; } if ( (scope_length + option_length) > size_left ) { LogError( "Process_v9: [%u] option template length error: size left %u too small for %u scopes length and %u options length", exporter->info.id, size_left, scope_length, option_length); return; } nr_scopes = scope_length >> 2; nr_options = option_length >> 2; dbg_printf("\n[%u] Option Template ID: %u\n", exporter->info.id, tableID); dbg_printf("Scope length: %u Option length: %u\n", scope_length, option_length); samplerOption = (samplerOption_t *)malloc(sizeof(samplerOption_t)); if ( !samplerOption ) { LogError("Error malloc(): %s in %s:%d", strerror (errno), __FILE__, __LINE__); return; } memset((void *)samplerOption, 0, sizeof(samplerOption_t)); samplerOption->tableID = tableID; int i; uint16_t offset = 0; p = option_template + 6; // start of length/type data for ( i=0; iinterval.length = length; samplerOption->interval.offset = offset; SetFlag(samplerOption->flags, STDSAMPLING34); break; case NF9_SAMPLING_ALGORITHM: // #35 samplerOption->mode.length = length; samplerOption->mode.offset = offset; SetFlag(samplerOption->flags, STDSAMPLING35); break; // individual samplers case NF9_FLOW_SAMPLER_ID: // #48 depricated - fall through case NF_SELECTOR_ID: // #302 samplerOption->id.length = length; samplerOption->id.offset = offset; SetFlag(samplerOption->flags, SAMPLER302); break; case FLOW_SAMPLER_MODE: // #49 depricated - fall through case NF_SELECTOR_ALGORITHM: // #304 samplerOption->mode.length = length; samplerOption->mode.offset = offset; SetFlag(samplerOption->flags, SAMPLER304); break; case NF9_FLOW_SAMPLER_RANDOM_INTERVAL: // #50 depricated - fall through case NF_SAMPLING_INTERVAL: // #305 samplerOption->interval.length = length; samplerOption->interval.offset = offset; SetFlag(samplerOption->flags, SAMPLER305); break; case 95: dbg_printf("Option template: Application ID found\n"); break; case 96: dbg_printf("Option template: Application Name found\n"); break; case 94: dbg_printf("Option template: Application Desc found\n"); break; } offset += length; } if ( (samplerOption->flags & SAMPLERMASK ) != 0) { dbg_printf("[%u] Sampler information found\n", exporter->info.id); InsertSamplerOption(exporter, samplerOption); } else if ( (samplerOption->flags & STDMASK ) != 0) { dbg_printf("[%u] Std sampling information found\n", exporter->info.id); InsertSamplerOption(exporter, samplerOption); } else { free(samplerOption); dbg_printf("[%u] No Sampling information found\n", exporter->info.id); } processed_records++; dbg_printf("\n"); } // End of Process_v9_option_templates static inline void Process_v9_data(exporterDomain_t *exporter, void *data_flowset, FlowSource_t *fs, input_translation_t *table ){ uint64_t start_time, end_time, sampling_rate; uint32_t size_left; uint8_t *in, *out; int i; char *string; size_left = GET_FLOWSET_LENGTH(data_flowset) - 4; // -4 for data flowset header -> id and length // map input buffer as a byte array in = (uint8_t *)(data_flowset + 4); // skip flowset header dbg_printf("[%u] Process data flowset size: %u\n", exporter->info.id, size_left); if ( table->sampler_offset ) dbg_printf("table sampler offset: %u\n", table->sampler_offset); dbg_printf("[%u] Exporter is 0x%llu\n", exporter->info.id, (long long unsigned)exporter); dbg_printf("[%u] Exporter has sampler: %s\n", exporter->info.id, exporter->sampler ? "yes" : "no"); // Check if sampling is announced if ( table->sampler_offset && exporter->sampler ) { sampler_t *sampler = exporter->sampler; uint32_t sampler_id; if ( table->sampler_size == 2 ) { sampler_id = Get_val16((void *)&in[table->sampler_offset]); } else { sampler_id = in[table->sampler_offset]; } dbg_printf("Extract sampler: %u\n", sampler_id); // usually not that many samplers, so following a chain is not too expensive. while ( sampler && sampler->info.id != sampler_id ) sampler = sampler->next; if ( sampler ) { sampling_rate = sampler->info.interval; dbg_printf("[%u] Sampling ID %u available\n", exporter->info.id, sampler_id); dbg_printf("[%u] Sampler_offset : %u\n", exporter->info.id, table->sampler_offset); dbg_printf("[%u] Sampler Data : %s\n", exporter->info.id, exporter->sampler == NULL ? "not available" : "available"); dbg_printf("[%u] Sampling rate: %llu\n", exporter->info.id, (long long unsigned)sampling_rate); } else { sampling_rate = default_sampling; dbg_printf("[%u] Sampling ID %u not (yet) available\n", exporter->info.id, sampler_id); } } else { sampler_t *sampler = exporter->sampler; while ( sampler && sampler->info.id != -1 ) sampler = sampler->next; if ( sampler ) { sampling_rate = sampler->info.interval; dbg_printf("[%u] Std sampling available for this flow source: Rate: %llu\n", exporter->info.id, (long long unsigned)sampling_rate); } else { sampling_rate = default_sampling; dbg_printf("[%u] No Sampling record found\n", exporter->info.id); } } if ( overwrite_sampling > 0 ) { sampling_rate = overwrite_sampling; dbg_printf("[%u] Hard overwrite sampling rate: %llu\n", exporter->info.id, (long long unsigned)sampling_rate); } if ( sampling_rate != 1 ) SetFlag(table->flags, FLAG_SAMPLED); while (size_left) { common_record_t *data_record; if ( table->input_record_size == 0 ) { dbg_printf("Process_v9: Corrupt data flowset? table input_record_sizei = 0\n"); LogError("Process_v9: Corrupt data flowset? table input_record_sizei = 0 "); size_left = 0; continue; } if ( size_left < table->input_record_size ) { if ( size_left > 3 ) { exporter->padding_errors++; dbg_printf("Process_v9: Corrupt data flowset? Pad bytes: %u, table record_size: %u\n", size_left, table->input_record_size); } size_left = 0; continue; } // check for enough space in output buffer if ( !CheckBufferSpace(fs->nffile, table->output_record_size) ) { // this should really never occur, because the buffer gets flushed ealier LogError("Process_v9: output buffer size error. Abort v9 record processing"); dbg_printf("Process_v9: output buffer size error. Abort v9 record processing"); return; } processed_records++; // map file record to output buffer data_record = (common_record_t *)fs->nffile->buff_ptr; // map output buffer as a byte array out = (uint8_t *)data_record; dbg_printf("[%u] Process data record: %u addr: %llu, in record size: %u, buffer size_left: %u\n", exporter->info.id, processed_records, (long long unsigned)((ptrdiff_t)in - (ptrdiff_t)data_flowset), table->input_record_size, size_left); // fill the data record data_record->flags = table->flags; data_record->size = table->output_record_size; data_record->type = CommonRecordType; data_record->ext_map = table->extension_info.map->map_id; data_record->exporter_sysid = exporter->info.sysid; data_record->nfversion = 9; data_record->biFlowDir = 0; data_record->flowEndReason = 0; table->packets = 0; table->bytes = 0; table->out_packets = 0; table->out_bytes = 0; dbg_printf("[%u] Process data record: MapID: %u\n", exporter->info.id, table->extension_info.map->map_id); // apply copy and processing sequence for ( i=0; inumber_of_sequences; i++ ) { int input_offset = table->sequence[i].input_offset; int output_offset = table->sequence[i].output_offset; void *stack = table->sequence[i].stack; switch (table->sequence[i].id) { case nop: break; case move8: out[output_offset] = in[input_offset]; break; case move16: *((uint16_t *)&out[output_offset]) = Get_val16((void *)&in[input_offset]); break; case move32: *((uint32_t *)&out[output_offset]) = Get_val32((void *)&in[input_offset]); break; case move40: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val40((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move48: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val48((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move56: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val56((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move64: { type_mask_t t; t.val.val64 = Get_val64((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move64_32: { type_mask_t t; t.val.val64 = Get_val64((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[1]; } break; case move96: { *((uint32_t *)&out[output_offset]) = Get_val32((void *)&in[input_offset]); *((uint32_t *)&out[output_offset+4]) = Get_val32((void *)&in[input_offset+4]); *((uint32_t *)&out[output_offset+8]) = Get_val32((void *)&in[input_offset+8]); } break; case move128: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val64((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; t.val.val64 = Get_val64((void *)&in[input_offset+8]); *((uint32_t *)&out[output_offset+8]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+12]) = t.val.val32[1]; } break; case move32_sampling: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val32((void *)&in[input_offset]); t.val.val64 *= sampling_rate; *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; *(uint64_t *)stack = t.val.val64; } break; case move64_sampling: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val64((void *)&in[input_offset]); t.val.val64 *= sampling_rate; *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; *(uint64_t *)stack = t.val.val64; } break; case move_mac: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val64 = Get_val48((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move_mpls: *((uint32_t *)&out[output_offset]) = Get_val24((void *)&in[input_offset]); break; case move_ulatency: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val32[0] = *((uint32_t *)&out[output_offset]); t.val.val32[1] = *((uint32_t *)&out[output_offset+4]); t.val.val64 += Get_val32((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move_slatency: /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ { type_mask_t t; t.val.val32[0] = *((uint32_t *)&out[output_offset]); t.val.val32[1] = *((uint32_t *)&out[output_offset+4]); // update sec to usec t.val.val64 += 1000000 * Get_val32((void *)&in[input_offset]); *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; case move_user_20: memcpy((void *)&out[output_offset],(void *)&in[input_offset],20); out[output_offset+20] = 0; // trailing 0 for string break; case move_user_65: memcpy((void *)&out[output_offset],(void *)&in[input_offset],65); out[output_offset+65] = 0; // trailing 0 for string break; case TimeMsec: { uint64_t DateMiliseconds = Get_val64((void *)&in[input_offset]); *(uint64_t *)stack = DateMiliseconds; } break; case PushTimeMsec: { type_mask_t t; t.val.val64 = Get_val64((void *)&in[input_offset]); *(uint64_t *)stack = t.val.val64; *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; } break; // zero sequences for unavailable elements case zero8: out[output_offset] = 0; break; case zero16: *((uint16_t *)&out[output_offset]) = 0; break; case zero32: *((uint32_t *)&out[output_offset]) = 0; break; case zero64: { *((uint32_t *)&out[output_offset]) = 0; *((uint32_t *)&out[output_offset+4]) = 0; } break; case zero96: { *((uint32_t *)&out[output_offset]) = 0; *((uint32_t *)&out[output_offset+4]) = 0; *((uint32_t *)&out[output_offset+8]) = 0; } break; case zero128: { *((uint32_t *)&out[output_offset]) = 0; *((uint32_t *)&out[output_offset+4]) = 0; *((uint32_t *)&out[output_offset+8]) = 0; *((uint32_t *)&out[output_offset+12]) = 0; } break; default: LogError( "Process_v9: Software bug! Unknown Sequence: %u. at %s line %d", table->sequence[i].id, __FILE__, __LINE__); dbg_printf("Software bug! Unknown Sequence: %u. at %s line %d", table->sequence[i].id, __FILE__, __LINE__); } } // Ungly ICMP hack for v9, because some IOS version are lazzy // most of them send ICMP in dst port field some don't some have both if ( data_record->prot == IPPROTO_ICMP || data_record->prot == IPPROTO_ICMPV6 ) { if ( table->ICMP_offset ) { data_record->dstport = Get_val16((void *)&in[table->ICMP_offset]); } if ( data_record->dstport == 0 && data_record->srcport != 0 ) { // some IOSes are even lazzier and map ICMP code in src port - ughh data_record->dstport = data_record->srcport; data_record->srcport = 0; } } // Check for NSEL/NEL Event time if ( table->flow_start ) { data_record->first = table->flow_start / 1000; data_record->msec_first = table->flow_start % 1000; start_time = table->flow_start; // test for tags 152/153 if ( table->flow_end ) { data_record->last = table->flow_end / 1000; data_record->msec_last = table->flow_end % 1000; end_time = table->flow_end; } else { data_record->last = data_record->first; data_record->msec_last = data_record->msec_first; end_time = table->flow_start; } dbg_printf("Found time flow start MSEC: %llu\n", table->EventTimeMsec); } else if ( table->EventTimeMsec && data_record->first == 0 ) { data_record->first = table->EventTimeMsec / 1000; data_record->msec_first = table->EventTimeMsec % 1000; data_record->last = data_record->first; data_record->msec_last = data_record->msec_first; start_time = table->EventTimeMsec; end_time = table->EventTimeMsec; dbg_printf("Found Time Event MSEC: %llu\n", table->EventTimeMsec); } else if ( data_record->first == 0 && data_record->last == 0 ) { // hmm - a record with no time at all .. data_record->first = 0; data_record->msec_last = 0; start_time = 0; end_time = 0; } else { uint32_t First = data_record->first; uint32_t Last = data_record->last; if ( First > Last ) /* First in msec, in case of msec overflow, between start and end */ start_time = exporter->boot_time - 0x100000000LL + (uint64_t)First; else start_time = (uint64_t)First + exporter->boot_time; /* end time in msecs */ end_time = (uint64_t)Last + exporter->boot_time; if ( (end_time - start_time) > 0xffc00000 && table->bytes < 2000 ) { dbg_printf("CISCO bugfix!\n"); start_time += 0xffc00000; } data_record->first = start_time/1000; data_record->msec_first = start_time - data_record->first*1000; data_record->last = end_time/1000; data_record->msec_last = end_time - data_record->last*1000; if ( data_record->first == 0 && data_record->last == 0 ) data_record->last = 0; } // update first_seen, last_seen if ( start_time < fs->first_seen ) fs->first_seen = start_time; if ( end_time > fs->last_seen ) fs->last_seen = end_time; // check if we need to record the router IP address if ( table->router_ip_offset ) { int output_offset = table->router_ip_offset; if ( exporter->info.sa_family == PF_INET6 ) { /* 64bit access to potentially unaligned output buffer. use 2 x 32bit for _LP64 CPUs */ type_mask_t t; t.val.val64 = exporter->info.ip.V6[0]; *((uint32_t *)&out[output_offset]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+4]) = t.val.val32[1]; t.val.val64 = exporter->info.ip.V6[1]; *((uint32_t *)&out[output_offset+8]) = t.val.val32[0]; *((uint32_t *)&out[output_offset+12]) = t.val.val32[1]; } else { *((uint32_t *)&out[output_offset]) = exporter->info.ip.V4; } } // Ugly hack. CISCO never really implemented #38/#39 tags in the records - so take it from the // header, unless some data is filled in if ( table->engine_offset ) { if ( *((uint32_t *)&out[table->engine_offset]) == 0 ) { tpl_ext_25_t *tpl = (tpl_ext_25_t *)&out[table->engine_offset]; tpl->engine_type = ( exporter->info.id >> 8 ) & 0xFF; tpl->engine_id = exporter->info.id & 0xFF; } } // check, if we need to store the packet received time if ( table->received_offset ) { type_mask_t t; t.val.val64 = (uint64_t)((uint64_t)fs->received.tv_sec * 1000LL) + (uint64_t)((uint64_t)fs->received.tv_usec / 1000LL); *((uint32_t *)&out[table->received_offset]) = t.val.val32[0]; *((uint32_t *)&out[table->received_offset+4]) = t.val.val32[1]; } switch (data_record->prot ) { // switch protocol of case IPPROTO_ICMP: fs->nffile->stat_record->numflows_icmp++; fs->nffile->stat_record->numpackets_icmp += table->packets; fs->nffile->stat_record->numbytes_icmp += table->bytes; fs->nffile->stat_record->numpackets_icmp += table->out_packets; fs->nffile->stat_record->numbytes_icmp += table->out_bytes; break; case IPPROTO_TCP: fs->nffile->stat_record->numflows_tcp++; fs->nffile->stat_record->numpackets_tcp += table->packets; fs->nffile->stat_record->numbytes_tcp += table->bytes; fs->nffile->stat_record->numpackets_tcp += table->out_packets; fs->nffile->stat_record->numbytes_tcp += table->out_bytes; break; case IPPROTO_UDP: fs->nffile->stat_record->numflows_udp++; fs->nffile->stat_record->numpackets_udp += table->packets; fs->nffile->stat_record->numbytes_udp += table->bytes; fs->nffile->stat_record->numpackets_udp += table->out_packets; fs->nffile->stat_record->numbytes_udp += table->out_bytes; break; default: fs->nffile->stat_record->numflows_other++; fs->nffile->stat_record->numpackets_other += table->packets; fs->nffile->stat_record->numbytes_other += table->bytes; fs->nffile->stat_record->numpackets_other += table->out_packets; fs->nffile->stat_record->numbytes_other += table->out_bytes; } exporter->flows++; fs->nffile->stat_record->numflows++; fs->nffile->stat_record->numpackets += table->packets; fs->nffile->stat_record->numbytes += table->bytes; fs->nffile->stat_record->numpackets += table->out_packets; fs->nffile->stat_record->numbytes += table->out_bytes; if ( verbose ) { master_record_t master_record; memset((void *)&master_record, 0, sizeof(master_record_t)); ExpandRecord_v2((common_record_t *)data_record, &(table->extension_info), &(exporter->info), &master_record); flow_record_to_raw(&master_record, &string, 0); printf("%s\n", string); } fs->nffile->block_header->size += data_record->size; fs->nffile->block_header->NumRecords++; fs->nffile->buff_ptr = (common_record_t *)((pointer_addr_t)data_record + data_record->size); // advance input size_left -= table->input_record_size; in += table->input_record_size; // buffer size sanity check if ( fs->nffile->block_header->size > BUFFSIZE ) { // should never happen LogError("### Software error ###: %s line %d", __FILE__, __LINE__); LogError("Process v9: Output buffer overflow! Flush buffer and skip records."); LogError("Buffer size: %u > %u", fs->nffile->block_header->size, BUFFSIZE); // reset buffer fs->nffile->block_header->size = 0; fs->nffile->block_header->NumRecords = 0; fs->nffile->buff_ptr = (void *)((pointer_addr_t)fs->nffile->block_header + sizeof(data_block_header_t) ); return; } } } // End of Process_v9_data static inline void Process_v9_option_data(exporterDomain_t *exporter, void *data_flowset, FlowSource_t *fs) { samplerOption_t *samplerOption; uint32_t tableID = GET_FLOWSET_ID(data_flowset); uint32_t size_left = GET_FLOWSET_LENGTH(data_flowset) - 4; // -4 for data flowset header -> id and length dbg_printf("[%u] Process option data flowset size: %u\n", exporter->info.id, size_left); samplerOption = exporter->samplerOption; while ( samplerOption && samplerOption->tableID != tableID ) samplerOption = samplerOption->next; if ( !samplerOption ) { // should never happen - catch it anyway LogError( "Process_v9: Panic! - No Offset table found! : %s line %d", __FILE__, __LINE__); return; } // map input buffer as a byte array uint8_t *in = (uint8_t *)(data_flowset + 4); // skip flowset header if ( (samplerOption->flags & SAMPLERMASK ) == SAMPLERFLAGS) { int32_t id; uint16_t mode; uint32_t interval; if ( CHECK_OPTION_DATA(size_left, samplerOption->id) && CHECK_OPTION_DATA(size_left, samplerOption->mode) && CHECK_OPTION_DATA(size_left, samplerOption->interval)) { id = Get_val(in, samplerOption->id.offset, samplerOption->id.length); mode = Get_val(in, samplerOption->mode.offset, samplerOption->mode.length); interval = Get_val(in, samplerOption->interval.offset, samplerOption->interval.length); } else { LogError("Process_v9_option: %s line %d: Not enough data for option data", __FILE__, __LINE__); return; } dbg_printf("Extracted Sampler data:\n"); dbg_printf("Sampler ID : %u\n", id); dbg_printf("Sampler mode : %u\n", mode); dbg_printf("Sampler interval: %u\n", interval); InsertSampler(fs, exporter, id, mode, interval); } if ( (samplerOption->flags & STDMASK ) == STDFLAGS) { int32_t id; uint16_t mode; uint32_t interval; id = -1; if ( CHECK_OPTION_DATA(size_left, samplerOption->mode) && CHECK_OPTION_DATA(size_left, samplerOption->interval)) { mode = Get_val(in, samplerOption->mode.offset, samplerOption->mode.length); interval = Get_val(in, samplerOption->interval.offset, samplerOption->interval.length); } else { LogError("Process_v9_option: %s line %d: Not enough data for option data", __FILE__, __LINE__); return; } InsertSampler(fs, exporter, id, mode, interval); dbg_printf("Extracted Std Sampler data:\n"); dbg_printf("Sampler ID : %i\n", id); dbg_printf("Sampler algorithm: %u\n", mode); dbg_printf("Sampler interval : %u\n", interval); dbg_printf("Set std sampler: algorithm: %u, interval: %u\n", mode, interval); } processed_records++; } // End of Process_v9_option_data void Process_v9(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs) { exporterDomain_t *exporter; void *flowset_header; netflow_v9_header_t *v9_header; int64_t distance; uint32_t flowset_id, flowset_length, exporter_id; ssize_t size_left; #ifdef DEVEL static int pkg_num = 0; pkg_num++; printf("Process_v9: Next packet: %i\n", pkg_num); #endif size_left = in_buff_cnt; if ( size_left < NETFLOW_V9_HEADER_LENGTH ) { LogError( "Process_v9: Too little data for v9 packet: '%lli'", (long long)size_left); return; } // map v9 data structure to input buffer v9_header = (netflow_v9_header_t *)in_buff; exporter_id = ntohl(v9_header->source_id); exporter = GetExporter(fs, exporter_id); if ( !exporter ) { LogError("Process_v9: Exporter NULL: Abort v9 record processing"); return; } exporter->packets++; /* calculate boot time in msec */ v9_header->SysUptime = ntohl(v9_header->SysUptime); v9_header->unix_secs = ntohl(v9_header->unix_secs); exporter->boot_time = (uint64_t)1000 * (uint64_t)(v9_header->unix_secs) - (uint64_t)v9_header->SysUptime; flowset_header = (void *)v9_header + NETFLOW_V9_HEADER_LENGTH; size_left -= NETFLOW_V9_HEADER_LENGTH; #ifdef DEVEL uint32_t expected_records = ntohs(v9_header->count); uint32_t skip = 0; printf("\n[%u] Process next packet: %i records: %u, buffer: %li \n", exporter_id, pkg_num, expected_records, size_left); printf("SourceID: %u, Sysuptime: %u.%u\n", v9_header->source_id, v9_header->SysUptime, v9_header->unix_secs); #endif // sequence check if ( exporter->first ) { exporter->last_sequence = ntohl(v9_header->sequence); exporter->sequence = exporter->last_sequence; exporter->first = 0; } else { exporter->last_sequence = exporter->sequence; exporter->sequence = ntohl(v9_header->sequence); distance = exporter->sequence - exporter->last_sequence; // handle overflow if (distance < 0) { distance = 0xffffffff + distance +1; } if (distance != 1) { exporter->sequence_failure++; fs->nffile->stat_record->sequence_failure++; dbg_printf("[%u] Sequence error: last seq: %lli, seq %lli dist %lli\n", exporter->info.id, (long long)exporter->last_sequence, (long long)exporter->sequence, (long long)distance); /* if ( report_seq ) LogError("Flow sequence mismatch. Missing: %lli packets", delta(last_count,distance)); */ } } dbg_printf("Sequence: %llu\n", exporter->sequence); processed_records = 0; // iterate over all flowsets in export packet, while there are bytes left flowset_length = 0; while (size_left) { flowset_header = flowset_header + flowset_length; flowset_id = GET_FLOWSET_ID(flowset_header); flowset_length = GET_FLOWSET_LENGTH(flowset_header); dbg_printf("[%u] Next flowset: %u, length: %u buffersize: %li addr: %llu\n", exporter->info.id, flowset_id, flowset_length, size_left, (long long unsigned)(flowset_header - in_buff) ); if ( flowset_length == 0 ) { /* this should never happen, as 4 is an empty flowset and smaller is an illegal flowset anyway ... if it happends, we can't determine the next flowset, so skip the entire export packet */ LogError("Process_v9: flowset zero length error."); dbg_printf("Process_v9: flowset zero length error.\n"); return; } // possible padding if ( flowset_length <= 4 ) { return; } if ( flowset_length > size_left ) { dbg_printf("flowset length error. Expected bytes: %u > buffersize: %lli", flowset_length, (long long)size_left); LogError("Process_v9: flowset length error. Expected bytes: %u > buffersize: %lli", flowset_length, (long long)size_left); size_left = 0; continue; } #ifdef DEVEL if ( (ptrdiff_t)fs->nffile->buff_ptr & 0x3 ) { fprintf(stderr, "PANIC: alignment error!! \n"); exit(255); } #endif switch (flowset_id) { case NF9_TEMPLATE_FLOWSET_ID: Process_v9_templates(exporter, flowset_header, fs); break; case NF9_OPTIONS_FLOWSET_ID: { #ifdef DEVEL option_template_flowset_t *option_flowset; option_flowset = (option_template_flowset_t *)flowset_header; dbg_printf("Process_v9: Found options flowset: template %u", ntohs(option_flowset->template_id)); #endif Process_v9_option_templates(exporter, flowset_header, fs); } break; default: { input_translation_t *table; if ( flowset_id < NF9_MIN_RECORD_FLOWSET_ID ) { dbg_printf("Invalid flowset id: %u\n", flowset_id); LogError("Process_v9: Invalid flowset id: %u", flowset_id); } else { dbg_printf("[%u] ID %u Data flowset\n", exporter->info.id, flowset_id); table = GetTranslationTable(exporter, flowset_id); if ( table ) { Process_v9_data(exporter, flowset_header, fs, table); } else if ( HasOptionTable(exporter, flowset_id) ) { Process_v9_option_data(exporter, flowset_header, fs); } else { // maybe a flowset with option data dbg_printf("Process v9: [%u] No table for id %u -> Skip record\n", exporter->info.id, flowset_id); #ifdef DEVEL skip = 1; #endif } } } } // next flowset size_left -= flowset_length; } // End of while #ifdef DEVEL if ( skip == 0 && processed_records != expected_records ) { LogError( "Process_v9: Processed records %u, expected %u", processed_records, expected_records); printf("Process_v9: Processed records %u, expected %u\n", processed_records, expected_records); } #endif return; } /* End of Process_v9 */ /* * functions for sending netflow v9 records */ void Init_v9_output(send_peer_t *peer) { int i; v9_output_header = (netflow_v9_header_t *)peer->send_buffer; v9_output_header->version = htons(9); v9_output_header->SysUptime = 0; v9_output_header->unix_secs = 0; v9_output_header->count = 0; v9_output_header->source_id = htonl(1); template_id = NF9_MIN_RECORD_FLOWSET_ID; peer->buff_ptr = (void *)((pointer_addr_t)v9_output_header + (pointer_addr_t)sizeof(netflow_v9_header_t)); // set the max number of v9 tags, we support. Max_num_v9_tags = 0; for (i=1; v9_element_map[i].id != 0; i++ ) { if ( v9_element_map[i].id != v9_element_map[i-1].id ) Max_num_v9_tags++; } } // End of Init_v9_output static output_template_t *GetOutputTemplate(uint32_t flags, extension_map_t *extension_map) { output_template_t **t; template_record_t *fields; uint32_t i, count, record_length; t = &output_templates; // search for the template, which corresponds to our flags and extension map while ( *t ) { if ( (*t)->flags == flags && (*t)->extension_map == extension_map ) return *t; t = &((*t)->next); } // nothing found, otherwise we would not get here *t = (output_template_t *)malloc(sizeof(output_template_t)); if ( !(*t)) { fprintf(stderr, "Panic! malloc() %s line %d: %s", __FILE__, __LINE__, strerror (errno)); exit(255); } memset((void *)(*t), 0, sizeof(output_template_t)); (*t)->next = NULL; (*t)->flags = flags; (*t)->extension_map = extension_map; (*t)->time_sent = 0; (*t)->template_flowset = malloc(sizeof(template_flowset_t) + ((Max_num_v9_tags * 4))); // 4 for 2 x uint16_t: type/length count = 0; record_length = 0; fields = (*t)->template_flowset->fields; // Fill the template flowset in the order of the common_record_t // followed be the available extensions fields->record[count].type = htons(NF9_FIRST_SWITCHED); fields->record[count].length = htons(4); record_length += 4; count++; fields->record[count].type = htons(NF9_LAST_SWITCHED); fields->record[count].length = htons(4); record_length += 4; count++; fields->record[count].type = htons(NF9_FORWARDING_STATUS); fields->record[count].length = htons(1); record_length += 1; count++; fields->record[count].type = htons(NF9_TCP_FLAGS); fields->record[count].length = htons(1); record_length += 1; count++; fields->record[count].type = htons(NF9_IN_PROTOCOL); fields->record[count].length = htons(1); record_length += 1; count++; fields->record[count].type = htons(NF9_SRC_TOS); fields->record[count].length = htons(1); record_length += 1; count++; fields->record[count].type = htons(NF9_L4_SRC_PORT); fields->record[count].length = htons(2); record_length += 2; count++; fields->record[count].type = htons(NF9_L4_DST_PORT); fields->record[count].length = htons(2); record_length += 2; count++; fields->record[count].type = htons(NF9_ICMP_TYPE); fields->record[count].length = htons(2); record_length += 2; count++; // common record processed // fill in IP address tags if ( (flags & FLAG_IPV6_ADDR) != 0 ) { // IPv6 addresses fields->record[count].type = htons(NF9_IPV6_SRC_ADDR); fields->record[count].length = htons(16); record_length += 16; count++; fields->record[count].type = htons(NF9_IPV6_DST_ADDR); fields->record[count].length = htons(16); record_length += 16; } else { // IPv4 addresses fields->record[count].type = htons(NF9_IPV4_SRC_ADDR); fields->record[count].length = htons(4); record_length += 4; count++; fields->record[count].type = htons(NF9_IPV4_DST_ADDR); fields->record[count].length = htons(4); record_length += 4; } count++; // packet counter fields->record[count].type = htons(NF9_IN_PACKETS); if ( (flags & FLAG_PKG_64) != 0 ) { // 64bit packet counter fields->record[count].length = htons(8); record_length += 8; } else { fields->record[count].length = htons(4); record_length += 4; } count++; // bytes counter fields->record[count].type = htons(NF9_IN_BYTES); if ( (flags & FLAG_BYTES_64) != 0 ) { // 64bit byte counter fields->record[count].length = htons(8); record_length += 8; } else { fields->record[count].length = htons(4); record_length += 4; } count++; // process extension map i = 0; while ( extension_map->ex_id[i] ) { switch (extension_map->ex_id[i++]) { // 0 - 3 should never be in an extension table so - ignore it case 0: case 1: case 2: case 3: break; case EX_IO_SNMP_2: fields->record[count].type = htons(NF9_INPUT_SNMP); fields->record[count].length = htons(2); record_length += 2; count++; fields->record[count].type = htons(NF9_OUTPUT_SNMP); fields->record[count].length = htons(2); record_length += 2; count++; break; case EX_IO_SNMP_4: // input/output SNMP 4 byte fields->record[count].type = htons(NF9_INPUT_SNMP); fields->record[count].length = htons(4); record_length += 4; count++; fields->record[count].type = htons(NF9_OUTPUT_SNMP); fields->record[count].length = htons(4); record_length += 4; count++; break; case EX_AS_2: // srcas/dstas 2 byte fields->record[count].type = htons(NF9_SRC_AS); fields->record[count].length = htons(2); record_length += 2; count++; fields->record[count].type = htons(NF9_DST_AS); fields->record[count].length = htons(2); record_length += 2; count++; break; case EX_AS_4: // srcas/dstas 4 byte fields->record[count].type = htons(NF9_SRC_AS); fields->record[count].length = htons(4); record_length += 4; count++; fields->record[count].type = htons(NF9_DST_AS); fields->record[count].length = htons(4); record_length += 4; count++; break; case EX_MULIPLE: { uint16_t src_mask, dst_mask; fields->record[count].type = htons(NF9_DST_TOS); fields->record[count].length = htons(1); record_length += 1; count++; fields->record[count].type = htons(NF9_DIRECTION); fields->record[count].length = htons(1); record_length += 1; count++; if ( (flags & FLAG_IPV6_ADDR) != 0 ) { // IPv6 addresses src_mask = NF9_IPV6_SRC_MASK; dst_mask = NF9_IPV6_DST_MASK; } else { // IPv4 addresses src_mask = NF9_SRC_MASK; dst_mask = NF9_DST_MASK; } fields->record[count].type = htons(src_mask); fields->record[count].length = htons(1); record_length += 1; count++; fields->record[count].type = htons(dst_mask); fields->record[count].length = htons(1); record_length += 1; count++; } break; case EX_NEXT_HOP_v4: fields->record[count].type = htons(NF9_V4_NEXT_HOP); fields->record[count].length = htons(4); record_length += 4; count++; break; case EX_NEXT_HOP_v6: fields->record[count].type = htons(NF9_V6_NEXT_HOP); fields->record[count].length = htons(16); record_length += 16; count++; break; case EX_NEXT_HOP_BGP_v4: fields->record[count].type = htons(NF9_BGP_V4_NEXT_HOP); fields->record[count].length = htons(4); record_length += 4; count++; break; case EX_NEXT_HOP_BGP_v6: fields->record[count].type = htons(NF9_BPG_V6_NEXT_HOP); fields->record[count].length = htons(16); record_length += 16; count++; break; case EX_VLAN: fields->record[count].type = htons(NF9_SRC_VLAN); fields->record[count].length = htons(2); record_length += 2; count++; fields->record[count].type = htons(NF9_DST_VLAN); fields->record[count].length = htons(2); record_length += 2; count++; break; case EX_OUT_PKG_4: fields->record[count].type = htons(NF9_OUT_PKTS); fields->record[count].length = htons(4); record_length += 4; count++; break; case EX_OUT_PKG_8: fields->record[count].type = htons(NF9_OUT_PKTS); fields->record[count].length = htons(8); record_length += 8; count++; break; case EX_OUT_BYTES_4: fields->record[count].type = htons(NF9_OUT_BYTES); fields->record[count].length = htons(4); record_length += 4; count++; break; case EX_OUT_BYTES_8: fields->record[count].type = htons(NF9_OUT_BYTES); fields->record[count].length = htons(8); record_length += 8; count++; break; case EX_AGGR_FLOWS_4: fields->record[count].type = htons(NF9_FLOWS_AGGR); fields->record[count].length = htons(4); record_length += 4; count++; break; case EX_AGGR_FLOWS_8: fields->record[count].type = htons(NF9_FLOWS_AGGR); fields->record[count].length = htons(8); record_length += 8; count++; break; case EX_MAC_1: fields->record[count].type = htons(NF9_IN_SRC_MAC); fields->record[count].length = htons(6); record_length += 6; count++; fields->record[count].type = htons(NF9_OUT_DST_MAC); fields->record[count].length = htons(6); record_length += 6; count++; break; case EX_MAC_2: fields->record[count].type = htons(NF9_IN_DST_MAC); fields->record[count].length = htons(6); record_length += 6; count++; fields->record[count].type = htons(NF9_OUT_SRC_MAC); fields->record[count].length = htons(6); record_length += 6; count++; break; case EX_MPLS: fields->record[count].type = htons(NF9_MPLS_LABEL_1); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_2); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_3); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_4); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_5); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_6); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_7); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_8); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_9); fields->record[count].length = htons(3); record_length += 3; count++; fields->record[count].type = htons(NF9_MPLS_LABEL_10); fields->record[count].length = htons(3); record_length += 3; count++; break; case EX_ROUTER_ID: fields->record[count].type = htons(NF9_ENGINE_TYPE); fields->record[count].length = htons(1); record_length += 1; count++; fields->record[count].type = htons(NF9_ENGINE_ID); fields->record[count].length = htons(1); record_length += 1; count++; break; case EX_BGPADJ: fields->record[count].type = htons(NF9_BGP_ADJ_NEXT_AS); fields->record[count].length = htons(4); record_length += 4; count++; fields->record[count].type = htons(NF9_BGP_ADJ_PREV_AS); fields->record[count].length = htons(4); record_length += 4; count++; break; // default: other extensions are not (yet) recognised } } (*t)->template_flowset->flowset_id = htons(NF9_TEMPLATE_FLOWSET_ID); (*t)->flowset_length = 4 * (2+count); // + 2 for the header // add proper padding for 32bit boundary if ( ((*t)->flowset_length & 0x3 ) != 0 ) (*t)->flowset_length += (4 - ((*t)->flowset_length & 0x3 )); (*t)->template_flowset->length = htons((*t)->flowset_length); (*t)->record_length = record_length; fields->template_id = htons(template_id++); fields->count = htons(count); return *t; } // End of GetOutputTemplate static void Append_Record(send_peer_t *peer, master_record_t *master_record) { extension_map_t *extension_map = master_record->map_ref; uint32_t i, t1, t2; uint16_t icmp; t1 = (uint32_t)(1000LL * (uint64_t)master_record->first + master_record->msec_first - boot_time); t2 = (uint32_t)(1000LL * (uint64_t)master_record->last + master_record->msec_last - boot_time); master_record->first = htonl(t1); master_record->last = htonl(t2); master_record->srcport = htons(master_record->srcport); master_record->dstport = htons(master_record->dstport); // if it's an ICMP send it in the appropriate v9 tag if ( master_record->prot == IPPROTO_ICMP || master_record->prot == IPPROTO_ICMPV6 ) { // it's an ICMP icmp = master_record->dstport; master_record->dstport = 0; } else { icmp = 0; } // write the first 16 bytes of the master_record starting with first up to and including dst port memcpy(peer->buff_ptr, (void *)&master_record->first, 16); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + 16); // write ICMP type/code memcpy(peer->buff_ptr, (void *)&icmp,2); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + 2); // IP address info if ((master_record->flags & FLAG_IPV6_ADDR) != 0 ) { // IPv6 master_record->V6.srcaddr[0] = htonll(master_record->V6.srcaddr[0]); master_record->V6.srcaddr[1] = htonll(master_record->V6.srcaddr[1]); master_record->V6.dstaddr[0] = htonll(master_record->V6.dstaddr[0]); master_record->V6.dstaddr[1] = htonll(master_record->V6.dstaddr[1]); // keep compiler happy // memcpy(peer->buff_ptr, master_record->V6.srcaddr, 4 * sizeof(uint64_t)); memcpy(peer->buff_ptr, master_record->ip_union._ip_64.addr, 4 * sizeof(uint64_t)); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + 4 * sizeof(uint64_t)); } else { Put_val32(htonl(master_record->V4.srcaddr), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); Put_val32(htonl(master_record->V4.dstaddr), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); } // packet counter if ((master_record->flags & FLAG_PKG_64) != 0 ) { // 64bit counters Put_val64(htonll(master_record->dPkts), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); } else { Put_val32(htonl((uint32_t)master_record->dPkts), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); } // bytes counter if ((master_record->flags & FLAG_BYTES_64) != 0 ) { // 64bit counters Put_val64(htonll(master_record->dOctets),peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); } else { Put_val32(htonl((uint32_t)master_record->dOctets),peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); } // send now optional extensions according the extension map i=0; while ( extension_map->ex_id[i] ) { switch (extension_map->ex_id[i++]) { // 0 - 3 should never be in an extension table so - ignore it case 0: case 1: case 2: case 3: break; case EX_IO_SNMP_2: { uint16_t in, out; in = htons(master_record->input); Put_val16(in, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint16_t)); out = htons(master_record->output); Put_val16(out, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint16_t)); } break; case EX_IO_SNMP_4: Put_val32(htonl(master_record->input), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); Put_val32(htonl(master_record->output), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); break; case EX_AS_2: { // srcas/dstas 2 byte uint16_t src, dst; src = htons(master_record->srcas); Put_val16(src, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint16_t)); dst = htons(master_record->dstas); Put_val16(dst, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint16_t)); } break; case EX_AS_4: // srcas/dstas 4 byte Put_val32(htonl(master_record->srcas), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); Put_val32(htonl(master_record->dstas), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); break; case EX_MULIPLE: { tpl_ext_8_t *tpl = (tpl_ext_8_t *)peer->buff_ptr; tpl->dst_tos = master_record->dst_tos; tpl->dir = master_record->dir; tpl->src_mask = master_record->src_mask; tpl->dst_mask = master_record->dst_mask; peer->buff_ptr = (void *)tpl->data; } break; case EX_NEXT_HOP_v4: Put_val32(htonl(master_record->ip_nexthop.V4), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); break; case EX_NEXT_HOP_v6: Put_val64(htonll(master_record->ip_nexthop.V6[0]), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); Put_val64(htonll(master_record->ip_nexthop.V6[1]), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); break; case EX_NEXT_HOP_BGP_v4: Put_val32(htonl(master_record->bgp_nexthop.V4), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); break; case EX_NEXT_HOP_BGP_v6: Put_val64(htonll(master_record->bgp_nexthop.V6[0]), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); Put_val64(htonll(master_record->bgp_nexthop.V6[1]), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); break; case EX_VLAN: Put_val16(htons(master_record->src_vlan), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint16_t)); Put_val16(htons(master_record->dst_vlan), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint16_t)); break; case EX_OUT_PKG_4: Put_val32(htonl((uint32_t)master_record->out_pkts), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); break; case EX_OUT_PKG_8: Put_val64(htonll(master_record->out_pkts), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); break; case EX_OUT_BYTES_4: Put_val32(htonl((uint32_t)master_record->out_bytes), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); break; case EX_OUT_BYTES_8: Put_val64(htonll(master_record->out_bytes), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); break; case EX_AGGR_FLOWS_4: Put_val32(htonl(master_record->aggr_flows), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); break; case EX_AGGR_FLOWS_8: Put_val64(htonll(master_record->aggr_flows), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint64_t)); break; case EX_MAC_1: { uint64_t val64; val64 = htonll(master_record->in_src_mac); Put_val48(val64, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + 6); // 48 bits val64 = htonll(master_record->out_dst_mac); Put_val48(val64, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + 6); // 48 bits } break; case EX_MAC_2: { uint64_t val64; val64 = htonll(master_record->in_dst_mac); Put_val48(val64, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + 6); // 48 bits val64 = htonll(master_record->out_src_mac); Put_val48(val64, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + 6); // 48 bits } break; case EX_MPLS: { uint32_t val32, i; for ( i=0; i<10; i++ ) { val32 = htonl(master_record->mpls_label[i]); Put_val24(val32, peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + 3); // 24 bits } } break; case EX_ROUTER_ID: { uint8_t *u = (uint8_t *)peer->buff_ptr; *u++ = master_record->engine_type; *u++ = master_record->engine_id; peer->buff_ptr = (void *)u; } break; case EX_BGPADJ: Put_val32(htonl(master_record->bgpNextAdjacentAS), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); Put_val32(htonl(master_record->bgpPrevAdjacentAS), peer->buff_ptr); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + sizeof(uint32_t)); break; // default: ignore all other extension, as we do not understand them } } } // End of Append_Record int Add_v9_output_record(master_record_t *master_record, send_peer_t *peer) { static data_flowset_t *data_flowset; static output_template_t *template; static uint32_t last_flags = 0; static extension_map_t *last_map = NULL; static int record_count, template_count, flowset_count, packet_count; uint32_t required_size; void *endwrite; time_t now = time(NULL); #ifdef DEVEL char *string; flow_record_to_raw(master_record, &string, 0); printf("%s\n", string); #endif if ( !v9_output_header->unix_secs ) { // first time a record is added // boot time is set one day back - assuming that the start time of every flow does not start ealier boot_time = (uint64_t)(master_record->first - 86400)*1000; v9_output_header->unix_secs = htonl(master_record->first - 86400); v9_output_header->sequence = 0; peer->buff_ptr = (void *)((pointer_addr_t)peer->send_buffer + NETFLOW_V9_HEADER_LENGTH); record_count = 0; template_count = 0; flowset_count = 0; packet_count = 0; data_flowset = NULL; // write common blocksize from frst up to including dstas for one write (memcpy) // common_block_size = (pointer_addr_t)&master_record->fill - (pointer_addr_t)&master_record->first; } else if ( flowset_count == 0 ) { // after a buffer flush packet_count++; v9_output_header->sequence = htonl(packet_count); } if ( data_flowset ) { // output buffer contains already a data flowset if ( last_flags == master_record->flags && last_map == master_record->map_ref ) { // same id as last record // if ( now - template->time_sent > MAX_LIFETIME ) if ( (record_count & 0xFFF) == 0 ) { // every 4096 flow records uint16_t length = (pointer_addr_t)peer->buff_ptr - (pointer_addr_t)data_flowset; uint8_t align = length & 0x3; if ( align != 0 ) { length += ( 4 - align ); data_flowset->length = htons(length); peer->buff_ptr += align; } // template refresh is needed // terminate the current data flowset data_flowset = NULL; if ( (pointer_addr_t)peer->buff_ptr + template->flowset_length > (pointer_addr_t)peer->endp ) { // not enough space for template flowset => flush buffer first record_count = 0; flowset_count = 0; template_count = 0; peer->flush = 1; return 1; // return to flush buffer } memcpy(peer->buff_ptr, (void *)template->template_flowset, template->flowset_length); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + template->flowset_length); template->time_sent = now; flowset_count++; template_count++; // open a new data flow set at this point in the output buffer data_flowset = (data_flowset_t *)peer->buff_ptr; data_flowset->flowset_id = template->template_flowset->fields[0].template_id; peer->buff_ptr = (void *)data_flowset->data; flowset_count++; } // else Add record } else { // record with different template id // terminate the current data flowset uint16_t length = (pointer_addr_t)peer->buff_ptr - (pointer_addr_t)data_flowset; uint8_t align = length & 0x3; if ( align != 0 ) { length += ( 4 - align ); data_flowset->length = htons(length); peer->buff_ptr += align; } data_flowset = NULL; last_flags = master_record->flags; last_map = master_record->map_ref; template = GetOutputTemplate(last_flags, master_record->map_ref); if ( now - template->time_sent > MAX_LIFETIME ) { // refresh template is needed endwrite= (void *)((pointer_addr_t)peer->buff_ptr + template->flowset_length + sizeof(data_flowset_t)); if ( endwrite > peer->endp ) { // not enough space for template flowset => flush buffer first record_count = 0; flowset_count = 0; template_count = 0; peer->flush = 1; return 1; // return to flush the buffer } memcpy(peer->buff_ptr, (void *)template->template_flowset, template->flowset_length); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + template->flowset_length); template->time_sent = now; flowset_count++; template_count++; } // open a new data flow set at this point in the output buffer data_flowset = (data_flowset_t *)peer->buff_ptr; data_flowset->flowset_id = template->template_flowset->fields[0].template_id; peer->buff_ptr = (void *)data_flowset->data; flowset_count++; } } else { // output buffer does not contain a data flowset peer->buff_ptr = (void *)((pointer_addr_t)v9_output_header + (pointer_addr_t)sizeof(netflow_v9_header_t)); last_flags = master_record->flags; last_map = master_record->map_ref; template = GetOutputTemplate(last_flags, master_record->map_ref); if ( now - template->time_sent > MAX_LIFETIME ) { // refresh template endwrite= (void *)((pointer_addr_t)peer->buff_ptr + template->flowset_length + sizeof(data_flowset_t)); if ( endwrite > peer->endp ) { // this must never happen! fprintf(stderr, "Panic: Software error in %s line %d\n", __FILE__, __LINE__); fprintf(stderr, "buffer %p, buff_ptr %p template length %x, endbuff %p\n", peer->send_buffer, peer->buff_ptr, template->flowset_length + (uint32_t)sizeof(data_flowset_t), peer->endp ); exit(255); } memcpy(peer->buff_ptr, (void *)template->template_flowset, template->flowset_length); peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + template->flowset_length); template->time_sent = now; flowset_count++; template_count++; } // open a new data flow set at this point in the output buffer data_flowset = (data_flowset_t *)peer->buff_ptr; data_flowset->flowset_id = template->template_flowset->fields[0].template_id; peer->buff_ptr = (void *)data_flowset->data; flowset_count++; } // now add the record required_size = template->record_length; endwrite = (void *)((pointer_addr_t)peer->buff_ptr + required_size); if ( endwrite > peer->endp ) { uint16_t length = (pointer_addr_t)peer->buff_ptr - (pointer_addr_t)data_flowset; // flush the buffer data_flowset->length = htons(length); if ( length == 4 ) { // empty flowset peer->buff_ptr = (void *)data_flowset; } data_flowset = NULL; v9_output_header->count = htons(record_count+template_count); record_count = 0; template_count = 0; flowset_count = 0; peer->flush = 1; return 1; // return to flush buffer } // this was a long way up to here, now we can add the data Append_Record(peer, master_record); data_flowset->length = htons((pointer_addr_t)peer->buff_ptr - (pointer_addr_t)data_flowset); record_count++; v9_output_header->count = htons(record_count+template_count); return 0; } // End of Add_v9_output_record static void InsertSampler( FlowSource_t *fs, exporterDomain_t *exporter, int32_t id, uint16_t mode, uint32_t interval) { sampler_t *sampler; dbg_printf("[%u] Insert Sampler: Exporter is 0x%llu\n", exporter->info.id, (long long unsigned)exporter); if ( !exporter->sampler ) { // no samplers so far sampler = (sampler_t *)malloc(sizeof(sampler_t)); if ( !sampler ) { LogError( "Process_v9: Panic! malloc(): %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return; } sampler->info.header.type = SamplerInfoRecordype; sampler->info.header.size = sizeof(sampler_info_record_t); sampler->info.exporter_sysid = exporter->info.sysid; sampler->info.id = id; sampler->info.mode = mode; sampler->info.interval = interval; sampler->next = NULL; exporter->sampler = sampler; FlushInfoSampler(fs, &(sampler->info)); LogInfo( "Add new sampler: ID: %i, mode: %u, interval: %u\n", id, mode, interval); dbg_printf("Add new sampler: ID: %i, mode: %u, interval: %u\n", id, mode, interval); } else { sampler = exporter->sampler; while ( sampler ) { // test for update of existing sampler if ( sampler->info.id == id ) { // found same sampler id - update record dbg_printf("Update existing sampler id: %i, mode: %u, interval: %u\n", id, mode, interval); // we update only on changes if ( mode != sampler->info.mode || interval != sampler->info.interval ) { FlushInfoSampler(fs, &(sampler->info)); sampler->info.mode = mode; sampler->info.interval = interval; LogInfo( "Update existing sampler id: %i, mode: %u, interval: %u\n", id, mode, interval); } else { dbg_printf("Sampler unchanged!\n"); } break; } // test for end of chain if ( sampler->next == NULL ) { // end of sampler chain - insert new sampler sampler->next = (sampler_t *)malloc(sizeof(sampler_t)); if ( !sampler->next ) { LogError( "Process_v9: Panic! malloc(): %s line %d: %s", __FILE__, __LINE__, strerror (errno)); return; } sampler = sampler->next; sampler->info.header.type = SamplerInfoRecordype; sampler->info.header.size = sizeof(sampler_info_record_t); sampler->info.exporter_sysid = exporter->info.sysid; sampler->info.id = id; sampler->info.mode = mode; sampler->info.interval = interval; sampler->next = NULL; FlushInfoSampler(fs, &(sampler->info)); LogInfo( "Append new sampler: ID: %u, mode: %u, interval: %u\n", id, mode, interval); dbg_printf("Append new sampler: ID: %u, mode: %u, interval: %u\n", id, mode, interval); break; } // advance sampler = sampler->next; } } } // End of InsertSampler nfdump-1.6.23/bin/netflow_v9.h000066400000000000000000000243721404501030700161130ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ /* v9 structures */ /* Packet Header Field Descriptions * * Version * Version of Flow Record format exported in this packet. The * value of this field is 9 for the current version. * * Count * The total number of records in the Export Packet, which is the * sum of Options FlowSet records, Template FlowSet records, and * Data FlowSet records. * * sysUpTime * Time in milliseconds since this device was first booted. * * UNIX Secs * Time in seconds since 0000 UTC 1970, at which the Export Packet * leaves the Exporter. * * Sequence Number * Incremental sequence counter of all Export Packets sent from * the current Observation Domain by the Exporter. This value * MUST be cumulative, and SHOULD be used by the Collector to * identify whether any Export Packets have been missed. * * Source ID * A 32-bit value that identifies the Exporter Observation Domain. * NetFlow Collectors SHOULD use the combination of the source IP * address and the Source ID field to separate different export * streams originating from the same Exporter. */ #ifndef _NETFLOW_V9_H #define _NETFLOW_V9_H 1 #include "config.h" #include #ifdef HAVE_STDINT_H #include #endif #include "collector.h" #include "nfnet.h" #include "nffile.h" typedef struct netflow_v9_header { uint16_t version; uint16_t count; uint32_t SysUptime; uint32_t unix_secs; uint32_t sequence; uint32_t source_id; } netflow_v9_header_t; #define NETFLOW_V9_HEADER_LENGTH sizeof(netflow_v9_header_t) /* FlowSet ID * FlowSet ID value of 0 is reserved for the Template FlowSet. * Length * Total length of this FlowSet. Because an individual Template * FlowSet MAY contain multiple Template Records, the Length value * MUST be used to determine the position of the next FlowSet * record, which could be any type of FlowSet. Length is the sum * of the lengths of the FlowSet ID, the Length itself, and all * Template Records within this FlowSet. * * Template ID * Each of the newly generated Template Records is given a unique * Template ID. This uniqueness is local to the Observation * Domain that generated the Template ID. Template IDs 0-255 are * reserved for Template FlowSets, Options FlowSets, and other * reserved FlowSets yet to be created. Template IDs of Data * FlowSets are numbered from 256 to 65535. * * Field Count * Number of fields in this Template Record. Because a Template * FlowSet usually contains multiple Template Records, this field * allows the Collector to determine the end of the current * Template Record and the start of the next. * * Field Type * A numeric value that represents the type of the field. Refer * to the "Field Type Definitions" section. * * Field Length * The length of the corresponding Field Type, in bytes. Refer to * the "Field Type Definitions" section. */ typedef struct template_record_s { uint16_t template_id; uint16_t count; struct { uint16_t type; uint16_t length; } record[1]; } template_record_t; typedef struct template_flowset_s { uint16_t flowset_id; uint16_t length; template_record_t fields[1]; } template_flowset_t; typedef struct data_flowset_s { uint16_t flowset_id; uint16_t length; uint8_t data[4]; } data_flowset_t; typedef struct option_template_flowset_s { uint16_t flowset_id; uint16_t length; uint16_t template_id; uint16_t option_scope_length; uint16_t option_length; struct { uint16_t type; uint16_t length; } record[1]; } option_template_flowset_t; typedef struct common_header_s { uint16_t flowset_id; uint16_t length; } common_header_t; #define _1byte 1 #define _2bytes 2 #define _3bytes 3 #define _4bytes 4 #define _6bytes 6 #define _8bytes 8 #define _12bytes 12 #define _16bytes 16 #define _20bytes 20 #define _24bytes 24 #define _65bytes 65 #define _72bytes 72 #define NF9_TEMPLATE_FLOWSET_ID 0 #define NF9_OPTIONS_FLOWSET_ID 1 #define NF9_MIN_RECORD_FLOWSET_ID 256 // Flowset record types #define NF9_IN_BYTES 1 #define NF9_IN_PACKETS 2 #define NF9_FLOWS_AGGR 3 #define NF9_IN_PROTOCOL 4 #define NF9_SRC_TOS 5 #define NF9_TCP_FLAGS 6 #define NF9_L4_SRC_PORT 7 #define NF9_IPV4_SRC_ADDR 8 #define NF9_SRC_MASK 9 #define NF9_INPUT_SNMP 10 #define NF9_L4_DST_PORT 11 #define NF9_IPV4_DST_ADDR 12 #define NF9_DST_MASK 13 #define NF9_OUTPUT_SNMP 14 #define NF9_V4_NEXT_HOP 15 #define NF9_SRC_AS 16 #define NF9_DST_AS 17 #define NF9_BGP_V4_NEXT_HOP 18 #define NF9_LAST_SWITCHED 21 #define NF9_FIRST_SWITCHED 22 #define NF9_OUT_BYTES 23 #define NF9_OUT_PKTS 24 #define NF9_IPV6_SRC_ADDR 27 #define NF9_IPV6_DST_ADDR 28 #define NF9_IPV6_SRC_MASK 29 #define NF9_IPV6_DST_MASK 30 #define NF9_IPV6_FLOW_LABEL 31 #define NF9_ICMP_TYPE 32 #define NF9_SAMPLING_INTERVAL 34 #define NF9_SAMPLING_ALGORITHM 35 #define NF9_ENGINE_TYPE 38 #define NF9_ENGINE_ID 39 #define NF9_FLOW_SAMPLER_ID 48 #define FLOW_SAMPLER_MODE 49 #define NF9_FLOW_SAMPLER_RANDOM_INTERVAL 50 #define NF_SELECTOR_ID 302 #define NF_SELECTOR_ALGORITHM 304 #define NF_SAMPLING_INTERVAL 305 // #define NF9_MIN_TTL 52 // #define NF9_MAX_TTL 53 // #define NF9_IPV4_IDENT 54 #define NF9_DST_TOS 55 #define NF9_IN_SRC_MAC 56 #define NF9_OUT_DST_MAC 57 #define NF9_SRC_VLAN 58 #define NF9_DST_VLAN 59 #define NF9_DIRECTION 61 #define NF9_V6_NEXT_HOP 62 #define NF9_BPG_V6_NEXT_HOP 63 // #define NF9_V6_OPTION_HEADERS 64 #define NF9_MPLS_LABEL_1 70 #define NF9_MPLS_LABEL_2 71 #define NF9_MPLS_LABEL_3 72 #define NF9_MPLS_LABEL_4 73 #define NF9_MPLS_LABEL_5 74 #define NF9_MPLS_LABEL_6 75 #define NF9_MPLS_LABEL_7 76 #define NF9_MPLS_LABEL_8 77 #define NF9_MPLS_LABEL_9 78 #define NF9_MPLS_LABEL_10 79 #define NF9_IN_DST_MAC 80 #define NF9_OUT_SRC_MAC 81 #define NF9_FORWARDING_STATUS 89 #define NF9_BGP_ADJ_NEXT_AS 128 #define NF9_BGP_ADJ_PREV_AS 129 #define NF9_dot1qVlanId 243 #define NF9_postDot1qVlanId 254 // CISCO ASA NSEL extension - Network Security Event Logging #define NF_F_FLOW_BYTES 85 #define NF_F_CONN_ID 148 #define NF_F_FLOW_CREATE_TIME_MSEC 152 #define NF_F_FLOW_END_TIME_MSEC 153 #define NF_F_ICMP_TYPE 176 #define NF_F_ICMP_CODE 177 #define NF_F_ICMP_TYPE_IPV6 178 #define NF_F_ICMP_CODE_IPV6 179 #define NF_F_FWD_FLOW_DELTA_BYTES 231 #define NF_F_REV_FLOW_DELTA_BYTES 232 #define NF_F_EVENT_TIME_MSEC 323 #define NF_F_INGRESS_ACL_ID 33000 #define NF_F_EGRESS_ACL_ID 33001 #define NF_F_FW_EXT_EVENT 33002 #define NF_F_USERNAME 40000 #define NF_F_XLATE_SRC_ADDR_IPV4 225 #define NF_F_XLATE_DST_ADDR_IPV4 226 #define NF_F_XLATE_SRC_PORT 227 #define NF_F_XLATE_DST_PORT 228 #define NF_F_XLATE_SRC_ADDR_IPV6 281 #define NF_F_XLATE_DST_ADDR_IPV6 282 #define NF_F_FW_EVENT 233 // ASA 8.4 compat elements #define NF_F_XLATE_SRC_ADDR_84 40001 #define NF_F_XLATE_DST_ADDR_84 40002 #define NF_F_XLATE_SRC_PORT_84 40003 #define NF_F_XLATE_DST_PORT_84 40004 #define NF_F_FW_EVENT_84 40005 // ASA 9.x packet counters: initiatorPackets and responderPackets // see https://www.iana.org/assignments/ipfix/ipfix.xhtml #define NF_F_INITIATORPACKETS 298 #define NF_F_RESPONDERPACKETS 299 // Zone-Based Firewall Logging #define NF_FW_CTS_SRC_SGT 34000 // Cisco ASR 1000 series NEL extension - Nat Event Logging #define NF_N_NAT_EVENT 230 #define NF_N_INGRESS_VRFID 234 #define NF_N_EGRESS_VRFID 235 #define NF_F_XLATE_PORT_BLOCK_START 361 #define NF_F_XLATE_PORT_BLOCK_END 362 #define NF_F_XLATE_PORT_BLOCK_STEP 363 #define NF_F_XLATE_PORT_BLOCK_SIZE 364 // nprobe latency extensions #define NF9_NPROBE_CLIENT_NW_DELAY_SEC 57554 #define NF9_NPROBE_CLIENT_NW_DELAY_USEC 57555 #define NF9_NPROBE_SERVER_NW_DELAY_SEC 57556 #define NF9_NPROBE_SERVER_NW_DELAY_USEC 57557 #define NF9_NPROBE_APPL_LATENCY_SEC 57558 #define NF9_NPROBE_APPL_LATENCY_USEC 57559 /* prototypes */ int Init_v9(int v, uint32_t sampling, uint32_t overwrite); void Process_v9(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs); void Init_v9_output(send_peer_t *peer); int Add_v9_output_record(master_record_t *master_record, send_peer_t *peer); #endif //_NETFLOW_V9_H 1 nfdump-1.6.23/bin/nfanon.c000077500000000000000000000310541404501030700152670ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "exporter.h" #include "flist.h" #include "panonymizer.h" // module limited globals extension_map_list_t *extension_map_list; /* Function Prototypes */ static void usage(char *name); static inline void AnonRecord(master_record_t *master_record); static void process_data(void *wfile); /* Functions */ #define NEED_PACKRECORD 1 #include "nffile_inline.c" #undef NEED_PACKRECORD static void usage(char *name) { printf("usage %s [options] \n" "-h\t\tthis text you see right here\n" "-K \tAnonymize IP addresses using CryptoPAn with key .\n" "-r\t\tread input from file\n" "-M \tRead input from multiple directories.\n" "-R \tRead input from sequence of files.\n" "-w \tName of output file. Defaults to input file.\n" , name); } /* usage */ static inline void AnonRecord(master_record_t *master_record) { extension_map_t *extension_map = master_record->map_ref; int i; // Required extension 1 - IP addresses if ( (master_record->flags & FLAG_IPV6_ADDR) != 0 ) { // IPv6 // IPv6 uint64_t anon_ip[2]; anonymize_v6(master_record->V6.srcaddr, anon_ip); master_record->V6.srcaddr[0] = anon_ip[0]; master_record->V6.srcaddr[1] = anon_ip[1]; anonymize_v6(master_record->V6.dstaddr, anon_ip); master_record->V6.dstaddr[0] = anon_ip[0]; master_record->V6.dstaddr[1] = anon_ip[1]; } else { // IPv4 master_record->V4.srcaddr = anonymize(master_record->V4.srcaddr); master_record->V4.dstaddr = anonymize(master_record->V4.dstaddr); } // Process optional extensions i=0; while ( extension_map->ex_id[i] ) { switch (extension_map->ex_id[i++]) { case EX_AS_2: // srcas/dstas 2 byte master_record->srcas = 0; master_record->dstas = 0; break; case EX_AS_4: // srcas/dstas 4 byte master_record->srcas = 0; master_record->dstas = 0; break; case EX_NEXT_HOP_v4: master_record->ip_nexthop.V4 = anonymize(master_record->ip_nexthop.V4); break; case EX_NEXT_HOP_v6: { uint64_t anon_ip[2]; anonymize_v6(master_record->ip_nexthop.V6, anon_ip); master_record->ip_nexthop.V6[0] = anon_ip[0]; master_record->ip_nexthop.V6[1] = anon_ip[1]; } break; case EX_NEXT_HOP_BGP_v4: master_record->bgp_nexthop.V4 = anonymize(master_record->bgp_nexthop.V4); break; case EX_NEXT_HOP_BGP_v6: { uint64_t anon_ip[2]; anonymize_v6(master_record->bgp_nexthop.V6, anon_ip); master_record->bgp_nexthop.V6[0] = anon_ip[0]; master_record->bgp_nexthop.V6[1] = anon_ip[1]; } break; case EX_ROUTER_IP_v4: master_record->ip_router.V4 = anonymize(master_record->ip_router.V4); break; case EX_ROUTER_IP_v6: { uint64_t anon_ip[2]; anonymize_v6(master_record->ip_router.V6, anon_ip); master_record->ip_router.V6[0] = anon_ip[0]; master_record->ip_router.V6[1] = anon_ip[1]; } break; #ifdef NSEL case EX_NSEL_XLATE_IP_v4: master_record->xlate_src_ip.V4 = anonymize(master_record->xlate_src_ip.V4); master_record->xlate_dst_ip.V4 = anonymize(master_record->xlate_dst_ip.V4); break; case EX_NSEL_XLATE_IP_v6: { uint64_t anon_ip[2]; anonymize_v6(master_record->xlate_src_ip.V6, anon_ip); master_record->xlate_src_ip.V6[0] = anon_ip[0]; master_record->xlate_src_ip.V6[1] = anon_ip[1]; anonymize_v6(master_record->xlate_dst_ip.V6, anon_ip); master_record->xlate_dst_ip.V6[0] = anon_ip[0]; master_record->xlate_dst_ip.V6[1] = anon_ip[1]; } break; #endif } } } // End of AnonRecord static void process_data(void *wfile) { master_record_t master_record; common_record_t *flow_record; nffile_t *nffile_r; nffile_t *nffile_w; int i, done, ret, cnt, verbose; char outfile[MAXPATHLEN], *cfile; cnt = 1; verbose = 1; // Get the first file handle nffile_r = GetNextFile(NULL, 0, 0); if ( !nffile_r ) { LogError("GetNextFile() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return; } if ( nffile_r == EMPTY_LIST ) { LogError("Empty file list. No files to process\n"); return; } cfile = GetCurrentFilename(); if ( !cfile ) { if ( nffile_r->fd == 0 ) { // stdin outfile[0] = '-'; outfile[1] = '\0'; verbose = 0; } else { LogError("(NULL) input file name error in %s line %d\n", __FILE__, __LINE__); return; } } else { // prepare output file snprintf(outfile,MAXPATHLEN-1, "%s-tmp", cfile); outfile[MAXPATHLEN-1] = '\0'; if ( verbose ) LogInfo(" %i Processing %s", cnt++, cfile); } if ( wfile ) nffile_w = OpenNewFile(wfile, NULL, FILE_COMPRESSION(nffile_r), 1, NULL); else nffile_w = OpenNewFile(outfile, NULL, FILE_COMPRESSION(nffile_r), 1, NULL); if ( !nffile_w ) { if ( nffile_r ) { CloseFile(nffile_r); DisposeFile(nffile_r); } return; } memcpy((void *)nffile_w->stat_record, (void *)nffile_r->stat_record, sizeof(stat_record_t)); done = 0; while ( !done ) { // get next data block from file ret = ReadBlock(nffile_r); switch (ret) { case NF_CORRUPT: case NF_ERROR: if ( ret == NF_CORRUPT ) LogError("Skip corrupt data file '%s'\n",GetCurrentFilename()); else LogError("Read error in file '%s': %s\n",GetCurrentFilename(), strerror(errno) ); // fall through - get next file in chain case NF_EOF: { nffile_t *next; if ( nffile_w->block_header->NumRecords ) { if ( WriteBlock(nffile_w) <= 0 ) { LogError("Failed to write output buffer to disk: '%s'" , strerror(errno)); } } if ( wfile == NULL ) { CloseUpdateFile(nffile_w, nffile_r->file_header->ident); if ( rename(outfile, cfile) < 0 ) { LogError("\nrename() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); LogError("Abort processing.\n"); return; } } next = GetNextFile(nffile_r, 0, 0); if ( next == EMPTY_LIST ) { done = 1; continue; } if ( next == NULL ) { LogError("Unexpected end of file list\n"); done = 1; continue; } cfile = GetCurrentFilename(); if ( !cfile ) { LogError("(NULL) input file name error in %s line %d\n", __FILE__, __LINE__); return; } printf(" %i Processing %s\r", cnt++, cfile); if ( wfile == NULL ) { snprintf(outfile,MAXPATHLEN-1, "%s-tmp", cfile); outfile[MAXPATHLEN-1] = '\0'; nffile_w = OpenNewFile(outfile, nffile_w, FILE_COMPRESSION(nffile_r), 1, NULL); if ( !nffile_w ) { if ( nffile_r ) { CloseFile(nffile_r); DisposeFile(nffile_r); } return; } memcpy((void *)nffile_w->stat_record, (void *)nffile_r->stat_record, sizeof(stat_record_t)); } else { SumStatRecords(nffile_w->stat_record, nffile_r->stat_record); } // continue with next file continue; } break; // not really needed } if ( nffile_r->block_header->id != DATA_BLOCK_TYPE_2 ) { LogError("Can't process block type %u. Skip block", nffile_r->block_header->id); continue; } flow_record = nffile_r->buff_ptr; uint32_t sumSize = 0; for ( i=0; i < nffile_r->block_header->NumRecords; i++ ) { if ( (sumSize + flow_record->size) > ret ) { LogError("Corrupt data file. Inconsistent block size in %s line %d\n", __FILE__, __LINE__); exit(255); } sumSize += flow_record->size; switch ( flow_record->type ) { case CommonRecordType: { uint32_t map_id = flow_record->ext_map; if ( extension_map_list->slot[map_id] == NULL ) { LogError("Corrupt data file! No such extension map id: %u. Skip record", flow_record->ext_map ); } else { ExpandRecord_v2( flow_record, extension_map_list->slot[flow_record->ext_map], NULL, &master_record); // update number of flows matching a given map extension_map_list->slot[map_id]->ref_count++; AnonRecord(&master_record); PackRecord(&master_record, nffile_w); } } break; case ExtensionMapType: { extension_map_t *map = (extension_map_t *)flow_record; int ret = Insert_Extension_Map(extension_map_list, map); switch (ret) { case 0: break; // map already known and flushed case 1: AppendToBuffer(nffile_w, (void *)map, map->size); break; default: LogError("Corrupt data file. Unable to decode at %s line %d\n", __FILE__, __LINE__); exit(255); } } break; case LegacyRecordType1: case LegacyRecordType2: case ExporterInfoRecordType: case ExporterStatRecordType: case SamplerInfoRecordype: // Silently skip exporter/sampler records break; default: { LogError("Skip unknown record type %i", flow_record->type); } } // Advance pointer by number of bytes for netflow record flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size); } // for all records } // while PackExtensionMapList(extension_map_list); if ( wfile != NULL ) CloseUpdateFile(nffile_w, nffile_r->file_header->ident); if ( nffile_r ) { CloseFile(nffile_r); DisposeFile(nffile_r); } DisposeFile(nffile_w); LogError("Processed %i files", --cnt); } // End of process_data int main( int argc, char **argv ) { char *rfile, *Rfile, *wfile, *Mdirs; int c; char CryptoPAnKey[32]; memset((void *)CryptoPAnKey, 0, sizeof(CryptoPAnKey)); rfile = Rfile = Mdirs = wfile = NULL; while ((c = getopt(argc, argv, "K:L:r:M:R:w:")) != EOF) { switch (c) { case 'h': usage(argv[0]); exit(0); break; break; case 'K': if ( !ParseCryptoPAnKey(optarg, CryptoPAnKey) ) { LogError("Invalid key '%s' for CryptoPAn", optarg); exit(255); } PAnonymizer_Init((uint8_t *)CryptoPAnKey); break; case 'L': if ( !InitLog(0, "argv[0]", optarg, 0) ) exit(255); break; case 'r': rfile = optarg; if ( strcmp(rfile, "-") == 0 ) rfile = NULL; break; case 'M': Mdirs = optarg; break; case 'R': Rfile = optarg; break; case 'w': wfile = optarg; break; default: usage(argv[0]); exit(0); } } if ( CryptoPAnKey[0] == '\0' ) { LogError("Expect -K - 32 bytes key"); exit(255); } if ( rfile && Rfile ) { LogError("-r and -R are mutually exclusive. Please specify either -r or -R"); exit(255); } if ( Mdirs && !(rfile || Rfile) ) { LogError("-M needs either -r or -R to specify the file or file list. Add '-R .' for all files in the directories"); exit(255); } extension_map_list = InitExtensionMaps(NEEDS_EXTENSION_LIST); SetupInputFileSequence(Mdirs, rfile, Rfile); process_data(wfile); FreeExtensionMaps(extension_map_list); return 0; } nfdump-1.6.23/bin/nfcapd.c000066400000000000000000001031421404501030700152360ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef PCAP #include "pcap_reader.h" #endif #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "exporter.h" #include "nfnet.h" #include "flist.h" #include "nfstatfile.h" #include "bookkeeper.h" #include "launch.h" #include "collector.h" #include "netflow_v1.h" #include "netflow_v5_v7.h" #include "netflow_v9.h" #include "ipfix.h" #ifdef HAVE_FTS_H # include #else # include "fts_compat.h" #define fts_children fts_children_compat #define fts_close fts_close_compat #define fts_open fts_open_compat #define fts_read fts_read_compat #define fts_set fts_set_compat #endif #include "expire.h" #define DEFAULTCISCOPORT "9995" #define DEFAULTHOSTNAME "127.0.0.1" #define SENDSOCK_BUFFSIZE 200000 static void *shmem = NULL; static int verbose = 0; static uint32_t default_sampling = 1; static uint32_t overwrite_sampling = 0; extern uint32_t default_sampling; // the default sampling rate when nothing else applies. set by -S extern uint32_t overwrite_sampling; // unconditionally overwrite sampling rate with given sampling rate -S // Define a generic type to get data from socket or pcap file typedef ssize_t (*packet_function_t)(int, void *, size_t, int, struct sockaddr *, socklen_t *); /* module limited globals */ static FlowSource_t *FlowSource; static int done, launcher_alive, periodic_trigger, launcher_pid; static const char *nfdump_version = VERSION; /* Local function Prototypes */ static void usage(char *name); static void kill_launcher(int pid); static void IntHandler(int signal); static inline FlowSource_t *GetFlowSource(struct sockaddr_storage *ss); static void daemonize(void); static void SetPriv(char *userid, char *groupid ); static void run(packet_function_t receive_packet, int socket, repeater_t *repeater, time_t twin, time_t t_begin, int report_seq, int use_subdirs, char *time_extension, int compress); /* Functions */ static void usage(char *name) { printf("usage %s [options] \n" "-h\t\tthis text you see right here\n" "-u userid\tChange user to username\n" "-g groupid\tChange group to groupname\n" "-w\t\tSync file rotation with next 5min (default) interval\n" "-t interval\tset the interval to rotate nfcapd files\n" "-b host\t\tbind socket to host/IP addr\n" "-J mcastgroup\tJoin multicast group \n" "-p portnum\tlisten on port portnum\n" "-l basdir \tset the output directory. (no default) \n" "-S subdir\tSub directory format. see nfcapd(1) for format\n" "-I Ident\tset the ident string for stat file. (default 'none')\n" "-H Add port histogram data to flow file.(default 'no')\n" "-n Ident,IP,logdir\tAdd this flow source - multiple streams\n" "-N sourceFile\tAdd flows from sourceFile\n" "-M dir \t\tSet the output directory for dynamic sources.\n" "-P pidfile\tset the PID file\n" "-R IP[/port]\tRepeat incoming packets to IP address/port. Max 8 repeaters.\n" "-s rate\tset default sampling rate (default 1)\n" "-x process\tlaunch process after a new file becomes available\n" "-z\t\tLZO compress flows in output file.\n" "-y\t\tLZ4 compress flows in output file.\n" "-j\t\tBZ2 compress flows in output file.\n" "-B bufflen\tSet socket buffer to bufflen bytes\n" "-e\t\tExpire data at each cycle.\n" "-D\t\tFork to background\n" "-E\t\tPrint extended format of netflow data. For debugging purpose only.\n" "-T\t\tInclude extension tags in records.\n" "-4\t\tListen on IPv4 (default).\n" "-6\t\tListen on IPv6.\n" "-V\t\tPrint version and exit.\n" "-Z\t\tAdd timezone offset to filename.\n" , name); } // End of usage void kill_launcher(int pid) { int stat, i; pid_t ret; if ( pid == 0 ) return; if ( launcher_alive ) { LogInfo("Signal launcher[%i] to terminate.", pid); kill(pid, SIGTERM); // wait for launcher to teminate for ( i=0; i= LAUNCHER_TIMEOUT ) { LogError("Launcher does not want to terminate - signal again"); kill(pid, SIGTERM); sleep(1); } } else { LogError("launcher[%i] already dead.", pid); } if ( (ret = waitpid (pid, &stat, 0)) == -1 ) { LogError("wait for launcher failed: %s %i", strerror(errno), ret); } else { if ( WIFEXITED(stat) ) { LogInfo("launcher exit status: %i", WEXITSTATUS(stat)); } if ( WIFSIGNALED(stat) ) { LogError("launcher terminated due to signal %i", WTERMSIG(stat)); } } } // End of kill_launcher static void IntHandler(int signal) { switch (signal) { case SIGALRM: periodic_trigger = 1; break; case SIGHUP: case SIGINT: case SIGTERM: done = 1; break; case SIGCHLD: launcher_alive = 0; break; default: // ignore everything we don't know break; } } /* End of IntHandler */ static void daemonize(void) { int fd; switch (fork()) { case 0: // child break; case -1: // error fprintf(stderr, "fork() error: %s\n", strerror(errno)); exit(0); break; default: // parent _exit(0); } if (setsid() < 0) { fprintf(stderr, "setsid() error: %s\n", strerror(errno)); exit(0); } // Double fork switch (fork()) { case 0: // child break; case -1: // error fprintf(stderr, "fork() error: %s\n", strerror(errno)); exit(0); break; default: // parent _exit(0); } fd = open("/dev/null", O_RDONLY); if (fd != 0) { dup2(fd, 0); close(fd); } fd = open("/dev/null", O_WRONLY); if (fd != 1) { dup2(fd, 1); close(fd); } fd = open("/dev/null", O_WRONLY); if (fd != 2) { dup2(fd, 2); close(fd); } } // End of daemonize static void SetPriv(char *userid, char *groupid ) { struct passwd *pw_entry; struct group *gr_entry; uid_t myuid, newuid, newgid; int err; if ( userid == 0 && groupid == 0 ) return; newuid = newgid = 0; myuid = getuid(); if ( myuid != 0 ) { LogError("Only root wants to change uid/gid"); fprintf(stderr, "ERROR: Only root wants to change uid/gid\n"); exit(255); } if ( userid ) { pw_entry = getpwnam(userid); newuid = pw_entry ? pw_entry->pw_uid : atol(userid); if ( newuid == 0 ) { fprintf (stderr,"Invalid user '%s'\n", userid); exit(255); } } if ( groupid ) { gr_entry = getgrnam(groupid); newgid = gr_entry ? gr_entry->gr_gid : atol(groupid); if ( newgid == 0 ) { fprintf (stderr,"Invalid group '%s'\n", groupid); exit(255); } err = setgid(newgid); if ( err ) { LogError("Can't set group id %ld for group '%s': %s", (long)newgid, groupid, strerror(errno)); fprintf (stderr,"Can't set group id %ld for group '%s': %s\n", (long)newgid, groupid, strerror(errno)); exit(255); } } if ( newuid ) { err = setuid(newuid); if ( err ) { LogError("Can't set user id %ld for user '%s': %s", (long)newuid, userid, strerror(errno)); fprintf (stderr,"Can't set user id %ld for user '%s': %s\n", (long)newuid, userid, strerror(errno)); exit(255); } } } // End of SetPriv static void format_file_block_header(data_block_header_t *header) { printf("\n" "File Block Header: \n" " NumBlocks = %10u\n" " Size = %10u\n" " id = %10u\n", header->NumRecords, header->size, header->id); } // End of format_file_block_header #include "nffile_inline.c" #include "collector_inline.c" static void run(packet_function_t receive_packet, int socket, repeater_t *repeater, time_t twin, time_t t_begin, int report_seq, int use_subdirs, char *time_extension, int compress) { common_flow_header_t *nf_header; FlowSource_t *fs; struct sockaddr_storage nf_sender; socklen_t nf_sender_size = sizeof(nf_sender); time_t t_start, t_now; uint64_t export_packets; uint32_t blast_cnt, blast_failures, ignored_packets; uint16_t version; ssize_t cnt; void *in_buff; int err; srecord_t *commbuff; if ( !Init_v1(verbose) || !Init_v5_v7_input(verbose, default_sampling, overwrite_sampling) || !Init_v9(verbose, default_sampling, overwrite_sampling) || !Init_IPFIX(verbose, default_sampling, overwrite_sampling) ) return; in_buff = malloc(NETWORK_INPUT_BUFF_SIZE); if ( !in_buff ) { LogError("malloc() allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) ); return; } // init vars commbuff = (srecord_t *)shmem; nf_header = (common_flow_header_t *)in_buff; // Init each netflow source output data buffer fs = FlowSource; while ( fs ) { // prepare file fs->nffile = OpenNewFile(fs->current, NULL, compress, 0, NULL); if ( !fs->nffile ) { return; } // init vars fs->bad_packets = 0; fs->first_seen = 0xffffffffffffLL; fs->last_seen = 0; // next source fs = fs->next; } export_packets = blast_cnt = blast_failures = 0; t_start = t_begin; cnt = 0; periodic_trigger = 0; ignored_packets = 0; // wake up at least at next time slot (twin) + 1s alarm(t_start + twin + 1 - time(NULL)); /* * Main processing loop: * this loop, continues until done = 1, set by the signal handler * The while loop will be breaked by the periodic file renaming code * for proper cleanup */ while ( 1 ) { struct timeval tv; int i; /* read next bunch of data into beginn of input buffer */ if ( !done) { #ifdef PCAP // Debug code to read from pcap file, or from socket cnt = receive_packet(socket, in_buff, NETWORK_INPUT_BUFF_SIZE , 0, (struct sockaddr *)&nf_sender, &nf_sender_size); // in case of reading from file EOF => -2 if ( cnt == -2 ) done = 1; #else cnt = recvfrom (socket, in_buff, NETWORK_INPUT_BUFF_SIZE , 0, (struct sockaddr *)&nf_sender, &nf_sender_size); #endif if ( cnt == -1 && errno != EINTR ) { LogError("ERROR: recvfrom: %s", strerror(errno)); continue; } i = 0; while ( repeater[i].hostname && (i < MAX_REPEATERS)) { ssize_t len; len = sendto(repeater[i].sockfd, in_buff, cnt, 0, (struct sockaddr *)&(repeater[i].addr), repeater[i].addrlen); if ( len < 0 ) { LogError("ERROR: sendto(): %s", strerror(errno)); } i++; } } /* Periodic file renaming, if time limit reached or if we are done. */ // t_now = time(NULL); gettimeofday(&tv, NULL); t_now = tv.tv_sec; if ( ((t_now - t_start) >= twin) || done ) { struct tm *now; char *subdir, fmt[MAXTIMESTRING]; alarm(0); now = localtime(&t_start); strftime(fmt, sizeof(fmt), time_extension, now); // prepare sub dir hierarchy if ( use_subdirs ) { subdir = GetSubDir(now); if ( !subdir ) { // failed to generate subdir path - put flows into base directory LogError("Failed to create subdir path!"); // failed to generate subdir path - put flows into base directory subdir = NULL; } } else { subdir = NULL; } // for each flow source update the stats, close the file and re-initialize the new file fs = FlowSource; while ( fs ) { char nfcapd_filename[MAXPATHLEN]; char error[255]; nffile_t *nffile = fs->nffile; if ( verbose ) { // Dump to stdout format_file_block_header(nffile->block_header); } if ( nffile->block_header->NumRecords ) { // flush current buffer to disc if ( WriteBlock(nffile) <= 0 ) LogError("Ident: %s, failed to write output buffer to disk: '%s'" , fs->Ident, strerror(errno)); } // else - no new records in current block // prepare filename if ( subdir ) { if ( SetupSubDir(fs->datadir, subdir, error, 255) ) { snprintf(nfcapd_filename, MAXPATHLEN-1, "%s/%s/nfcapd.%s", fs->datadir, subdir, fmt); } else { LogError("Ident: %s, Failed to create sub hier directories: %s", fs->Ident, error ); // skip subdir - put flows directly into current directory snprintf(nfcapd_filename, MAXPATHLEN-1, "%s/nfcapd.%s", fs->datadir, fmt); } } else { snprintf(nfcapd_filename, MAXPATHLEN-1, "%s/nfcapd.%s", fs->datadir, fmt); } nfcapd_filename[MAXPATHLEN-1] = '\0'; // update stat record // if no flows were collected, fs->last_seen is still 0 // set first_seen to start of this time slot, with twin window size. if ( fs->last_seen == 0 ) { fs->first_seen = (uint64_t)1000 * (uint64_t)t_start; fs->last_seen = (uint64_t)1000 * (uint64_t)(t_start + twin); } nffile->stat_record->first_seen = fs->first_seen/1000; nffile->stat_record->msec_first = fs->first_seen - nffile->stat_record->first_seen*1000; nffile->stat_record->last_seen = fs->last_seen/1000; nffile->stat_record->msec_last = fs->last_seen - nffile->stat_record->last_seen*1000; // Flush Exporter Stat to file FlushExporterStats(fs); // Close file CloseUpdateFile(nffile, fs->Ident); // if rename fails, we are in big trouble, as we need to get rid of the old .current file // otherwise, we will loose flows and can not continue collecting new flows err = rename(fs->current, nfcapd_filename); if ( err ) { LogError("Ident: %s, Can't rename dump file: %s", fs->Ident, strerror(errno)); LogError("Ident: %s, Serious Problem! Fix manually", fs->Ident); if ( launcher_pid ) commbuff->failed = 1; // we do not update the books here, as the file failed to rename properly // otherwise the books may be wrong } else { struct stat fstat; if ( launcher_pid ) commbuff->failed = 0; // Update books stat(nfcapd_filename, &fstat); UpdateBooks(fs->bookkeeper, t_start, 512*fstat.st_blocks); } // log stats LogInfo("Ident: '%s' Flows: %llu, Packets: %llu, Bytes: %llu, Sequence Errors: %u, Bad Packets: %u", fs->Ident, (unsigned long long)nffile->stat_record->numflows, (unsigned long long)nffile->stat_record->numpackets, (unsigned long long)nffile->stat_record->numbytes, nffile->stat_record->sequence_failure, fs->bad_packets); // reset stats fs->bad_packets = 0; fs->first_seen = 0xffffffffffffLL; fs->last_seen = 0; if ( !done ) { nffile = OpenNewFile(fs->current, nffile, compress, 0, NULL); if ( !nffile ) { LogError("killed due to fatal error: ident: %s", fs->Ident); break; } } // Dump all extension maps and exporters to the buffer FlushStdRecords(fs); // next flow source fs = fs->next; } // end of while (fs) // trigger launcher if required if ( launcher_pid ) { // Signal launcher strncpy(commbuff->tstring, fmt, MAXTIMESTRING); commbuff->tstring[MAXTIMESTRING-1] = '\0'; commbuff->tstamp = t_start; if ( subdir ) { snprintf(commbuff->fname, MAXPATHLEN-1, "%s/nfcapd.%s", subdir, fmt); } else { snprintf(commbuff->fname, MAXPATHLEN-1, "nfcapd.%s", fmt); } commbuff->fname[MAXPATHLEN-1] = '\0'; if ( launcher_alive ) { LogInfo("Signal launcher"); kill(launcher_pid, SIGHUP); } else LogError("ERROR: Launcher died unexpectedly!"); } LogInfo("Total ignored packets: %u", ignored_packets); ignored_packets = 0; if ( done ) break; // update alarm for next cycle t_start += twin; /* t_start = filename time stamp: begin of slot * + twin = end of next time interval * + 1 = act at least 1s after time window expired * - t_now = difference value to now */ alarm(t_start + twin + 1 - t_now); } /* check for error condition or done . errno may only be EINTR */ if ( cnt < 0 ) { if ( periodic_trigger ) { // alarm triggered, no new flow data periodic_trigger = 0; continue; } if ( done ) // signaled to terminate - exit from loop break; else { /* this should never be executed as it should be caught in other places */ LogError("error condition in '%s', line '%d', cnt: %i", __FILE__, __LINE__ ,(int)cnt); continue; } } /* enough data? */ if ( cnt == 0 ) continue; // get flow source record for current packet, identified by sender IP address fs = GetFlowSource(&nf_sender); if ( fs == NULL ) { fs = AddDynamicSource(&FlowSource, &nf_sender); if ( fs == NULL ) { LogError("Skip UDP packet. Ignored packets so far %u packets", ignored_packets); ignored_packets++; continue; } if ( InitBookkeeper(&fs->bookkeeper, fs->datadir, getpid(), launcher_pid) != BOOKKEEPER_OK ) { LogError("Failed to initialise bookkeeper for new source"); // fatal error return; } fs->nffile = OpenNewFile(fs->current, NULL, compress, 0, NULL); if ( !fs->nffile ) { LogError("Failed to open new collector file"); return; } } /* check for too little data - cnt must be > 0 at this point */ if ( cnt < sizeof(common_flow_header_t) ) { LogError("Ident: %s, Data length error: too little data for common netflow header. cnt: %i",fs->Ident, (int)cnt); fs->bad_packets++; continue; } fs->received = tv; /* Process data - have a look at the common header */ version = ntohs(nf_header->version); switch (version) { case 1: Process_v1(in_buff, cnt, fs); break; case 5: // fall through case 7: Process_v5_v7(in_buff, cnt, fs); break; case 9: Process_v9(in_buff, cnt, fs); break; case 10: Process_IPFIX(in_buff, cnt, fs); break; case 255: // blast test header if ( verbose ) { uint16_t count = ntohs(nf_header->count); if ( blast_cnt != count ) { // LogError("Mismatch blast check: Expected %u got %u\n", blast_cnt, count); blast_cnt = count; blast_failures++; } else { blast_cnt++; } if ( blast_cnt == 65535 ) { fprintf(stderr, "Total missed packets: %u\n", blast_failures); done = 1; } break; } default: // data error, while reading data from socket LogError("Ident: %s, Error reading netflow header: Unexpected netflow version %i", fs->Ident, version); fs->bad_packets++; continue; // not reached break; } // each Process_xx function has to process the entire input buffer, therefore it's empty now. export_packets++; // flush current buffer to disc if ( fs->nffile->block_header->size > BUFFSIZE ) { // fishy! - we already wrote into someone elses memory! - I'm sorry // reset output buffer - data may be lost, as we don not know, where it happen fs->nffile->block_header->size = 0; fs->nffile->block_header->NumRecords = 0; fs->nffile->buff_ptr = (void *)((pointer_addr_t)fs->nffile->block_header + sizeof(data_block_header_t) ); LogError("### Software bug ### Ident: %s, output buffer overflow: expect memory inconsitency", fs->Ident); } } if ( verbose && blast_failures ) { fprintf(stderr, "Total missed packets: %u\n", blast_failures); } free(in_buff); fs = FlowSource; while ( fs ) { DisposeFile(fs->nffile); fs = fs->next; } } /* End of run */ int main(int argc, char **argv) { char *bindhost, *datadir, pidstr[32], *launch_process; char *userid, *groupid, *checkptr, *listenport, *mcastgroup, *extension_tags; char *Ident, *dynsrcdir, *time_extension, pidfile[MAXPATHLEN]; struct stat fstat; packet_function_t receive_packet; repeater_t repeater[MAX_REPEATERS]; FlowSource_t *fs; struct sigaction act; int family, bufflen; time_t twin, t_start; int sock, do_daemonize, expire, spec_time_extension, report_sequence; int subdir_index, sampling_rate, compress; int c, i; #ifdef PCAP char *pcap_file = NULL; #endif receive_packet = recvfrom; verbose = do_daemonize = 0; bufflen = 0; family = AF_UNSPEC; launcher_pid = 0; launcher_alive = 0; report_sequence = 0; listenport = DEFAULTCISCOPORT; bindhost = NULL; mcastgroup = NULL; pidfile[0] = 0; launch_process = NULL; userid = groupid = NULL; twin = TIME_WINDOW; datadir = NULL; subdir_index = 0; time_extension = "%Y%m%d%H%M"; spec_time_extension = 0; expire = 0; sampling_rate = 1; compress = NOT_COMPRESSED; memset((void *)&repeater, 0, sizeof(repeater)); for ( i = 0; i < MAX_REPEATERS; i++ ) { repeater[i].family = AF_UNSPEC; } Ident = "none"; FlowSource = NULL; extension_tags = DefaultExtensions; dynsrcdir = NULL; while ((c = getopt(argc, argv, "46ef:whEVI:DB:b:jl:J:M:n:N:p:P:R:S:s:T:t:x:Xru:g:yzZ")) != EOF) { switch (c) { case 'h': usage(argv[0]); exit(0); break; case 'u': userid = optarg; break; case 'g': groupid = optarg; break; case 'e': expire = 1; break; case 'f': { #ifdef PCAP struct stat fstat; pcap_file = optarg; stat(pcap_file, &fstat); if ( !S_ISREG(fstat.st_mode) ) { fprintf(stderr, "Not a regular file: %s\n", pcap_file); exit(254); } #else fprintf(stderr, "PCAP reader not compiled! Option ignored!\n"); #endif } break; case 'E': verbose = 1; Setv6Mode(1); break; case 'V': printf("%s: Version: %s\n",argv[0], nfdump_version); exit(0); break; case 'D': do_daemonize = 1; break; case 'I': Ident = strdup(optarg); break; case 'M': dynsrcdir = strdup(optarg); if ( strlen(dynsrcdir) > MAXPATHLEN ) { fprintf(stderr, "ERROR: Path too long!\n"); exit(255); } if ( stat(dynsrcdir, &fstat) < 0 ) { fprintf(stderr, "stat() failed on %s: %s\n", dynsrcdir, strerror(errno)); exit(255); } if ( !(fstat.st_mode & S_IFDIR) ) { fprintf(stderr, "No such directory: %s\n", dynsrcdir); break; } if ( !SetDynamicSourcesDir(&FlowSource, dynsrcdir) ) { fprintf(stderr, "-l, -M and -n are mutually exclusive\n"); break; } break; case 'n': if ( AddFlowSource(&FlowSource, optarg) != 1 ) exit(255); break; case 'N': if ( AddFlowSourceFromFile(&FlowSource, optarg) ) exit(255); break; case 'w': // allow for compatibility - always sync timeslot break; case 'B': bufflen = strtol(optarg, &checkptr, 10); if ( (checkptr != NULL && *checkptr == 0) && bufflen > 0 ) break; fprintf(stderr,"Argument error for -B\n"); exit(255); case 'b': bindhost = optarg; break; case 'J': mcastgroup = optarg; break; case 'p': listenport = optarg; break; case 'P': if ( optarg[0] == '/' ) { // absolute path given strncpy(pidfile, optarg, MAXPATHLEN-1); } else { // path relative to current working directory char tmp[MAXPATHLEN]; if ( !getcwd(tmp, MAXPATHLEN-1) ) { fprintf(stderr, "Failed to get current working directory: %s\n", strerror(errno)); exit(255); } tmp[MAXPATHLEN-1] = 0; if ( (strlen(tmp) + strlen(optarg) + 3) < MAXPATHLEN ) { snprintf(pidfile, MAXPATHLEN - 3 - strlen(tmp), "%s/%s", tmp, optarg); } else { fprintf(stderr, "pidfile MAXPATHLEN error:\n"); exit(255); } } // pidfile now absolute path pidfile[MAXPATHLEN-1] = 0; break; case 'R': { char *port, *hostname; char *p = strchr(optarg, '/'); int i = 0; if ( p ) { *p++ = '\0'; port = strdup(p); } else { port = DEFAULTCISCOPORT; } hostname = strdup(optarg); while ( repeater[i].hostname && (i < MAX_REPEATERS) ) i++; if ( i == MAX_REPEATERS ) { fprintf(stderr, "Too many packet repeaters! Max: %i repeaters allowed.\n", MAX_REPEATERS); exit(255); } repeater[i].hostname = hostname; repeater[i].port = port; break; } case 'r': report_sequence = 1; break; case 's': // a negative sampling rate is set as the overwrite sampling rate sampling_rate = (int)strtol(optarg, (char **)NULL, 10); if ( (sampling_rate == 0 ) || (sampling_rate < 0 && sampling_rate < -10000000) || (sampling_rate > 0 && sampling_rate > 10000000) ) { fprintf(stderr, "Invalid sampling rate: %s\n", optarg); exit(255); } break; case 'T': { size_t len = strlen(optarg); extension_tags = optarg; if ( len == 0 || len > 128 ) { fprintf(stderr, "Extension length error. Unexpected option '%s'\n", extension_tags); exit(255); } break; } case 'l': datadir = optarg; if ( strlen(datadir) > MAXPATHLEN ) { LogError("ERROR: Path too long!"); exit(255); } if ( stat(datadir, &fstat) < 0 ) { LogError("stat() failed on %s: %s", datadir, strerror(errno)); exit(255); } if ( !(fstat.st_mode & S_IFDIR) ) { LogError("No such directory: %s", datadir); exit(255); } datadir = realpath(datadir, NULL); if ( !datadir ) { LogError("Can not resolve realpath of %s", optarg); exit(255); } break; case 'S': subdir_index = atoi(optarg); break; case 't': twin = atoi(optarg); if ( twin < 2 ) { LogError("time interval <= 2s not allowed"); exit(255); } if (twin < 60) { time_extension = "%Y%m%d%H%M%S"; } break; case 'x': launch_process = optarg; break; case 'j': if ( compress ) { LogError("Use one compression: -z for LZO, -j for BZ2 or -y for LZ4 compression\n"); exit(255); } compress = BZ2_COMPRESSED; break; case 'y': if ( compress ) { LogError("Use one compression: -z for LZO, -j for BZ2 or -y for LZ4 compression\n"); exit(255); } compress = LZ4_COMPRESSED; break; case 'z': if ( compress ) { LogError("Use one compression: -z for LZO, -j for BZ2 or -y for LZ4 compression\n"); exit(255); } compress = LZO_COMPRESSED; break; case 'Z': time_extension = "%Y%m%d%H%M%z"; spec_time_extension = 1; break; case '4': if ( family == AF_UNSPEC ) family = AF_INET; else { fprintf(stderr, "ERROR, Accepts only one protocol IPv4 or IPv6!\n"); exit(255); } break; case '6': if ( family == AF_UNSPEC ) family = AF_INET6; else { fprintf(stderr, "ERROR, Accepts only one protocol IPv4 or IPv6!\n"); exit(255); } break; default: usage(argv[0]); exit(255); } } if ( FlowSource == NULL && datadir == NULL && dynsrcdir == NULL ) { fprintf(stderr, "ERROR, Missing -n (-l/-I) or -M source definitions\n"); exit(255); } if ( FlowSource == NULL && datadir != NULL && !AddDefaultFlowSource(&FlowSource, Ident, datadir) ) { fprintf(stderr, "Failed to add default data collector directory\n"); exit(255); } if ( bindhost && mcastgroup ) { fprintf(stderr, "ERROR, -b and -j are mutually exclusive!!\n"); exit(255); } if ( !InitLog(do_daemonize, argv[0], SYSLOG_FACILITY, verbose) ) { exit(255); } if ( expire && spec_time_extension ) { fprintf(stderr, "ERROR, -Z timezone extension breaks expire -e\n"); exit(255); } InitExtensionMaps(NO_EXTENSION_LIST); SetupExtensionDescriptors(strdup(extension_tags)); // Debug code to read from pcap file #ifdef PCAP sock = 0; if ( pcap_file ) { printf("Setup pcap reader\n"); setup_packethandler(pcap_file, NULL); receive_packet = NextPacket; } else #endif if ( mcastgroup ) sock = Multicast_receive_socket (mcastgroup, listenport, family, bufflen); else sock = Unicast_receive_socket(bindhost, listenport, family, bufflen ); if ( sock == -1 ) { fprintf(stderr,"Terminated due to errors.\n"); exit(255); } i = 0; while ( repeater[i].hostname && (i < MAX_REPEATERS) ) { repeater[i].sockfd = Unicast_send_socket (repeater[i].hostname, repeater[i].port, repeater[i].family, bufflen, &repeater[i].addr, &repeater[i].addrlen ); if ( repeater[i].sockfd <= 0 ) exit(255); LogInfo("Replay flows to host: %s port: %s", repeater[i].hostname, repeater[i].port); i++; } if ( sampling_rate < 0 ) { default_sampling = -sampling_rate; overwrite_sampling = default_sampling; } else { default_sampling = sampling_rate; } SetPriv(userid, groupid); if ( subdir_index && !InitHierPath(subdir_index) ) { close(sock); exit(255); } // check if pid file exists and if so, if a process with registered pid is running if ( strlen(pidfile) ) { int pidf; pidf = open(pidfile, O_RDONLY, 0); if ( pidf > 0 ) { // pid file exists char s[32]; ssize_t len; len = read(pidf, (void *)s, 31); close(pidf); s[31] = '\0'; if ( len < 0 ) { fprintf(stderr, "read() error existing pid file: %s\n", strerror(errno)); exit(255); } else { unsigned long pid = atol(s); if ( pid == 0 ) { // garbage - use this file unlink(pidfile); } else { if ( kill(pid, 0) == 0 ) { // process exists fprintf(stderr, "A process with pid %lu registered in pidfile %s is already running!\n", pid, strerror(errno)); exit(255); } else { // no such process - use this file unlink(pidfile); } } } } else { if ( errno != ENOENT ) { fprintf(stderr, "open() error existing pid file: %s\n", strerror(errno)); exit(255); } // else errno == ENOENT - no file - this is fine } } if (argc - optind > 1) { usage(argv[0]); close(sock); exit(255); } t_start = time(NULL); t_start = t_start - ( t_start % twin); if ( do_daemonize ) { verbose = 0; daemonize(); } if (strlen(pidfile)) { pid_t pid = getpid(); int pidf = open(pidfile, O_RDWR|O_TRUNC|O_CREAT, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); if ( pidf == -1 ) { LogError("Error opening pid file: '%s' %s", pidfile, strerror(errno)); close(sock); exit(255); } snprintf(pidstr,31,"%lu\n", (unsigned long)pid); if ( write(pidf, pidstr, strlen(pidstr)) <= 0 ) { LogError("Error write pid file: '%s' %s", pidfile, strerror(errno)); } close(pidf); } done = 0; if ( launch_process || expire ) { // create laucher comm memory struct shmem = mmap(0, sizeof(srecord_t), PROT_READ | PROT_WRITE, MAP_ANON | MAP_SHARED, -1, 0); if ( shmem == MAP_FAILED) { LogError("mmap() error in %s:%i: %s", __FILE__, __LINE__ , strerror(errno)); close(sock); exit(255); } launcher_pid = fork(); switch (launcher_pid) { case 0: // child close(sock); launcher(shmem, FlowSource, launch_process, expire); _exit(0); break; case -1: LogError("fork() error: %s", strerror(errno)); if ( strlen(pidfile) ) unlink(pidfile); exit(255); break; default: // parent launcher_alive = 1; LogInfo("Launcher[%i] forked", launcher_pid); } } fs = FlowSource; while ( fs ) { if ( InitBookkeeper(&fs->bookkeeper, fs->datadir, getpid(), launcher_pid) != BOOKKEEPER_OK ) { LogError("initialize bookkeeper failed."); // release all already allocated bookkeepers fs = FlowSource; while ( fs && fs->bookkeeper ) { ReleaseBookkeeper(fs->bookkeeper, DESTROY_BOOKKEEPER); fs = fs->next; } close(sock); if ( launcher_pid ) kill_launcher(launcher_pid); if ( strlen(pidfile) ) unlink(pidfile); exit(255); } // Init the extension map list if ( !InitExtensionMapList(fs) ) { // error message goes to syslog exit(255); } fs = fs->next; } /* Signal handling */ memset((void *)&act,0,sizeof(struct sigaction)); act.sa_handler = IntHandler; sigemptyset(&act.sa_mask); act.sa_flags = 0; sigaction(SIGTERM, &act, NULL); sigaction(SIGINT, &act, NULL); sigaction(SIGHUP, &act, NULL); sigaction(SIGALRM, &act, NULL); sigaction(SIGCHLD, &act, NULL); LogInfo("Startup."); run(receive_packet, sock, repeater, twin, t_start, report_sequence, subdir_index, time_extension, compress); close(sock); kill_launcher(launcher_pid); fs = FlowSource; while ( fs && fs->bookkeeper ) { dirstat_t *dirstat; // if we do not auto expire and there is a stat file, update the stats before we leave if ( expire == 0 && ReadStatInfo(fs->datadir, &dirstat, LOCK_IF_EXISTS) == STATFILE_OK ) { UpdateBookStat(dirstat, fs->bookkeeper); WriteStatInfo(dirstat); LogInfo("Updating statinfo in directory '%s'", datadir); } ReleaseBookkeeper(fs->bookkeeper, DESTROY_BOOKKEEPER); fs = fs->next; } LogInfo("Terminating nfcapd."); EndLog(); free(datadir); if ( strlen(pidfile) ) unlink(pidfile); return 0; } /* End of main */ nfdump-1.6.23/bin/nfdump.c000066400000000000000000001114621404501030700153000ustar00rootroot00000000000000/* * Copyright (c) 2009-2020, Peter Haag * Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * * Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * Neither the name of the author nor the names of its contributors may be * used to endorse or promote products derived from this software without * specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ #include "config.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_STDINT_H #include #endif #include "util.h" #include "nfdump.h" #include "nffile.h" #include "nfx.h" #include "flist.h" #include "nfnet.h" #include "bookkeeper.h" #include "collector.h" #include "exporter.h" #include "output_util.h" #include "output_raw.h" #include "output_pipe.h" #include "output_csv.h" #include "output_json.h" #include "netflow_v5_v7.h" #include "netflow_v9.h" #include "nftree.h" #include "nfprof.h" #include "nflowcache.h" #include "nfstat.h" #include "nfexport.h" #include "ipconv.h" /* hash parameters */ #define NumPrealloc 128000 #define AGGR_SIZE 7 /* Global Variables */ FilterEngine_t *Engine; extern char *FilterFilename; extern uint32_t loopcnt; /* Local Variables */ const char *nfdump_version = VERSION; static uint64_t total_bytes; static uint32_t recordCount; static uint32_t skipped_blocks; static uint32_t is_anonymized; static time_t t_first_flow, t_last_flow; static char Ident[IDENTLEN]; int hash_hit = 0; int hash_miss = 0; int hash_skip = 0; extension_map_list_t *extension_map_list; extern exporter_t **exporter_list; /* * Output Formats: * User defined output formats can be compiled into nfdump, for easy access * The format has the same syntax as describe in nfdump(1) -o fmt: * * A format description consists of a single line containing arbitrary strings * and format specifier as described below: * * %ts // Start Time - first seen * %te // End Time - last seen * %td // Duration * %pr // Protocol * %sa // Source Address * %da // Destination Address * %sap // Source Address:Port * %dap // Destination Address:Port * %sp // Source Port * %dp // Destination Port * %nh // Next-hop IP Address * %nhb // BGP Next-hop IP Address * %sas // Source AS * %das // Destination AS * %in // Input Interface num * %out // Output Interface num * %pkt // Packets - default input * %ipkt // Input Packets * %opkt // Output Packets * %byt // Bytes - default input * %ibyt // Input Bytes * %obyt // Output Bytes * %fl // Flows * %flg // TCP Flags * %tos // Tos - Default src * %stos // Src Tos * %dtos // Dst Tos * %dir // Direction: ingress, egress * %smk // Src mask * %dmk // Dst mask * %fwd // Forwarding Status * %svln // Src Vlan * %dvln // Dst Vlan * %ismc // Input Src Mac Addr * %odmc // Output Dst Mac Addr * %idmc // Output Src Mac Addr * %osmc // Input Dst Mac Addr * %mpls1 // MPLS label 1 * %mpls2 // MPLS label 2 * %mpls3 // MPLS label 3 * %mpls4 // MPLS label 4 * %mpls5 // MPLS label 5 * %mpls6 // MPLS label 6 * %mpls7 // MPLS label 7 * %mpls8 // MPLS label 8 * %mpls9 // MPLS label 9 * %mpls10 // MPLS label 10 * * %bps // bps - bits per second * %pps // pps - packets per second * %bpp // bps - Bytes per package * * The nfdump standard output formats line, long and extended are defined as follows: */ #define FORMAT_line "%ts %td %pr %sap -> %dap %pkt %byt %fl" #define FORMAT_long "%ts %td %pr %sap -> %dap %flg %tos %pkt %byt %fl" #define FORMAT_extended "%ts %td %pr %sap -> %dap %flg %tos %pkt %byt %pps %bps %bpp %fl" #define FORMAT_biline "%ts %td %pr %sap <-> %dap %opkt %ipkt %obyt %ibyt %fl" #define FORMAT_bilong "%ts %td %pr %sap <-> %dap %flg %tos %opkt %ipkt %obyt %ibyt %fl" #define FORMAT_nsel "%ts %evt %xevt %pr %sap -> %dap %xsap -> %xdap %ibyt %obyt" #define FORMAT_nel "%ts %nevt %pr %sap -> %dap %nsap -> %ndap" #ifdef NSEL # define DefaultMode "nsel" #else # define DefaultMode "line" #endif /* The appropriate header line is compiled automatically. * * For each defined output format a v6 long format automatically exists as well e.g. * line -> line6, long -> long6, extended -> extended6 * v6 long formats need more space to print IP addresses, as IPv6 addresses are printed in full length, * where as in standard output format IPv6 addresses are condensed for better readability. * * Define your own output format and compile it into nfdumnp: * 1. Define your output format string. * 2. Test the format using standard syntax -o "fmt:" * 3. Create a #define statement for your output format, similar than the standard output formats above. * 4. Add another line into the printmap[] struct below BEFORE the last NULL line for you format: * { "formatname", format_special, FORMAT_definition, NULL }, * The first parameter is the name of your format as recognized on the command line as -o * The second parameter is always 'format_special' - the printing function. * The third parameter is your format definition as defined in #define. * The forth parameter is always NULL for user defined formats. * 5. Recompile nfdump */ static void flow_record_to_null(void *record, char ** s, int tag); // Assign print functions for all output options -o // Teminated with a NULL record printmap_t printmap[] = { { "raw", flow_record_to_raw, raw_prolog, raw_epilog, NULL }, { "line", format_special, text_prolog, text_epilog, FORMAT_line }, { "long", format_special, text_prolog, text_epilog, FORMAT_long }, { "extended", format_special, text_prolog, text_epilog, FORMAT_extended }, { "biline", format_special, text_prolog, text_epilog, FORMAT_biline }, { "bilong", format_special, text_prolog, text_epilog, FORMAT_bilong }, { "pipe", flow_record_to_pipe, pipe_prolog, pipe_epilog, NULL }, { "json", flow_record_to_json, json_prolog, json_epilog, NULL }, { "csv", flow_record_to_csv, csv_prolog, csv_epilog, NULL }, { "null", flow_record_to_null, text_prolog, text_epilog, NULL }, #ifdef NSEL { "nsel", format_special, text_prolog, text_epilog, FORMAT_nsel }, { "nel", format_special, text_prolog, text_epilog, FORMAT_nel }, #endif // add your formats here // This is always the last line { NULL, NULL, NULL, NULL, "" } }; // For automatic output format generation in case of custom aggregation #define AggrPrependFmt "%ts %td " #define AggrAppendFmt "%pkt %byt %bps %bpp %fl" // compare at most 16 chars #define MAXMODELEN 16 /* Function Prototypes */ static void usage(char *name); static void PrintSummary(stat_record_t *stat_record, outputParams_t *outputParams); static stat_record_t process_data(char *wfile, int element_stat, int flow_stat, int sort_flows, printer_t print_record, time_t twin_start, time_t twin_end, uint64_t limitRecords, outputParams_t *outputParams, int compress); /* Functions */ #include "nfdump_inline.c" #include "nffile_inline.c" static void usage(char *name) { printf("usage %s [options] [\"filter\"]\n" "-h\t\tthis text you see right here\n" "-V\t\tPrint version and exit.\n" "-a\t\tAggregate netflow data.\n" "-A [/net]\tHow to aggregate: ',' sep list of tags see nfdump(1)\n" "\t\tor subnet aggregation: srcip4/24, srcip6/64.\n" "-b\t\tAggregate netflow records as bidirectional flows.\n" "-B\t\tAggregate netflow records as bidirectional flows - Guess direction.\n" "-r \tread input from file\n" "-w \twrite output to file\n" "-f\t\tread netflow filter from file\n" "-n\t\tDefine number of top N for stat or sorted output.\n" "-c\t\tLimit number of matching records\n" "-D \tUse nameserver for host lookup.\n" "-N\t\tPrint plain numbers\n" "-s [/]\tGenerate statistics for any valid record element.\n" "\t\tand ordered by : packets, bytes, flows, bps pps and bpp.\n" "-q\t\tQuiet: Do not print the header and bottom stat lines.\n" "-i \tChange Ident to in file given by -r.\n" "-J \tModify file compression: 0: uncompressed - 1: LZO - 2: BZ2 - 3: LZ4 compressed.\n" "-z\t\tLZO compress flows in output file. Used in combination with -w.\n" "-y\t\tLZ4 compress flows in output file. Used in combination with -w.\n" "-j\t\tBZ2 compress flows in output file. Used in combination with -w.\n" "-l \tSet limit on packets for line and packed output format.\n" "\t\tkey: 32 character string or 64 digit hex string starting with 0x.\n" "-L \tSet limit on bytes for line and packed output format.\n" "-I \t\tPrint netflow summary statistics info from file, specified by -r.\n" "-M \tRead input from multiple directories.\n" "\t\t/dir/dir1:dir2:dir3 Read the same files from '/dir/dir1' '/dir/dir2' and '/dir/dir3'.\n" "\t\trequests either -r filename or -R firstfile:lastfile without pathnames\n" "-m\t\tdeprecated\n" "-O Sort order for aggregated flows - tstart, tend, flows, packets bps pps bbp etc.\n" "-R \tRead input from sequence of files.\n" "\t\t/any/dir Read all files in that directory.\n" "\t\t/dir/file Read all files beginning with 'file'.\n" "\t\t/dir/file1:file2: Read all files from 'file1' to file2.\n" "-o \tUse to print out netflow records:\n" "\t\t raw Raw record dump.\n" "\t\t line Standard output line format.\n" "\t\t long Standard output line format with additional fields.\n" "\t\t extended Even more information.\n" "\t\t csv ',' separated, machine parseable output format.\n" "\t\t json json output format.\n" "\t\t pipe '|' separated legacy machine parseable output format.\n" "\t\t null no flow records, but statistics output.\n" "\t\t\tmode may be extended by '6' for full IPv6 listing. e.g.long6, extended6.\n" "-E \tPrint exporter and sampling info for collected flows.\n" "-v \tverify netflow data file. Print version and blocks.\n" "-x \tverify extension records in netflow data file.\n" "-X\t\tDump Filtertable and exit (debug option).\n" "-Z\t\tCheck filter syntax and exit.\n" "-t