pax_global_header00006660000000000000000000000064146224765130014524gustar00rootroot0000000000000052 comment=52fd91350a26d21ab9b16c601fe94753273103ac dnsdbq-2.6.7/000077500000000000000000000000001462247651300130135ustar00rootroot00000000000000dnsdbq-2.6.7/.gitignore000066400000000000000000000002721462247651300150040ustar00rootroot00000000000000# Object files *.o *.ko # Libraries *.lib *.a # Shared objects (inc. Windows DLLs) *.dll *.so *.so.* *.dylib # Executables *.exe *.out *.app dnsdbq valgrind.suppress dnsdbq.core TAGS dnsdbq-2.6.7/Apache-2.0000066400000000000000000000261361462247651300144240ustar00rootroot00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. dnsdbq-2.6.7/Makefile000066400000000000000000000064531462247651300144630ustar00rootroot00000000000000# # Copyright (c) 2014-2020 by Farsight Security, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Base directory for jansson header and libraries JANSBASE=/usr/local # For macOS on M1, use this instead of the above line: #JANSBASE=/opt/homebrew JANSINCL = -I$(JANSBASE)/include JANSLIBS = -L$(JANSBASE)/lib -ljansson # For almost static builds on macOS, use this instead of the above line: #JANSLIBS = $(JANSBASE)/lib/libjansson.a CURLINCL = `curl-config --cflags` CURLLIBS = `[ ! -z "$$(curl-config --libs)" ] && curl-config --libs || curl-config --static-libs` CWARN =-W -Wall -Wextra -Wcast-qual -Wpointer-arith -Wwrite-strings \ -Wmissing-prototypes -Wbad-function-cast -Wnested-externs \ -Wunused -Wshadow -Wmissing-noreturn -Wswitch-enum -Wconversion # try shipping without any warnings CWARN +=-Werror # warning about bad indentation, only for clang 6.x+ #CWARN +=-Werror=misleading-indentation CDEFS = -DWANT_PDNS_DNSDB=1 -DWANT_PDNS_CIRCL=1 CGPROF = CDEBUG = -g -O3 CFLAGS += $(CGPROF) $(CDEBUG) $(CWARN) $(CDEFS) INCL= $(CURLINCL) $(JANSINCL) LIBS= $(CURLLIBS) $(JANSLIBS) -lresolv # For freebsd, it requires that -lresolv _not_ be used here, use this instead of the above line: #LIBS= $(CURLLIBS) $(JANSLIBS) TOOL = dnsdbq TOOL_OBJ = $(TOOL).o ns_ttl.o netio.o \ pdns.o pdns_circl.o pdns_dnsdb.o \ sort.o time.o asinfo.o deduper.o \ tokstr.o TOOL_SRC = $(TOOL).c ns_ttl.c netio.c \ pdns.c pdns_circl.c pdns_dnsdb.c \ sort.c time.c asinfo.c deduper.c \ tokstr.c all: $(TOOL) install: all rm -f /usr/local/bin/$(TOOL) mkdir -p /usr/local/bin cp $(TOOL) /usr/local/bin/$(TOOL) rm -f /usr/local/share/man/man1/$(TOOL).1 mkdir -p /usr/local/share/man/man1 cp $(TOOL).man /usr/local/share/man/man1/$(TOOL).1 clean: rm -f $(TOOL) rm -f $(TOOL_OBJ) dnsdbq: $(TOOL_OBJ) Makefile $(CC) $(CDEBUG) -o $(TOOL) $(CGPROF) $(TOOL_OBJ) $(LIBS) .c.o: $(CC) $(CFLAGS) $(INCL) -c $< $(TOOL_OBJ): Makefile # BSD only depend: mkdep $(CURLINCL) $(JANSINCL) $(CDEFS) $(TOOL_SRC) # these were made by mkdep on BSD but are now staticly edited deduper.o: deduper.c deduper.h asinfo.o: asinfo.c \ asinfo.h globals.h defs.h sort.h pdns.h netio.h dnsdbq.o: dnsdbq.c \ defs.h netio.h \ pdns.h tokstr.h \ pdns_dnsdb.h pdns_circl.h sort.h \ time.h globals.h ns_ttl.o: ns_ttl.c \ ns_ttl.h netio.o: netio.c \ defs.h netio.h \ pdns.h \ globals.h sort.h pdns.o: pdns.c defs.h \ asinfo.h \ netio.h \ pdns.h \ time.h \ globals.h sort.h tokstr.h pdns_circl.o: pdns_circl.c \ defs.h \ pdns.h \ netio.h \ pdns_circl.h globals.h sort.h pdns_dnsdb.o: pdns_dnsdb.c \ defs.h \ pdns.h \ netio.h \ pdns_dnsdb.h time.h globals.h sort.h sort.o: sort.c \ defs.h sort.h pdns.h \ netio.h \ globals.h time.o: time.c \ defs.h time.h \ globals.h sort.h pdns.h \ netio.h \ ns_ttl.h tokstr.o: tokstr.c \ tokstr.h dnsdbq-2.6.7/README000066400000000000000000000206131462247651300136750ustar00rootroot00000000000000/* * Copyright (c) 2014-2021 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ Table of Contents: * Introduction * Dependencies needed * Installing dependencies * Building and installing * Getting Started * Background on ASINFO/CIDR lookups and annotations Introduction: This is a pure C99 program that accesses passive DNS database systems such as: * the DNSDB API server at Farsight Security * the CIRCL pDNS server at Computer Incident Response Center (LU) An API key is required for operation. The command syntax was inspired by a python script called dnsdb_query, but significant departure has occured, largely inspired by a modern understanding of "time fencing" and a desire for new features such as CSV output and JSON reprocessing. NOTE: Prior to version 2.5.3, the default pDNS system supported was Farsight Security APIv1, and it was called "dnsdb". As of version 2.5.3, the default system is Farsight Security APIv2 and system "dnsdb2" is synonymous with "dnsdb". For APIv1, specify "dnsdb1" using the command line -u option or the DNSDBQ_SYSTEM variable. Dependencies needed: jansson (2.5 or later) libcurl (7.28 or later) modern compiler (clang or GCC) Installing dependencies: On Debian 8 Linux: apt-get install libcurl4-openssl-dev apt-get install libjansson-dev On CentOS 6 Linux: # Based on PHP instructions for installing libcurl... wget http://curl.haxx.se/download/curl-7.28.1.tar.gz tar xvzf curl-7.28.1.tar.gz cd curl-7.28.1/ ./configure --with-libssh2 --enable-ares=/usr/local/ --enable-ipv6 make make install # lib jansson wget http://www.digip.org/jansson/releases/jansson-2.5.tar.gz tar -xpzf jansson-2.5.tar.gz cd jansson-2.5 ./configure make make install echo /usr/local/lib >> /etc/ld.so.conf.d/local.conf ldconfig On CentOS 7 Linux: yum install libcurl-devel.x86_64 yum install jansson-devel.x86_64 yum install centos-release-scl yum install devtoolset-8 scl enable devtoolset-8 bash # do the build from that bash On CentOS / AlmaLinux / Rocky Linux 8: dnf install gcc jansson.x86_64 jansson-devel.x86_64 libcurl-devel.x86_64 On FreeBSD 10: pkg install curl jansson On macOS: brew install jansson On Amazon Linux 2023 (AWS Ec2): # A plain vanilla Amazon Linux AMI comes with with curl-minimal, and 'make' will # fail with an error that curl/curl.h is not found. Installing the following packages # will allow dnsdbq to compile on a fresh Amazon Linux 2023 AMI: yum install make yum install git yum install gcc yum install libcurl-devel yum install jansson-devel Building and installing: (Assumes you have "git") git clone https://github.com/dnsdb/dnsdbq.git cd dnsdbq make install clean On FreeBSD, you may need to remove -lresolv in the LIBS line of the Makefile. On macOS on Apple M1 processors, Homebrew now defaults to be installed in /opt/homebrew instead of /usr/local. If that is the case on your system, in the Makefile, uncomment the line #JANSBASE=/opt/homebrew On macOS, if you want an almost static dnsdbq binary on macOS, that is, one with minimal non-System library dependencies, you can rebuild dnsdbq with a static jansson library. That binary could then be deployed on any identical macOS version and architecture. 1. Find the static jansson library, probably as installed by brew /usr/local/lib/libjansson.a or /opt/homebrew/lib/libjansson.a 2. Change the Makefile's line JANSLIBS = -L$(JANSBASE)/lib -ljansson to instead specify the static library location, probably to: JANSLIBS = $(JANSBASE)/lib/libjansson.a 3. Then run make Getting Started: Add the API key to ~/.dnsdb-query.conf in the below given format, APIKEY="YOURAPIKEYHERE" If you're interested in purchasing a Farsight DNSDB subscription, please contact sales@farsightsecurity.com. Farsight also has a grant program for selected researchers, investigative journalists, and cybersecurity workers at some public benefit non-profits. See https://www.farsightsecurity.com/grant-access/ Here's an example query and output after dnsdbq is compiled: $ ./dnsdbq -r farsightsecurity.com/A -l 1 ;; record times: 2013-09-25 15:37:03 .. 2015-04-01 06:17:25 (~1y ~187d) ;; count: 6350; bailiwick: farsightsecurity.com. farsightsecurity.com. A 66.160.140.81 Background on ASINFO/CIDR lookups and annotations: Annotating IP addresses with ASN information can help an analyst focus their attention on unusual or unexpected ASNs (for example, perhaps a domestic US corporation's IP address inexplicably ended up being originated by a foreign consumer ISP). This code has been tested against three sources of ASN information, each of which are valid arguments to the -D parameter. 1. asn.routeviews.org (the default value for the -D parameter). Given an IPv4 in reverse order, this returns a space separated three-tuple: "ASN" "CIDR prefix" "prefix length" For badly formatted addresses, it returns: "4294967295" "0" "0" For unknown addresses, it returns no answers. Examples: $ dig +short 34.168.254.125.asn.routeviews.org TXT "23724" "125.254.168.0" "21" $ dig +short a.b.c.d.asn.routeviews.org TXT "4294967295" "0" "0" $ dig +short 0.0.0.128.aspath.routeviews.org TXT $ 2. aspath.routeviews.org The same as asn.routeviews.org, except it returns a three-tuple: AS path, CIDR prefix, prefix length. Examples: $ dig +short 0.0.0.4.aspath.routeviews.org TXT "3303 3356" "4.0.0.0" "9" 3. origin.asn.cymru.com Given an IPv4 in reverse order, this returns a pipe-separated five-tuple: ASN | CIDR prefix/prefix length | CC | Registry | Allocated date Example: $ dig +short 0.0.0.4.origin.asn.cymru.com TXT "3356 | 4.0.0.0/9 | US | arin | 1992-12-01" Function asinfo_from_dns() in asinfo.c has specific code to parse those formats. asn.routeviews.org and aspath.routeviews.org do not currently handle IPv6 addresses properly, so dnsdbq does not support IPv6 annotation now. There is a complication that arises when doing those sort of IP to ASN mappings, however: publicly available IP to ASN zones are based on currently observed IP to ASN mappings, while the IPs that are being mapped may have been seen in passive DNS months or even years earlier, when that IP may have been originated by a different ASN. Often the IP to ASN mappings are quite static, in which case historical IPs will map just fine using the current IP to ASN data. On the other hand, some IPs may have been hijacked and used without authorization, or transferred, or otherwise ended up going from one ASN to another. Therefore the IP to ASN mapping should be viewed as an experimental best effort feature, and interpreted with care. In addition to the issue of potential ASN misalignment, the size and origin of the reported origin IP and prefix length may also have changed over time. For example, a /19 may have been de-aggregated into a set of more specific /24's. Again, we report the state of the world as it is currently seen by the service used for the IP to ASN mapping. Other miscellaneous notes: * dnsdbq does not support mapping IPv6 addresses to ASNs at this time. * In the case of Multiple Origin ASNs, typically IP to ASN services will report one of the multiple ASNs; other origin ASNs may also exist but not be reported. * We do not map ASNs to their owner or the owner description string. Please see WHOIS for information about the entity currently assigned a given ASN. We welcome feedback on this feature. Do you use it and find it useful? Are there IP-to-ASN service providers we've inadvertently omitted? Share your feedback by writing . dnsdbq-2.6.7/asinfo.c000066400000000000000000000215671462247651300144510ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ /* Note that cygwin has a crippled libresolv that does not * include the not so recent ns_initparse() function, etc. * This AS info functionality is thus not available * in cygwin. */ /* external. */ /* asprintf() does not appear on linux without this */ #define _GNU_SOURCE #include #include #include #include #include #include #include #include #include "globals.h" #ifndef CRIPPLED_LIBC /* must be after globals.h - which includes defs.h */ #include "asinfo.h" /* private. */ static struct __res_state res; /* forward. */ static char *asinfo_from_ipv4(const char *, char **, char **); #ifdef asinfo_ipv6 static const char *asinfo_from_ipv6(const char *, char **, char **); #endif static char *asinfo_from_dns(const char *, char **, char **); static const char *keep_best(char **, char **, char *, char *); /* public. */ /* asinfo_from_rr(rrtype, rdata, asnum, cidr) -- find ASINFO for A/AAAA string * * return NULL on success, or else, reason (malloc'd string) for failure. * * side effect: on success, *asnum and *cidr will be heap-allocated strings. */ char * asinfo_from_rr(const char *rrtype, const char *rdata, char **asnum, char **cidr) { if (asinfo_lookup) { if (strcmp(rrtype, "A") == 0) return asinfo_from_ipv4(rdata, asnum, cidr); #ifdef asinfo_ipv6 if (strcmp(rrtype, "AAAA") == 0) return asinfo_from_ipv6(rdata, asnum, cidr); #endif } return NULL; } /* asinfo_domain_exists(domain) -- verify DNS-level existence of a domain * * return boolean -- does this domain exist in some form? */ bool asinfo_domain_exists(const char *domain) { u_char buf[NS_PACKETSZ]; return res_query(domain, ns_c_in, ns_t_txt, buf, sizeof buf) > 0 || _res.res_h_errno != HOST_NOT_FOUND; } /* asinfo_shutdown() -- deallocate underlying library's heap resources */ void asinfo_shutdown(void) { if ((res.options & RES_INIT) != 0) res_nclose(&res); } /* static. */ /* asinfo_from_ipv4(addr, asnum, cidr) -- prepare and use ASINFO IPv4 name * * return NULL on success, or else, reason (malloc'd string) for failure. * * side effect: on success, *asnum and *cidr will be heap-allocated strings. */ static char * asinfo_from_ipv4(const char *addr, char **asnum, char **cidr) { u_char a4[32/8]; char *dname; if (inet_pton(AF_INET, addr, a4) < 0) return strdup(strerror(errno)); int n = asprintf(&dname, "%d.%d.%d.%d.%s", a4[3], a4[2], a4[1], a4[0], asinfo_domain); if (n < 0) return strdup(strerror(errno)); char *result = asinfo_from_dns(dname, asnum, cidr); free(dname); return result; } #ifdef asinfo_ipv6 /* asinfo_from_ipv6(addr, asnum, cidr) -- prepare and use ASINFO IPv6 name * * return NULL on success, or else, reason (malloc'd string) for failure. * * side effect: on success, *asnum and *cidr will be heap-allocated strings. * * NOTE WELL: this is a placeholder, since no ASINFO source has working IPv6. */ static char * asinfo_from_ipv6(const char *addr, char **asnum, char **cidr) { char *result, *dname, *p; u_char a6[128/8]; int i; if (inet_pton(AF_INET6, addr, &a6) < 0) return strdup(strerror(errno)); dname = malloc(strlen(asinfo_domain) + (128/4)*2); if (dname == NULL) return strdup(strerror(errno)); result = NULL; p = dname; for (i = (128/8) - 1; i >= 0; i--) { int n = sprintf(p, "%x.%x.", a6[i] & 0xf, a6[i] >> 4); if (n < 0) { result = strdup(strerror(errno)); break; } p += n; } if (result == NULL) { strcpy(p, asinfo_domain); result = asinfo_from_dns(dname, asnum, cidr); } p = NULL; free(dname); return result; } #endif /* asinfo_from_dns(dname, asnum, cidr) -- retrieve and parse a ASINFO DNS TXT * * return NULL on success, or else, reason (malloc'd string) for failure. * * side effect: on success, *asnum and *cidr will be heap-allocated strings. */ static char * asinfo_from_dns(const char *dname, char **asnum, char **cidr) { u_char buf[NS_PACKETSZ]; int n, an, rrn, rcode; char *result; ns_msg msg; ns_rr rr; DEBUG(1, true, "asinfo_from_dns(%s)\n", dname); if ((res.options & RES_INIT) == 0) res_ninit(&res); n = res_nquery(&res, dname, ns_c_in, ns_t_txt, buf, sizeof buf); if (n < 0) { if (res.res_h_errno == HOST_NOT_FOUND) return NULL; else return strdup(hstrerror(res.res_h_errno)); } if (ns_initparse(buf, n, &msg) < 0) return strdup(strerror(errno)); rcode = ns_msg_getflag(msg, ns_f_rcode); if (rcode != ns_r_noerror) { if (asprintf(&result, "DNS RCODE 0x%x", rcode) < 0) return strdup(strerror(errno)); return result; } an = ns_msg_count(msg, ns_s_an); if (an == 0) return strdup("ANCOUNT == 0"); /* some ASINFO data sources return multiple TXT RR's, each having * a prefix length measured in bits. we will select the best * (longest match) prefix offered. */ for (result = NULL, rrn = 0; result == NULL && rrn < an; rrn++) { const u_char *rdata; int rdlen, ntxt; char *txt[3]; if (ns_parserr(&msg, ns_s_an, rrn, &rr) < 0) { result = strdup(strerror(errno)); break; } if (ns_rr_type(rr) != ns_t_txt) goto next_rr; rdata = ns_rr_rdata(rr); rdlen = ns_rr_rdlen(rr); ntxt = 0; while (rdlen > 0) { /* no current ASINFO source has a TXT schema having * more than three TXT segments (). */ if (ntxt == 3) { result = strdup("len(TXT[]) > 3"); break; } n = *rdata++; rdlen--; if (n > rdlen) { result = strdup("TXT overrun"); break; } txt[ntxt] = strndup((const char *)rdata, (size_t)n); if (txt[ntxt] == NULL) { result = strdup("strndup FAIL"); break; } DEBUG(2, true, "TXT[%d] \"%s\"\n", ntxt, txt[ntxt]); rdata += n; rdlen -= n; ntxt++; } if (result == NULL) { const int seplen = sizeof " | " - 1; const char *t1 = NULL, *t2 = NULL; if (ntxt == 1 && (t1 = strstr(txt[0], " | ")) != NULL && (t2 = strstr(t1 + seplen, " | ")) != NULL) { /* team-cymru.com format: * * one TXT segment per TXT RR, having * internal structure of vertical bar (|) * separated fields, of which the first * two are our desired output values * (AS number or path or set; CIDR prefix). */ char *new_asnum, *new_cidr; new_asnum = strndup(txt[0], (size_t) (t1 - txt[0])); new_cidr = strndup(t1 + seplen, (size_t) (t2 - (t1 + seplen))); t1 = t2 = NULL; const char *t = keep_best(asnum, cidr, new_asnum, new_cidr); if (t != NULL) result = strdup(t); } else if (ntxt == 3) { /* routeviews.org format: * * three TXT segments per TXT RR, which are * the AS number or path or set, and the * prefix mantissa, and the prefix length. * we use the first directly, and combine * the second and third to form CIDR prefix. */ char *new_asnum, *new_cidr; if (asprintf(&new_cidr, "%s/%s", txt[1], txt[2]) >= 0) { new_asnum = strdup(txt[0]); const char *t = keep_best(asnum, cidr, new_asnum, new_cidr); if (t != NULL) result = strdup(t); } else { result = strdup(strerror(errno)); } } else { result = strdup("unrecognized TXT format"); } } for (n = 0; n < ntxt; n++) { free(txt[n]); txt[n] = NULL; } next_rr:; } return result; } /* keep_best(asnum, cidr, new_asnum, new_cidr) -- select/keep "best" ASINFO * * return NULL on success, or else, reason (string) for failure. * * side effect: on success, *asnum and *cidr will be heap-allocated strings. */ static const char * keep_best(char **asnum, char **cidr, char *new_asnum, char *new_cidr) { if (*asnum != NULL && *cidr != NULL) { int pfxlen = -1, new_pfxlen = -1; char *cp; if ((cp = strchr(*cidr, '/')) == NULL || (pfxlen = atoi(cp+1)) <= 0 || pfxlen > 128) return "bad CIDR syntax (old)"; if ((cp = strchr(new_cidr, '/')) == NULL || (new_pfxlen = atoi(cp+1)) <= 0 || new_pfxlen > 128) return "bad CIDR syntax (new)"; if (new_pfxlen <= pfxlen) { free(new_asnum); free(new_cidr); return NULL; } free(*asnum); *asnum = NULL; free(*cidr); *cidr = NULL; } if (strcmp(new_asnum, "4294967295") == 0) { /* in routeviews.org, this is how they signal "unknown". */ free(new_asnum); free(new_cidr); } else { *asnum = new_asnum; *cidr = new_cidr; } return NULL; } #endif /*CRIPPLED_LIBC*/ dnsdbq-2.6.7/asinfo.h000066400000000000000000000016211462247651300144430ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef ASINFO_H_INCLUDED #define ASINFO_H_INCLUDED 1 #include #ifndef CRIPPLED_LIBC char * asinfo_from_rr(const char *rrtype, const char *rdata, char **asn, char **cidr); #endif bool asinfo_domain_exists(const char *); void asinfo_shutdown(void); #endif /*ASINFO_H_INCLUDED*/ dnsdbq-2.6.7/deduper.c000066400000000000000000000066251462247651300146200ustar00rootroot00000000000000/* * Copyright (c) 2021 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #include #include #include #include #include "deduper.h" struct chainlink; typedef struct chainlink *chainlink_t; struct chainlink { chainlink_t next; char str[]; }; /* keep this adjacent to the related 'struct' (chainlink). */ static inline size_t chainlink_size(size_t length) { return sizeof(struct chainlink) + length + 1; } struct deduper { size_t buckets; chainlink_t chains[]; }; /* keep this adjacent to the related 'struct' (deduper). */ static inline size_t deduper_size(size_t buckets) { return sizeof(struct deduper) + buckets * sizeof(chainlink_t); } static unsigned long hash_djb2(const char *); /* deduper_new(buckets) -- create a deduper having a set number of buckets */ deduper_t deduper_new(size_t buckets) { deduper_t ret = malloc(deduper_size(buckets)); if (ret == NULL) abort(); memset(ret, 0x00, deduper_size(buckets)); ret->buckets = buckets; return ret; } /* deduper_tas(str) -- test and maybe set this string in a deduper */ bool deduper_tas(deduper_t me, const char *str) { size_t bucket = hash_djb2(str) % me->buckets; for (chainlink_t chainlink = me->chains[bucket]; chainlink != NULL; chainlink = chainlink->next) if (strcmp(str, chainlink->str) == 0) return true; size_t len = strlen(str); chainlink_t chainlink = malloc(chainlink_size(len)); if (chainlink == NULL) abort(); memset(chainlink, 0, chainlink_size(len)); chainlink->next = me->chains[bucket]; strcpy(chainlink->str, str); me->chains[bucket] = chainlink; return false; } /* deduper_dump(out) -- for debugging, render a deduper's contents to an output */ void deduper_dump(deduper_t me, FILE *out) { for (size_t bucket = 0; bucket < me->buckets; bucket++) if (me->chains[bucket] != NULL) { fprintf(out, "[%zu]", bucket); for (chainlink_t chainlink = me->chains[bucket]; chainlink != NULL; chainlink = chainlink->next) fprintf(out, " \"%s\"", chainlink->str); fprintf(out, ".\n"); } } /* deduper_destroy() -- release all heap storage used by a deduper */ void deduper_destroy(deduper_t *me) { for (size_t bucket = 0; bucket < (*me)->buckets; bucket++) { chainlink_t next = (*me)->chains[bucket]; if (next != NULL) { (*me)->chains[bucket] = NULL; for (chainlink_t chainlink = next; chainlink != NULL; chainlink = next) { next = chainlink->next; chainlink->next = NULL; chainlink->str[0] = '\0'; free(chainlink); } } } memset(*me, 0, deduper_size((*me)->buckets)); free(*me); *me = NULL; } /* hash_djb2() -- compute daniel j. bernstein (djb2) hash #2 over a string */ static unsigned long hash_djb2(const char *str) { unsigned long hash = 5381; unsigned int c; while ((c = (unsigned char) *str++) != 0) hash = ((hash << 5) + hash) + c; /* hash * 33 + c */ return hash; } dnsdbq-2.6.7/deduper.h000066400000000000000000000016131462247651300146150ustar00rootroot00000000000000/* * Copyright (c) 2021 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef __DEDUPER_H_INCLUDED #define __DEDUPER_H_INCLUDED 1 struct deduper; typedef struct deduper *deduper_t; deduper_t deduper_new(size_t); bool deduper_tas(deduper_t, const char *); void deduper_dump(deduper_t, FILE *); void deduper_destroy(deduper_t *); #endif /*__DEDUPER_H_INCLUDED*/ dnsdbq-2.6.7/defs.h000066400000000000000000000051011462247651300141020ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef DEFS_H_INCLUDED #define DEFS_H_INCLUDED 1 #include #include #include #include #include #include "time.h" /* Note that cygwin has a crippled libresolv that does not * include the not so recent ns_initparse() function, etc. * This AS info functionality is thus not available * in cygwin. */ #ifdef __CYGWIN__ #define CRIPPLED_LIBC 1 #endif /* __CYGWIN__ */ #if WANT_PDNS_DNSDB #define DEFAULT_SYS "dnsdb2" #elif WANT_PDNS_CIRL #define DEFAULT_SYS "circl" #else #error "No passive DNS system defined" #endif #define DEFAULT_VERB 0 /* maximum number of concurrent fetches. * must not be greater than any pDNS system's concurrent connection limit. */ #define MAX_FETCHES 8 #define DNSDBQ_SYSTEM "DNSDBQ_SYSTEM" #define CREATE(p, s) if ((p) != NULL) { my_panic(false, "non-NULL ptr"); } \ else if (((p) = malloc(s)) == NULL) { my_panic(true, "malloc"); } \ else { memset((p), 0, s); } #define DESTROY(p) { if ((p) != NULL) { free(p); (p) = NULL; } } #define DEBUG(ge, ...) { if (debug_level >= (ge)) debug(__VA_ARGS__); } typedef enum { pres_none, pres_text, pres_json, pres_csv, pres_minimal } present_e; typedef enum { batch_none, batch_terse, batch_verbose } batch_e; #define TRANS_REVERSE 0x01 #define TRANS_DATEFIX 0x02 #define TRANS_CHOMP 0x04 #define TRANS_QDETAIL 0x08 /* or_else -- return one pointer or else the other. */ static inline const char * or_else(const char *p, const char *or_else) { if (p != NULL) return p; return or_else; } /* debug -- at the moment, dump to stderr. */ static inline void debug(bool want_header, const char *fmtstr, ...) { va_list ap; va_start(ap, fmtstr); if (want_header) fprintf(stderr, "debug [%s]: ", timeval_str(NULL, true)); vfprintf(stderr, fmtstr, ap); va_end(ap); } /* a query mode. not all pdns systems support all of these. */ typedef enum { no_mode = 0, rrset_mode, name_mode, ip_mode, raw_rrset_mode, raw_name_mode } mode_e; #endif /*DEFS_H_INCLUDED*/ dnsdbq-2.6.7/dnsdbq.c000066400000000000000000001206721462247651300144420ustar00rootroot00000000000000/* * Copyright (c) 2014-2021 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ /* External. */ /* asprintf() does not appear on linux without this */ #define _GNU_SOURCE /* gettimeofday() does not appear on linux without this. */ #define _BSD_SOURCE /* modern glibc will complain about the above if it doesn't see this. */ #define _DEFAULT_SOURCE #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* Types. */ #define MAIN_PROGRAM #include "asinfo.h" #include "defs.h" #include "netio.h" #include "pdns.h" #include "sort.h" #include "time.h" #include "tokstr.h" #include "globals.h" #undef MAIN_PROGRAM #define QPARAM_GETOPT "A:B:L:l:O:cgG" /* Forward. */ static void help(void); static void qdesc_debug(const char *, qdesc_ct); static void qparam_debug(const char *, qparam_ct); static __attribute__((noreturn)) void usage(const char *, ...); static bool parse_long(const char *, long *); static void set_timeout(const char *, const char *); static const char *qparam_ready(qparam_t); static const char *qparam_option(int, const char *, qparam_t); static verb_ct find_verb(const char *); static char *select_config(void); static void do_batch(FILE *, qparam_ct); static const char *batch_options(const char *, qparam_t, qparam_ct); static const char *batch_parse(char *, qdesc_t); static char *makepath(qdesc_ct); static query_t query_launcher(qdesc_ct, qparam_ct, writer_t); static const char *rrtype_correctness(const char *); static void launch_fetch(query_t, const char *, pdns_fence_ct); static void ruminate_json(int, qparam_ct); static const char *lookup_ok(void); static const char *summarize_ok(void); static const char *check_7bit(const char *); /* Constants. */ static const char * const conf_files[] = { "~/.isc-dnsdb-query.conf", "~/.dnsdb-query.conf", "/etc/isc-dnsdb-query.conf", "/etc/dnsdb-query.conf", NULL }; const struct presenter pres_text_lookup = { present_text_lookup, true }; const struct presenter pres_json_lookup = { present_json_lookup, true }; const struct presenter pres_csv_lookup = { present_csv_lookup, true }; const struct presenter pres_minimal_lookup = { present_minimal_lookup, false }; const struct presenter pres_text_summarize = { present_text_summarize, true }; const struct presenter pres_json_summarize = { present_json_summarize, true }; const struct presenter pres_csv_summarize = { present_csv_summarize, true }; const struct verb verbs[] = { /* note: element [0] of this array is the DEFAULT_VERB. */ { "lookup", "/lookup", lookup_ok, &pres_text_lookup, &pres_json_lookup, &pres_csv_lookup, &pres_minimal_lookup }, { "summarize", "/summarize", summarize_ok, &pres_text_summarize, &pres_json_summarize, &pres_csv_summarize, NULL }, { NULL, NULL, NULL, NULL, NULL, NULL, NULL } }; /* Private. */ static size_t ideal_buffer; static bool allow_8bit = false; /* Public. */ int main(int argc, char *argv[]) { struct qdesc qd = { .mode = no_mode }; struct qparam qp = qparam_empty; char *picked_system = NULL; bool info = false; int json_fd = -1; const char *msg; char *value; int ch; /* global dynamic initialization. */ ideal_buffer = 4 * (size_t) sysconf(_SC_PAGESIZE); gettimeofday(&startup_time, NULL); if ((program_name = strrchr(argv[0], '/')) == NULL) program_name = argv[0]; else program_name++; if ((value = getenv(env_time_fmt)) != NULL && strcasecmp(value, "iso") == 0) iso8601 = true; if ((value = getenv(env_config_file)) != NULL) config_file = strdup(value); if ((value = getenv(env_timeout)) != NULL) set_timeout(value, env_timeout); pverb = &verbs[DEFAULT_VERB]; /* process the command line options. */ while ((ch = getopt(argc, argv, "C:D:R:r:N:n:i:M:u:p:t:b:k:J:V:T:0:o:" "adfhIjmqSsUv468" QPARAM_GETOPT)) != -1) { switch (ch) { /* keep these in-sync with QPARAM_GETOPT. */ case 'A': case 'B': case 'c': case 'g': case 'G': case 'l': case 'L': case 'O': if ((msg = qparam_option(ch, optarg, &qp)) != NULL) usage(msg); break; case '0': { const char *equal = strchr(optarg, '='); if (equal == NULL) usage("-0 requires 'function='"); size_t length = (size_t) (equal - optarg); const char *thing = optarg + length + 1; if (strncmp(optarg, "countoff", length) == 0) { struct counted *c = countoff(thing); countoff_debug("main", thing, c); DESTROY(c); } else { usage("-0 function unrecognized"); } my_exit(0); } case 'a': asinfo_lookup = true; break; case 'C': cookie_file = strdup(optarg); break; case 'D': asinfo_domain = optarg; break; case 'R': { if (qd.mode != no_mode) usage("-r, -n, -i, -N, or -R " "can only appear once"); assert(qd.thing == NULL); qd.mode = raw_rrset_mode; const char *p = strchr(optarg, '/'); if (p != NULL) { if (qd.rrtype != NULL || qd.bailiwick != NULL) usage("if -b or -t are specified then " "-R cannot contain a slash"); const char *q = strchr(p + 1, '/'); if (q != NULL) { qd.bailiwick = strdup(q + 1); qd.rrtype = strndup(p + 1, (size_t) (q - p - 1)); } else { qd.rrtype = strdup(p + 1); } qd.thing = strndup(optarg, (size_t)(p - optarg)); } else { qd.thing = strdup(optarg); } break; } case 'r': { if (qd.mode != no_mode) usage("-r, -n, -i, -N, or -R " "can only appear once"); assert(qd.thing == NULL); qd.mode = rrset_mode; const char *p = strchr(optarg, '/'); if (p != NULL) { if (qd.rrtype != NULL || qd.bailiwick != NULL) usage("if -b or -t are specified then " "-r cannot contain a slash"); const char *q = strchr(p + 1, '/'); if (q != NULL) { qd.bailiwick = strdup(q + 1); qd.rrtype = strndup(p + 1, (size_t) (q - p - 1)); } else { qd.rrtype = strdup(p + 1); } qd.thing = strndup(optarg, (size_t)(p - optarg)); } else { qd.thing = strdup(optarg); } break; } case 'N': { if (qd.mode != no_mode) usage("-r, -n, -i, -N, or -R " "can only appear once"); assert(qd.thing == NULL); qd.mode = raw_name_mode; const char *p = strchr(optarg, '/'); if (p != NULL) { if (qd.rrtype != NULL || qd.bailiwick != NULL) usage("if -b or -t are specified then " "-N cannot contain a slash"); const char *q = strchr(p + 1, '/'); if (q != NULL) { qd.bailiwick = strdup(q + 1); qd.rrtype = strndup(p + 1, (size_t) (q - p - 1)); } else { qd.rrtype = strdup(p + 1); } qd.thing = strndup(optarg, (size_t)(p - optarg)); } else { qd.thing = strdup(optarg); } break; } case 'n': { if (qd.mode != no_mode) usage("-r, -n, -i, -N, or -R " "can only appear once"); assert(qd.thing == NULL); qd.mode = name_mode; const char *p = strchr(optarg, '/'); if (p != NULL) { if (qd.rrtype != NULL || qd.bailiwick != NULL) usage("if -b or -t are specified then " "-n cannot contain a slash"); const char *q = strchr(p + 1, '/'); if (q != NULL) { qd.bailiwick = strdup(q + 1); qd.rrtype = strndup(p + 1, (size_t) (q - p - 1)); } else { qd.rrtype = strdup(p + 1); } qd.thing = strndup(optarg, (size_t)(p - optarg)); } else { qd.thing = strdup(optarg); } break; } case 'i': { if (qd.mode != no_mode) usage("-r, -n, -i, -N, or -R " "can only appear once"); assert(qd.thing == NULL); qd.mode = ip_mode; const char *p = strchr(optarg, '/'); if (p != NULL) { qd.thing = strndup(optarg, (size_t)(p - optarg)); qd.pfxlen = strdup(p + 1); } else { qd.thing = strdup(optarg); } break; } case 'V': { pverb = find_verb(optarg); if (pverb == NULL) usage("Unsupported verb for -V argument"); break; } case 'M': if (!parse_long(optarg, &max_count) || (max_count <= 0)) usage("-M must be positive"); break; case 'o': set_timeout(optarg, "-o"); break; case 'u': picked_system = strdup(optarg); break; case 'U': donotverify = true; break; case 'p': if (strcasecmp(optarg, "json") == 0) presentation = pres_json; else if (strcasecmp(optarg, "csv") == 0) presentation = pres_csv; else if (strcasecmp(optarg, "text") == 0 || strcasecmp(optarg, "dns") == 0) presentation = pres_text; else if (strcasecmp(optarg, "minimal") == 0) presentation = pres_minimal; else usage("-p must specify " "json, text, csv, or minimal"); DESTROY(presentation_name); presentation_name = strdup(optarg); break; case 't': if (qd.rrtype != NULL) usage("can only specify rrtype(s) once"); qd.rrtype = strdup(optarg); break; case 'b': if (qd.bailiwick != NULL) usage("can only specify bailiwick one way"); qd.bailiwick = strdup(optarg); break; case 'k': { if (sorting == no_sort) usage("-k must be preceded by -s or -S"); struct tokstr *ts = tokstr_string(optarg); for (char *tok; (tok = tokstr_next(ts, ",")) != NULL; free(tok)) { if (find_sort_key(tok) != NULL) usage("Each sort key may only be " "specified once"); if ((msg = add_sort_key(tok)) != NULL) usage(msg); } tokstr_last(&ts); break; } case 'J': if (strcmp(optarg, "-") == 0) json_fd = STDIN_FILENO; else json_fd = open(optarg, O_RDONLY); if (json_fd < 0) my_panic(true, optarg); break; case 'd': debug_level++; break; case 'j': presentation = pres_json; DESTROY(presentation_name); presentation_name = strdup("json"); break; case 'f': switch (batching) { case batch_none: batching = batch_terse; break; case batch_terse: batching = batch_verbose; break; case batch_verbose: /* FALLTHROUGH */ default: usage("too many -f options"); } break; case 'T': { struct tokstr *ts = tokstr_string(optarg); for (char *token; (token = tokstr_next(ts, ",")) != NULL; free(token)) { if (strcasecmp(token, "reverse") == 0) transforms |= TRANS_REVERSE; else if (strcasecmp(token, "datefix") == 0) transforms |= TRANS_DATEFIX; else if (strcasecmp(token, "chomp") == 0) transforms |= TRANS_CHOMP; else if (strcasecmp(token, "qdetail") == 0) transforms |= TRANS_QDETAIL; else { usage("unrecognized transform in -T"); } } tokstr_last(&ts); break; } case 'm': multiple = true; break; case 's': sorting = normal_sort; break; case 'S': sorting = reverse_sort; break; case 'I': info = true; break; case 'v': printf("%s: version %s\n", program_name, id_version); my_exit(0); case 'q': quiet = true; break; case 'h': help(); my_exit(0); case '4': curl_ipresolve = CURL_IPRESOLVE_V4; break; case '6': curl_ipresolve = CURL_IPRESOLVE_V6; break; case '8': allow_8bit = true; break; default: usage("unrecognized option"); } } argc -= optind; if (argc != 0) usage("there are no non-option arguments to this program"); argv = NULL; if (allow_8bit == false && batching == batch_none && (qd.mode == name_mode || qd.mode == rrset_mode)) { msg = check_7bit(qd.thing); if (msg != NULL) usage(msg); } if (asinfo_lookup) { #ifdef CRIPPLED_LIBC usage("the -a option requires a modern functional C library."); #else if (!asinfo_domain_exists(asinfo_domain)) { my_logf("ASINFO domain (%s) does not exist", asinfo_domain); my_exit(1); } #endif } if (presentation == pres_none) { presentation = pres_text; assert(presentation_name == NULL); presentation_name = strdup("text"); } if (presentation == pres_minimal) minimal_deduper = deduper_new(minimal_modulus); if ((msg = qparam_ready(&qp)) != NULL) usage(msg); /* optionally dump program options as interpreted. */ if (debug_level >= 1) { qdesc_debug("main", &qd); qparam_debug("main", &qp); debug(true, "batching=%d, multiple=%d\n", batching != false, multiple != false); } /* select presenter. */ switch (presentation) { case pres_text: presenter = pverb->text; break; case pres_json: presenter = pverb->json; break; case pres_csv: presenter = pverb->csv; break; case pres_minimal: presenter = pverb->minimal; break; case pres_none: /* FALLTHROUGH */ default: abort(); } assert(presentation != pres_none); assert(presentation_name != NULL); if (presenter == NULL) { char *errmsg = NULL; int x = asprintf(&errmsg, "that verb (%s) has no presenter for \"%s\"", pverb->name, presentation_name); if (x < 0) my_panic(true, "asprintf"); usage(errmsg); } if (presentation != pres_json && (transforms & TRANS_QDETAIL) != 0) usage("'-T qdetail' currently requires '-j' or '-p json'"); /* get to final readiness; in particular, get psys set. */ if (sorting != no_sort) { if (!presenter->sortable) { char *errmsg = NULL; int x = asprintf(&errmsg, "that presentation format (%s) " "cannot be sorted", presentation_name); if (x < 0) my_panic(true, "asprintf"); usage(errmsg); } if ((transforms & TRANS_QDETAIL) != 0) usage("\"-T qdetail\" is incompatible with sorting"); sort_ready(); } if (config_file == NULL) config_file = select_config(); if (picked_system != NULL) { psys_specified = true; pick_system(picked_system, "-u option"); DESTROY(picked_system); } else { pick_system(DEFAULT_SYS, "default system"); psys_specified = true; } if (json_fd != -1) { #if WANT_PDNS_DNSDB /* the json output files are in COF format, never SAF. */ if (strcmp(psys->name, "dnsdb2") == 0) pick_system("dnsdb1", "downgrade for -J"); #endif NULL; } else { make_curl(); assert(psys_specified); } /* validate some interrelated options. */ if (multiple && batching == batch_none) usage("using -m without -f makes no sense."); if ((msg = (*pverb->ok)()) != NULL) usage(msg); if ((msg = psys->verb_ok(pverb->name, &qp)) != NULL) usage(msg); /* get some input from somewhere, and use it to drive our output. */ if (json_fd != -1) { /* read a JSON file. */ if (qd.mode != no_mode) usage("can't mix -n, -r, -i, or -R with -J"); if (batching != batch_none) usage("can't mix -f with -J"); if (qd.bailiwick != NULL) usage("can't mix -b with -J"); if (info) usage("can't mix -I with -J"); if (qd.rrtype != NULL) usage("can't mix -t with -J"); if (pverb != &verbs[DEFAULT_VERB]) usage("can't mix -V with -J"); if (max_count > 0) usage("can't mix -M with -J"); if (qp.gravel) usage("can't mix -g with -J"); if (qp.offset != 0) usage("can't mix -O with -J"); ruminate_json(json_fd, &qp); close(json_fd); } else if (batching != batch_none) { /* drive via a batch file. */ if (qd.mode != no_mode) usage("can't mix -n, -r, -i, or -R with -f"); if (qd.bailiwick != NULL) usage("can't mix -b with -f"); if (qd.rrtype != NULL) usage("can't mix -t with -f"); if (info) usage("can't mix -I with -f"); do_batch(stdin, &qp); } else if (info) { /* use the "info" verb. */ if (qd.mode != no_mode) usage("can't mix -n, -r, -i, or -R with -I"); if (presentation != pres_text && presentation != pres_json) usage("info must be presented in json or text format"); if (qd.bailiwick != NULL) usage("can't mix -b with -I"); if (qd.rrtype != NULL) usage("can't mix -t with -I"); if (psys->info == NULL) usage("there is no 'info' for this service"); psys->info(); } else { /* do a LHS or RHS lookup of some kind. */ if (qd.mode == no_mode) usage("must specify -r, -n, -i, or -R" " unless -f or -J is used"); if (qd.bailiwick != NULL) { if (qd.mode == ip_mode) usage("can't mix -b with -i"); if (qd.mode == raw_rrset_mode) usage("can't mix -b with -R"); if (qd.mode == raw_name_mode) usage("can't mix -b with -N"); if (qd.mode == name_mode) usage("can't mix -b with -n"); } if (qd.mode == ip_mode && qd.rrtype != NULL) usage("can't mix -i with -t"); writer_t writer = writer_init(qp.output_limit, ps_stdout, false); (void) query_launcher(&qd, &qp, writer); io_engine(0); writer_fini(writer); writer = NULL; } if (json_fd == -1) { unmake_curl(); } /* clean up and go home. */ DESTROY(qd.thing); DESTROY(qd.rrtype); DESTROY(qd.bailiwick); DESTROY(qd.pfxlen); my_exit(exit_code); } /* my_exit -- close or destroy global objects, then exit. */ __attribute__((noreturn)) void my_exit(int code) { DESTROY(presentation_name); /* deduper state if any can be trashed. */ if (minimal_deduper != NULL) deduper_destroy(&minimal_deduper); /* writers and readers which are still known, must be freed. */ unmake_writers(); /* if curl is operating, it must be shut down. */ unmake_curl(); /* globals which may have been initialized, are to be freed. */ DESTROY(config_file); DESTROY(cookie_file); if (psys != NULL) psys->destroy(); /* sort key specifications and computations, are to be freed. */ sort_destroy(); #ifndef CRIPPLED_LIBC /* asinfo logic has an internal DNS resolver context. */ asinfo_shutdown(); #endif /* terminate process. */ DEBUG(1, true, "about to call exit(%d)\n", code); exit(code); } /* my_panic -- display an error on diagnostic output stream, exit ungracefully */ __attribute__((noreturn)) void my_panic(bool want_perror, const char *s) { my_logf("panic (%s): ", s, want_perror ? strerror(errno) : "generic"); my_exit(1); } /* Private. */ /* help -- display a brief usage-help text; then exit. * * this goes to stdout since we can expect it not to be piped unless to $PAGER. */ static void help(void) { verb_ct v; printf("usage: %s [-acdfGghIjmqSsUv468] [-p dns|json|csv|minimal]\n", program_name); puts("\t[-u SYSTEM] [-V VERB] [-0 FUNCTION=INPUT]\n" "\t[-k (first|last|duration|count|name|type|data)[,...]]\n" "\t[-l QUERY-LIMIT] [-L OUTPUT-LIMIT]\n" "\t[-O OFFSET] [-M MAX_COUNT]\n" "\t[-A AFTER] [-B BEFORE]\n" "\t[-D ASINFO_DOMAIN] [-T (datefix|reverse|chomp|qdetail)[,...] {\n" "\t\t-f |\n" "\t\t-J INPUTFILE |\n" "\t\t[-t RRTYPE[,...]] [-b BAILIWICK] {\n" "\t\t\t-r OWNER[/RRTYPE[,...][/BAILIWICK]] |\n" "\t\t\t-n NAME[/RRTYPE[,...]] |\n" "\t\t\t-i IP[/PFXLEN] |\n" "\t\t\t-N RAW-NAME-DATA[/RRTYPE[,...]]\n" "\t\t\t-R RAW-OWNER-DATA[/RRTYPE[,...][/BAILIWICK]]\n" "\t\t}\n" "\t}"); printf("for -A and -B, use absolute format YYYY-MM-DD[ HH:MM:SS],\n" "\tor relative format %%dw%%dd%%dh%%dm%%ds.\n" "use -a to get ASNs associated with reported IP addresses\n" "use -c to get complete (strict) time matching for -A and -B.\n" "for -D, the default is \"%s\"\n" "use -d one or more times to ramp up the diagnostic output.\n" "for -0, the function must be \"countoff\"\n" "for -f, stdin must contain lines of the following forms:\n" "\trrset/name/NAME[/RRTYPE[,...][/BAILIWICK]]\n" "\trrset/raw/HEX-PAIRS[/RRTYPE[,...][/BAILIWICK]]\n" "\trdata/name/NAME[/RRTYPE[,...]]\n" "\trdata/ip/ADDR[,PFXLEN]\n" "\trdata/raw/HEX-PAIRS[/RRTYPE[,...]]\n" "\t(output format will depend on -p or -j, framed by '--'.)\n" "\t(with -ff, framing will be '++ $cmd', '-- $stat ($code)'.\n" "use -g to get graveled results (default is -G, rocks).\n" "use -h to reliably display this helpful text.\n" "use -I to see a system-specific account/key summary.\n" "for -J, input format is newline-separated JSON, " "as from -j output.\n" "use -j as a synonym for -p json.\n" "use -M # to end a summarize op when count exceeds threshold.\n" "use -m with -f for multiple upstream queries in single result.\n" "use -m with -f -f for multiple upstream queries out of order.\n" "use -O # to skip this many results in what is returned.\n" "use -q for warning reticence.\n" "use -s to sort in ascending order, " "or -S for descending order.\n" "\t-s/-S can be repeated before several -k arguments.\n" "for -T, transforms are datefix, reverse, chomp, and qdetail.\n" "use -U to turn off SSL certificate verification.\n" "use -v to show the program version.\n" "use -4 to force connecting to the server via IPv4.\n" "use -6 to force connecting to the server via IPv6.\n" "use -8 to allow 8-bit values in -r and -n arguments.\n", asinfo_domain); puts("for -u, system must be one of:"); #if WANT_PDNS_DNSDB puts("\tdnsdb"); puts("\tdnsdb2"); #endif #if WANT_PDNS_CIRCL puts("\tcircl"); #endif puts("for -V, verb must be one of:"); for (v = verbs; v->name != NULL; v++) printf("\t%s\n", v->name); puts("\nGetting Started:\n" "\tAdd your API key to ~/.dnsdb-query.conf like this:\n" "\t\tDNSDB_API_KEY=\"YOURAPIKEYHERE\""); printf("\nTry man %s for full documentation.\n", program_name); } /* qdesc_debug -- dump a qdesc. */ static void qdesc_debug(const char *where, qdesc_ct qdp) { debug(true, "qdesc(%s)[", where); const char *sep = "\040"; if (qdp->mode != no_mode) { debug(false, "%smo %d", sep, (int)qdp->mode); sep = ",\040"; } if (qdp->thing != NULL) { debug(false, "%sth '%s'", sep, qdp->thing); sep = ",\040"; } if (qdp->rrtype != NULL) { debug(false, "%srr '%s'", sep, qdp->rrtype); sep = ",\040"; } if (qdp->bailiwick != NULL) { debug(false, "%sbw '%s'", sep, qdp->bailiwick); sep = ",\040"; } if (qdp->pfxlen != NULL) { debug(false, "%spfx '%s'", sep, qdp->pfxlen); sep = ",\040"; } debug(false, " ]\n"); } /* qparam_debug -- dump a qparam. */ static void qparam_debug(const char *where, qparam_ct qpp) { debug(true, "qparam(%s)[", where); const char *sep = "\040"; if (qpp->after != 0) { debug(false, "%s-A%ld(%s)", sep, qpp->after, time_str(qpp->after, false)); sep = "\n\t"; } if (qpp->before != 0) { debug(false, "%s-B%ld(%s)", sep, qpp->before, time_str(qpp->before, false)); sep = "\n\t"; } if (qpp->query_limit != -1) { debug(false, "%s-l%ld", sep, qpp->query_limit); sep = "\040"; } if (qpp->output_limit != -1) { debug(false, "%s-L%ld", sep, qpp->output_limit); sep = "\040"; } if (qpp->complete) { debug(false, "%s-c", sep); sep = "\040"; } if (qpp->gravel) { debug(false, "%s-g", sep); sep = "\040"; } debug(false, "\040]\n"); } /* usage -- display a usage error message, brief usage help text; then exit. * * this goes to stderr in case stdout has been piped or redirected. */ static __attribute__((noreturn)) void usage(const char *fmtstr, ...) { va_list ap; va_start(ap, fmtstr); fputs("error: ", stderr); vfprintf(stderr, fmtstr, ap); va_end(ap); fputs("\n\n", stderr); fprintf(stderr, "try %s -h for a short description of program usage.\n", program_name); my_exit(1); } /* parse_long -- parse a base 10 long value. * * Return true if ok, else return false. */ static bool parse_long(const char *in, long *out) { char *ep; long result; /* "The strtol() function shall not change the setting of errno * if successful." (IEEE Std 1003.1, 2004 Edition) */ errno = 0; result = strtol(in, &ep, 10); if ((errno == ERANGE && (result == LONG_MAX || result == LONG_MIN)) || (errno != 0 && result == 0) || (ep == in)) return false; *out = result; return true; } /* set_timeout -- ingest a setting for curl_timeout * * exits through usage() if the value is invalid. */ static void set_timeout(const char *value, const char *source) { if (!parse_long(value, &curl_timeout) || (curl_timeout < 0)) usage("%s must be non-negative", source); } /* qparam_ready -- check and possibly adjust the contents of a qparam. */ static const char * qparam_ready(qparam_t qpp) { if (qpp->output_limit == -1 && qpp->query_limit != -1 && !multiple) qpp->output_limit = qpp->query_limit; if (qpp->complete && qpp->after != 0 && qpp->before != 0) { if (qpp->after > qpp->before) return "-A value must be before -B value" " if using complete time matching"; } if (qpp->complete && qpp->after == 0 && qpp->before == 0) return "-c without -A or -B makes no sense."; return NULL; } /* qparam_option -- process one command line option related to a qparam */ static const char * qparam_option(int opt, const char *arg, qparam_t qpp) { switch (opt) { case 'A': if (!time_get(arg, &qpp->after) || qpp->after == 0UL) return "bad -A timestamp"; break; case 'B': if (!time_get(arg, &qpp->before) || qpp->before == 0UL) return "bad -B timestamp"; break; case 'c': qpp->complete = true; break; case 'g': qpp->gravel = true; break; case 'G': qpp->gravel = false; break; case 'l': if (!parse_long(arg, &qpp->query_limit) || (qpp->query_limit < 0)) return "-l must be zero or positive"; break; case 'L': if (!parse_long(arg, &qpp->output_limit) || (qpp->output_limit <= 0)) return "-L must be positive"; qpp->explicit_output_limit = qpp->output_limit; break; case 'O': if (!parse_long(optarg, &qpp->offset) || (qpp->offset < 0)) return "-O must be zero or positive"; break; } return NULL; } /* lookup_ok -- validate command line options for 'lookup'. */ static const char * lookup_ok(void) { /* TODO too many local variables would need to be global to check * more here. */ if (max_count > 0) return "max_count is not allowed for the lookup verb"; return NULL; } /* summarize_ok -- validate commandline options for 'summarize'. */ static const char * summarize_ok(void) { if (sorting != no_sort) return "Sorting with a summarize verb makes no sense"; return NULL; } /* find_verb -- locate a verb by option parameter */ static verb_ct find_verb(const char *option) { verb_ct v; for (v = verbs; v->name != NULL; v++) if (strcasecmp(option, v->name) == 0) return v; return NULL; } /* select_config -- try to find a config file in static path. */ static char * select_config(void) { const char * const *conf; char *cf = NULL; for (conf = conf_files; *conf != NULL; conf++) { wordexp_t we; wordexp(*conf, &we, WRDE_NOCMD); cf = strdup(we.we_wordv[0]); wordfree(&we); if (access(cf, R_OK) == 0) { DEBUG(1, true, "conf found: '%s'\n", cf); return cf; } DESTROY(cf); } return NULL; } /* do_batch -- implement "filter" mode, reading commands from a batch file. */ static void do_batch(FILE *f, qparam_ct qpp) { struct qparam qp = *qpp; writer_t writer = NULL; char *command = NULL; size_t n = 0; /* if doing multiple parallel upstreams, start a writer. */ bool one_writer = multiple && batching != batch_verbose; if (one_writer) writer = writer_init(qp.output_limit, ps_stdout, false); while (getline(&command, &n, f) > 0) { const char *msg; struct qdesc qd; char *nl; /* the last line of the file may not have a newline. */ nl = strchr(command, '\n'); if (nl != NULL) *nl = '\0'; /* allow # as a comment syntax */ if (command[0] == '#') continue; DEBUG(1, true, "do_batch(%s)\n", command); /* if this is a $OPTIONS, parse it and change our qparams. */ if (strncasecmp(command, "$options", (sizeof "$options") - 1) == 0) { if ((msg = batch_options(command, &qp, qpp)) != NULL) my_logf("warning: batch option parse error: %s", msg); continue; } /* if not parallelizing, start a writer here instead. */ if (!one_writer) writer = writer_init(qp.output_limit, ps_stdout, false); /* crack the batch line if possible. */ msg = batch_parse(command, &qd); if (msg != NULL) { my_logf("batch entry parse error: %s", msg); } else { /* start one or more curl fetches based on this entry. */ query_t query = query_launcher(&qd, &qp, writer); /* if merging, drain some jobs; else, drain all jobs. */ if (one_writer) io_engine(MAX_FETCHES); else io_engine(0); /* if one of our fetches already failed, say so now. */ if (query != NULL && query->status != NULL && batching != batch_verbose) { assert(query->message != NULL); my_logf("batch line status: %s (%s)", query->status, query->message); } } if (!one_writer) { /* think about showing the end-of-object separator. * We reach here after all the queries from * this batch line have finished. */ switch (batching) { case batch_none: break; case batch_terse: assert(writer->ps_buf == NULL && writer->ps_len == 0); writer->ps_buf = strdup("--\n"); writer->ps_len = strlen(writer->ps_buf); break; case batch_verbose: /* query_done() will do this. */ break; default: abort(); } writer_fini(writer); writer = NULL; fflush(stdout); } } DESTROY(command); /* if parallelized, run remaining jobs to completion, then finish up. */ if (one_writer) { io_engine(0); writer_fini(writer); writer = NULL; } } /* batch_options -- parse a $OPTIONS line out of a batch file. */ static const char * batch_options(const char *optstr, qparam_t options, qparam_ct dflt) { char **optv = calloc(strlen(optstr) + 1, sizeof(char *)); struct qparam save = *options; char **opt = optv; const char *msg; int optc, ch; /* crack the option string based on space or tab delimiters. */ struct tokstr *ts = tokstr_string(optstr); for (char *tok; (tok = tokstr_next(ts, "\040\011")) != NULL; free(tok)) { /* dispense with extra spaces and tabs (empty fields). */ if (*tok == '\0') continue; *opt++ = strdup(tok); } tokstr_last(&ts); /* if no options were specified (e.g., $options\n), restore defaults. */ msg = NULL; optc = (int) (opt - optv); if (optc == 1) { DEBUG(2, true, "default options restored\n"); *options = *dflt; } else { /* use getopt() to parse the cracked array. */ #if defined __GLIBC__ /* glibc needs to have optind set to 0 instead of the * "traditional value" of 1. */ optind = 0; #else /* 1 is the value that optind should be initialized to, * according to IEEE Std 1003.1. */ optind = 1; #if defined __FreeBSD__ || defined __OpenBSD__ || defined __NetBSD__ || \ defined __APPLE__ || defined __DragonFly__ /* BSD-like libc also needs to have optreset set to 1. */ optreset = 1; #endif /*BSD*/ #endif while ((ch = getopt(optc, optv, QPARAM_GETOPT)) != -1) { if ((msg = qparam_option(ch, optarg, options)) != NULL) break; } optc -= optind; if (msg == NULL && optc != 0) msg = "superfluous non-arguments in $OPTIONS"; } /* if an error occured, reset options to saved values. */ if (msg != NULL) { *options = save; } else { /* otherwise consider reporting the new options. */ if (debug_level >= 1) qparam_debug("batch", options); } /* done. */ for (int i = 0; i < opt - optv; i++) free(optv[i]); DESTROY(optv); return msg; } /* batch_parse -- turn one line from a -f batch into a qdesc_t. */ static const char * batch_parse(char *line, qdesc_t qdp) { struct qdesc qd = (struct qdesc) { }; char *saveptr = NULL; const char *msg; char *t; if ((t = strtok_r(line, "/", &saveptr)) == NULL) return "too few terms"; if (strcmp(t, "rrset") == 0) { if ((t = strtok_r(NULL, "/", &saveptr)) == NULL) return "missing term after 'rrset/'"; if (strcmp(t, "name") == 0) { qd.mode = rrset_mode; if ((t = strtok_r(NULL, "/", &saveptr)) == NULL) return "missing term after 'rrset/name/'"; if (allow_8bit == false && ((msg = check_7bit(t)) != NULL)) return msg; qd.thing = t; if ((t = strtok_r(NULL, "/", &saveptr)) != NULL) { qd.rrtype = t; if ((t = strtok_r(NULL, "/", &saveptr)) != NULL) { qd.bailiwick = t; } } } else if (strcmp(t, "raw") == 0) { qd.mode = raw_rrset_mode; if ((t = strtok_r(NULL, "/", &saveptr)) == NULL) return "missing term after 'rrset/raw/'"; qd.thing = t; if ((t = strtok_r(NULL, "/", &saveptr)) != NULL) { qd.rrtype = t; if ((t = strtok_r(NULL, "/", &saveptr)) != NULL) { qd.bailiwick = t; } } } else { return "unrecognized term after 'rrset/'"; } } else if (strcmp(t, "rdata") == 0) { if ((t = strtok_r(NULL, "/", &saveptr)) == NULL) return "missing term after 'rdata/'"; if (strcmp(t, "name") == 0) { qd.mode = name_mode; if ((t = strtok_r(NULL, "/", &saveptr)) == NULL) return "missing term after 'rdata/name/'"; if (allow_8bit == false && ((msg = check_7bit(t)) != NULL)) return msg; qd.thing = t; if ((t = strtok_r(NULL, "/", &saveptr)) != NULL) { qd.rrtype = t; } } else if (strcmp(t, "raw") == 0) { qd.mode = raw_name_mode; if ((t = strtok_r(NULL, "/", &saveptr)) == NULL) return "missing term after 'rdata/raw/'"; qd.thing = t; if ((t = strtok_r(NULL, "/", &saveptr)) != NULL) { qd.rrtype = t; } } else if (strcmp(t, "ip") == 0) { qd.mode = ip_mode; if ((t = strtok_r(NULL, "/", &saveptr)) == NULL) return "missing term after 'rdata/ip/'"; qd.thing = t; } else { return "unrecognized term after 'rdata/'"; } } else { return "unrecognized initial term"; } t = strtok_r(NULL, "/", &saveptr); if (t != NULL) return "extra garbage"; *qdp = qd; return NULL; } /* makepath -- make a RESTful URI that describes these query parameters. * * Returns a string that must be free()d. */ static char * makepath(qdesc_ct qdp) { /* recondition various options for HTML use. */ char *thing = escape(qdp->thing); char *rrtype = escape(qdp->rrtype); char *bailiwick = escape(qdp->bailiwick); char *pfxlen = escape(qdp->pfxlen); char *path = NULL; switch (qdp->mode) { int x; case rrset_mode: if (rrtype != NULL && bailiwick != NULL) x = asprintf(&path, "rrset/name/%s/%s/%s", thing, rrtype, bailiwick); else if (rrtype != NULL) x = asprintf(&path, "rrset/name/%s/%s", thing, rrtype); else if (bailiwick != NULL) x = asprintf(&path, "rrset/name/%s/ANY/%s", thing, bailiwick); else x = asprintf(&path, "rrset/name/%s", thing); if (x < 0) my_panic(true, "asprintf"); break; case name_mode: if (rrtype != NULL) x = asprintf(&path, "rdata/name/%s/%s", thing, rrtype); else x = asprintf(&path, "rdata/name/%s", thing); if (x < 0) my_panic(true, "asprintf"); break; case ip_mode: if (pfxlen != NULL) x = asprintf(&path, "rdata/ip/%s,%s", thing, pfxlen); else x = asprintf(&path, "rdata/ip/%s", thing); if (x < 0) my_panic(true, "asprintf"); break; case raw_rrset_mode: if (rrtype != NULL) x = asprintf(&path, "rrset/raw/%s/%s", thing, rrtype); else x = asprintf(&path, "rrset/raw/%s", thing); if (x < 0) my_panic(true, "asprintf"); break; case raw_name_mode: if (rrtype != NULL) x = asprintf(&path, "rdata/raw/%s/%s", thing, rrtype); else x = asprintf(&path, "rdata/raw/%s", thing); if (x < 0) my_panic(true, "asprintf"); break; case no_mode: /*FALLTHROUGH*/ default: abort(); } DESTROY(thing); DESTROY(rrtype); DESTROY(bailiwick); DESTROY(pfxlen); return path; } /* query_launcher -- fork off some curl jobs via launch() for this query. * * can write to STDERR and return NULL if a query cannot be launched. */ static query_t query_launcher(qdesc_ct qdp, qparam_ct qpp, writer_t writer) { struct pdns_fence fence = {}; query_t query = NULL; const char *msg; /* ready player one. */ CREATE(query, sizeof(struct query)); query->descr = makepath(qdp); query->mode = qdp->mode; query->qp = *qpp; qpp = NULL; /* define the fence. */ if (query->qp.after != 0) { if (query->qp.complete) { /* each db tuple must begin after the fence-start. */ fence.first_after = query->qp.after; } else { /* each db tuple must end after the fence-start. */ fence.last_after = query->qp.after; } } if (query->qp.before != 0) { if (query->qp.complete) { /* each db tuple must end before the fence-end. */ fence.last_before = query->qp.before; } else { /* each db tuple must begin before the fence-end. */ fence.first_before = query->qp.before; } } /* branch on rrtype; launch (or queue) nec'y fetches. */ if (qdp->rrtype == NULL) { /* no rrtype string given, let makepath set it to "any". */ char *path = makepath(qdp); launch_fetch(query, path, &fence); DESTROY(path); } else if ((msg = rrtype_correctness(qdp->rrtype)) != NULL) { my_logf("rrtype incorrect: %s", msg); DESTROY(query); return NULL; } else { /* rrtype string was given, parse comma separated list. */ int nfetches = 0; struct tokstr *ts = tokstr_string(qdp->rrtype); for (char *rrtype; (rrtype = tokstr_next(ts, ",")) != NULL; free(rrtype)) { struct qdesc qd = { .mode = qdp->mode, .thing = qdp->thing, .rrtype = rrtype, .bailiwick = qdp->bailiwick, .pfxlen = qdp->pfxlen }; char *path = makepath(&qd); launch_fetch(query, path, &fence); nfetches++; DESTROY(path); } tokstr_last(&ts); if (nfetches > 1) query->multitype = true; } /* finish query initialization, link it up, and return it. */ query->writer = writer; writer = NULL; query->next = query->writer->queries; query->writer->queries = query; return query; } /* rrtype_correctness -- return an error text if the rrtypes are senseless */ static const char * rrtype_correctness(const char *input) { char **rrtypeset = calloc(MAX_FETCHES, sizeof(char *)); const char *ret = NULL; int nrrtypeset = 0; bool some = false, any = false, some_dnssec = false, any_dnssec = false; struct tokstr *ts = tokstr_string(input); for (char *rrtype; (rrtype = tokstr_next(ts, ",")) != NULL; free(rrtype)) { for (char *p = rrtype; *p != '\0'; p++) if (isupper((int)*p)) *p = (char) tolower((int)*p); if (nrrtypeset == MAX_FETCHES) { ret = "too many rrtypes specified"; goto done; } for (int i = 0; i < nrrtypeset; i++) if (strcmp(rrtype, rrtypeset[i]) == 0) { ret = "duplicate rrtype encountered"; goto done; } rrtypeset[nrrtypeset++] = strdup(rrtype); if (strcmp(rrtype, "any") == 0) any = true; else if (strcmp(rrtype, "any-dnssec") == 0) any_dnssec = true; else if (strcmp(rrtype, "ds") == 0 || strcmp(rrtype, "rrsig") == 0 || strcmp(rrtype, "nsec") == 0 || strcmp(rrtype, "dnskey") == 0 || strcmp(rrtype, "cdnskey") == 0 || strcmp(rrtype, "cds") == 0 || strcmp(rrtype, "ta") == 0 || strcmp(rrtype, "nsec3") == 0 || strcmp(rrtype, "nsec3param") == 0 || strcmp(rrtype, "dlv") == 0) { some_dnssec = true; } else { some = true; } if (any && some) { ret = "ANY is redundant when mixed like this"; goto done; } if (any_dnssec && some_dnssec) { ret = "ANY-DNSSEC is redundant when mixed like this"; goto done; } } tokstr_last(&ts); done: for (int i = 0; i < nrrtypeset; i++) DESTROY(rrtypeset[i]); DESTROY(rrtypeset); return ret; } /* launch_fetch -- actually launch a query job, given a path and time fences. */ static void launch_fetch(query_t query, const char *path, pdns_fence_ct fp) { char *url = psys->url(path, NULL, &query->qp, fp, false); if (url == NULL) my_exit(1); DEBUG(1, true, "url [%s]\n", url); create_fetch(query, url); } /* ruminate_json -- process a json file from the filesys rather than the API. */ static void ruminate_json(int json_fd, qparam_ct qpp) { fetch_t fetch = NULL; query_t query = NULL; void *buf = NULL; writer_t writer; ssize_t len; writer = writer_init(qpp->output_limit, NULL, false); CREATE(query, sizeof(struct query)); query->writer = writer; query->qp = *qpp; CREATE(fetch, sizeof(struct fetch)); fetch->query = query; query->fetches = fetch; writer->queries = query; CREATE(buf, ideal_buffer); while ((len = read(json_fd, buf, ideal_buffer)) > 0) { writer_func(buf, 1, (size_t)len, query->fetches); } DESTROY(buf); writer_fini(writer); writer = NULL; } /* check_7bit -- check if its argument is 7 bit clean ASCII. * * returns NULL on success, else an error message. */ static const char * check_7bit(const char *name) { int ch; while ((ch = *name++) != '\0') if ((ch & 0x80) != 0) return "search argument is not 7-bit clean"; return NULL; } dnsdbq-2.6.7/dnsdbq.man000066400000000000000000000543301462247651300147700ustar00rootroot00000000000000.\" Copyright (c) 2014-2021 by Farsight Security, Inc. .\" .\" Licensed under the Apache License, Version 2.0 (the "License"); .\" you may not use this file except in compliance with the License. .\" You may obtain a copy of the License at .\" .\" http://www.apache.org/licenses/LICENSE-2.0 .\" .\" Unless required by applicable law or agreed to in writing, software .\" distributed under the License is distributed on an "AS IS" BASIS, .\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .\" See the License for the specific language governing permissions and .\" limitations under the License. .\" .Dd 2018-01-30 .Dt dnsdbq 1 DNSDB .Os " " .Sh NAME .Nm dnsdbq .Nd DNSDB query tool .Sh SYNOPSIS .Nm dnsdbq .Op Fl acdfgGhIjmqSsUv468 .Op Fl A Ar timestamp .Op Fl B Ar timestamp .Op Fl b Ar bailiwick .Op Fl D Ar asn_domain .Op Fl i Ar ip .Op Fl J Ar input_file .Op Fl k Ar sort_keys .Op Fl L Ar output_limit .Op Fl l Ar query_limit .Op Fl M Ar max_count .Op Fl C Ar cookie_file .Op Fl N Ar hex[/rrtype[,...]] .Op Fl n Ar name[/rrtype[,...]] .Op Fl O Ar offset .Op Fl o Ar timeout .Op Fl p Ar output_type .Op Fl R Ar hex[/rrtype[,...][/bailiwick]] .Op Fl r Ar name[/rrtype[,...][/bailiwick]] .Op Fl T Ar transform[,...] .Op Fl t Ar rrtype[,...] .Op Fl u Ar server_sys .Op Fl V Ar verb .Op Fl 0 Ar function=thing .Sh DESCRIPTION .Nm dnsdbq constructs and issues queries to Passive DNS systems which return data in the IETF Passive DNS Common Output Format. It is commonly used as a production command line interface to such systems. .Pp .Nm dnsdbq displays responses in various formats. Its default query type is a "lookup" query. As an option, it can issue a "summarize" query type. Different Passive DNS systems or versions of those systems may implement different query features. .Sh FARSIGHT SECURITY'S "DNSDB" Farsight Security's "DNSDB" is one such Passive DNS system which is accessed by specifying system "dnsdb". .Pp You'll need to get an API key from Farsight to use .Ic dnsdbq with DNSDB. .Pp Farsight's passive DNS infrastructure performs a complex process of "bailiwick reconstruction" where an RRset's position within the DNS hierarchy is approximated. This serves two purposes: .Bl -enum -offset indent .It Provide context of the location of a given DNS record within the DNS hierarchy. .It Prevent "untrustworthy" records that are a result of intentional or unintentional cache poisoning attempts from being replicated by downstream consumers. .El .Pp For example, given the fully qualified domain name .Ic www.dachshund.example.com , valid bailiwicks would be .Ic dachshund.example.com , .Ic example.com , or .Ic com . .Sh OPTIONS .Bl -tag -width 3n .It Fl a enables ASINFO/CIDR annotation for IP addresses in A (IPv4 address) RRsets. The metadata thus appended depends on which data source is given by .Fl D . .It Fl A Ar timestamp Specify a forward time fence. Only results seen by the passive DNS on or after this time will be selected. See also .Fl c . See the TIMESTAMP FORMATS section for more information about this. .It Fl B Ar timestamp Specify a backward time fence. Only results seen by the passive DNS sensor network on or before this time will be selected. See also .Fl c . See the TIMESTAMP FORMATS section for more information about this. .It Fl b Ar bailiwick specify bailiwick (only valid with .Fl r queries). .It Fl c by default, .Fl A and .Fl B (separately or together) will select partial overlaps of database tuples and time search criteria. To match only completely bracketed tuples, add the .Fl c ("completeness") flag (this is also known as "strict" mode). Can only be specified once, and for reasons of geometry, affects both .Fl A and .Fl B if both are specified. .It Fl C specify a cookie file to be used by libcurl (read-only; analogous to .Xr curl 1 -b). Typically this will be used for service authentication. .It Fl D specify the data source for ASINFO/CIDR annotations, if enabled by .Fl a . Default is .Ic "asn.routeviews.org" , but you may wish to try .Ic "aspath.routeviews.org" . .It Fl d enable debug mode. Repeat for more debug output. .It Fl f specify batch lookup mode allowing one or more queries to be performed. Queries will be read from standard input and are expected to be in one of the following formats: .Bl -dash -offset indent .It RRset (name) query: .Ic rrset/name/NAME[/RRTYPE[,...][/BAILIWICK]] .It RRset (raw) query: .Ic rrset/raw/HEX[/RRTYPE[,...][/BAILIWICK]] .It Rdata (name) query: .Ic rdata/name/NAME[/RRTYPE][,...] .It Rdata (IP address) query: .Ic rdata/ip/ADDR[,PFXLEN] .It Rdata (raw) query: .Ic rdata/raw/HEX[/RRTYPE][,...] .It Change query options: .Ic $OPTIONS {options} .El .Pp Where .Ic options :== .Bd -literal -offset 4n [-A\ timestamp] [-B\ timestamp] [-c] [-g] [-G] [-l\ query_limit] [-L\ output_limit] [-O\ offset] .Ed .Pp $OPTIONS alone on a line allows command line options to be changed mid-batch. If no options are given, the query parameters will be reset to those given on the command line, if any, or else to defaults. .Pp A line starting with a # will be ignored as a comment. .Pp Any internal slash (/) or comma (,) characters within the search names of a batch entry must be URL-encoded (for example, %2F or %2C). So, to search for the domain "212.0/24.150.104.24.in-addr.arpa" the search string would be specified as "212.0%2F150.104.24.in-addr.arpa". .Pp For raw queries, the hex value is an even number of hexadecimal digits specifying a raw octet string. The "raw" wire-format encodings are standardized. The embedding of these in dnstable is documented in the .Xr dnstable-encoding 5 manual page. This topic is explained in detail at . .Pp In batch lookup mode, each answer will be followed by a '--' marker, so that programmatic users will know when it is safe to send the next lookup, or if lookups are pipelined, to know when one answer has ended and another begun. This option cannot be mixed with .Fl n , .Fl r , .Fl R , or .Fl i . See the EXAMPLES section for more information on how to use .Fl f . .Pp If two .Fl f options are given, then each answer will also be preceded by a '++' marker giving the query string (as read from the batch input) in order to identify each answer when a very large batch input is given, and the '--' marker will include an error/noerror indicator and a short message describing the outcome. With two .Fl f options and also .Fl m , answers can appear in a different order than the batched questions, and the '--' and '++' markers, which are not valid JSON, are therefore suppressed. .It Fl g return graveled results if available. The default is to return aggregated results ("rocks"). Gravel is a feature for providing Volume Across Time. Note that not all Passive DNS system APIs support this feature, and not all time ranges contain granular results ("gravel"). .It Fl G undo the effect of .Fl g , this returning rocks rather than gravel. (Used in $OPTIONS in batch files.) .It Fl h emit usage and quit. .It Fl I request information from the API server concerning the API key itself, which may include rate limit, query quota, query allowance, or privilege levels; the output format and content is dependent on the server_sys argument (see .Ic -u ) and upon the .Fl p argument. .Ic -I -p json prints the raw info. .Ic -I -p text prints the information in a more understandable textual form, including converting any epoch integer times into UTC formatted times. .It Fl i Ar ip specify rdata ip ("right-hand side") query. The value is one of an IPv4 address, an IPv6 address, an IPv4 network with prefix length, an IPv4 address range, or an IPv6 network with prefix length. If a network lookup is being performed, the delimiter between network address and prefix length is a single comma (",") character rather than the usual slash ("/") character to avoid clashing with the HTTP URI path name separator. See EXAMPLES section for more information about separator substitution rules. .It Fl J Ar input_file opens input_file and reads newline-separated JSON objects therefrom, in preference to -f (batch mode) or query mode. This can be used to reprocess the output from a prior invocation which used .Fl j (or .Fl p json). Sorting, limits, and time fences will work. Specification of a domain name, RRtype, Rdata, or offset is not supported at this time. If input_file is "-" then standard input (stdin) will be read. .It Fl j synonym for .Fl p json. .It Fl k Ar sort_keys when sorting with -s or -S, selects one or more comma separated sort keys, among "first", "last", "duration", "count", "name", "type", and/or "data". The default order is be "first,last,duration,count,name,type,data" (if sorting is requested.) Names are sorted right to left (by TLD then 2LD etc). Data is sorted either by name if present, or else by numeric value (e.g., for A and AAAA RRsets.) Several .Fl k options can be given after different .Fl s and .Fl S options, to sort in ascending order for some keys, descending for others. .It Fl l Ar query_limit query for that limit's number of responses. If specified as 0 then the DNSDB API server will return the maximum limit of results allowed. If .Fl l , is not specified, then the query will not specify a limit, and the DNSDB API server may use its default limit. .It Fl L Ar output_limit clamps the number of objects per response (under .Fl [R|r|N|n|i|f] ) or for all responses (under .Fl [fm|ff|ffm] ) output to .Ic output_limit . If unset, and if batch and merge modes have not been selected with the .Fl f and .Fl m options, then the .Fl L output limit defaults to the .Fl l limit's value. Otherwise the default is no output limit. .It Fl M Ar max_count for the summarize verb, stops summarizing when the count reaches that max_count, which must be a positive integer. The resulting total count may exceed max_count as it will include the entire count from the last RRset examined. The default is to not constrain the maximum count. The number of RRsets summarized is also limited by the query_limit. .It Fl m used only with .Fl f , this causes multiple (up to ten) API queries to execute in parallel. In this mode there will be no "--" marker, and the combined output of all queries is what will be subject to sorting, if any. If two .Fl f flags are specified with .Fl m , the output will not be merged, can appear in any order, will be sorted separately for each response, and will have normal '--' and '++' markers. (See .Fl f option above.) .It Fl N Ar hex[/rrtype[,...]] specify raw .Ic rdata data ("right-hand side") query. Hex is as described above for .Fl f . .It Fl n Ar name specify .Ic rdata name ("right-hand side") query. The value is a DNS domain name in presentation format, or a left-hand ("*.example.com") or right-hand ("www.example.*") wildcard domain name. Note that left-hand wildcard queries are somewhat more expensive than right-hand wildcard queries. .It Fl O Ar offset to offset by #offset the results returned by the query. This gives you incremental results transfers. Cannot be negative. The default is 0 which means no offset. .It Fl o Ar timeout specifies the timeout, in seconds, for initial connection to database server and for each transaction made to that server. .It Fl p Ar output_type select output type. Specify: .Bl -tag -width "minimal" .It Cm text for presentation output meant to be human-readable. This is the default. .Cm dns is a synonym, for compatibility with older programmatic callers. .It Cm json for newline delimited JSON output. See also . .It Cm csv for comma separated value output. This format is information losing, since it cannot express multiple resource records that are in a single RRset. Instead, each resource record is expressed in a separate line of output. See the .Ic DNSDBQ_TIME_FORMAT environment variable for controlling how timestamps are formatted for this option. .It Cm minimal outputs only the owner name or rdata, one per line and deduplicated; for use by shell scripts. This is incompatible with sorting. .El .It Fl q makes the program reticent about warnings. .It Fl R Ar hex[/rrtype[,...][/bailiwick]] specify raw .Ic rrset owner data ("left-hand side") query. Hex is as described above for .Fl f . .It Fl r Ar name[/type[,...][/bailiwick]] specify RRset ("left-hand side") name query. See discussion in .Fl n above as to the format of and limitations on query names. .It Fl s sort output in ascending key order. Limits (if any) specified by .Fl l and .Fl L will be applied before and after sorting, respectively. In batch mode, the .Fl f , .Fl ff , and .Fl ffm option sets will cause each batch entry's result to be sorted independently, whereas with .Fl fm , all outputs will be combined before sorting. This means with .Fl fm there will be no output until after the last batch entry has been processed, due to store and forward by the sort process. .It Fl S sort output in descending key order. See discussion for .Fl s above. .It Fl T Ar transform[,...] specify one or more transforms to be applied to the output: .Bl -tag -width "datefix" .It Cm datefix always show dates in the format selected by the DNSDBQ_TIME_FORMAT environment variable, not in database format. .It Cm reverse show the DNS owner name (rrname) in TLD-first order (so, COM.EXAMPLE rather than EXAMPLE.COM). .It Cm chomp strip away the trailing dot (.) from the DNS owner name (rrname). .It Cm qdetail annotate the response to include query details, mostly for use by .Fl V Ar summarize. This is incompatible with sorting. .El .It Fl t Ar rrtype[,...] specify the resource record type(s) desired. Default is ANY. If present, this option should precede any .Fl R , .Fl r , .Fl N , or .Fl n options. This option is not allowed if the .Fl i option is present. Valid values include those defined in DNS RFCs, including ANY. A special-case supported in DNSDB is ANY-DNSSEC, which matches on DS, RRSIG, NSEC, DNSKEY, NSEC3, NSEC3PARAM, and DLV resource record types. .Pp If multiple .Ar rrtype values are specified, each will be sent separately to the database server, consuming quota if there is a quota. Such queries will be sent simultaneously in parallel, which may have a load impact on the server. .It Fl u Ar server_sys specifies the Passive DNS system and thus its syntax for RESTful URLs. Can be "dnsdb" or "circl". The default is "dnsdb". See also environment variable DNSDBQ_SYSTEM. .It Fl V Ar verb The verb to perform, i.e. the type of query, either "lookup" or "summarize". The default is the "lookup" verb. As an option, you can specify the "summarize" verb, which gives you an estimate of result size. At a glance, it provides information on when a given domain name, IP address or other DNS asset was first-seen and last-seen by the global sensor network, as well as the total sensor network observation count. This verb respects the database limit (see .Fl l ) in that the resulting summary will only be of rows that would have been returned by the "lookup" verb. See also .Fl M . .It Fl 0 Ar function=thing This is a developer tool meant to feed automated testing systems. .It Fl U turns off TLS certificate verification (unsafe). .It Fl v report the version of dnsdbq and exit. .It Fl 4 use to force connecting to the DNSDB server via IPv4. .It Fl 6 use to force connecting to the DNSDB server via IPv6. .It Fl 8 Normally dnsdbq requires that .Fl n or .Fl r arguments are 7-bit ASCII clean. Non-ASCII values should be queried using PUNYCODE IDN encoding. This .Fl 8 option allows using arbitrary 8 bit values. .El .Sh "TIMESTAMP FORMATS" Timestamps may be one of following forms. .Bl -dash -offset indent .It positive unsigned integer : in Unix epoch format. .It negative unsigned integer : negative offset in seconds from now. .It YYYY-MM-DD [HH:MM:SS] : in absolute form, in UTC time, as DNSDB does its fencing using UTC time. .It %uw%ud%uh%um%us : the relative form with explicit labels (w=weeks, d=days, h=hours, m=minutes, s=seconds). Calculates offset from UTC time, as DNSDB does its fencing using UTC time. .Pp .El When using batch mode with the second or forth cases, using relative times to now, the value for "now" is set when dnsdbq starts. .Pp A few examples of how to use timefencing options. .Bd -literal -offset 4n # tuples ending after Aug 22, 2015 (midnight) $ dnsdbq ... -A 2015-08-22 # tuples starting before Jan 22, 2013 (midnight) $ dnsdbq ... -B 2013-01-22 # tuples starting or ending from 2015 (midnight to midnight) $ dnsdbq ... -B 2016-01-01 -A 2015-01-01 # tuples ending after 2015-08-22 14:36:10 $ dnsdbq ... -A "2015-08-22 14:36:10" # tuples ending within the last 60 minutes $ dnsdbq ... -A "-3600" # tuples ending after "just now" $ date +%s 1485284066 $ dnsdbq ... -A 1485284066 # batch mode with only tuples ending within last 60 minutes, # even if feeding inputs to dnsdbq in batch mode takes hours. $ dnsdbq -f ... -A "-3600" .Ed .Sh EXAMPLES .Pp A few examples of how to specify IP address information. .Bd -literal -offset 4n # specify a single IPv4 address $ dnsdbq ... -i 128.223.32.35 # specify an IPv4 CIDR $ dnsdbq ... -i 128.223.32.0/24 # specify a range of IPv4 addresses $ dnsdbq ... -i 128.223.32.0-128.223.32.32 .Ed .Pp Perform an RRset query for a single A record for .Ic farsightsecurity.com . The output is serialized as JSON and is piped to the .Ic jq program (a command-line JSON processor, see ) for pretty printing. .Bd -literal -offset 4n $ dnsdbq -r farsightsecurity.com/A -l 1 -j -a | jq . { "count": 6350, "time_first": 1380123423, "time_last": 1427869045, "rrname": "farsightsecurity.com.", "rrtype": "A", "bailiwick": "farsightsecurity.com.", "rdata": [ "66.160.140.81" ], "dnsdbq-rdata": [ { "asinfo": [ 6939 ], "cidr": "66.160.128.0/18", "rdata": "66.160.140.81" } ] } .Ed .Pp Note the "dnsdbq-rdata" element added due to the use of the .Fl a option. .Pp Perform a batched operation for a several different .Ic rrset and .Ic rdata queries. Output is again serialized as JSON and redirected to a file. .Bd -literal -offset 4n $ cat batch.txt rrset/name/\*.wikipedia.org rrset/name/\*.dmoz.org rrset/raw/0366736902696f00/A rdata/name/\*.pbs.org rdata/name/\*.opb.org rdata/ip/198.35.26.96 rdata/ip/23.21.237.0,24 rdata/raw/0b763d73706631202d616c6c $ dnsdbq -j -f < batch.txt > batch-output.json $ head -1 batch-output.json | jq . { "count": 2411, "zone_time_first": 1275401003, "zone_time_last": 1484841664, "rrname": "wikipedia.org.", "rrtype": "NS", "bailiwick": "org.", "rdata": [ "ns0.wikimedia.org.", "ns1.wikimedia.org.", "ns2.wikimedia.org." ] } .Ed .Sh ASINFO/CIDR LOOKUPS When the .Fl a option is used, every address seen in a response will cause a DNS lookup under the domain specified by the .Fl D option. This stream of DNS queries might be an intolerable information leak depending on the nature of the underlying research, and it could also lead to unusably bad performance depending on the placement of your configured recursive DNS service. .Pp For best results, always use an on-server or on-LAN recursive DNS service, and consider whether to configure that recursive DNS service to be a "stealth secondary" of the zone denoted by the .Fl D option. For the default .Fl D value, more information can be found online at .Ic "http://archive.routeviews.org/dnszones/" . .Pp Use of DNS lookups to retrieve ASINFO/CIDR metadata can be extremely fast and surveillance-free, but some attention must be paid in order to obtain that outcome. For occasional low-volume use, your current recursive DNS placement and configuration is probably good enough. .Pp Note that while Passive DNS information is historical, the ASINFO/CIDR annotations made possible using the .Fl a and .Fl D options are based on current information. Internet routing system information may have changed since the DNS data was recorded. More information about this can be found online at .Ic "https://github.com/dnsdb/dnsdbq/blob/master/README" . .Sh FILES .Ic ~/.isc-dnsdb-query.conf , .Ic ~/.dnsdb-query.conf , .Ic /etc/isc-dnsdb-query.conf , or .Ic /etc/dnsdb-query.conf : configuration file which can specify the API key and other variables. The first of these files which is readable will be used, alone, in its entirety. See the .Ic DNSDBQ_CONFIG_FILE environment variable which can specify a different configuration file to use. .Pp The variables which can be set in the configuration file are as follows: .Bl -tag -width ".Ev DNSDB_API_KEY , APIKEY" .It Ev DNSDBQ_SYSTEM contains the default value for the .Fl u option described above. The last setting found for any given variable will prevail. .It Ev DNSDB_API_KEY , APIKEY contains the user's DNSDB apikey (no default). .It Ev DNSDB_SERVER contains the URL of the DNSDB API server (default is https://api.dnsdb.info), and optionally the URI prefix for the database. The default URI prefix for system "dnsdb2" is "/dnsdb/v2/lookup"; the default for "dnsdb1" is "/lookup". .It Ev CIRCL_AUTH , CIRCL_SERVER enable access to a passive DNS system compatible with the CIRCL.LU system. .El .Sh ENVIRONMENT .Bl -tag -width ".Ev DNSDBQ_CONFIG_FILE" .It Ev DNSDBQ_CONFIG_FILE specifies the configuration file to use, overriding the internal search list. .It Ev DNSDB_API_KEY contains the user's apikey. The older APIKEY environment variable has been retired, though it can still be used in the configuration file. Note that environment variables are unprotected, and putting one's API key in an unprotected place could cause inadvertant sharing. .It Ev DNSDB_SERVER contains the URL of the DNSDB API server, and optionally a URI prefix to be used (default is "/lookup"). If not set, the configuration file is consulted. .It Ev DNSDBQ_TIME_FORMAT controls how human readable date times are presented in the output. If "iso" (the default) then ISO8601 (RFC3339) format is used, for example; "2018-09-06T22:48:00Z". If "csv" then an Excel CSV compatible format is used; for example, "2018-09-06 22:48:00". .It Ev HTTPS_PROXY contains the URL of the HTTPS proxy that you wish to use. See .Ic "https://curl.se/libcurl/c/CURLOPT_PROXY.html" for information on its values. .El .Sh "EXIT STATUS" Success (exit status zero) occurs if a connection could be established to the back end database server, even if no records matched the search criteria. Failure (exit status nonzero) occurs if no connection could be established, perhaps due to a network or service failure, or a configuration error such as specifying the wrong server hostname. .Sh "SEE ALSO" .Xr dig 1 , .Xr jq 1 , .Xr libcurl 3 , .Xr dnstable-encoding 5 , .Ic "https://curl.se/docs/http-cookies.html" dnsdbq-2.6.7/globals.h000066400000000000000000000064601462247651300146150ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef GLOBALS_H_INCLUDED #define GLOBALS_H_INCLUDED 1 #include #include "defs.h" #include "deduper.h" #include "sort.h" #include "time.h" #ifdef MAIN_PROGRAM #define EXTERN #define INIT(...) = __VA_ARGS__ #else #define EXTERN extern #define INIT(...) #endif #ifndef MAIN_PROGRAM extern const struct verb verbs[]; #endif EXTERN const char id_swclient[] INIT("dnsdbq"); EXTERN const char id_version[] INIT("2.6.7"); EXTERN const char *program_name INIT(NULL); EXTERN const char path_sort[] INIT("/usr/bin/sort"); EXTERN const char json_header[] INIT("Accept: application/json"); EXTERN const char jsonl_header[] INIT("Accept: application/x-ndjson"); EXTERN const char env_time_fmt[] INIT("DNSDBQ_TIME_FORMAT"); EXTERN const char env_config_file[] INIT("DNSDBQ_CONFIG_FILE"); EXTERN const char env_timeout[] INIT("DNSDBQ_TIMEOUT"); EXTERN const char status_noerror[] INIT("NOERROR"); EXTERN const char status_error[] INIT("ERROR"); EXTERN const char *asinfo_domain INIT("asn.routeviews.org"); EXTERN struct qparam qparam_empty INIT({ .query_limit = -1L, .explicit_output_limit = -1L, .output_limit = -1L }); EXTERN char *cookie_file INIT(NULL); EXTERN char *config_file INIT(NULL); EXTERN verb_ct pverb INIT(NULL); EXTERN pdns_system_ct psys INIT(NULL); EXTERN int debug_level INIT(0); EXTERN bool asinfo_lookup INIT(false); EXTERN bool donotverify INIT(false); EXTERN bool quiet INIT(false); EXTERN bool iso8601 INIT(false); EXTERN bool multiple INIT(false); EXTERN bool psys_specified INIT(false); EXTERN int transforms INIT(0); EXTERN long max_count INIT(0L); EXTERN sort_e sorting INIT(no_sort); EXTERN batch_e batching INIT(batch_none); EXTERN present_e presentation INIT(pres_none); EXTERN char *presentation_name INIT(NULL); EXTERN presenter_ct presenter INIT(NULL); EXTERN struct timeval startup_time INIT({}); EXTERN int exit_code INIT(0); EXTERN long curl_ipresolve INIT(CURL_IPRESOLVE_WHATEVER); EXTERN long curl_timeout INIT(0L); EXTERN deduper_t minimal_deduper INIT(NULL); /* deduplication table size. trades memory efficiency (an array of this many * pointers will be allocated under '-p minimal' conditions) against run time * efficiency (the string's hash will be modulo this size). */ EXTERN const size_t minimal_modulus INIT(10000); #undef INIT #undef EXTERN __attribute__((noreturn)) void my_exit(int); __attribute__((noreturn)) void my_panic(bool, const char *); /* my_logf -- annotate to stderr with program name and current time */ static inline void my_logf(const char *fmtstr, ...) { va_list ap; va_start(ap, fmtstr); fprintf(stderr, "%s [%s]: ", program_name, timeval_str(NULL, true)); vfprintf(stderr, fmtstr, ap); putc('\n', stderr); } #endif /*GLOBALS_H_INCLUDED*/ dnsdbq-2.6.7/netio.c000066400000000000000000000505721462247651300143060ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #define _GNU_SOURCE #define _BSD_SOURCE #define _DEFAULT_SOURCE #include #include #include #include #include #include #include "defs.h" #include "netio.h" #include "pdns.h" #include "globals.h" #include "time.h" static void io_drain(void); static void fetch_reap(fetch_t); static void fetch_done(fetch_t); static void fetch_unlink(fetch_t); static void last_fetch(fetch_t); static writer_t writers = NULL; static CURLM *multi = NULL; static bool curl_cleanup_needed = false; static query_t paused[MAX_FETCHES]; static int npaused = 0; const char saf_begin[] = "begin"; const char saf_ongoing[] = "ongoing"; const char saf_succeeded[] = "succeeded"; const char saf_limited[] = "limited"; const char saf_failed[] = "failed"; const char *saf_valid_conds[] = { saf_begin, saf_ongoing, saf_succeeded, saf_limited, saf_failed }; /* make_curl -- perform global initializations of libcurl. */ void make_curl(void) { curl_global_init(CURL_GLOBAL_DEFAULT); curl_cleanup_needed = true; multi = curl_multi_init(); if (multi == NULL) { my_logf("curl_multi_init() failed"); my_exit(1); } } /* unmake_curl -- clean up and discard libcurl's global state. */ void unmake_curl(void) { if (multi != NULL) { curl_multi_cleanup(multi); multi = NULL; } if (curl_cleanup_needed) { curl_global_cleanup(); curl_cleanup_needed = false; } } /* fetch -- given a url, tell libcurl to go fetch it, attach fetch to query. */ fetch_t create_fetch(query_t query, char *url) { fetch_t fetch = NULL; CURLMcode res; DEBUG(2, true, "fetch(%s)\n", url); CREATE(fetch, sizeof *fetch); fetch->query = query; query = NULL; fetch->easy = curl_easy_init(); if (fetch->easy == NULL) { /* an error will have been output by libcurl in this case. */ DESTROY(fetch); DESTROY(url); my_exit(1); } fetch->url = url; url = NULL; curl_easy_setopt(fetch->easy, CURLOPT_URL, fetch->url); if (donotverify) { curl_easy_setopt(fetch->easy, CURLOPT_SSL_VERIFYPEER, 0L); curl_easy_setopt(fetch->easy, CURLOPT_SSL_VERIFYHOST, 0L); } /* if user specified a prefence for IPv4 or IPv6, use it. */ if (curl_ipresolve != CURL_IPRESOLVE_WHATEVER) curl_easy_setopt(fetch->easy, CURLOPT_IPRESOLVE, curl_ipresolve); /* if user specified a timeout, use for connection and transactions. */ if (curl_timeout != 0L) { curl_easy_setopt(fetch->easy, CURLOPT_CONNECTTIMEOUT, curl_timeout); curl_easy_setopt(fetch->easy, CURLOPT_TIMEOUT, curl_timeout); } if (psys->auth != NULL) psys->auth(fetch); /* if user specified a cookie file, tell libcurl about it. */ if (cookie_file != NULL) curl_easy_setopt(fetch->easy, CURLOPT_COOKIEFILE, cookie_file); if (psys->encap == encap_saf) fetch->hdrs = curl_slist_append(fetch->hdrs, jsonl_header); else fetch->hdrs = curl_slist_append(fetch->hdrs, json_header); curl_easy_setopt(fetch->easy, CURLOPT_HTTPHEADER, fetch->hdrs); curl_easy_setopt(fetch->easy, CURLOPT_WRITEFUNCTION, writer_func); curl_easy_setopt(fetch->easy, CURLOPT_WRITEDATA, fetch); curl_easy_setopt(fetch->easy, CURLOPT_PRIVATE, fetch); #ifdef CURL_AT_LEAST_VERSION /* If CURL_AT_LEAST_VERSION is not defined then the curl is probably too old */ #if CURL_AT_LEAST_VERSION(7,42,0) /* do not allow curl to swallow /./ and /../ in our URLs */ curl_easy_setopt(fetch->easy, CURLOPT_PATH_AS_IS, 1L); #endif #endif /* CURL_AT_LEAST_VERSION */ if (debug_level >= 3) curl_easy_setopt(fetch->easy, CURLOPT_VERBOSE, 1L); /* linked-list insert. */ fetch->next = fetch->query->fetches; fetch->query->fetches = fetch; res = curl_multi_add_handle(multi, fetch->easy); if (res != CURLM_OK) { my_logf("curl_multi_add_handle() failed: %s", curl_multi_strerror(res)); my_exit(1); } return fetch; } /* fetch_reap -- reap one fetch. */ static void fetch_reap(fetch_t fetch) { if (fetch->easy != NULL) { curl_multi_remove_handle(multi, fetch->easy); curl_easy_cleanup(fetch->easy); fetch->easy = NULL; } if (fetch->hdrs != NULL) { curl_slist_free_all(fetch->hdrs); fetch->hdrs = NULL; } DESTROY(fetch->saf_msg); DESTROY(fetch->url); DESTROY(fetch->buf); DESTROY(fetch); } /* fetch_done -- deal with consequences of end-of-fetch. */ static void fetch_done(fetch_t fetch) { query_t query = fetch->query; /* if this was the last fetch on some query, signal. */ if (query->fetches == fetch && fetch->next == NULL) last_fetch(fetch); } /* fetch_unlink -- disconnect a fetch from its writer. */ static void fetch_unlink(fetch_t fetch) { fetch_t cur, prev; for (cur = fetch->query->fetches, prev = NULL; cur != NULL && cur != fetch; prev = cur, cur = cur->next) { } assert(cur == fetch); if (prev == NULL) fetch->query->fetches = fetch->next; else prev->next = fetch->next; fetch->query = NULL; } /* writer_init -- instantiate a writer, which may involve forking a "sort". */ writer_t writer_init(long output_limit, ps_user_t ps_user, bool meta_query) { writer_t writer = NULL; CREATE(writer, sizeof(struct writer)); writer->output_limit = output_limit; writer->ps_user = ps_user; writer->meta_query = meta_query; if (sorting != no_sort) { /* sorting involves a subprocess (POSIX sort(1) command), * which will by definition not output anything until * after it receives EOF. this means we can pipe both * to its stdin and from its stdout, without risk of * deadlock. it also means a full store-and-forward of * the result, which increases latency to the first * output for our user. */ int p1[2], p2[2]; if (pipe(p1) < 0 || pipe(p2) < 0) my_panic(true, "pipe"); if ((writer->sort_pid = fork()) < 0) my_panic(true, "fork"); if (writer->sort_pid == 0) exec_sort(p1, p2); close(p1[0]); writer->sort_stdin = fdopen(p1[1], "w"); writer->sort_stdout = fdopen(p2[0], "r"); close(p2[1]); } writer->next = writers; writers = writer; return writer; } void ps_stdout(writer_t writer) { fwrite(writer->ps_buf, 1, writer->ps_len, stdout); } /* query_status -- install a status code and description in a query. */ void query_status(query_t query, const char *status, const char *message) { assert((query->status == NULL) == (query->message == NULL)); if (query->multitype && query->status != NULL) { DESTROY(query->status); DESTROY(query->message); } else { assert(query->status == NULL); } query->status = strdup(status); query->message = strdup(message); } /* writer_func -- process a block of json text, from filesys or API socket. * * This function's signature must conform to write_callback() in * CURLOPT_WRITEFUNCTION. * Returns the number of bytes actually taken care of or returns * CURL_WRITEFUNC_PAUSE to pause this query's connection until * curl_easy_pause(..., CURLPAUSE_CONT) is called. */ size_t writer_func(char *ptr, size_t size, size_t nmemb, void *blob) { fetch_t fetch = (fetch_t) blob; query_t query = fetch->query; writer_t writer = query->writer; qparam_ct qp = &query->qp; size_t bytes = size * nmemb; char *nl; DEBUG(3, true, "writer_func(%d, %d): %d\n", (int)size, (int)nmemb, (int)bytes); /* if we're in asynchronous batch mode, only one query can reach * the writer at a time. fetches within a query can interleave. */ if (batching == batch_verbose) { if (multiple) { if (writer->active == NULL) { /* grab the token. */ writer->active = query; DEBUG(2, true, "active (%d) %s\n", npaused, query->descr); } else if (writer->active != query) { /* pause the query. */ paused[npaused++] = query; DEBUG(2, true, "pause (%d) %s\n", npaused, query->descr); return CURL_WRITEFUNC_PAUSE; } } if (!query->hdr_sent) { printf("++ %s\n", query->descr); query->hdr_sent = true; } } fetch->buf = realloc(fetch->buf, fetch->len + bytes); memcpy(fetch->buf + fetch->len, ptr, bytes); fetch->len += bytes; /* when the fetch is a live web result, emit * !2xx errors and info payloads as reports. */ if (fetch->easy != NULL) { if (fetch->rcode == 0) curl_easy_getinfo(fetch->easy, CURLINFO_RESPONSE_CODE, &fetch->rcode); if (fetch->rcode != HTTP_OK) { char *message = strndup(fetch->buf, fetch->len); /* only report the first line of data. */ char *eol = strpbrk(message, "\r\n"); if (eol != NULL) *eol = '\0'; /* only remember the first response status. */ if (query->status == NULL) query_status(query, psys->status(fetch), message); if (!quiet) { char *url; curl_easy_getinfo(fetch->easy, CURLINFO_EFFECTIVE_URL, &url); my_logf("warning: libcurl %ld [%s] %s", fetch->rcode, url, message); } DESTROY(message); fetch->buf[0] = '\0'; fetch->len = 0; return bytes; } } /* deblock. */ while ((nl = memchr(fetch->buf, '\n', fetch->len)) != NULL) { size_t pre_len = (size_t)(nl - fetch->buf), post_len = (fetch->len - pre_len) - 1; if (sorting == no_sort && writer->output_limit > 0 && writer->count >= writer->output_limit) { DEBUG(9, true, "hit output limit %ld\n", qp->output_limit); /* cause CURLE_WRITE_ERROR for this transfer. */ bytes = 0; if (psys->encap == encap_saf) fetch->saf_cond = sc_we_limited; /* inform io_engine() that the abort is intentional. */ fetch->stopped = true; } else if (writer->meta_query) { /* concatenate this fragment (incl \n) to ps_buf. */ writer->ps_buf = realloc(writer->ps_buf, writer->ps_len + pre_len + 1); memcpy(writer->ps_buf + writer->ps_len, fetch->buf, pre_len + 1); writer->ps_len += pre_len + 1; } else { query->writer->count += pdns_blob(fetch, pre_len); if (psys->encap == encap_saf) switch (fetch->saf_cond) { case sc_init: case sc_begin: case sc_ongoing: case sc_missing: break; case sc_succeeded: case sc_limited: case sc_failed: case sc_we_limited: /* inform io_engine() that the abort * is intentional. */ fetch->stopped = true; break; } } memmove(fetch->buf, nl + 1, post_len); fetch->len = post_len; } return bytes; } /* last_fetch -- do something with leftover buffer data when a query ends. */ static void last_fetch(fetch_t fetch) { query_t query = fetch->query; assert(query->fetches == fetch && fetch->next == NULL); DEBUG(2, true, "query_done(%s), meta=%d\n", query->descr, query->writer->meta_query); if (query->writer->meta_query) return; if (batching == batch_none && !quiet) { const char *msg = or_else(fetch->saf_msg, ""); if (fetch->saf_cond == sc_limited) my_logf("Database API limit: %s", msg); else if (fetch->saf_cond == sc_failed) my_logf("Database result: %s", msg); else if (fetch->saf_cond == sc_missing) my_logf("API response missing: %s", msg); else if (query->status != NULL && !query->multitype) my_logf("API status: %s (%s)", query->status, query->message); } else if (batching == batch_verbose) { /* if this was an actively written query, unpause another. */ writer_t writer = query->writer; if (multiple) { assert(writer->active == query); writer->active = NULL; } assert(writer->ps_buf == NULL && writer->ps_len == 0); writer->ps_len = (size_t) asprintf(&writer->ps_buf, "-- %s (%s)\n", or_else(query->status, status_noerror), or_else(query->message, or_else(fetch->saf_msg, "no error"))); if (npaused > 0) { query_t unpause; fetch_t ufetch; int i; /* unpause the next query's fetches. */ unpause = paused[0]; npaused--; for (i = 0; i < npaused; i++) paused[i] = paused[i + 1]; for (ufetch = unpause->fetches; ufetch != NULL; ufetch = ufetch->next) { DEBUG(2, true, "unpause (%d) %s\n", npaused, unpause->descr); curl_easy_pause(ufetch->easy, CURLPAUSE_CONT); } } } } /* writer_fini -- stop a writer's fetches, and perhaps execute a POSIX "sort". */ void writer_fini(writer_t writer) { /* unlink this writer from the global chain. */ if (writers == writer) { writers = writer->next; } else { writer_t prev = NULL; writer_t temp; for (temp = writers; temp != NULL; temp = temp->next) { if (temp->next == writer) { prev = temp; break; } } assert(prev != NULL); prev->next = writer->next; } /* finish and close any fetches still cooking. */ while (writer->queries != NULL) { query_t query = writer->queries, query_next = query->next; while (query->fetches != NULL) { fetch_t fetch = query->fetches, fetch_next = fetch->next; /* release any buffered info. */ DESTROY(fetch->buf); if (fetch->len != 0) { my_logf("warning: stranding %d octets!", (int)fetch->len); fetch->len = 0; } /* release any fetch-specific data. */ DESTROY(fetch->saf_msg); /* tear down any curl infrastructure on the fetch. */ fetch_reap(fetch); fetch = NULL; query->fetches = fetch_next; } assert((query->status != NULL) == (query->message != NULL)); DESTROY(query->status); DESTROY(query->message); DESTROY(query->descr); DESTROY(query); writer->queries = query_next; } /* drain the sort if there is one. */ if (writer->sort_pid != 0) { int status, count; char *line = NULL; size_t n = 0; /* when sorting, there has been no output yet. gather the * intermediate representation from the POSIX sort stdout, * skip over the sort keys we added earlier, and process. */ fclose(writer->sort_stdin); DEBUG(1, true, "closed sort_stdin, wrote %d objs\n", writer->count); count = 0; while (getline(&line, &n, writer->sort_stdout) > 0) { /* if we're above the limit, ignore remaining output. * this is nec'y to avoid SIGPIPE from sort if we were * to close its stdout pipe without emptying it first. */ if (writer->output_limit > 0 && count >= writer->output_limit) { if (!writer->sort_killed) { kill(writer->sort_pid, SIGTERM); writer->sort_killed = true; } continue; } struct pdns_tuple tup; char *nl, *linep; const char *msg; size_t len; if ((nl = strchr(line, '\n')) == NULL) { my_logf("warning: no \\n found in '%s'", line); continue; } linep = line; DEBUG(2, true, "sort1: '%*.*s'\n", (int)(nl - linep), (int)(nl - linep), linep); /* skip sort key: first */ if ((linep = strchr(linep, ' ')) == NULL) { my_logf("warning: no SP found in '%s'", line); continue; } linep += strspn(linep, " "); /* skip sort key: last */ if ((linep = strchr(linep, ' ')) == NULL) { my_logf("warning: no second SP in '%s'", line); continue; } linep += strspn(linep, " "); /* skip sort key: duration */ if ((linep = strchr(linep, ' ')) == NULL) { my_logf("warning: no third SP in '%s'", line); continue; } linep += strspn(linep, " "); /* skip sort key: count */ if ((linep = strchr(linep, ' ')) == NULL) { my_logf("warning: no fourth SP in '%s'", line); continue; } linep += strspn(linep, " "); /* skip sort key: rrname */ if ((linep = strchr(linep, ' ')) == NULL) { my_logf("warning: no fifth SP in '%s'", line); continue; } linep += strspn(linep, " "); /* skip sort key: rrtype */ if ((linep = strchr(linep, ' ')) == NULL) { my_logf("warning: no sixth SP in '%s'", line); continue; } linep += strspn(linep, " "); /* skip sort key: rdata */ if ((linep = strchr(linep, ' ')) == NULL) { my_logf("warning: no seventh SP in '%s'", line); continue; } linep += strspn(linep, " "); /* recover the json */ DEBUG(2, true, "sort2: '%*.*s'\n", (int)(nl - linep), (int)(nl - linep), linep); len = (size_t)(nl - linep); msg = tuple_make(&tup, linep, len); if (msg != NULL) { my_logf("warning: tuple_make: %s", msg); continue; } /* after the sort, we don't know what query * caused any given tuple. */ (*presenter->output)(&tup, NULL, writer); tuple_unmake(&tup); count++; } DESTROY(line); fclose(writer->sort_stdout); DEBUG(1, true, "closed sort_stdout, read %d objs (lim %ld)\n", count, writer->output_limit); if (waitpid(writer->sort_pid, &status, 0) < 0) { perror("waitpid"); } else { if (!writer->sort_killed && status != 0) my_logf("warning: sort exit status is %u", (unsigned)status); } } /* burp out the stored postscript, if any, and destroy it. */ if (writer->ps_len > 0) { assert(writer->ps_user != NULL); writer->ps_user(writer); DESTROY(writer->ps_buf); writer->ps_len = 0; } DESTROY(writer); } void unmake_writers(void) { while (writers != NULL) writer_fini(writers); } /* io_engine -- let libcurl run until there are few enough outstanding jobs. */ void io_engine(int jobs) { int still, repeats, numfds; DEBUG(2, true, "io_engine(%d)\n", jobs); /* let libcurl run while there are too many jobs remaining. */ still = 0; repeats = 0; while (curl_multi_perform(multi, &still) == CURLM_OK && still > jobs) { DEBUG(3, true, "...waiting (still %d)\n", still); numfds = 0; if (curl_multi_wait(multi, NULL, 0, 0, &numfds) != CURLM_OK) break; if (numfds == 0) { /* curl_multi_wait() can return 0 fds for no reason. */ if (++repeats > 1) { struct timespec req, rem; req = (struct timespec){ .tv_sec = 0, .tv_nsec = 100*1000*1000 // 100ms }; while (nanosleep(&req, &rem) == EINTR) { /* as required by nanosleep(3). */ req = rem; } } } else { repeats = 0; } io_drain(); } io_drain(); } /* io_drain -- drain the response code reports. */ static void io_drain(void) { struct CURLMsg *cm; int still = 0; while ((cm = curl_multi_info_read(multi, &still)) != NULL) { fetch_t fetch; query_t query; char *private; curl_easy_getinfo(cm->easy_handle, CURLINFO_PRIVATE, &private); fetch = (fetch_t) private; query = fetch->query; if (cm->msg == CURLMSG_DONE) { if (fetch->rcode == 0) curl_easy_getinfo(fetch->easy, CURLINFO_RESPONSE_CODE, &fetch->rcode); DEBUG(2, true, "io_drain(%s) DONE rcode=%d\n", query->descr, fetch->rcode); if (psys->encap == encap_saf) { if (fetch->saf_cond == sc_begin || fetch->saf_cond == sc_ongoing) { /* stream ended without a terminating * SAF value, so override stale value * we received before the problem. */ fetch->saf_cond = sc_missing; fetch->saf_msg = strdup( "Data transfer failed " "-- No SAF terminator " "at end of stream"); query_status(query, status_error, fetch->saf_msg); } DEBUG(2, true, "... saf_cond %d saf_msg %s\n", fetch->saf_cond, or_else(fetch->saf_msg, "")); } if (cm->data.result == CURLE_COULDNT_RESOLVE_HOST) { my_logf("libcurl failed since " "could not resolve host"); exit_code = 1; } else if (cm->data.result == CURLE_COULDNT_CONNECT) { my_logf("libcurl failed since " "could not connect"); exit_code = 1; } else if (cm->data.result != CURLE_OK && !fetch->stopped) { my_logf("libcurl failed with " "curl error %d (%s)", cm->data.result, curl_easy_strerror(cm->data.result)); exit_code = 1; } /* record emptiness as status if nothing else. */ if (psys->encap == encap_saf && query->writer != NULL && !query->writer->meta_query && query->writer->count == 0 && query->status == NULL) { query_status(query, status_noerror, "no results found for query."); } fetch_done(fetch); fetch_unlink(fetch); fetch_reap(fetch); } DEBUG(3, true, "...info read (still %d)\n", still); } } /* escape -- HTML-encode a string, returns a string which must be free()'d. */ char * escape(const char *str) { char *escaped, *ret; if (str == NULL) return NULL; escaped = curl_escape(str, (int)strlen(str)); if (escaped == NULL) { my_logf("curl_escape(%s) failed", str); my_exit(1); } ret = strdup(escaped); curl_free(escaped); escaped = NULL; return ret; } dnsdbq-2.6.7/netio.h000066400000000000000000000061071462247651300143060ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef NETIO_H_INCLUDED #define NETIO_H_INCLUDED 1 #include #include /* encapsulation protocol. ruminate, DNBDB APIv1 and CIRCL use encap_cof. */ typedef enum { encap_cof = 0, encap_saf } encap_e; /* official SAF condition values, plus sc_init, sc_we_limited, and sc_missing. */ typedef enum { sc_init = 0, /* initial condition */ /* official */ sc_begin, sc_ongoing, sc_succeeded, sc_limited, sc_failed, sc_we_limited, /* we noticed we hit the output limit */ sc_missing /* cond was missing at end of input stream */ } saf_cond_e; /* search parameters, per query and globally. */ struct qparam { u_long after; u_long before; long query_limit; /* actually set on the command line or in OPTIONS */ long explicit_output_limit; /* inferred and used in output code */ long output_limit; long offset; bool complete; bool gravel; }; typedef struct qparam *qparam_t; typedef const struct qparam *qparam_ct; /* one API fetch; several may be needed for complex (multitype) queries. */ struct fetch { struct fetch *next; struct query *query; CURL *easy; struct curl_slist *hdrs; char *url; char *buf; size_t len; long rcode; bool stopped; saf_cond_e saf_cond; char *saf_msg; }; typedef struct fetch *fetch_t; /* one query; one per invocation (or per batch line.) */ struct query { struct query *next; struct fetch *fetches; struct writer *writer; struct qparam qp; char *descr; mode_e mode; bool multitype; /* invariant: (status == NULL) == (writer == NULL) */ char *status; char *message; bool hdr_sent; }; typedef struct query *query_t; typedef const struct query *query_ct; typedef void (*ps_user_t)(struct writer *); /* one output stream, having one or several queries merging into it. */ struct writer { struct writer *next; struct query *queries; struct query *active; FILE *sort_stdin; FILE *sort_stdout; pid_t sort_pid; bool sort_killed; bool csv_headerp; bool meta_query; char *ps_buf; size_t ps_len; ps_user_t ps_user; long output_limit; int count; }; typedef struct writer *writer_t; void make_curl(void); void unmake_curl(void); fetch_t create_fetch(query_t, char *); writer_t writer_init(long, ps_user_t, bool); void ps_stdout(writer_t); void query_status(query_t, const char *, const char *); size_t writer_func(char *ptr, size_t size, size_t nmemb, void *blob); void writer_fini(writer_t); void unmake_writers(void); void io_engine(int); char *escape(const char *); #endif /*NETIO_H_INCLUDED*/ dnsdbq-2.6.7/ns_ttl.c000066400000000000000000000063201462247651300144630ustar00rootroot00000000000000/* * Copyright (c) 2004 by Internet Systems Consortium, Inc. ("ISC") * Copyright (c) 1996,1999 by Internet Software Consortium. * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL ISC BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT * OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ /* ISC "Id: ns_ttl.c,v 8.10 2005/03/31 02:02:29 marka Exp $" */ /* Import. */ #include #include #include #include #include #include "ns_ttl.h" /* Forward. */ static int fmt1(u_long t, char s, char **buf, size_t *buflen); /* Macros. */ #define T(x) {if ((x) < 0) return -1; else (void)NULL;} #define A(c) {if (dstlen < 2) return -1; else *dst++ = c, *dst = '\0';} /* Public. */ int ns_format_ttl(u_long src, char *dst, size_t dstlen) { u_long secs, mins, hours, days, years; char *odst = dst; secs = src % 60; src /= 60; mins = src % 60; src /= 60; hours = src % 24; src /= 24; days = src % 365; src /= 365; years = src; src = 0; if (years != 0) { A('~'); T(fmt1(years, 'y', &dst, &dstlen)); } if (days != 0) { if (dst != odst) A('\040'); A('~'); T(fmt1(days, 'd', &dst, &dstlen)); } if (years == 0 && hours != 0) { if (dst != odst) A('\040'); T(fmt1(hours, 'h', &dst, &dstlen)); } if (years == 0 && mins != 0) { if (dst != odst) A('\040'); T(fmt1(mins, 'm', &dst, &dstlen)); } if ((years == 0 && days == 0 && secs != 0) || (years == 0 && days == 0 && hours == 0 && mins == 0)) { if (dst != odst) A('\040'); T(fmt1(secs, 's', &dst, &dstlen)); } return (int)(dst - odst); } int ns_parse_ttl(const char *src, u_long *dst) { u_long ttl, tmp; int ch, digits, dirty; ttl = 0; tmp = 0; digits = 0; dirty = 0; while ((ch = *src++) != '\0') { if (!isascii(ch) || !isprint(ch)) goto einval; if (isdigit(ch)) { tmp *= 10; tmp += (unsigned)(ch - '0'); digits++; continue; } if (digits == 0) goto einval; if (islower(ch)) ch = toupper(ch); switch (ch) { case 'W': tmp *= 7; /* FALLTHROUGH */ case 'D': tmp *= 24; /* FALLTHROUGH */ case 'H': tmp *= 60; /* FALLTHROUGH */ case 'M': tmp *= 60; /* FALLTHROUGH */ case 'S': break; default: goto einval; } ttl += tmp; tmp = 0; digits = 0; dirty = 1; } if (digits > 0) { if (dirty) goto einval; else ttl += tmp; } else if (!dirty) goto einval; *dst = ttl; return 0; einval: errno = EINVAL; return -1; } /* Private. */ static int fmt1(u_long t, char s, char **buf, size_t *buflen) { char tmp[50]; size_t len; len = (size_t) sprintf(tmp, "%lu%c", t, s); if (len + 1 > *buflen) return -1; strcpy(*buf, tmp); *buf += len; *buflen -= len; return 0; } dnsdbq-2.6.7/ns_ttl.h000066400000000000000000000013671462247651300144760ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef __NS_TTL_H #define __NS_TTL_H 1 int ns_format_ttl(u_long, char *, size_t); int ns_parse_ttl(const char *src, u_long *dst); #endif dnsdbq-2.6.7/pdns.c000066400000000000000000000747421462247651300141410ustar00rootroot00000000000000/* * Copyright (c) 2014-2021 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ /* asprintf() does not appear on linux without this */ #define _GNU_SOURCE #include #include #include #include #include #include "asinfo.h" #include "defs.h" #include "netio.h" #include "ns_ttl.h" #include "pdns.h" #include "time.h" #include "tokstr.h" #include "globals.h" static void present_text_line(const char *, const char *, const char *); static void present_csv_line(pdns_tuple_ct, const char *); static void present_minimal_thing(const char *thing); static void present_json(pdns_tuple_ct, query_ct, bool); static json_t *annotate_json(pdns_tuple_ct, query_ct, bool); static json_t *annotation_json(query_ct query, json_t *annoRD); static json_t *annotate_one(json_t *, const char *, const char *, json_t *); #ifndef CRIPPLED_LIBC static json_t *annotate_asinfo(const char *, const char *); #endif static struct counted *countoff_r(const char *, int); /* present_text_lookup -- render one pdns tuple in "dig" style ascii text. */ void present_text_lookup(pdns_tuple_ct tup, query_ct query __attribute__ ((unused)), writer_t writer __attribute__ ((unused))) { bool pflag, ppflag; const char *prefix; ppflag = false; /* Timestamps. */ if (tup->obj.time_first != NULL && tup->obj.time_last != NULL) { char duration[50]; if (ns_format_ttl(tup->time_last - tup->time_first + 1, //non-0 duration, sizeof duration) < 0) strcpy(duration, "?"); printf(";; record times: %s", time_str(tup->time_first, iso8601)); printf(" .. %s (%s)\n", time_str(tup->time_last, iso8601), duration); ppflag = true; } if (tup->obj.zone_first != NULL && tup->obj.zone_last != NULL) { char duration[50]; if (ns_format_ttl(tup->zone_last - tup->zone_first, // no +1 duration, sizeof duration) < 0) strcpy(duration, "?"); printf(";; zone times: %s", time_str(tup->zone_first, iso8601)); printf(" .. %s (%s)\n", time_str(tup->zone_last, iso8601), duration); ppflag = true; } /* Count and Bailiwick. */ prefix = ";;"; pflag = false; if (tup->obj.count != NULL) { printf("%s count: %lld", prefix, (long long)tup->count); prefix = ";"; pflag = true; ppflag = true; } if (tup->obj.bailiwick != NULL) { printf("%s bailiwick: %s", prefix, tup->bailiwick); prefix = NULL; pflag = true; ppflag = true; } if (pflag) putchar('\n'); /* Records. */ if (json_is_array(tup->obj.rdata)) { size_t index; json_t *rr; json_array_foreach(tup->obj.rdata, index, rr) { const char *rdata = NULL; if (json_is_string(rr)) rdata = json_string_value(rr); else rdata = "[bad value]"; present_text_line(tup->rrname, tup->rrtype, rdata); ppflag = true; } } else { present_text_line(tup->rrname, tup->rrtype, tup->rdata); ppflag = true; } /* Cleanup. */ if (ppflag) putchar('\n'); } /* present_text_line -- render one RR in "dig" style ascii text. */ static void present_text_line(const char *rrname, const char *rrtype, const char *rdata) { char *asnum = NULL, *cidr = NULL, *comment = NULL, *result = NULL; #ifndef CRIPPLED_LIBC result = asinfo_from_rr(rrtype, rdata, &asnum, &cidr); #endif if (result != NULL) { comment = result; result = NULL; } else if (asnum != NULL && cidr != NULL) { const char *src = asnum; bool wordbreak = true; char ch, *dst; dst = comment = malloc(strlen(asnum) * 3 + strlen(cidr) + 1); while ((ch = *src++) != '\0') { if (wordbreak) { *dst++ = 'A'; *dst++ = 'S'; } *dst++ = ch; wordbreak = (ch == '\040'); } *dst++ = '\040'; dst = stpcpy(dst, cidr); free(asnum); free(cidr); } printf("%s %s %s", rrname, rrtype, rdata); if (comment != NULL) { printf(" ; %s", comment); free(comment); } putchar('\n'); } /* present_text_summ -- render summarize object in "dig" style ascii text. */ void present_text_summarize(pdns_tuple_ct tup, query_ct query __attribute__ ((unused)), writer_t writer __attribute__ ((unused))) { const char *prefix; /* Timestamps. */ if (tup->obj.time_first != NULL && tup->obj.time_last != NULL) { printf(";; record times: %s", time_str(tup->time_first, iso8601)); printf(" .. %s\n", time_str(tup->time_last, iso8601)); } if (tup->obj.zone_first != NULL && tup->obj.zone_last != NULL) { printf(";; zone times: %s", time_str(tup->zone_first, iso8601)); printf(" .. %s\n", time_str(tup->zone_last, iso8601)); putchar('\n'); } /* Count and Num_Results. */ prefix = ";;"; if (tup->obj.count != NULL) { printf("%s count: %lld", prefix, (long long)tup->count); prefix = ";"; } if (tup->obj.num_results != NULL) { printf("%s num_results: %lld", prefix, (long long)tup->num_results); prefix = NULL; } putchar('\n'); } /* pprint_json -- pretty-print a JSON buffer after validation. * * returns true if could parse the json ok, otherwise returns false. */ bool pprint_json(const char *buf, size_t len, FILE *outf) { json_error_t error; json_t *js = json_loadb(buf, len, 0, &error); if (js == NULL) { my_logf("JSON parsing error %d:%d: %s %s", error.line, error.column, error.text, error.source); return false; } json_dumpf(js, outf, JSON_INDENT(2)); fputc('\n', outf); json_decref(js); return true; } /* present_json_lookup -- render one DNSDB tuple as newline-separated JSON. */ void present_json_lookup(pdns_tuple_ct tup, query_ct query __attribute__ ((unused)), writer_t writer __attribute__ ((unused))) { present_json(tup, query, true); } /* present_json_summarize -- render one DNSDB tuple as newline-separated JSON. */ void present_json_summarize(pdns_tuple_ct tup, query_ct query __attribute__ ((unused)), writer_t writer __attribute__ ((unused))) { present_json(tup, query, false); } /* present_json -- shared renderer for DNSDB JSON tuples (lookup and summarize) */ static void present_json(pdns_tuple_ct tup, query_ct query, bool rd) { json_t *copy = annotate_json(tup, query, rd); if (copy != NULL) { json_dumpf(copy, stdout, JSON_INDENT(0) | JSON_COMPACT); json_decref(copy); } else { json_dumpf(tup->obj.cof_obj, stdout, JSON_INDENT(0) | JSON_COMPACT); } putchar('\n'); } /* annotate_json -- create a temporary copy of a tuple; apply transforms. */ static json_t * annotate_json(pdns_tuple_ct tup, query_ct query, bool rd) { json_t *annoRD = NULL, *annoTF = NULL, *annoTL = NULL, *annoZF = NULL, *annoZL = NULL; /* annotate zone first/last? */ if ((transforms & TRANS_DATEFIX) != 0 && tup->obj.zone_first != NULL && tup->obj.zone_last != NULL) { annoZF = json_string_nocheck(time_str(tup->zone_first, iso8601)); annoZL = json_string_nocheck(time_str(tup->zone_last, iso8601)); } /* annotate time first/last? */ if ((transforms & TRANS_DATEFIX) != 0 && tup->obj.time_first != NULL && tup->obj.time_last != NULL) { annoTF = json_string_nocheck(time_str(tup->time_first, iso8601)); annoTL = json_string_nocheck(time_str(tup->time_last, iso8601)); } /* annotate rdata? */ if (rd) { if (json_is_array(tup->obj.rdata)) { size_t index; json_t *rr; json_array_foreach(tup->obj.rdata, index, rr) { const char *rdata = json_string_value(rr); json_t *asinfo = NULL; #ifndef CRIPPLED_LIBC asinfo = annotate_asinfo(tup->rrtype, rdata); #endif if (asinfo != NULL) annoRD = annotate_one(annoRD, rdata, "asinfo", asinfo); } } else { json_t *asinfo = NULL; #ifndef CRIPPLED_LIBC asinfo = annotate_asinfo(tup->rrtype, tup->rdata); #endif if (asinfo != NULL) annoRD = annotate_one(annoRD, tup->rdata, "asinfo", asinfo); } } //rd? /* anything annotated? */ if ((annoZF != NULL && annoZL != NULL) || (annoTF != NULL && annoTL != NULL) || (transforms & (TRANS_REVERSE|TRANS_CHOMP|TRANS_QDETAIL)) != 0 || annoRD != NULL) { json_t *copy = json_deep_copy(tup->obj.cof_obj); if (annoZF != NULL || annoZL != NULL) { json_object_set_new_nocheck(copy, "zone_time_first", annoZF); json_object_set_new_nocheck(copy, "zone_time_last", annoZL); } if (annoTF != NULL || annoTL != NULL) { json_object_set_new_nocheck(copy, "time_first", annoTF); json_object_set_new_nocheck(copy, "time_last", annoTL); } if ((transforms & (TRANS_REVERSE|TRANS_CHOMP)) != 0) json_object_set_new_nocheck(copy, "rrname", json_string(tup->rrname)); if ((transforms & TRANS_QDETAIL) != 0 || annoRD != NULL) { json_t *obj = annotation_json(query, annoRD); if (obj != NULL) json_object_set_new_nocheck(copy, "_dnsdbq", obj); } return copy; } return NULL; } static inline void instantiate_json(json_t **objptr) { if (*objptr == NULL) *objptr = json_object(); } static json_t * annotation_json(query_ct query, json_t *annoRD) { json_t *obj = NULL; if (query != NULL && (transforms & TRANS_QDETAIL) != 0) { instantiate_json(&obj); if ((transforms & TRANS_QDETAIL) != 0) json_object_set_new_nocheck(obj, "descr", json_string(query->descr)); if (query->qp.after != 0) json_object_set_new_nocheck(obj, "after", json_string_nocheck( time_str(query->qp.after, iso8601))); if (query->qp.before != 0) json_object_set_new_nocheck(obj, "before", json_string_nocheck( time_str(query->qp.before, iso8601))); if (query->qp.query_limit != -1) json_object_set_new_nocheck(obj, "limit", json_integer((json_int_t) query->qp.query_limit)); if (query->qp.offset != 0) json_object_set_new_nocheck(obj, "offset", json_integer((json_int_t) query->qp.offset)); json_object_set_new_nocheck(obj, "gravel", json_boolean(query->qp.gravel)); json_object_set_new_nocheck(obj, "complete", json_boolean(query->qp.complete)); } if (annoRD != NULL) { instantiate_json(&obj); json_object_set_new_nocheck(obj, "anno", annoRD); } return obj; } static json_t * annotate_one(json_t *anno, const char *rdata, const char *name, json_t *obj) { json_t *this = NULL; bool new = false; if (anno == NULL) anno = json_object(); if ((this = json_object_get(anno, rdata)) == NULL) { this = json_object(); new = true; } json_object_set_new_nocheck(this, name, obj); if (new) json_object_set_new_nocheck(anno, rdata, this); else json_decref(this); return anno; } #ifndef CRIPPLED_LIBC static json_t * annotate_asinfo(const char *rrtype, const char *rdata) { char *asnum = NULL, *cidr = NULL, *result = NULL; json_t *asinfo = NULL; if ((result = asinfo_from_rr(rrtype, rdata, &asnum, &cidr)) != NULL) { asinfo = json_object(); json_object_set_new_nocheck(asinfo, "comment", json_string(result)); free(result); } else if (asnum != NULL && cidr != NULL) { json_t *array = json_array(); struct tokstr *ts = tokstr_string(asnum); for (char *t; (t = tokstr_next(ts, "\040")) != NULL; free(t)) json_array_append_new(array, json_integer(atol(t))); tokstr_last(&ts); asinfo = json_object(); json_object_set_new_nocheck(asinfo, "as", array); json_object_set_new_nocheck(asinfo, "cidr", json_string(cidr)); } DESTROY(asnum); DESTROY(cidr); return asinfo; } #endif /* present_csv_lookup -- render one DNSDB tuple as comma-separated values (CSV) */ void present_csv_lookup(pdns_tuple_ct tup, query_ct query __attribute__ ((unused)), writer_t writer) { if (!writer->csv_headerp) { printf("time_first,time_last,zone_first,zone_last," "count,bailiwick," "rrname,rrtype,rdata"); if (asinfo_lookup) fputs(",asnum,cidr", stdout); putchar('\n'); writer->csv_headerp = true; } if (json_is_array(tup->obj.rdata)) { size_t index; json_t *rr; json_array_foreach(tup->obj.rdata, index, rr) { const char *rdata = NULL; if (json_is_string(rr)) rdata = json_string_value(rr); else rdata = "[bad value]"; present_csv_line(tup, rdata); } } else { present_csv_line(tup, tup->rdata); } } /* present_csv_line -- display a CSV for one rdatum out of an rrset. */ static void present_csv_line(pdns_tuple_ct tup, const char *rdata) { /* Timestamps. */ if (tup->obj.time_first != NULL) printf("\"%s\"", time_str(tup->time_first, iso8601)); putchar(','); if (tup->obj.time_last != NULL) printf("\"%s\"", time_str(tup->time_last, iso8601)); putchar(','); if (tup->obj.zone_first != NULL) printf("\"%s\"", time_str(tup->zone_first, iso8601)); putchar(','); if (tup->obj.zone_last != NULL) printf("\"%s\"", time_str(tup->zone_last, iso8601)); putchar(','); /* Count and bailiwick. */ if (tup->obj.count != NULL) printf("%lld", (long long) tup->count); putchar(','); if (tup->obj.bailiwick != NULL) printf("\"%s\"", tup->bailiwick); putchar(','); /* Records. */ if (tup->obj.rrname != NULL) printf("\"%s\"", tup->rrname); putchar(','); if (tup->obj.rrtype != NULL) printf("\"%s\"", tup->rrtype); putchar(','); if (tup->obj.rdata != NULL) printf("\"%s\"", rdata); if (asinfo_lookup && tup->obj.rrtype != NULL && tup->obj.rdata != NULL) { char *asnum = NULL, *cidr = NULL, *result = NULL; #ifndef CRIPPLED_LIBC result = asinfo_from_rr(tup->rrtype, rdata, &asnum, &cidr); #endif if (result != NULL) { asnum = strdup(result); cidr = result; result = NULL; } putchar(','); if (asnum != NULL) { printf("\"%s\"", asnum); free(asnum); } putchar(','); if (cidr != NULL) { printf("\"%s\"", cidr); free(cidr); } } putchar('\n'); } /* present_minimal_lookup -- render one DNSDB tuple as a "line" */ void present_minimal_lookup(pdns_tuple_ct tup, query_ct query, writer_t writer __attribute__ ((unused))) { /* here is why this presenter is incompatible with sorting. */ assert(query != NULL); /* did this tuple come from a left hand or right hand query? */ bool left = true; switch (query->mode) { case no_mode: abort(); case rrset_mode: /* FALLTHROUGH */ case raw_rrset_mode: break; case name_mode: /* FALLTHROUGH */ case ip_mode: /* FALLTHROUGH */ case raw_name_mode: left = false; } /* for RHS queries, output the LHS once, and exit. */ if (!left) { present_minimal_thing(tup->rrname); return; } /* for LHS queries, output each RHS found. */ if (json_is_array(tup->obj.rdata)) { size_t index; json_t *rr; json_array_foreach(tup->obj.rdata, index, rr) { const char *rdata = NULL; if (json_is_string(rr)) rdata = json_string_value(rr); else rdata = "[bad value]"; present_minimal_thing(rdata); } } else { present_minimal_thing(tup->rdata); } } static void present_minimal_thing(const char *thing) { if (!deduper_tas(minimal_deduper, thing)) puts(thing); } /* present_csv_summarize -- render a summarize result as CSV. */ void present_csv_summarize(pdns_tuple_ct tup, query_ct query __attribute__ ((unused)), writer_t writer __attribute__ ((unused))) { printf("time_first,time_last,zone_first,zone_last," "count,num_results\n"); /* Timestamps. */ if (tup->obj.time_first != NULL) printf("\"%s\"", time_str(tup->time_first, iso8601)); putchar(','); if (tup->obj.time_last != NULL) printf("\"%s\"", time_str(tup->time_last, iso8601)); putchar(','); if (tup->obj.zone_first != NULL) printf("\"%s\"", time_str(tup->zone_first, iso8601)); putchar(','); if (tup->obj.zone_last != NULL) printf("\"%s\"", time_str(tup->zone_last, iso8601)); putchar(','); /* Count and num_results. */ if (tup->obj.count != NULL) printf("%lld", (long long) tup->count); putchar(','); if (tup->obj.num_results != NULL) printf("%lld", tup->num_results); putchar('\n'); } /* tuple_make -- create one DNSDB tuple object out of a JSON object. */ const char * tuple_make(pdns_tuple_t tup, const char *buf, size_t len) { const char *msg = NULL; json_error_t error; memset(tup, 0, sizeof *tup); DEBUG(4, true, "[%d] '%-*.*s'\n", (int)len, (int)len, (int)len, buf); tup->obj.main = json_loadb(buf, len, 0, &error); if (tup->obj.main == NULL) { my_logf("warning: json_loadb: %d:%d: %s %s", error.line, error.column, error.text, error.source); abort(); } if (debug_level >= 4) { char *pretty = json_dumps(tup->obj.main, JSON_INDENT(2)); my_logf("%s", pretty); free(pretty); } switch (psys->encap) { case encap_cof: /* the COF just is the JSON object. */ tup->obj.cof_obj = tup->obj.main; break; case encap_saf: /* the COF is embedded in the JSONL object. */ tup->obj.saf_cond = json_object_get(tup->obj.main, "cond"); if (tup->obj.saf_cond != NULL) { if (!json_is_string(tup->obj.saf_cond)) { msg = "cond must be a string"; goto ouch; } tup->cond = json_string_value(tup->obj.saf_cond); } tup->obj.saf_msg = json_object_get(tup->obj.main, "msg"); if (tup->obj.saf_msg != NULL) { if (!json_is_string(tup->obj.saf_msg)) { msg = "msg must be a string"; goto ouch; } tup->msg = json_string_value(tup->obj.saf_msg); } tup->obj.saf_obj = json_object_get(tup->obj.main, "obj"); if (tup->obj.saf_obj != NULL) { if (!json_is_object(tup->obj.saf_obj)) { msg = "obj must be an object"; goto ouch; } tup->obj.cof_obj = tup->obj.saf_obj; } break; default: /* we weren't prepared for this -- unknown program state. */ abort(); } /* Timestamps. */ tup->obj.zone_first = json_object_get(tup->obj.cof_obj, "zone_time_first"); if (tup->obj.zone_first != NULL) { if (!json_is_integer(tup->obj.zone_first)) { msg = "zone_time_first must be an integer"; goto ouch; } tup->zone_first = (u_long) json_integer_value(tup->obj.zone_first); } tup->obj.zone_last = json_object_get(tup->obj.cof_obj, "zone_time_last"); if (tup->obj.zone_last != NULL) { if (!json_is_integer(tup->obj.zone_last)) { msg = "zone_time_last must be an integer"; goto ouch; } tup->zone_last = (u_long) json_integer_value(tup->obj.zone_last); } tup->obj.time_first = json_object_get(tup->obj.cof_obj, "time_first"); if (tup->obj.time_first != NULL) { if (!json_is_integer(tup->obj.time_first)) { msg = "time_first must be an integer"; goto ouch; } tup->time_first = (u_long) json_integer_value(tup->obj.time_first); } tup->obj.time_last = json_object_get(tup->obj.cof_obj, "time_last"); if (tup->obj.time_last != NULL) { if (!json_is_integer(tup->obj.time_last)) { msg = "time_last must be an integer"; goto ouch; } tup->time_last = (u_long) json_integer_value(tup->obj.time_last); } /* Count. */ tup->obj.count = json_object_get(tup->obj.cof_obj, "count"); if (tup->obj.count != NULL) { if (!json_is_integer(tup->obj.count)) { msg = "count must be an integer"; goto ouch; } tup->count = json_integer_value(tup->obj.count); } /* Bailiwick. */ tup->obj.bailiwick = json_object_get(tup->obj.cof_obj, "bailiwick"); if (tup->obj.bailiwick != NULL) { if (!json_is_string(tup->obj.bailiwick)) { msg = "bailiwick must be a string"; goto ouch; } tup->bailiwick = json_string_value(tup->obj.bailiwick); } /* num_results -- just for a summarize. */ tup->obj.num_results = json_object_get(tup->obj.cof_obj, "num_results"); if (tup->obj.num_results != NULL) { if (!json_is_integer(tup->obj.num_results)) { msg = "num_results must be an integer"; goto ouch; } tup->num_results = json_integer_value(tup->obj.num_results); } /* Records. */ tup->obj.rrname = json_object_get(tup->obj.cof_obj, "rrname"); if (tup->obj.rrname != NULL) { if (!json_is_string(tup->obj.rrname)) { msg = "rrname must be a string"; goto ouch; } char *r = strdup(json_string_value(tup->obj.rrname)); int dot = 0; if ((transforms & TRANS_REVERSE) != 0) { char *t = reverse(r); DESTROY(r); r = t; t = NULL; /* leading dot comes from reverse() */ if ((transforms & TRANS_CHOMP) != 0) dot = 1; } else if ((transforms & TRANS_CHOMP) != 0) { /* unescaped trailing dot? */ size_t l = strlen(r); if (l > 0 && r[l-1] == '.' && (l == 1 || r[l-2] != '\\')) r[l-1] = '\0'; } if (dot) { /* in chomp+reverse, the dot to chomp is now leading. */ tup->rrname = strdup(r + dot); DESTROY(r); } else { tup->rrname = r; } } tup->obj.rrtype = json_object_get(tup->obj.cof_obj, "rrtype"); if (tup->obj.rrtype != NULL) { if (!json_is_string(tup->obj.rrtype)) { msg = "rrtype must be a string"; goto ouch; } tup->rrtype = json_string_value(tup->obj.rrtype); } tup->obj.rdata = json_object_get(tup->obj.cof_obj, "rdata"); if (tup->obj.rdata != NULL) { if (json_is_string(tup->obj.rdata)) { tup->rdata = json_string_value(tup->obj.rdata); } else if (!json_is_array(tup->obj.rdata)) { msg = "rdata must be a string or array"; goto ouch; } /* N.b., the array case is for the consumer to iterate over. */ } assert(msg == NULL); return NULL; ouch: assert(msg != NULL); tuple_unmake(tup); return msg; } /* tuple_unmake -- deallocate the heap storage associated with one tuple. */ void tuple_unmake(pdns_tuple_t tup) { DESTROY(tup->rrname); json_decref(tup->obj.main); } /* countoff{_r,_debug} -- count and map the labels in a DNS name. */ static struct counted * countoff_r(const char *src, int nlabel) { const char *sp = src; bool slash = false; struct counted *c; int ch; /* count and map the alnums in the facing dns label. */ size_t nalnum = 0; while ((ch = *sp++) != '\0') { if (isalnum(ch)) nalnum++; if (!slash) { if (ch == '\\') slash = true; else if (ch == '.') break; } else { slash = false; } } size_t len = (size_t) (sp - src); if (ch == '.') { /* end of label, recurse to reach rest of name. */ c = countoff_r(sp, nlabel+1); /* fill in output structure on the way back up. */ c->nchar += len; c->nalnum += nalnum; c->lens[nlabel] = len; } else if (ch == '\0') { /* end of name, and perhaps of a unterminated label. */ len--; /*'\0'*/ if (len != 0) nlabel++; c = (struct counted *)malloc(COUNTED_SIZE(nlabel)); memset(c, 0, COUNTED_SIZE(nlabel)); c->nlabel = nlabel; c->nalnum = nalnum; if (len != 0) { c->nchar = len; c->lens[nlabel-1] = c->nchar; } } else { abort(); } return c; } struct counted * countoff(const char *src) { return countoff_r(src, 0); } void countoff_debug(const char *place, const char *thing, const struct counted *c) { printf("\"%s\" -> {nlabel %d, nchar %zd, nalnum %zd, lens [", thing, c->nlabel, c->nchar, c->nalnum); const char *sep = ""; for (int i = 0; i < c->nlabel; i++) { printf("%s%zd", sep, c->lens[i]); sep = ", "; } printf("]} (%s)\n", place); } /* reverse -- put a domain name into TLD-first order. * * returns NULL if errno is set, else, a heap string. */ char * reverse(const char *src) { struct counted *c = countoff(src); char *ret = malloc(c->nchar + 1/*'.'*/ + 1/*'\0'*/); char *p = ret; size_t nchar = 0; for (ssize_t i = (ssize_t)c->nlabel-1; i >= 0; i--) { size_t dot = (src[c->nchar - nchar - 1] == '.'); *p++ = '.'; memcpy(p, src + c->nchar - nchar - c->lens[i], c->lens[i] - dot); p += c->lens[i] - dot; nchar += c->lens[i]; } *p = '\0'; DESTROY(c); return ret; } /* pdns_blob -- process one deblocked json pdns blob as a counted string. * * presents, or outputs to POSIX sort(1), the blob, and then frees it. * returns number of tuples processed (for now, 1 or 0). */ int pdns_blob(fetch_t fetch, size_t len) { query_t query = fetch->query; writer_t writer = query->writer; struct pdns_tuple tup; u_long first, last; const char *msg; int ret = 0; msg = tuple_make(&tup, fetch->buf, len); if (msg != NULL) { my_logf("%s", msg); goto more; } if (psys->encap == encap_saf) { if (tup.msg != NULL) { DEBUG(5, true, "data_blob tup.msg = %s\n", tup.msg); fetch->saf_msg = strdup(tup.msg); } if (tup.cond != NULL) { DEBUG(5, true, "pdns_blob tup.cond = %s\n", tup.cond); /* if we goto next now, this line will not be counted. */ if (strcmp(tup.cond, "begin") == 0) { fetch->saf_cond = sc_begin; goto next; } else if (strcmp(tup.cond, "ongoing") == 0) { /* "cond":"ongoing" key vals should * be ignored but the rest of line used. */ fetch->saf_cond = sc_ongoing; } else if (strcmp(tup.cond, "succeeded") == 0) { fetch->saf_cond = sc_succeeded; goto next; } else if (strcmp(tup.cond, "limited") == 0) { fetch->saf_cond = sc_limited; goto next; } else if (strcmp(tup.cond, "failed") == 0) { fetch->saf_cond = sc_failed; goto next; } else { /* use sc_missing for an invalid cond value */ fetch->saf_cond = sc_missing; my_logf("Unknown value for \"cond\": %s", tup.cond); } } /* A COF keepalive will have no "obj" * but may have a "cond" or "msg". */ if (tup.obj.saf_obj == NULL) { DEBUG(4, true, "COF object is empty, i.e. a keepalive\n"); goto next; } } /* there are two sets of timestamps in a tuple. we prefer * the on-the-wire times to the zone times, when available. */ if (tup.time_first != 0 && tup.time_last != 0) { first = (u_long)tup.time_first; last = (u_long)tup.time_last; } else { first = (u_long)tup.zone_first; last = (u_long)tup.zone_last; } if (sorting != no_sort) { /* POSIX sort(1) is given six extra fields at the front * of each line (first,last,duration,count,name,data) * which are accessed as -k1 .. -k7 on the * sort command line. we strip them off later * when reading the result back. the reason * for all this PDP11-era logic is to avoid * having to store the full result in memory. */ char *dyn_rrname = sortable_rrname(&tup), *dyn_rdata = sortable_rdata(&tup); DEBUG(3, true, "dyn_rrname = '%s'\n", dyn_rrname); DEBUG(3, true, "dyn_rdata = '%s'\n", dyn_rdata); fprintf(writer->sort_stdin, "%lu %lu %lu %lu %s %s %s %*.*s\n", (unsigned long)first, (unsigned long)last, (unsigned long)(last - first), (unsigned long)tup.count, or_else(dyn_rrname, "n/a"), tup.rrtype, or_else(dyn_rdata, "n/a"), (int)len, (int)len, fetch->buf); DEBUG(2, true, "sort0: '%lu %lu %lu %lu %s %s %s %*.*s'\n", (unsigned long)first, (unsigned long)last, (unsigned long)(last - first), (unsigned long)tup.count, or_else(dyn_rrname, "n/a"), tup.rrtype, or_else(dyn_rdata, "n/a"), (int)len, (int)len, fetch->buf); DESTROY(dyn_rrname); DESTROY(dyn_rdata); } else { /* before the sort, we know the query that caused the tuple. */ (*presenter->output)(&tup, query, writer); } ret = 1; next: tuple_unmake(&tup); more: return ret; } /* pick_system -- find a named system descriptor, return t/f as to "found?" * * returns if psys != NULL, or exits fatally otherwise. */ void pick_system(const char *name, const char *context) { pdns_system_ct tsys = NULL; char *msg = NULL; DEBUG(1, true, "pick_system(%s)\n", name); #if WANT_PDNS_DNSDB if (strcmp(name, "dnsdb1") == 0) tsys = pdns_dnsdb1(); /* "dnsdb" is an alias for "dnsdb2". */ if (strcmp(name, "dnsdb2") == 0 || strcmp(name, "dnsdb") == 0) tsys = pdns_dnsdb2(); #endif #if WANT_PDNS_CIRCL if (strcmp(name, "circl") == 0) tsys = pdns_circl(); #endif if (tsys == NULL) { if (asprintf(&msg, "unrecognized system name (%s)", name) < 0) my_panic(true, "asprintf"); } else if (tsys == psys) { /* likely recursion via read_config due to DNSDBQ_SYSTEM. */ return; } else { if (psys != NULL) { psys->destroy(); psys = NULL; } psys = tsys; tsys = NULL; if (config_file != NULL) read_config(); const char *tmsg = psys->ready(); if (tmsg != NULL) { msg = strdup(tmsg); tmsg = NULL; } } if (msg != NULL) { my_logf("%s (in %s)\n", msg, context); DESTROY(msg); my_exit(1); } } /* read_config -- parse a given config file. */ void read_config(void) { char *cmd, *line; size_t n; int x, l; FILE *f; /* in the "echo dnsdb server..." lines, the * first parameter is the pdns system to which to dispatch * the key and value (i.e. second the third parameters). */ x = asprintf(&cmd, "set -e; . '%s';" "echo dnsdbq system ${" DNSDBQ_SYSTEM ":-" DEFAULT_SYS "};" #if WANT_PDNS_DNSDB "echo dnsdb1 apikey ${DNSDB_API_KEY:-$APIKEY};" "echo dnsdb1 server $DNSDB_SERVER;" "echo dnsdb2 apikey ${DNSDB_API_KEY:-$APIKEY};" "echo dnsdb2 server $DNSDB_SERVER;" #endif #if WANT_PDNS_CIRCL "echo circl apikey $CIRCL_AUTH;" "echo circl server $CIRCL_SERVER;" #endif "exit", config_file); if (x < 0) my_panic(true, "asprintf"); // this variable can be set in the config file but not the environ. unsetenv("APIKEY"); f = popen(cmd, "r"); if (f == NULL) { my_logf("[%s]: %s", cmd, strerror(errno)); DESTROY(cmd); my_exit(1); } DEBUG(1, true, "conf cmd = '%s'\n", cmd); DESTROY(cmd); line = NULL; n = 0; l = 0; while (getline(&line, &n, f) > 0) { char *tok1, *tok2, *tok3; char *saveptr = NULL; const char *msg; l++; if (strchr(line, '\n') == NULL) { my_logf("conf line #%d: too long", l); my_exit(1); } tok1 = strtok_r(line, "\040\012", &saveptr); tok2 = strtok_r(NULL, "\040\012", &saveptr); tok3 = strtok_r(NULL, "\040\012", &saveptr); if (tok1 == NULL || tok2 == NULL) { my_logf("conf line #%d: malformed", l); my_exit(1); } if (tok3 == NULL || *tok3 == '\0') { /* variable wasn't set, ignore the line. */ continue; } /* some env/conf variables are dnsdbq-specific. */ if (strcmp(tok1, "dnsdbq") == 0) { /* env/config psys does not override -u. */ if (strcmp(tok2, "system") == 0 && !psys_specified) { pick_system(tok3, config_file); if (psys == NULL) { my_logf("unknown %s %s\n", DNSDBQ_SYSTEM, tok3); my_exit(1); } } continue; } /* if this variable is for this system, consume it. */ if (debug_level >= 1) { char *t = NULL; if (strcmp(tok2, "apikey") == 0) { int ignored __attribute__((unused)); ignored = asprintf(&t, "[%zu]", strlen(tok3)); } else { t = strdup(tok3); } my_logf("line #%d: sets %s|%s|%s", l, tok1, tok2, t); DESTROY(t); } if (strcmp(tok1, psys->name) == 0) { msg = psys->setval(tok2, tok3); if (msg != NULL) { my_logf("setval: %s", msg); my_exit(1); } } } DESTROY(line); x = pclose(f); if (!WIFEXITED(x) || WEXITSTATUS(x) != 0) my_exit(1); assert(psys != NULL); } dnsdbq-2.6.7/pdns.h000066400000000000000000000141251462247651300141330ustar00rootroot00000000000000/* * Copyright (c) 2014-2021 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef PDNS_H_INCLUDED #define PDNS_H_INCLUDED 1 #include #include "netio.h" /* ".main" is the primary jansson library object from a json_loadb(). * all the other fields in this structure will point inside main, as * borrowed references, so const. main must be deallocated * by json_decref() which then invalidates all the other fields. * * ".cof_obj" points to the object that contains time_first...num_results. * If not using SAF encapulation, then cof_obj points to main. * If using SAF encapulation, then saf_cond, saf_msg, and saf_obj are * parsed from main and cof_obj is repointed to saf_obj. * * tup->obj.rrname is always original, tup->rrname is sometimes reversed. * as a result, tup->rrname is always heap-allocated, even if unreversed. */ struct pdns_json { json_t *main; const json_t *cof_obj, *saf_obj, *saf_cond, *saf_msg, *time_first, *time_last, *zone_first, *zone_last, *bailiwick, *rrname, *rrtype, *rdata, *count, *num_results; }; struct pdns_tuple { struct pdns_json obj; u_long time_first, time_last, zone_first, zone_last; const char *bailiwick, *rrtype, *rdata, *cond, *msg; char *rrname; json_int_t count, num_results; }; typedef struct pdns_tuple *pdns_tuple_t; typedef const struct pdns_tuple *pdns_tuple_ct; struct pdns_fence { u_long first_after, first_before, last_after, last_before; }; typedef const struct pdns_fence *pdns_fence_ct; struct pdns_system { /* name of this pdns system, as specifiable by the user. */ const char *name; /* default URL to reach this pdns API endpoint. May be overridden. */ const char *base_url; /* what encapsulation does this system speak? */ encap_e encap; /* start creating a URL corresponding to a command-path string. * First argument is the input URL path. * Second argument is an output parameter pointing to the * separator character (? or &) that the caller should use * between any further URL parameters. May be NULL if the * caller doesn't care. * Third argument is search parameters. * Fourth argument is time fencing parameters. * Fifth argument is true if the query is a meta query, which * doesn't get regular result processing. */ char * (*url)(const char *, char *, qparam_ct, pdns_fence_ct, bool); /* send a request for info, such as quota information. * may be NULL if info requests are not supported by this pDNS system. */ void (*info)(void); /* add authentication information to the fetch request being created. * may be NULL if auth is not needed by this pDNS system. */ void (*auth)(fetch_t); /* map a non-200 HTTP rcode from a fetch to an error indicator. */ const char * (*status)(fetch_t); /* verify that the specified verb is supported by this pdns system. * Returns NULL if supported; otherwise returns a static error message. * First argument is the verb. * Second argument is search parameters. */ const char * (*verb_ok)(const char *, qparam_ct); /* set a configuration key-value pair. * Returns NULL if ok; otherwise returns a static error message. * First argument is the key. * Second argument is the value. */ const char * (*setval)(const char *, const char *); /* check if ready with enough config settings to try API queries. * Returns NULL if ready; otherwise returns a static error message. */ const char * (*ready)(void); /* drop heap storage. */ void (*destroy)(void); }; typedef const struct pdns_system *pdns_system_ct; struct presenter { void (*output)(pdns_tuple_ct, query_ct, writer_t); bool sortable; }; typedef const struct presenter *presenter_ct; /* a verb is a specific type of request. See struct pdns_system * verb_ok() for that function that verifies if the verb and the options * provided is supported by that pDNS system. */ struct verb { const char *name; const char *url_fragment; /* review the command line options for constraints being met. * Returns NULL if ok; otherwise returns a static error message. */ const char * (*ok)(void); /* formatter function for each presentation format */ presenter_ct text, json, csv, minimal; }; typedef const struct verb *verb_ct; /* query parameters descriptor. */ struct qdesc { mode_e mode; char *thing; char *rrtype; char *bailiwick; char *pfxlen; }; typedef struct qdesc *qdesc_t; typedef const struct qdesc *qdesc_ct; struct counted { int nlabel; size_t nchar; size_t nalnum; size_t lens[]; }; #define COUNTED_SIZE(nlabel) \ (sizeof(struct counted) + (unsigned int) nlabel * sizeof(size_t)) bool pprint_json(const char *, size_t, FILE *); void present_json_lookup(pdns_tuple_ct, query_ct, writer_t); void present_json_summarize(pdns_tuple_ct, query_ct, writer_t); void present_text_lookup(pdns_tuple_ct, query_ct, writer_t); void present_csv_lookup(pdns_tuple_ct, query_ct, writer_t); void present_minimal_lookup(pdns_tuple_ct, query_ct, writer_t); void present_text_summarize(pdns_tuple_ct, query_ct, writer_t); void present_csv_summarize(pdns_tuple_ct, query_ct, writer_t); const char *tuple_make(pdns_tuple_t, const char *, size_t); void tuple_unmake(pdns_tuple_t); struct counted *countoff(const char *); void countoff_debug(const char *, const char *, const struct counted *); char *reverse(const char *); int pdns_blob(fetch_t, size_t); void pick_system(const char *, const char *); void read_config(void); /* Some HTTP status codes we handle specifically */ #define HTTP_OK 200 #define HTTP_NOT_FOUND 404 #if WANT_PDNS_DNSDB #include "pdns_dnsdb.h" #endif #if WANT_PDNS_CIRCL #include "pdns_circl.h" #endif #endif /*PDNS_H_INCLUDED*/ dnsdbq-2.6.7/pdns_circl.c000066400000000000000000000105201462247651300152750ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #if WANT_PDNS_CIRCL /* asprintf() does not appear on linux without this */ #define _GNU_SOURCE #include #include "defs.h" #include "pdns.h" #include "pdns_circl.h" #include "globals.h" static char *circl_url(const char *, char *, qparam_ct, pdns_fence_ct, bool); static void circl_auth(fetch_t); static const char *circl_status(fetch_t); static const char *circl_verb_ok(const char *, qparam_ct); static const char *circl_ready(void); static const char *circl_setval(const char *, const char *); static void circl_destroy(void); static char *circl_base_url = NULL; static char *circl_authinfo = NULL; static const struct pdns_system circl = { "circl", "https://www.circl.lu/pdns/query", encap_cof, circl_url, NULL, circl_auth, circl_status, circl_verb_ok, circl_setval, circl_ready, circl_destroy }; pdns_system_ct pdns_circl(void) { return &circl; } static const char * circl_setval(const char *key, const char *value) { if (strcmp(key, "apikey") == 0) { DESTROY(circl_authinfo); circl_authinfo = strdup(value); } else if (strcmp(key, "server") == 0) { DESTROY(circl_base_url); circl_base_url = strdup(value); } else { return "circl_setval() unrecognized key"; } return NULL; } static const char * circl_ready(void) { return NULL; } static void circl_destroy(void) { DESTROY(circl_base_url); DESTROY(circl_authinfo); } /* circl_url -- create a URL corresponding to a command-path string. * * the batch file and command line syntax are in native DNSDB API format. * this function has the opportunity to crack this into pieces, and re-form * those pieces into the URL format needed by some other DNSDB-like system * which might have the same JSON output format but a different REST syntax. * * CIRCL pDNS only "understands IP addresses, hostnames or domain names * (please note that CIDR block queries are not supported)". exit with an * error message if asked to do something the CIRCL server does not handle. * * 1. RRSet query: rrset/name/NAME[/TYPE[/BAILIWICK]] * 2. Rdata (name) query: rdata/name/NAME[/TYPE] * 3. Rdata (IP address) query: rdata/ip/ADDR[/PFXLEN] */ static char * circl_url(const char *path, char *sep, qparam_ct qp __attribute__((unused)), pdns_fence_ct fp __attribute__((unused)), bool meta_query __attribute__((unused))) { const char *val = NULL; char *ret; int x, pi; /* NULL-terminate array of valid query paths for CIRCL */ const char *valid_paths[] = { "rrset/name/", "rdata/name/", "rdata/ip/", NULL }; if (circl_base_url == NULL) circl_base_url = strdup(psys->base_url); for (pi = 0; valid_paths[pi] != NULL; pi++) if (strncasecmp(path, valid_paths[pi], strlen(valid_paths[pi])) == 0) { val = path + strlen(valid_paths[pi]); break; } if (val == NULL) { my_logf("unsupported type of query for CIRCL pDNS: %s", path); my_exit(1); } if (strchr(val, '/') != NULL) { my_logf("qualifiers not supported by CIRCL pDNS: %s", val); my_exit(1); } x = asprintf(&ret, "%s/%s", circl_base_url, val); if (x < 0) my_panic(true, "asprintf"); /* because we will NOT append query parameters, * tell the caller to use ? for its query parameters. */ if (sep != NULL) *sep = '?'; return ret; } static void circl_auth(fetch_t fetch) { if (fetch->easy != NULL) { curl_easy_setopt(fetch->easy, CURLOPT_USERPWD, circl_authinfo); curl_easy_setopt(fetch->easy, CURLOPT_HTTPAUTH, CURLAUTH_BASIC); } } static const char * circl_status(fetch_t fetch __attribute__((unused))) { return status_error; } static const char * circl_verb_ok(const char *verb_name, qparam_ct qpp __attribute__((unused))) { /* Only "lookup" is valid */ if (strcasecmp(verb_name, "lookup") != 0) return "the CIRCL system only understands 'lookup'"; return NULL; } #endif /*WANT_PDNS_CIRCL*/ dnsdbq-2.6.7/pdns_circl.h000066400000000000000000000014101462247651300153000ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef PDNS_CIRCL_H_INCLUDED #define PDNS_CIRCL_H_INCLUDED 1 #if WANT_PDNS_CIRCL pdns_system_ct pdns_circl(void); #endif #endif /*PDNS_CIRCL_H_INCLUDED*/ dnsdbq-2.6.7/pdns_dnsdb.c000066400000000000000000000356301462247651300153040ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #if WANT_PDNS_DNSDB /* asprintf() does not appear on linux without this */ #define _GNU_SOURCE #include #include #include "defs.h" #include "pdns.h" #include "pdns_dnsdb.h" #include "time.h" #include "globals.h" /* types. */ struct rate_json { json_t *main, *reset, *expires, *limit, *remaining, *burst_size, *burst_window, *results_max, *offset_max; }; struct rateval { enum { rk_naught = 0, /* not present. */ rk_na, /* "n/a". */ rk_unlimited, /* "unlimited". */ rk_int /* some integer in as_int. */ } rk; u_long as_int; /* only for rk == rk_int. */ }; typedef struct rateval *rateval_t; typedef const struct rateval *rateval_ct; struct rate_tuple { struct rate_json obj; struct rateval reset, expires, limit, remaining, burst_size, burst_window, results_max, offset_max; }; typedef struct rate_tuple *rate_tuple_t; /* forwards. */ static const char *dnsdb_setval(const char *, const char *); static const char *dnsdb_ready(void); static void dnsdb_destroy(void); static char *dnsdb_url(const char *, char *, qparam_ct, pdns_fence_ct, bool); static void dnsdb_info(void); static void dnsdb_auth(fetch_t); static const char *dnsdb_status(fetch_t); static const char *dnsdb_verb_ok(const char *, qparam_ct); static void print_rateval(const char *, rateval_ct, FILE *); static void print_burstrate(const char *, rateval_ct, rateval_ct, FILE *); static const char *rateval_make(rateval_t, const json_t *, const char *); static const char *rate_tuple_make(rate_tuple_t, const char *, size_t); static void rate_tuple_unmake(rate_tuple_t); /* variables. */ static const char env_api_key[] = "DNSDB_API_KEY"; static const char env_dnsdb_base_url[] = "DNSDB_SERVER"; static char *api_key = NULL; static char *dnsdb_base_url = NULL; static const char dnsdb2_url_prefix[] = "/dnsdb/v2"; static const struct pdns_system dnsdb1 = { "dnsdb1", "https://api.dnsdb.info", encap_cof, dnsdb_url, dnsdb_info, dnsdb_auth, dnsdb_status, dnsdb_verb_ok, dnsdb_setval, dnsdb_ready, dnsdb_destroy }; static const struct pdns_system dnsdb2 = { "dnsdb2", "https://api.dnsdb.info/dnsdb/v2", encap_saf, dnsdb_url, dnsdb_info, dnsdb_auth, dnsdb_status, dnsdb_verb_ok, dnsdb_setval, dnsdb_ready, dnsdb_destroy }; /*---------------------------------------------------------------- public */ pdns_system_ct pdns_dnsdb1(void) { return &dnsdb1; } pdns_system_ct pdns_dnsdb2(void) { return &dnsdb2; } /*---------------------------------------------------------------- private */ /* dnsdb_setval() -- install configuration element */ static const char * dnsdb_setval(const char *key, const char *value) { if (strcmp(key, "apikey") == 0) { DESTROY(api_key); api_key = strdup(value); } else if (strcmp(key, "server") == 0) { DESTROY(dnsdb_base_url); dnsdb_base_url = strdup(value); } else { return "dnsdb_setval() unrecognized key"; } return NULL; } /* dnsdb_ready() -- override the config file from environment variables? */ static const char * dnsdb_ready(void) { const char *value; if ((value = getenv(env_api_key)) != NULL) { dnsdb_setval("apikey", value); DEBUG(1, true, "conf env api_key was set\n"); } if ((value = getenv(env_dnsdb_base_url)) != NULL) { dnsdb_setval("server", value); DEBUG(1, true, "conf env dnsdb_server = '%s'\n", dnsdb_base_url); } if (dnsdb_base_url == NULL) dnsdb_base_url = strdup(psys->base_url); /* If SAF (aka APIv2) ensure URL contains special /dnsdb/v2 prefix. */ if (psys->encap == encap_saf && strstr(dnsdb_base_url, dnsdb2_url_prefix) == NULL) { char *temp; int x; x = asprintf(&temp, "%s%s", dnsdb_base_url, dnsdb2_url_prefix); if (x < 0) { perror("asprintf"); abort(); } DESTROY(dnsdb_base_url); dnsdb_base_url = temp; } if (api_key == NULL) return "no API key given"; return NULL; } /* dnsdb_destroy() -- drop heap storage */ static void dnsdb_destroy(void) { DESTROY(api_key); DESTROY(dnsdb_base_url); } /* dnsdb_url -- create a URL corresponding to a command-path string. * * the batch file and command line syntax are in native DNSDB API format. * this function has the opportunity to crack this into pieces, and re-form * those pieces into the URL format needed by some other DNSDB-like system * which might have the same JSON output format but a different REST syntax. * returns a string that must be freed. */ static char * dnsdb_url(const char *path, char *sep, qparam_ct qpp, pdns_fence_ct fp, bool meta_query) { const char *verb_path, *p, *scheme_if_needed, *aggr_if_needed; char *ret = NULL, *max_count_str = NULL, *offset_str = NULL, *first_after_str = NULL, *first_before_str = NULL, *last_after_str = NULL, *last_before_str = NULL, *query_limit_str = NULL; int x, num_slash; /* count the number of slashes in the base url, after the :// * if present. 1 or more means there's a /path after the host. * In that case, don't add /[verb] here, and also don't allow * selecting a verb that's not "lookup" since the /path could * include its own verb. (this is from an old python-era rule.) */ if ((p = strstr(dnsdb_base_url, "://")) != NULL) p += sizeof "://" - sizeof ""; else p = dnsdb_base_url; num_slash = 0; if (strstr(dnsdb_base_url, dnsdb2_url_prefix) == NULL) for (; *p != '\0'; p++) num_slash += (*p == '/'); verb_path = ""; if (num_slash == 0) { if (psys->encap == encap_saf && meta_query) verb_path = ""; else if (pverb->url_fragment != NULL) verb_path = pverb->url_fragment; else verb_path = "/lookup"; } /* supply a scheme if the server string did not. */ scheme_if_needed = ""; if (strstr(dnsdb_base_url, "://") == NULL) scheme_if_needed = "https://"; /* handle gravel vs. rocks. */ aggr_if_needed = ""; if (qpp->gravel) aggr_if_needed = "&aggr=f"; if (qpp->offset > 0) { x = asprintf(&offset_str, "&offset=%ld", qpp->offset); if (x < 0) { perror("asprintf"); goto done; } } if (max_count > 0) { x = asprintf(&max_count_str, "&max_count=%ld", max_count); if (x < 0) { perror("asprintf"); goto done; } } if (qpp->query_limit != -1) { x = asprintf(&query_limit_str, "&limit=%ld", qpp->query_limit); if (x < 0) { perror("asprintf"); goto done; } } if (fp->first_after != 0) { x = asprintf(&first_after_str, "&time_first_after=%lu", fp->first_after); if (x < 0) { perror("asprintf"); goto done; } } if (fp->first_before != 0) { x = asprintf(&first_before_str, "&time_first_before=%lu", fp->first_before); if (x < 0) { perror("asprintf"); goto done; } } if (fp->last_after != 0) { x = asprintf(&last_after_str, "&time_last_after=%lu", fp->last_after); if (x < 0) { perror("asprintf"); goto done; } } if (fp->last_before != 0) { x = asprintf(&last_before_str, "&time_last_before=%lu", fp->last_before); if (x < 0) { perror("asprintf"); goto done; } } x = asprintf(&ret, "%s%s%s/%s?swclient=%s&version=%s%s%s%s%s%s%s%s%s", scheme_if_needed, dnsdb_base_url, verb_path, path, id_swclient, id_version, aggr_if_needed, or_else(offset_str, ""), or_else(max_count_str, ""), or_else(query_limit_str, ""), or_else(first_after_str, ""), or_else(first_before_str, ""), or_else(last_after_str, ""), or_else(last_before_str, "")); if (x < 0) { perror("asprintf"); goto done; } /* because we append query parameters, tell the caller to use & for * any further query parameters. */ if (sep != NULL) *sep = '&'; done: DESTROY(offset_str); DESTROY(max_count_str); DESTROY(query_limit_str); DESTROY(first_after_str); DESTROY(first_before_str); DESTROY(last_after_str); DESTROY(last_before_str); return ret; } static void dnsdb_infoback(writer_t writer) { switch (presentation) { case pres_text: { struct rate_tuple tup; const char *msg; msg = rate_tuple_make(&tup, writer->ps_buf, writer->ps_len); if (msg != NULL) { /* there was an error */ puts(msg); } else { puts("rate:"); print_rateval("reset", &tup.reset, stdout); print_rateval("expires", &tup.expires, stdout); print_rateval("limit", &tup.limit, stdout); print_rateval("remaining", &tup.remaining, stdout); print_rateval("results_max", &tup.results_max, stdout); print_rateval("offset_max", &tup.offset_max, stdout); print_burstrate("burst rate", &tup.burst_size, &tup.burst_window, stdout); rate_tuple_unmake(&tup); } break; } case pres_json: /* Ignore any failure in pprint_json. */ (void) pprint_json(writer->ps_buf, writer->ps_len, stdout); break; case pres_csv: /* FALLTHROUGH */ case pres_none: /* FALLTHROUGH */ case pres_minimal: abort(); } } static void dnsdb_info(void) { query_t query = NULL; writer_t writer; DEBUG(1, true, "dnsdb_info()\n"); /* start a meta_query writer. */ writer = writer_init(qparam_empty.output_limit, dnsdb_infoback, true); /* create a rump query. */ CREATE(query, sizeof(struct query)); query->writer = writer; query->descr = strdup("rate_limit"); writer->queries = query; /* start a status fetch. */ create_fetch(query, dnsdb_url("rate_limit", NULL, &qparam_empty, &(struct pdns_fence){}, true)); /* run all jobs to completion. */ io_engine(0); /* stop the writer. */ writer_fini(writer); } static void dnsdb_auth(fetch_t fetch) { if (api_key != NULL) { char *key_header; if (asprintf(&key_header, "X-Api-Key: %s", api_key) < 0) my_panic(true, "asprintf"); fetch->hdrs = curl_slist_append(fetch->hdrs, key_header); DESTROY(key_header); } } static const char * dnsdb_status(fetch_t fetch) { /* APIv1 DNSDB returns 404 for "no rrs found". * APIv2 DNSDB returns 200 with no SAF lines for "no rrs found". */ if (psys->encap == encap_saf && fetch->rcode == HTTP_NOT_FOUND) return status_error; return status_noerror; } static const char * dnsdb_verb_ok(const char *verb_name, qparam_ct qpp __attribute__((unused))) { if (strcasecmp(verb_name, "lookup") != 0) { /* -O (offset) cannot be used except for verb "lookup". */ if (qpp->offset != 0) return "only 'lookup' understands offsets"; /* -L (output_limit) cannot be used except for verb "lookup". */ if (qpp->explicit_output_limit != -1) return "only 'lookup' understands output limits"; } return NULL; } /*---------------------------------------------------------------- private */ /* print_rateval -- output formatter for rateval. */ static void print_rateval(const char *key, rateval_ct tp, FILE *outf) { /* if unspecified, output nothing, not even the key name. */ if (tp->rk == rk_naught) return; fprintf(outf, "\t%s: ", key); switch (tp->rk) { case rk_na: fprintf(outf, "n/a"); break; case rk_unlimited: fprintf(outf, "unlimited"); break; case rk_int: if (strcmp(key, "reset") == 0 || strcmp(key, "expires") == 0) fputs(time_str(tp->as_int, iso8601), outf); else fprintf(outf, "%lu", tp->as_int); break; case rk_naught: /*FALLTHROUGH*/ default: abort(); } fputc('\n', outf); } /* print_burstrate -- output formatter for burst_size, burst_window ratevals. */ static void print_burstrate(const char *key, rateval_ct tp_size, rateval_ct tp_window, FILE *outf) { /* if unspecified, output nothing, not even the key name. */ if (tp_size->rk == rk_naught || tp_window->rk == rk_naught) return; assert(tp_size->rk == rk_int); assert(tp_window->rk == rk_int); u_long b_w = tp_window->as_int; u_long b_s = tp_size->as_int; fprintf(outf, "\t%s: ", key); if (b_w == 3600) fprintf(outf, "%lu per hour", b_s); else if (b_w == 60) fprintf(outf, "%lu per minute", b_s); else if ((b_w % 3600) == 0) fprintf(outf, "%lu per %lu hours", b_s, b_w / 3600); else if ((b_w % 60) == 0) fprintf(outf, "%lu per %lu minutes", b_s, b_w / 60); else fprintf(outf, "%lu per %lu seconds", b_s, b_w); fputc('\n', outf); } /* rateval_make: make an optional key value from the json object. * * note: a missing key means the corresponding key's value is a "no value". */ static const char * rateval_make(rateval_t tp, const json_t *obj, const char *key) { struct rateval rvalue = {.rk = rk_naught}; const json_t *jvalue = json_object_get(obj, key); if (jvalue != NULL) { if (json_is_integer(jvalue)) { rvalue.rk = rk_int; rvalue.as_int = (u_long)json_integer_value(jvalue); } else { const char *strvalue = json_string_value(jvalue); bool ok = false; if (strvalue != NULL) { if (strcasecmp(strvalue, "n/a") == 0) { rvalue.rk = rk_na; ok = true; } else if (strcasecmp(strvalue, "unlimited") == 0) { rvalue.rk = rk_unlimited; ok = true; } } if (!ok) return "value must be an integer " "or \"n/a\" or \"unlimited\""; } } *tp = rvalue; return NULL; } /* rate_tuple_make -- create one rate tuple object out of a JSON object. */ static const char * rate_tuple_make(rate_tuple_t tup, const char *buf, size_t len) { const char *msg = NULL; json_error_t error; json_t *rate; memset(tup, 0, sizeof *tup); DEBUG(3, true, "[%d] '%-*.*s'\n", (int)len, (int)len, (int)len, buf); tup->obj.main = json_loadb(buf, len, 0, &error); if (tup->obj.main == NULL) { my_logf("warning: json_loadb: %d:%d: %s %s", error.line, error.column, error.text, error.source); abort(); } if (debug_level >= 4) { char *pretty = json_dumps(tup->obj.main, JSON_INDENT(2)); my_logf("%s", pretty); free(pretty); } rate = json_object_get(tup->obj.main, "rate"); if (rate == NULL) { msg = "Missing \"rate\" object"; goto ouch; } msg = rateval_make(&tup->reset, rate, "reset"); if (msg != NULL) goto ouch; msg = rateval_make(&tup->expires, rate, "expires"); if (msg != NULL) goto ouch; msg = rateval_make(&tup->limit, rate, "limit"); if (msg != NULL) goto ouch; msg = rateval_make(&tup->remaining, rate, "remaining"); if (msg != NULL) goto ouch; msg = rateval_make(&tup->results_max, rate, "results_max"); if (msg != NULL) goto ouch; msg = rateval_make(&tup->offset_max, rate, "offset_max"); if (msg != NULL) goto ouch; msg = rateval_make(&tup->burst_size, rate, "burst_size"); if (msg != NULL) goto ouch; msg = rateval_make(&tup->burst_window, rate, "burst_window"); if (msg != NULL) goto ouch; assert(msg == NULL); return NULL; ouch: assert(msg != NULL); rate_tuple_unmake(tup); return msg; } /* rate_tuple_unmake -- deallocate heap storage associated with a rate tuple. */ static void rate_tuple_unmake(rate_tuple_t tup) { json_decref(tup->obj.main); } #endif /*WANT_PDNS_DNSDB*/ dnsdbq-2.6.7/pdns_dnsdb.h000066400000000000000000000014531462247651300153050ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef PDNS_DNSDB_H_INCLUDED #define PDNS_DNSDB_H_INCLUDED 1 #if WANT_PDNS_DNSDB pdns_system_ct pdns_dnsdb1(void); pdns_system_ct pdns_dnsdb2(void); #endif #endif /*PDNS_DNSDB_H_INCLUDED*/ dnsdbq-2.6.7/sort.c000066400000000000000000000173211462247651300141520ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ /* asprintf() does not appear on linux without this */ #define _GNU_SOURCE #include #include #include #include #include #include #include "defs.h" #include "sort.h" #include "globals.h" /* in the POSIX sort(1) intermediate format, the fields are: * #1 first * #2 last * #3 duration * #4 count * #5 rrname * #7 rrtype * #6 rdata * #8 mode * #9 json */ #define MAX_KEYS 7 extern char **environ; static struct sortkey keys[MAX_KEYS]; static int nkeys = 0; /* sort_ready -- finish initializing the sort related metadata. * * If sorting, all keys must be specified, to enable -u. * This adds every possible sort key, ignoring any errors from adding * a key in case the key was already added as specified by the user. */ void sort_ready(void) { (void) add_sort_key("first"); (void) add_sort_key("last"); (void) add_sort_key("duration"); (void) add_sort_key("count"); (void) add_sort_key("name"); (void) add_sort_key("type"); (void) add_sort_key("data"); } /* add_sort_key -- add a key for use by POSIX sort. * * Returns NULL if no error, otherwise a static error message. */ const char * add_sort_key(const char *key_name) { const char *key = NULL; char *computed; int x; if (nkeys == MAX_KEYS) return "too many sort keys given."; if (strcasecmp(key_name, "first") == 0) key = "-k1n"; else if (strcasecmp(key_name, "last") == 0) key = "-k2n"; else if (strcasecmp(key_name, "duration") == 0) key = "-k3n"; else if (strcasecmp(key_name, "count") == 0) key = "-k4n"; else if (strcasecmp(key_name, "name") == 0) key = "-k5"; else if (strcasecmp(key_name, "type") == 0) key = "-k6"; else if (strcasecmp(key_name, "data") == 0) key = "-k7"; else return "key must be in " "first|last|duration|count|name|type|data"; x = asprintf(&computed, "%s%s", key, sorting == reverse_sort ? "r" : ""); if (x < 0) my_panic(true, "asprintf"); keys[nkeys++] = (struct sortkey){strdup(key_name), computed}; return NULL; } /* find_sort_key -- return pointer to a sort key, or NULL if it's not specified */ sortkey_ct find_sort_key(const char *key_name) { int n; for (n = 0; n < nkeys; n++) { if (strcmp(keys[n].specified, key_name) == 0) return &keys[n]; } return NULL; } /* sort_destroy -- drop sort metadata from heap. */ void sort_destroy(void) { int n; for (n = 0; n < nkeys; n++) { DESTROY(keys[n].specified); DESTROY(keys[n].computed); } } /* exec_sort -- replace this fork with a POSIX sort program */ __attribute__((noreturn)) void exec_sort(int p1[], int p2[]) { char *sort_argv[3+MAX_KEYS], **sap; int n; if (dup2(p1[0], STDIN_FILENO) < 0 || dup2(p2[1], STDOUT_FILENO) < 0) { perror("dup2"); _exit(1); } close(p1[0]); close(p1[1]); close(p2[0]); close(p2[1]); sap = sort_argv; *sap++ = strdup("sort"); *sap++ = strdup("-u"); for (n = 0; n < nkeys; n++) *sap++ = strdup(keys[n].computed); *sap++ = NULL; putenv(strdup("LC_ALL=C")); DEBUG(1, true, "\"%s\" args:", path_sort); for (sap = sort_argv; *sap != NULL; sap++) DEBUG(1, false, " [%s]", *sap); DEBUG(1, false, "\n"); execve(path_sort, sort_argv, environ); perror("execve"); for (sap = sort_argv; *sap != NULL; sap++) DESTROY(*sap); _exit(1); } /* sortable_rrname -- return a POSIX-sort-collatable rendition of RR name+type. */ char * sortable_rrname(pdns_tuple_ct tup) { struct sortbuf buf = {}; sortable_dnsname(&buf, json_string_value(tup->obj.rrname)); buf.base = realloc(buf.base, buf.size+1); buf.base[buf.size++] = '\0'; return buf.base; } /* sortable_rdata -- return a POSIX-sort-collatable rendition of RR data set. */ char * sortable_rdata(pdns_tuple_ct tup) { struct sortbuf buf = {}; if (json_is_array(tup->obj.rdata)) { size_t index; json_t *rr; json_array_foreach(tup->obj.rdata, index, rr) { if (json_is_string(rr)) sortable_rdatum(&buf, tup->rrtype, json_string_value(rr)); else my_logf("warning: rdata slot is not a string"); } } else { sortable_rdatum(&buf, tup->rrtype, tup->rdata); } buf.base = realloc(buf.base, buf.size+1); buf.base[buf.size++] = '\0'; return buf.base; } /* sortable_rdatum -- called only by sortable_rdata(), realloc and normalize. * * this converts (lossily) addresses into hex strings, and extracts the * server-name component of a few other types like MX. all other rdata * are left in their normal string form, because it's hard to know what * to sort by with something like TXT, and extracting the serial number * from an SOA using a language like C is a bit ugly. */ void sortable_rdatum(sortbuf_t buf, const char *rrtype, const char *rdatum) { if (strcmp(rrtype, "A") == 0) { u_char a[4]; if (inet_pton(AF_INET, rdatum, a) != 1) memset(a, 0, sizeof a); sortable_hexify(buf, a, sizeof a); } else if (strcmp(rrtype, "AAAA") == 0) { u_char aaaa[16]; if (inet_pton(AF_INET6, rdatum, aaaa) != 1) memset(aaaa, 0, sizeof aaaa); sortable_hexify(buf, aaaa, sizeof aaaa); } else if (strcmp(rrtype, "NS") == 0 || strcmp(rrtype, "PTR") == 0 || strcmp(rrtype, "CNAME") == 0 || strcmp(rrtype, "DNAME") == 0) { sortable_dnsname(buf, rdatum); } else if (strcmp(rrtype, "MX") == 0 || strcmp(rrtype, "RP") == 0) { const char *space = strrchr(rdatum, ' '); if (space != NULL) sortable_dnsname(buf, space+1); else sortable_hexify(buf, (const u_char *)rdatum, strlen(rdatum)); } else { sortable_hexify(buf, (const u_char *)rdatum, strlen(rdatum)); } } /* sortable_hexify -- convert src into hex string in buffer */ void sortable_hexify(sortbuf_t buf, const u_char *src, size_t len) { buf->base = realloc(buf->base, buf->size + len*2); for (size_t i = 0; i < len; i++) { static const char hex[] = "0123456789abcdef"; unsigned int ch = src[i]; buf->base[buf->size++] = hex[ch >> 4]; buf->base[buf->size++] = hex[ch & 0xf]; } } /* sortable_dnsname -- make a sortable dns name; destructive and lossy. * * to be lexicographically sortable, a dnsname has to be converted to * TLD-first, all uppercase letters must be converted to lower case, * and all characters except dots then converted to hexadecimal. this * transformation is for POSIX sort's use, and is irreversibly lossy. */ void sortable_dnsname(sortbuf_t buf, const char *name) { struct counted *c = countoff(name); // ensure our result buffer is large enough. size_t new_size = buf->size + c->nalnum; if (new_size == 0) { DESTROY(c); buf->base = realloc(buf->base, 1); buf->base[0] = '.'; buf->size = 1; return; } if (new_size != buf->size) buf->base = realloc(buf->base, new_size); char *p = buf->base + buf->size; // collatable names are TLD-first, alphanumeric only, lower case. size_t nchar = 0; for (ssize_t i = (ssize_t)(c->nlabel-1); i >= 0; i--) { size_t dot = (name[c->nchar - nchar - 1] == '.'); ssize_t j = (ssize_t)(c->lens[i] - dot); ssize_t k = (ssize_t)(c->nchar - nchar - c->lens[i]); for (ssize_t l = k; l < j+k; l++) { int ch = name[l]; if (isalnum(ch)) *p++ = (char) tolower(ch); } nchar += c->lens[i]; } DESTROY(c); // update our counted-string output. buf->size = (size_t)(p - buf->base); assert(buf->size == new_size); } dnsdbq-2.6.7/sort.h000066400000000000000000000026441462247651300141610ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef SORT_H_INCLUDED #define SORT_H_INCLUDED 1 #include #include "pdns.h" struct sortbuf { char *base; size_t size; }; typedef struct sortbuf *sortbuf_t; struct sortkey { char *specified, *computed; }; typedef struct sortkey *sortkey_t; typedef const struct sortkey *sortkey_ct; typedef enum { no_sort = 0, normal_sort, reverse_sort } sort_e; const char *add_sort_key(const char *); sortkey_ct find_sort_key(const char *); void sort_ready(void); void sort_destroy(void); __attribute__((noreturn)) void exec_sort(int p1[], int p2[]); char *sortable_rrname(pdns_tuple_ct); char *sortable_rdata(pdns_tuple_ct); void sortable_rdatum(sortbuf_t, const char *, const char *); void sortable_dnsname(sortbuf_t, const char *); void sortable_hexify(sortbuf_t, const u_char *, size_t); #endif /*SORT_H_INCLUDED*/ dnsdbq-2.6.7/time.c000066400000000000000000000053501462247651300141200ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #define _GNU_SOURCE #define _BSD_SOURCE #define _DEFAULT_SOURCE #include #include #include #include #include #include "defs.h" #include "time.h" #include "globals.h" #include "ns_ttl.h" /* time_cmp -- compare two absolute timestamps, give -1, 0, or 1. */ int time_cmp(u_long a, u_long b) { if (a < b) return -1; if (a > b) return 1; return 0; } /* time_str -- format one (possibly zero) timestamp * * returns static string. always uses GMT. */ const char * time_str(u_long x, bool iso8601fmt) { static char ret[sizeof "yyyy-mm-ddThh:mm:ssZ"]; if (x == 0) { strcpy(ret, "0"); } else { time_t t = (time_t)x; struct tm result, *y = gmtime_r(&t, &result); strftime(ret, sizeof ret, iso8601fmt ? "%FT%TZ" : "%F %T", y); } return ret; } /* timeval_str -- format one timeval (NULL means current time) * * returns static string. always uses GMT. * * output format: yyyy-mm-dd hh:mm:ss.fff[fff] */ const char * timeval_str(const struct timeval *src, bool milliseconds) { static char ret[sizeof "yyyy-mm-dd hh:mm:ss.ffffff"]; char *dst; struct timeval now; if (src == NULL) { gettimeofday(&now, NULL); src = &now; } time_t t = (time_t)src->tv_sec; struct tm result, *y = gmtime_r(&t, &result); dst = ret + strftime(ret, sizeof ret, "%F %T", y); long usecs = (long)src->tv_usec; if (milliseconds) sprintf(dst, ".%03ld", usecs % 1000); else sprintf(dst, ".%06ld", usecs % 1000000); return ret; } /* time_get -- parse and return one (possibly relative) timestamp. */ int time_get(const char *src, u_long *dst) { struct tm tt; long long ll; u_long t; char *ep; memset(&tt, 0, sizeof tt); if (((ep = strptime(src, "%F %T", &tt)) != NULL && *ep == '\0') || ((ep = strptime(src, "%F", &tt)) != NULL && *ep == '\0')) { *dst = (u_long)(timegm(&tt)); return 1; } ll = strtoll(src, &ep, 10); if (*src != '\0' && *ep == '\0') { if (ll < 0) *dst = (u_long)startup_time.tv_sec - (u_long)imaxabs(ll); else *dst = (u_long)ll; return 1; } if (ns_parse_ttl(src, &t) == 0) { *dst = (u_long)startup_time.tv_sec - t; return 1; } return 0; } dnsdbq-2.6.7/time.h000066400000000000000000000016341462247651300141260ustar00rootroot00000000000000/* * Copyright (c) 2014-2020 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef TIME_H_INCLUDED #define TIME_H_INCLUDED 1 #include #include #include int time_cmp(u_long, u_long); const char *time_str(u_long, bool); const char *timeval_str(const struct timeval *, bool); int time_get(const char *, u_long *); #endif /*TIME_H_INCLUDED*/ dnsdbq-2.6.7/tokstr.c000066400000000000000000000103201462247651300145010ustar00rootroot00000000000000// tokstr -- textual token iterator with some input independence // 2022-01-29 [revised during code review, add regions] // 2022-01-25 [initially released inside dnsdbq] /* externals. */ #define _GNU_SOURCE #include #include #include #include #include "tokstr.h" /* private data types. */ enum tokstr_type { ts_buffer, ts_string }; struct tokstr_class { enum tokstr_type type; }; struct tokstr_region { struct tokstr_class class; struct tokstr_reg source; }; struct tokstr_string { struct tokstr_class class; const char *source; }; struct tokstr_pvt { union { struct tokstr_class class; struct tokstr_region region; struct tokstr_string string; } data; }; /* forward. */ static struct tokstr_reg next_region(struct tokstr_region *, const char *); static struct tokstr_reg next_string(struct tokstr_string *, const char *); /* public. */ // tokstr_region -- create an iterator for a counted string struct tokstr * tokstr_region(struct tokstr_reg source) { struct tokstr_region *ts = malloc(sizeof(struct tokstr_region)); if (ts != NULL) { *ts = (struct tokstr_region) { .class = (struct tokstr_class) { .type = ts_buffer, }, .source = source, }; } return (struct tokstr *) ts; } // tokstr_string -- create an iterator for a nul-terminated string struct tokstr * tokstr_string(const char *source) { struct tokstr_string *ts = malloc(sizeof(struct tokstr_string)); if (ts != NULL) { *ts = (struct tokstr_string) { .class = (struct tokstr_class) { .type = ts_string, }, .source = source, }; } return (struct tokstr *) ts; } // tokstr_next -- return next token from an iterator (which must be free()'d) // (NULL means no more tokens are available.) char * tokstr_next(struct tokstr *ts_pub, const char *delims) { struct tokstr_reg reg = tokstr_next_region(ts_pub, delims); if (reg.base == NULL) return NULL; return strndup(reg.base, reg.size); } // tokstr_next_copy -- copy next token from an iterator; return size, 0, or -1 // (0 means no more tokens are available.) ssize_t tokstr_next_copy(struct tokstr *ts, const char *delims, char *buffer, size_t size) { struct tokstr_reg reg = tokstr_next_region(ts, delims); if (reg.base == NULL) return 0; if (reg.size >= size) return -1; memcpy(buffer, reg.base, reg.size); buffer[reg.size] = '\0'; return (ssize_t) reg.size; } // tokstr_next_region -- return next token from an iterator (zero-copy) // (.base == NULL means no more tokens are available.) struct tokstr_reg tokstr_next_region(struct tokstr *ts_pub, const char *delims) { struct tokstr_pvt *ts = (struct tokstr_pvt *) ts_pub; struct tokstr_reg reg = {}; switch (ts->data.class.type) { case ts_buffer: reg = next_region(&ts->data.region, delims); break; case ts_string: reg = next_string(&ts->data.string, delims); break; default: abort(); } assert((reg.base == NULL) == (reg.size == 0)); return reg; } // tokstr_last -- destroy an iterator and release all of its internal resources void tokstr_last(struct tokstr **pts) { free(*pts); *pts = NULL; } /* private functions. */ // next_buffer -- implement tokstr_next for counted string iterators static struct tokstr_reg next_region(struct tokstr_region *reg, const char *delims) { if (reg->source.size != 0) { while (reg->source.size != 0 && strchr(delims, *reg->source.base) != 0) reg->source.size--, reg->source.base++; const char *prev = reg->source.base; while (reg->source.size != 0 && strchr(delims, *reg->source.base) == 0) reg->source.size--, reg->source.base++; size_t size = (size_t) (reg->source.base - prev); if (size != 0) return (struct tokstr_reg) {prev, size}; } return (struct tokstr_reg) {}; } // next_string -- implement tokstr_next for nul-terminated string iterators static struct tokstr_reg next_string(struct tokstr_string *str, const char *delims) { int ch = *str->source; if (ch != '\0') { while (ch != '\0' && strchr(delims, ch) != NULL) ch = *++str->source; const char *prev = str->source; while (ch != '\0' && strchr(delims, ch) == NULL) ch = *++str->source; size_t size = (size_t) (str->source - prev); if (size != 0) return (struct tokstr_reg) {prev, size}; } return (struct tokstr_reg) {}; } dnsdbq-2.6.7/tokstr.h000066400000000000000000000041041462247651300145110ustar00rootroot00000000000000#ifndef __TOKSTR_H #define __TOKSTR_H // tokstr -- textual token iterator with some input independence // 2022-01-29 [revised during code review, add regions] // 2022-01-25 [initially released inside dnsdbq] /* example using heap-allocated strings: struct tokstr *ts = tokstr_string("this:is+-test"); for (char *t; (t = tokstr_next(ts, "-:+")) != NULL; free(t)) printf("\t\"%s\"\n", t); tokstr_last(&ts); * will output "this", "is", and "test". so will this: struct tokstr *ts = tokstr_string("this:is+-test"); for (char t[100]; tokstr_next_copy(ts, "-:+", t, sizeof t) > 0;) printf("\t\"%s\"\n", t); tokstr_last(&ts); * as will this: struct tokstr *ts = tokstr_string("this:is+-test"); for (;;) { struct tokstr_reg t = tokstr_next_region(ts, "-:+"); if (t.base == NULL) break; printf("\t\"%*s\"\n", t.size, t.base); } tokstr_last(&ts); */ // opaque type for iterator state -- never used struct tokstr; struct tokstr_reg { const char *base; size_t size; }; // tokstr_region -- create an iterator for a counted string struct tokstr *tokstr_region(struct tokstr_reg); // tokstr_string -- create an iterator for a nul-terminated string struct tokstr *tokstr_string(const char *); // tokstr_string -- create an iterator for a nul-terminated string struct tokstr *tokstr_string(const char *); // tokstr_next -- return next token from an iterator (which must be free()'d) // (NULL means no more tokens are available.) char *tokstr_next(struct tokstr *, const char *); // tokstr_next_copy -- copy next token from an iterator; return size, 0, or -1 // (0 means no more tokens are available.) ssize_t tokstr_next_copy(struct tokstr *, const char *, char *, size_t); // tokstr_next_region -- return next token from iterator (zero-copy) // (.base == NULL means no more tokens are available.) // NOTE WELL: if program state becomes undefined here, can assert() or abort() struct tokstr_reg tokstr_next_region(struct tokstr *, const char *); // tokstr_last -- destroy an iterator and release all of its internal resources void tokstr_last(struct tokstr **); #endif /*__TOKSTR_H*/