pax_global_header00006660000000000000000000000064125564435440014526gustar00rootroot0000000000000052 comment=28317091f99b8faba3aac17fa7c1ec258fb002de fnordmetric-0.1.0.20150502+gite736fba3/000077500000000000000000000000001255644354400166555ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/.gitignore000066400000000000000000000000121255644354400206360ustar00rootroot00000000000000.DS_Store fnordmetric-0.1.0.20150502+gite736fba3/.travis.yml000066400000000000000000000001161255644354400207640ustar00rootroot00000000000000rvm: 1.9.3 script: ./fnordmetric-core/run_specs.sh services: - redis-server fnordmetric-0.1.0.20150502+gite736fba3/Dockerfile000066400000000000000000000013751255644354400206550ustar00rootroot00000000000000FROM ubuntu:14.04 RUN apt-get update \ && apt-get install -y curl \ && rm -rf /var/lib/apt/lists/* RUN buildDeps='unzip git gcc make clang++-3.4 cmake libmysqlclient-dev'; \ set -x \ && apt-get update && apt-get install -y $buildDeps --no-install-recommends \ && rm -rf /var/lib/apt/lists/* RUN mkdir -p /usr/src/ \ && cd /usr/src \ && curl -R -O https://codeload.github.com/dolfly/fnordmetric/zip/master \ && unzip master \ && cd fnordmetric-master/fnordmetric-core \ && make \ && cp build/cmake/target/fnordmetric-server /usr/local/bin VOLUME /data/fnordmetric EXPOSE 8080 8125/udp CMD [ "/usr/local/bin/fnordmetric-server", "--http_port", "8080", "--statsd_port", "8125", "--storage_backend", "disk", "--datadir", "/data/fnordmetric"] fnordmetric-0.1.0.20150502+gite736fba3/LICENSE000066400000000000000000001045131255644354400176660ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . fnordmetric-0.1.0.20150502+gite736fba3/Makefile000066400000000000000000000011531255644354400203150ustar00rootroot00000000000000# This file is part of the "FnordMetric" project # Copyright (c) 2014 Paul Asmuth, Google Inc. # # Licensed under the MIT license (see LICENSE). all: build build: (cd fnordmetric-webui && make build) (cd fnordmetric-core && make build) install: (cd fnordmetric-core && make install) test: (cd fnordmetric-core && make test) clean: (cd fnordmetric-webui && make clean) (cd fnordmetric-core && make clean) doc: find doc/examples -name "*.sql" | while read file; do PATH=./build:$$PATH fnordmetric -f svg -o $${file/.sql/.svg}.html $$file; done (cd doc/web && rake build) .PHONY: all test clean doc build fnordmetric-0.1.0.20150502+gite736fba3/README.md000066400000000000000000000076021255644354400201410ustar00rootroot00000000000000FnordMetric =========== FnordMetric allows you collect and visualize timeseries data using only SQL. It enables you to build beautiful real-time analytics dashboards within minutes. Documentation: [fnordmetric.io](http://fnordmetric.io/) Demo Video: [http://fnordmetric.io/fnordmetric-server.mp4](http://fnordmetric.io/fnordmetric-server.mp4) [ ![Screenshot](https://raw.githubusercontent.com/paulasmuth/fnordmetric/master/fnordmetric-doc/web/assets/img/fnordmetric_server_screen1.png) ](http://github.com/paulasmuth/fnordmetric) #### FnordMetric ChartSQL FnordMetric ChartSQL allows you to write SQL queries that return charts instead of tables. The charts are rendered as SVG vector graphics and can easily be embedded into any website and customized with css in order to build beautiful dashboards. You can execute ChartSQL queries from the command line against a number of external sources like CSV files or a MySQL database. #### FnordMetric Server Fnordmetric Server is a standalone HTTP server application. It exposes a web UI and a HTTP API to run ChartSQL queries and collect timeseries data. You can use fnordmetric-server as a one-stop solution for metric collection and charting. Since fnordmetric-server aims to be a StatsD+graphite competitor, it implements a wire compatible StatsD API. FnordMetric Server can store the collected timeseries data on local disk or in external storage (HBase). Documentation ------------- You can find the full FnordMetric Documentation at http://fnordmetric.io/ Contributions ------------- #### Individual Contributors to FnordMetric + Laura Schlimmer (http://github.com/lauraschlimmer) + Henrik Muehe (http://github.com/henrik-muehe) + Philipp Bogensberger (http://github.com/bogensberger) + "Dolfly" (http://github.com/dolfly) + Christian Parpart (http://github.com/trapni) + Finn Zirngibl (https://github.com/finnomenon) + Simon Menke (http://github.com/fd) + Bruno Michel (http://github.com/nono) + Marco Borromeo (http://github.com/mborromeo) + Leo Lou (http://github.com/l4u) + Andy Lindeman (http://github.com/alindeman) + Jurriaan Pruis (http://github.com/jurriaan) + Kacper Bielecki (http://github.com/kazjote) + John Murray (http://github.com/JohnMurray) + Lars Gierth (http://github.com/lgierth) + Ross Kaffenberger (http://github.com/rossta) + Kunal Modi (http://github.com/kunalmodi) + Michael Fairchild (http://github.com/fairchild) + James Cox (http://github.com/imajes) + Pieter Noordhuis (http://github.com/pietern) + Tadas Ščerbinskas (http://github.com/tadassce) + Sebastian Korfmann (http://github.com/skorfmann) + Paul Asmuth (http://github.com/paulasmuth) To contribute, please fork this repository, make your changes and run the specs, commit them to your github repository and send me a pull request. Need help, head on over to our [Google Groups][1] page to discuss any ideas that you might have. #### Other Open Source Software included in FnordMetric + Marijn Haverbeke's CodeMirror --- javascript code editor (https://github.com/marijnh/codemirror) + FontAwesome --- icon font (http://fontawesome.io/) License ------- Copyright (c) 2011-2014 Paul Asmuth FnordMetric is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. FnordMetric is distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with FnordMetric. If not, see . [1]: http://groups.google.com/group/fnordmetric [2]: http://www.screenr.com/KiJs [3]: https://secure.travis-ci.org/paulasmuth/fnordmetric.png [4]: http://travis-ci.org/paulasmuth/fnordmetric fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-agents/000077500000000000000000000000001255644354400224505ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-agents/linux-agent/000077500000000000000000000000001255644354400247035ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-agents/linux-agent/fm-linux-agent.rb000066400000000000000000000110321255644354400300600ustar00rootroot00000000000000#!/usr/bin/env ruby require "timeout" require "socket" ############################################################################### # Prefix every metric name with this string METRIC_NAME_PREFIX = "/linux/" # Report the statistics every 1 second INTERVAL = 1.0 ############################################################################### if ARGV.length != 2 puts "usage: #{$0} " puts " Report Linux system stats to FnordMetric Server via statsd" puts "" puts "example: #{$0} 127.0.0.1 8125" exit 1 end udp = UDPSocket.new target_host = ARGV[0] target_port = ARGV[1].to_i loop do last_run = Time.now.to_f @samples = [] # hostname @hostname = `hostname`.strip # collect CPU load averages def report_cpu_load if File.exists?("/proc/loadavg") loadavg_data = IO::read("/proc/loadavg") loadavg = loadavg_data.scan(/([0-9]+[,\.][0-9]+)+/).flatten if loadavg.size == 3 @samples << { :metric => "load_avg_1m", :value => loadavg[0], :labels => { :host => @hostname } } @samples << { :metric => "load_avg_5m", :value => loadavg[1], :labels => { :host => @hostname } } @samples << { :metric => "load_avg_15m", :value => loadavg[2], :labels => { :host => @hostname } } end end end # gather basic memory statistics def report_memory_stats if File.exists?("/proc/meminfo") loadavg_data = IO::read("/proc/meminfo") memtotal = loadavg_data.scan(/MemTotal:\s+ (\d+)\skB/).flatten.first.to_i memfree = loadavg_data.scan(/MemFree:\s+ (\d+)\skB/).flatten.first.to_i swaptotal = loadavg_data.scan(/SwapTotal:\s+ (\d+)\skB/).flatten.first.to_i swapfree = loadavg_data.scan(/SwapFree:\s+ (\d+)\skB/).flatten.first.to_i @samples << { :metric => "memory_total", :value => memtotal, :labels => { :host => @hostname } } @samples << { :metric => "memory_free", :value => memfree, :labels => { :host => @hostname } } @samples << { :metric => "swap_total", :value => swaptotal, :labels => { :host => @hostname } } @samples << { :metric => "swap_free", :value => swapfree, :labels => { :host => @hostname } } end end # determine disk usage and available space def report_disk_stats df_input = `df -P`.lines[1..-1] if !df_input.empty? df_input.each do |single_line| elements = single_line.split " " @samples << { :metric => "disk_used", :value => elements[2], :labels => { :host => @hostname, :mount_name => elements[0] } } @samples << { :metric => "disk_available", :value => elements[3], :labels => { :host => @hostname, :mount_name => elements[0] } } end end end #count open TCP and UDP sockets def report_open_sockets if File.exists?("/proc/net/tcp") tcp_sockets = %x{wc -l "/proc/net/tcp"}.to_i - 1 @samples << { :metric => "open_tcp_sockets", :value => tcp_sockets, :labels => { :host => @hostname } } end if File.exists?("/proc/net/udp") tcp_sockets = %x{wc -l "/proc/net/udp"}.to_i - 1 @samples << { :metric => "open_udp_sockets", :value => tcp_sockets, :labels => { :host => @hostname } } end end # fill samples array with data report_memory_stats report_disk_stats report_cpu_load report_open_sockets # send the samples in a single udp packet to FnordMetric server (the combined # packet size must be strictly less than or equal to 65535 bytes packet = "" @samples.each do |sample| packet << METRIC_NAME_PREFIX + sample[:metric] sample[:labels].each do |k,v| packet << "[#{k}=#{v}]" end packet << ":" packet << sample[:value].to_s.gsub(",", ".") packet << "\n" end begin udp.send packet, 0, target_host, target_port rescue Exception => e puts e end # sleep if we completed executing faster than the requested interval sleep_for = (last_run + INTERVAL) - Time.now.to_f sleep(sleep_for) if sleep_for > 0 end fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-agents/osx-agent/000077500000000000000000000000001255644354400243555ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-agents/osx-agent/fm-osx-agent.rb000077500000000000000000000035771255644354400272260ustar00rootroot00000000000000#!/usr/bin/env ruby require "timeout" require "socket" ############################################################################### # Prefix every metric name with this string METRIC_NAME_PREFIX = "/osx/" # Report the statistics every 1 second INTERVAL = 1.0 ############################################################################### if ARGV.length != 2 puts "usage: #{$0} " puts " Report OSX system stats to FnordMetric Server via statsd" puts "" puts "example: #{$0} 127.0.0.1 8125" exit 1 end udp = UDPSocket.new target_host = ARGV[0] target_port = ARGV[1].to_i loop do last_run = Time.now.to_f samples = [] # hostname hostname = `hostname`.strip # uptime uptime = `uptime`.strip # gather load averages if uptime =~ /load averages?: ([0-9]+[,\.][0-9]+)\s+([0-9]+[,\.][0-9]+)\s+([0-9]+[,\.][0-9]+)/ samples << { :metric => "load_avg_1m", :value => $1, :labels => { :host => hostname } } samples << { :metric => "load_avg_5m", :value => $2, :labels => { :host => hostname } } samples << { :metric => "load_avg_15m", :value => $3, :labels => { :host => hostname } } end # send the samples in a single udp packet to FnordMetric server (the combined # packet size must be strictly less than or equal to 65535 bytes packet = "" samples.each do |sample| packet << METRIC_NAME_PREFIX + sample[:metric] sample[:labels].each do |k,v| packet << "[#{k}=#{v}]" end packet << ":" packet << sample[:value].to_s.gsub(",", ".") packet << "\n" end begin udp.send packet, 0, target_host, target_port rescue Exception => e puts e end # sleep if we completed executing faster than the requested interval sleep_for = (last_run + INTERVAL) - Time.now.to_f sleep(sleep_for) if sleep_for > 0 end fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/000077500000000000000000000000001255644354400221175ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/Makefile000066400000000000000000000010741255644354400235610ustar00rootroot00000000000000# This file is part of the "FnordMetric" project # Copyright (c) 2014 Paul Asmuth, Google Inc. # # Licensed under the MIT license (see LICENSE). all: build build: (cd ../fnordmetric-webui && make) (cd build/cmake && make) install: build (cd build/cmake && make install) devserver: build mkdir -p /tmp/fnordmetric-data DEV_ASSET_PATH=../ ./build/cmake/target/fnordmetric-server --datadir /tmp/fnordmetric-data --verbose test: (cd build/cmake && make test) clean: (cd build/cmake && make clean) rm -rf build/test/tmp* .PHONY: all test clean build devserver fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/autogen.sh000077500000000000000000000001161255644354400241160ustar00rootroot00000000000000#!/bin/bash if [[ "$1" == "clean" ]]; then make clean exit 0 fi make all fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/000077500000000000000000000000001255644354400232165ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/.gitignore000066400000000000000000000001131255644354400252010ustar00rootroot00000000000000fnordmetric libfnordmetric-static.a libfnordmetric.dylib libfnordmetric.so fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/000077500000000000000000000000001255644354400242765ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/.gitignore000066400000000000000000000000151255644354400262620ustar00rootroot00000000000000stage target fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/CMakeLists.txt000066400000000000000000000251301255644354400270370ustar00rootroot00000000000000# This file is part of the "FnordMetric" project # Copyright (c) 2014 Paul Asmuth, Google Inc. # # FnordMetric is free software: you can redistribute it and/or modify it under # the terms of the GNU General Public License v3.0. You should have received a # copy of the GNU General Public License along with this program. If not, see # . cmake_minimum_required(VERSION 2.6) include(FindPkgConfig) include(CheckIncludeFile) set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/modules/") project(fnordmetric) option(ENABLE_TESTS "Build unit tests [default: off]" OFF) set(FNORDMETRIC_SOURCES stage/src/fnordmetric/cli/cli.cc stage/src/fnordmetric/cli/flagparser.cc stage/src/fnordmetric/cli/cli.cc stage/src/fnordmetric/io/acceptor.cc stage/src/fnordmetric/io/eventloop.cc stage/src/fnordmetric/io/file.cc stage/src/fnordmetric/io/fileutil.cc stage/src/fnordmetric/io/filerepository.cc stage/src/fnordmetric/io/mmappedfile.cc stage/src/fnordmetric/io/pagemanager.cc stage/src/fnordmetric/net/udpserver.cc stage/src/fnordmetric/environment.cc stage/src/fnordmetric/http/httpinputstream.cc stage/src/fnordmetric/http/httpoutputstream.cc stage/src/fnordmetric/http/httpmessage.cc stage/src/fnordmetric/http/httprequest.cc stage/src/fnordmetric/http/httpresponse.cc stage/src/fnordmetric/http/httpserver.cc stage/src/fnordmetric/sstable/cursor.cc stage/src/fnordmetric/sstable/fileheaderreader.cc stage/src/fnordmetric/sstable/fileheaderwriter.cc stage/src/fnordmetric/sstable/index.cc stage/src/fnordmetric/sstable/indexprovider.cc stage/src/fnordmetric/sstable/rowoffsetindex.cc stage/src/fnordmetric/sstable/sstablereader.cc stage/src/fnordmetric/sstable/sstablerepair.cc stage/src/fnordmetric/sstable/sstablewriter.cc stage/src/fnordmetric/util/assets.cc stage/src/fnordmetric/util/binarymessagereader.cc stage/src/fnordmetric/util/binarymessagewriter.cc stage/src/fnordmetric/util/buffer.cc stage/src/fnordmetric/util/datetime.cc stage/src/fnordmetric/util/exceptionhandler.cc stage/src/fnordmetric/util/format.cc stage/src/fnordmetric/util/fnv.cc stage/src/fnordmetric/util/ieee754.cc stage/src/fnordmetric/util/inputstream.cc stage/src/fnordmetric/util/inspect.cc stage/src/fnordmetric/util/logger.cc stage/src/fnordmetric/util/logoutputstream.cc stage/src/fnordmetric/util/outputstream.cc stage/src/fnordmetric/util/jsonoutputstream.cc stage/src/fnordmetric/util/random.cc stage/src/fnordmetric/util/runtimeexception.cc stage/src/fnordmetric/util/signalhandler.cc stage/src/fnordmetric/util/stringutil.cc stage/src/fnordmetric/util/uri.cc stage/src/fnordmetric/util/wallclock.cc stage/src/fnordmetric/ui/axisdefinition.cc stage/src/fnordmetric/ui/areachart.cc stage/src/fnordmetric/ui/barchart.cc stage/src/fnordmetric/ui/linechart.cc stage/src/fnordmetric/ui/pointchart.cc stage/src/fnordmetric/ui/canvas.cc stage/src/fnordmetric/ui/domain.cc stage/src/fnordmetric/ui/domainprovider.cc stage/src/fnordmetric/ui/drawable.cc stage/src/fnordmetric/ui/griddefinition.cc stage/src/fnordmetric/ui/legenddefinition.cc stage/src/fnordmetric/ui/series.cc stage/src/fnordmetric/ui/timedomain.cc stage/src/fnordmetric/sql/backends/csv/csvbackend.cc stage/src/fnordmetric/sql/backends/csv/csvinputstream.cc stage/src/fnordmetric/sql/backends/csv/csvtableref.cc stage/src/fnordmetric/sql/backends/mysql/mysqlbackend.cc stage/src/fnordmetric/sql/backends/mysql/mysqlconnection.cc stage/src/fnordmetric/sql/backends/mysql/mysqltableref.cc stage/src/fnordmetric/sql/backends/postgres/postgresbackend.cc stage/src/fnordmetric/sql/backends/postgres/postgresconnection.cc stage/src/fnordmetric/sql/backends/postgres/postgrestableref.cc stage/src/fnordmetric/query/query.cc stage/src/fnordmetric/query/queryservice.cc stage/src/fnordmetric/sql/expressions/aggregate.cc stage/src/fnordmetric/sql/expressions/boolean.cc stage/src/fnordmetric/sql/expressions/datetime.cc stage/src/fnordmetric/sql/expressions/math.cc stage/src/fnordmetric/sql/parser/astnode.cc stage/src/fnordmetric/sql/parser/astutil.cc stage/src/fnordmetric/sql/parser/parser.cc stage/src/fnordmetric/sql/parser/token.cc stage/src/fnordmetric/sql/parser/tokenize.cc stage/src/fnordmetric/sql/runtime/compile.cc stage/src/fnordmetric/sql/runtime/defaultruntime.cc stage/src/fnordmetric/sql/runtime/execute.cc stage/src/fnordmetric/sql/runtime/groupovertimewindow.cc stage/src/fnordmetric/sql/runtime/orderby.cc stage/src/fnordmetric/sql/runtime/importstatement.cc stage/src/fnordmetric/sql/runtime/queryplan.cc stage/src/fnordmetric/sql/runtime/queryplanbuilder.cc stage/src/fnordmetric/sql/runtime/queryplannode.cc stage/src/fnordmetric/sql/runtime/runtime.cc stage/src/fnordmetric/sql/runtime/symboltable.cc stage/src/fnordmetric/sql/runtime/tablerepository.cc stage/src/fnordmetric/sql/runtime/tablescan.cc stage/src/fnordmetric/sql/svalue.cc stage/src/fnordmetric/sql_extensions/areachartbuilder.cc stage/src/fnordmetric/sql_extensions/barchartbuilder.cc stage/src/fnordmetric/sql_extensions/linechartbuilder.cc stage/src/fnordmetric/sql_extensions/pointchartbuilder.cc stage/src/fnordmetric/sql_extensions/domainconfig.cc stage/src/fnordmetric/sql_extensions/drawstatement.cc stage/src/fnordmetric/sql_extensions/seriesadapter.cc stage/src/fnordmetric/thread/threadpool.cc stage/src/fnordmetric/metricdb/adminui.cc stage/src/fnordmetric/metricdb/backends/disk/compactiontask.cc stage/src/fnordmetric/metricdb/backends/disk/metric.cc stage/src/fnordmetric/metricdb/backends/disk/metriccursor.cc stage/src/fnordmetric/metricdb/backends/disk/metricsnapshot.cc stage/src/fnordmetric/metricdb/backends/disk/metricrepository.cc stage/src/fnordmetric/metricdb/backends/disk/labelindex.cc stage/src/fnordmetric/metricdb/backends/disk/labelindexreader.cc stage/src/fnordmetric/metricdb/backends/disk/labelindexwriter.cc stage/src/fnordmetric/metricdb/backends/disk/samplereader.cc stage/src/fnordmetric/metricdb/backends/disk/samplewriter.cc stage/src/fnordmetric/metricdb/backends/disk/tableheaderreader.cc stage/src/fnordmetric/metricdb/backends/disk/tableheaderwriter.cc stage/src/fnordmetric/metricdb/backends/disk/tableref.cc stage/src/fnordmetric/metricdb/backends/disk/tokenindex.cc stage/src/fnordmetric/metricdb/backends/disk/tokenindexreader.cc stage/src/fnordmetric/metricdb/backends/disk/tokenindexwriter.cc stage/src/fnordmetric/metricdb/backends/inmemory/metric.cc stage/src/fnordmetric/metricdb/backends/inmemory/metricrepository.cc stage/src/fnordmetric/metricdb/httpapi.cc stage/src/fnordmetric/metricdb/metric.cc stage/src/fnordmetric/metricdb/metricrepository.cc stage/src/fnordmetric/metricdb/metrictableref.cc stage/src/fnordmetric/metricdb/metrictablerepository.cc stage/src/fnordmetric/metricdb/sample.cc stage/src/fnordmetric/metricdb/statsd.cc) include_directories(${PROJECT_BINARY_DIR}) include_directories(stage/src) include_directories(stage/assets) if(APPLE) set(CMAKE_CXX_FLAGS "-std=c++0x -stdlib=libc++ ${CMAKE_CXX_FLAGS}") else() set(CMAKE_CXX_FLAGS "-std=c++0x ${CMAKE_CXX_FLAGS}") endif() add_executable(fnordmetric-cli ${FNORDMETRIC_SOURCES} stage/src/fnordmetric/cli.cc) target_link_libraries(fnordmetric-cli m) install(TARGETS fnordmetric-cli DESTINATION bin) add_executable(fnordmetric-server ${FNORDMETRIC_SOURCES} stage/src/fnordmetric/server.cc) target_link_libraries(fnordmetric-server m) install(TARGETS fnordmetric-server DESTINATION bin) find_package(Threads) target_link_libraries(fnordmetric-cli ${CMAKE_THREAD_LIBS_INIT}) target_link_libraries(fnordmetric-server ${CMAKE_THREAD_LIBS_INIT}) find_package(MySQL) if(MYSQL_FOUND) set(FNORD_ENABLE_MYSQL true) target_link_libraries(fnordmetric-cli ${MYSQL_CLIENT_LIBS}) target_link_libraries(fnordmetric-server ${MYSQL_CLIENT_LIBS}) else() message("WARNING: libmysqlclient not found, FnordMetric will be compiled without MySQL support") endif() find_package(PostgreSQL) if(PostgreSQL_FOUND) set(FNORD_ENABLE_POSTGRES true) target_link_libraries(fnordmetric-cli ${PostgreSQL_LIBRARIES}) target_link_libraries(fnordmetric-server ${PostgreSQL_LIBRARIES}) else() message("WARNING: libpq not found, FnordMetric will be compiled without Postgres support") endif() configure_file(config.h.in config.h) if(ENABLE_TESTS) add_library(fnord SHARED ${FNORDMETRIC_SOURCES}) target_link_libraries(fnord m ${MYSQL_CLIENT_LIBS}) add_executable(tests/test-sql stage/src/fnordmetric/sql/sql_test.cc) target_link_libraries(tests/test-sql fnord) add_executable(tests/test-ui stage/src/fnordmetric/ui/ui_test.cc) target_link_libraries(tests/test-ui fnord) add_executable(tests/test-input-stream stage/src/fnordmetric/util/inputstream_test.cc) target_link_libraries(tests/test-input-stream fnord) add_executable(tests/test-uri stage/src/fnordmetric/util/uri_test.cc) target_link_libraries(tests/test-uri fnord) add_executable(tests/test-http stage/src/fnordmetric/http/http_test.cc) target_link_libraries(tests/test-http fnord) target_link_libraries(tests/test-uri fnord) add_executable(tests/test-fnv stage/src/fnordmetric/util/fnv_test.cc) target_link_libraries(tests/test-fnv fnord) add_executable(tests/test-csv-backend stage/src/fnordmetric/sql/backends/csv/csvbackend_test.cc) target_link_libraries(tests/test-csv-backend fnord) add_executable(tests/test-pagemanager stage/src/fnordmetric/io/pagemanager_test.cc) target_link_libraries(tests/test-pagemanager fnord) add_executable(tests/test-sstable stage/src/fnordmetric/sstable/sstable_test.cc) target_link_libraries(tests/test-sstable fnord) add_executable(tests/test-sql-extensions stage/src/fnordmetric/sql_extensions/sql_extensions_test.cc) target_link_libraries(tests/test-sql-extensions fnord) add_executable(tests/test-query stage/src/fnordmetric/query/query_test.cc) target_link_libraries(tests/test-query fnord) add_executable(tests/test-cli stage/src/fnordmetric/cli/cli_test.cc) target_link_libraries(tests/test-cli fnord) add_executable(tests/test-statsd stage/src/fnordmetric/metricdb/statsd_test.cc) target_link_libraries(tests/test-statsd fnord) add_executable(tests/test-disk-backend stage/src/fnordmetric/metricdb/backends/disk/diskbackend_test.cc) target_link_libraries(tests/test-disk-backend fnord) endif() fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/Makefile000066400000000000000000000013751255644354400257440ustar00rootroot00000000000000# This file is part of the "FnordMetric" project # Copyright (c) 2014 Paul Asmuth, Google Inc. # # Licensed under the MIT license (see LICENSE). all: assets mkdir -p target/tests mkdir -p stage/src test -e stage/src/fnordmetric || ln -s ../../../../src stage/src/fnordmetric || true (cd target && cmake .. -DCMAKE_BUILD_TYPE=Release && make) rm -f ../fnordmetric rm -f ../libfnordmetric-static.a rm -f ../libfnordmetric.dylib rm -f ../libfnordmetric.so ln -s cmake/target/fnordmetric ../fnordmetric install: (cd target && cmake .. && make install) assets: ./assets.sh test: all @find target/tests -iname "test-*" | while read t; do (cd ../../ && build/cmake/$$t) || exit 1; done clean: rm -rf target stage .PHONY: all test clean assets install fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/assets.sh000077500000000000000000000033331255644354400261410ustar00rootroot00000000000000#/bin/bash set -e mkdir -p stage/src echo "#include " > stage/src/asset_bundle.cc asset_file() { (cd ../../.. && cat $@ | xxd -i) echo "};" } asset_uniq() { echo "static const unsigned char __$1_data[] = {" } asset_name() { echo "static fnordmetric::util::Assets::AssetFile __$1(\"$2\", __$1_data, sizeof(__$1_data));" }; ( asset_uniq "fnordmetric_js_fnordmetric_js" asset_file "fnordmetric-js/fnordmetric.js" asset_name "fnordmetric_js_fnordmetric_js" "fnordmetric-js/fnordmetric.js" ) >> stage/src/asset_bundle.cc ( asset_uniq "fnordmetric_webui_fnordmetric_webui_css" asset_file "fnordmetric-webui/fnordmetric-webui.css" asset_name "fnordmetric_webui_fnordmetric_webui_css" "fnordmetric-webui/fnordmetric-webui.css" ) >> stage/src/asset_bundle.cc ( asset_uniq "fnordmetric_webui_fnordmetric_webui_html" asset_file "fnordmetric-webui/fnordmetric-webui.html" asset_name "fnordmetric_webui_fnordmetric_webui_html" "fnordmetric-webui/fnordmetric-webui.html" ) >> stage/src/asset_bundle.cc ( asset_uniq "fnordmetric_webui_fnordmetric_webui_js" asset_file "fnordmetric-webui/fnordmetric-webui.js" asset_name "fnordmetric_webui_fnordmetric_webui_js" "fnordmetric-webui/fnordmetric-webui.js" ) >> stage/src/asset_bundle.cc ( asset_uniq "fnordmetric_webui_fnordmetric_favicon_ico" asset_file "fnordmetric-webui/fnordmetric-favicon.ico" asset_name "fnordmetric_webui_fnordmetric_favicon_ico" "fnordmetric-webui/fnordmetric-favicon.ico" ) >> stage/src/asset_bundle.cc ( asset_uniq "fnordmetric_webui_fontawesome_woff" asset_file "fnordmetric-webui/fontawesome.woff" asset_name "fnordmetric_webui_fontawesome_woff" "fnordmetric-webui/fontawesome.woff" ) >> stage/src/asset_bundle.cc fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/config.h.in000066400000000000000000000001031255644354400263130ustar00rootroot00000000000000#cmakedefine FNORD_ENABLE_MYSQL #cmakedefine FNORD_ENABLE_POSTGRES fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/modules/000077500000000000000000000000001255644354400257465ustar00rootroot00000000000000CMakeParseArguments.cmake000066400000000000000000000134541255644354400325410ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/modules#.rst: # CMakeParseArguments # ------------------- # # # # CMAKE_PARSE_ARGUMENTS( # args...) # # CMAKE_PARSE_ARGUMENTS() is intended to be used in macros or functions # for parsing the arguments given to that macro or function. It # processes the arguments and defines a set of variables which hold the # values of the respective options. # # The argument contains all options for the respective macro, # i.e. keywords which can be used when calling the macro without any # value following, like e.g. the OPTIONAL keyword of the install() # command. # # The argument contains all keywords for this macro # which are followed by one value, like e.g. DESTINATION keyword of the # install() command. # # The argument contains all keywords for this # macro which can be followed by more than one value, like e.g. the # TARGETS or FILES keywords of the install() command. # # When done, CMAKE_PARSE_ARGUMENTS() will have defined for each of the # keywords listed in , and # a variable composed of the given # followed by "_" and the name of the respective keyword. These # variables will then hold the respective value from the argument list. # For the keywords this will be TRUE or FALSE. # # All remaining arguments are collected in a variable # _UNPARSED_ARGUMENTS, this can be checked afterwards to see # whether your macro was called with unrecognized parameters. # # As an example here a my_install() macro, which takes similar arguments # as the real install() command: # # :: # # function(MY_INSTALL) # set(options OPTIONAL FAST) # set(oneValueArgs DESTINATION RENAME) # set(multiValueArgs TARGETS CONFIGURATIONS) # cmake_parse_arguments(MY_INSTALL "${options}" "${oneValueArgs}" # "${multiValueArgs}" ${ARGN} ) # ... # # # # Assume my_install() has been called like this: # # :: # # my_install(TARGETS foo bar DESTINATION bin OPTIONAL blub) # # # # After the cmake_parse_arguments() call the macro will have set the # following variables: # # :: # # MY_INSTALL_OPTIONAL = TRUE # MY_INSTALL_FAST = FALSE (this option was not used when calling my_install() # MY_INSTALL_DESTINATION = "bin" # MY_INSTALL_RENAME = "" (was not used) # MY_INSTALL_TARGETS = "foo;bar" # MY_INSTALL_CONFIGURATIONS = "" (was not used) # MY_INSTALL_UNPARSED_ARGUMENTS = "blub" (no value expected after "OPTIONAL" # # # # You can then continue and process these variables. # # Keywords terminate lists of values, e.g. if directly after a # one_value_keyword another recognized keyword follows, this is # interpreted as the beginning of the new option. E.g. # my_install(TARGETS foo DESTINATION OPTIONAL) would result in # MY_INSTALL_DESTINATION set to "OPTIONAL", but MY_INSTALL_DESTINATION # would be empty and MY_INSTALL_OPTIONAL would be set to TRUE therefor. #============================================================================= # Copyright 2010 Alexander Neundorf # # Distributed under the OSI-approved BSD License (the "License"); # see accompanying file Copyright.txt for details. # # This software is distributed WITHOUT ANY WARRANTY; without even the # implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # See the License for more information. #============================================================================= # (To distribute this file outside of CMake, substitute the full # License text for the above reference.) if(__CMAKE_PARSE_ARGUMENTS_INCLUDED) return() endif() set(__CMAKE_PARSE_ARGUMENTS_INCLUDED TRUE) function(CMAKE_PARSE_ARGUMENTS prefix _optionNames _singleArgNames _multiArgNames) # first set all result variables to empty/FALSE foreach(arg_name ${_singleArgNames} ${_multiArgNames}) set(${prefix}_${arg_name}) endforeach() foreach(option ${_optionNames}) set(${prefix}_${option} FALSE) endforeach() set(${prefix}_UNPARSED_ARGUMENTS) set(insideValues FALSE) set(currentArgName) # now iterate over all arguments and fill the result variables foreach(currentArg ${ARGN}) list(FIND _optionNames "${currentArg}" optionIndex) # ... then this marks the end of the arguments belonging to this keyword list(FIND _singleArgNames "${currentArg}" singleArgIndex) # ... then this marks the end of the arguments belonging to this keyword list(FIND _multiArgNames "${currentArg}" multiArgIndex) # ... then this marks the end of the arguments belonging to this keyword if(${optionIndex} EQUAL -1 AND ${singleArgIndex} EQUAL -1 AND ${multiArgIndex} EQUAL -1) if(insideValues) if("${insideValues}" STREQUAL "SINGLE") set(${prefix}_${currentArgName} ${currentArg}) set(insideValues FALSE) elseif("${insideValues}" STREQUAL "MULTI") list(APPEND ${prefix}_${currentArgName} ${currentArg}) endif() else() list(APPEND ${prefix}_UNPARSED_ARGUMENTS ${currentArg}) endif() else() if(NOT ${optionIndex} EQUAL -1) set(${prefix}_${currentArg} TRUE) set(insideValues FALSE) elseif(NOT ${singleArgIndex} EQUAL -1) set(currentArgName ${currentArg}) set(${prefix}_${currentArgName}) set(insideValues "SINGLE") elseif(NOT ${multiArgIndex} EQUAL -1) set(currentArgName ${currentArg}) set(${prefix}_${currentArgName}) set(insideValues "MULTI") endif() endif() endforeach() # propagate the result variables to the caller: foreach(arg_name ${_singleArgNames} ${_multiArgNames} ${_optionNames}) set(${prefix}_${arg_name} ${${prefix}_${arg_name}} PARENT_SCOPE) endforeach() set(${prefix}_UNPARSED_ARGUMENTS ${${prefix}_UNPARSED_ARGUMENTS} PARENT_SCOPE) endfunction() fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/modules/FindMySQL.cmake000066400000000000000000000076741255644354400305340ustar00rootroot00000000000000#-------------------------------------------------------- # Copyright (C) 1995-2007 MySQL AB # # This program is free software; you can redistribute it and/or modify # it under the terms of version 2 of the GNU General Public License as # published by the Free Software Foundation. # # There are special exceptions to the terms and conditions of the GPL # as it is applied to this software. View the full text of the exception # in file LICENSE.exceptions in the top-level directory of this software # distribution. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # The MySQL Connector/ODBC is licensed under the terms of the # GPL, like most MySQL Connectors. There are special exceptions # to the terms and conditions of the GPL as it is applied to # this software, see the FLOSS License Exception available on # mysql.com. ########################################################################## #-------------- FIND MYSQL_INCLUDE_DIR ------------------ FIND_PATH(MYSQL_INCLUDE_DIR mysql.h $ENV{MYSQL_INCLUDE_DIR} $ENV{MYSQL_DIR}/include /usr/include/mysql /usr/local/include/mysql /opt/mysql/mysql/include /opt/mysql/mysql/include/mysql /opt/mysql/include /opt/local/include/mysql5 /usr/local/mysql/include /usr/local/mysql/include/mysql $ENV{ProgramFiles}/MySQL/*/include $ENV{SystemDrive}/MySQL/*/include) #----------------- FIND MYSQL_LIB_DIR ------------------- IF (WIN32) # Set lib path suffixes # dist = for mysql binary distributions # build = for custom built tree IF (CMAKE_BUILD_TYPE STREQUAL Debug) SET(libsuffixDist debug) SET(libsuffixBuild Debug) ELSE (CMAKE_BUILD_TYPE STREQUAL Debug) SET(libsuffixDist opt) SET(libsuffixBuild Release) ADD_DEFINITIONS(-DDBUG_OFF) ENDIF (CMAKE_BUILD_TYPE STREQUAL Debug) FIND_LIBRARY(MYSQL_LIB NAMES mysqlclient PATHS $ENV{MYSQL_DIR}/lib/${libsuffixDist} $ENV{MYSQL_DIR}/libmysql $ENV{MYSQL_DIR}/libmysql/${libsuffixBuild} $ENV{MYSQL_DIR}/client/${libsuffixBuild} $ENV{MYSQL_DIR}/libmysql/${libsuffixBuild} $ENV{ProgramFiles}/MySQL/*/lib/${libsuffixDist} $ENV{SystemDrive}/MySQL/*/lib/${libsuffixDist}) ELSE (WIN32) FIND_LIBRARY(MYSQL_LIB NAMES mysqlclient_r PATHS $ENV{MYSQL_DIR}/libmysql_r/.libs $ENV{MYSQL_DIR}/lib $ENV{MYSQL_DIR}/lib/mysql /usr/lib/mysql /usr/local/lib/mysql /usr/local/mysql/lib /usr/local/mysql/lib/mysql /opt/local/mysql5/lib /opt/local/lib/mysql5/mysql /opt/mysql/mysql/lib/mysql /opt/mysql/lib/mysql) ENDIF (WIN32) IF(MYSQL_LIB) GET_FILENAME_COMPONENT(MYSQL_LIB_DIR ${MYSQL_LIB} PATH) ENDIF(MYSQL_LIB) IF (MYSQL_INCLUDE_DIR AND MYSQL_LIB_DIR) SET(MYSQL_FOUND TRUE) INCLUDE_DIRECTORIES(${MYSQL_INCLUDE_DIR}) LINK_DIRECTORIES(${MYSQL_LIB_DIR}) FIND_LIBRARY(MYSQL_ZLIB zlib PATHS ${MYSQL_LIB_DIR}) FIND_LIBRARY(MYSQL_YASSL yassl PATHS ${MYSQL_LIB_DIR}) FIND_LIBRARY(MYSQL_TAOCRYPT taocrypt PATHS ${MYSQL_LIB_DIR}) SET(MYSQL_CLIENT_LIBS mysqlclient_r) IF (MYSQL_ZLIB) SET(MYSQL_CLIENT_LIBS ${MYSQL_CLIENT_LIBS} zlib) ENDIF (MYSQL_ZLIB) IF (MYSQL_YASSL) SET(MYSQL_CLIENT_LIBS ${MYSQL_CLIENT_LIBS} yassl) ENDIF (MYSQL_YASSL) IF (MYSQL_TAOCRYPT) SET(MYSQL_CLIENT_LIBS ${MYSQL_CLIENT_LIBS} taocrypt) ENDIF (MYSQL_TAOCRYPT) # Added needed mysqlclient dependencies on Windows IF (WIN32) SET(MYSQL_CLIENT_LIBS ${MYSQL_CLIENT_LIBS} ws2_32) ENDIF (WIN32) MESSAGE(STATUS "MySQL Include dir: ${MYSQL_INCLUDE_DIR} library dir: ${MYSQL_LIB_DIR}") MESSAGE(STATUS "MySQL client libraries: ${MYSQL_CLIENT_LIBS}") ENDIF (MYSQL_INCLUDE_DIR AND MYSQL_LIB_DIR) FindPackageHandleStandardArgs.cmake000066400000000000000000000347421255644354400344510ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/modules#.rst: # FindPackageHandleStandardArgs # ----------------------------- # # # # FIND_PACKAGE_HANDLE_STANDARD_ARGS( ... ) # # This function is intended to be used in FindXXX.cmake modules files. # It handles the REQUIRED, QUIET and version-related arguments to # find_package(). It also sets the _FOUND variable. The # package is considered found if all variables ... listed contain # valid results, e.g. valid filepaths. # # There are two modes of this function. The first argument in both # modes is the name of the Find-module where it is called (in original # casing). # # The first simple mode looks like this: # # :: # # FIND_PACKAGE_HANDLE_STANDARD_ARGS( # (DEFAULT_MSG|"Custom failure message") ... ) # # If the variables to are all valid, then # _FOUND will be set to TRUE. If DEFAULT_MSG is given # as second argument, then the function will generate itself useful # success and error messages. You can also supply a custom error # message for the failure case. This is not recommended. # # The second mode is more powerful and also supports version checking: # # :: # # FIND_PACKAGE_HANDLE_STANDARD_ARGS( # [FOUND_VAR ] # [REQUIRED_VARS ...] # [VERSION_VAR ] # [HANDLE_COMPONENTS] # [CONFIG_MODE] # [FAIL_MESSAGE "Custom failure message"] ) # # In this mode, the name of the result-variable can be set either to # either _FOUND or _FOUND using the # FOUND_VAR option. Other names for the result-variable are not # allowed. So for a Find-module named FindFooBar.cmake, the two # possible names are FooBar_FOUND and FOOBAR_FOUND. It is recommended # to use the original case version. If the FOUND_VAR option is not # used, the default is _FOUND. # # As in the simple mode, if through are all valid, # _FOUND will be set to TRUE. After REQUIRED_VARS the # variables which are required for this package are listed. Following # VERSION_VAR the name of the variable can be specified which holds the # version of the package which has been found. If this is done, this # version will be checked against the (potentially) specified required # version used in the find_package() call. The EXACT keyword is also # handled. The default messages include information about the required # version and the version which has been actually found, both if the # version is ok or not. If the package supports components, use the # HANDLE_COMPONENTS option to enable handling them. In this case, # find_package_handle_standard_args() will report which components have # been found and which are missing, and the _FOUND variable # will be set to FALSE if any of the required components (i.e. not the # ones listed after OPTIONAL_COMPONENTS) are missing. Use the option # CONFIG_MODE if your FindXXX.cmake module is a wrapper for a # find_package(... NO_MODULE) call. In this case VERSION_VAR will be # set to _VERSION and the macro will automatically check whether # the Config module was found. Via FAIL_MESSAGE a custom failure # message can be specified, if this is not used, the default message # will be displayed. # # Example for mode 1: # # :: # # find_package_handle_standard_args(LibXml2 DEFAULT_MSG # LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR) # # # # LibXml2 is considered to be found, if both LIBXML2_LIBRARY and # LIBXML2_INCLUDE_DIR are valid. Then also LIBXML2_FOUND is set to # TRUE. If it is not found and REQUIRED was used, it fails with # FATAL_ERROR, independent whether QUIET was used or not. If it is # found, success will be reported, including the content of . On # repeated Cmake runs, the same message won't be printed again. # # Example for mode 2: # # :: # # find_package_handle_standard_args(LibXslt # FOUND_VAR LibXslt_FOUND # REQUIRED_VARS LibXslt_LIBRARIES LibXslt_INCLUDE_DIRS # VERSION_VAR LibXslt_VERSION_STRING) # # In this case, LibXslt is considered to be found if the variable(s) # listed after REQUIRED_VAR are all valid, i.e. LibXslt_LIBRARIES and # LibXslt_INCLUDE_DIRS in this case. The result will then be stored in # LibXslt_FOUND . Also the version of LibXslt will be checked by using # the version contained in LibXslt_VERSION_STRING. Since no # FAIL_MESSAGE is given, the default messages will be printed. # # Another example for mode 2: # # :: # # find_package(Automoc4 QUIET NO_MODULE HINTS /opt/automoc4) # find_package_handle_standard_args(Automoc4 CONFIG_MODE) # # In this case, FindAutmoc4.cmake wraps a call to find_package(Automoc4 # NO_MODULE) and adds an additional search directory for automoc4. Here # the result will be stored in AUTOMOC4_FOUND. The following # FIND_PACKAGE_HANDLE_STANDARD_ARGS() call produces a proper # success/error message. #============================================================================= # Copyright 2007-2009 Kitware, Inc. # # Distributed under the OSI-approved BSD License (the "License"); # see accompanying file Copyright.txt for details. # # This software is distributed WITHOUT ANY WARRANTY; without even the # implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # See the License for more information. #============================================================================= # (To distribute this file outside of CMake, substitute the full # License text for the above reference.) include(${CMAKE_CURRENT_LIST_DIR}/FindPackageMessage.cmake) include(${CMAKE_CURRENT_LIST_DIR}/CMakeParseArguments.cmake) # internal helper macro macro(_FPHSA_FAILURE_MESSAGE _msg) if (${_NAME}_FIND_REQUIRED) message(FATAL_ERROR "${_msg}") else () if (NOT ${_NAME}_FIND_QUIETLY) message(STATUS "${_msg}") endif () endif () endmacro() # internal helper macro to generate the failure message when used in CONFIG_MODE: macro(_FPHSA_HANDLE_FAILURE_CONFIG_MODE) # _CONFIG is set, but FOUND is false, this means that some other of the REQUIRED_VARS was not found: if(${_NAME}_CONFIG) _FPHSA_FAILURE_MESSAGE("${FPHSA_FAIL_MESSAGE}: missing: ${MISSING_VARS} (found ${${_NAME}_CONFIG} ${VERSION_MSG})") else() # If _CONSIDERED_CONFIGS is set, the config-file has been found, but no suitable version. # List them all in the error message: if(${_NAME}_CONSIDERED_CONFIGS) set(configsText "") list(LENGTH ${_NAME}_CONSIDERED_CONFIGS configsCount) math(EXPR configsCount "${configsCount} - 1") foreach(currentConfigIndex RANGE ${configsCount}) list(GET ${_NAME}_CONSIDERED_CONFIGS ${currentConfigIndex} filename) list(GET ${_NAME}_CONSIDERED_VERSIONS ${currentConfigIndex} version) set(configsText "${configsText} ${filename} (version ${version})\n") endforeach() if (${_NAME}_NOT_FOUND_MESSAGE) set(configsText "${configsText} Reason given by package: ${${_NAME}_NOT_FOUND_MESSAGE}\n") endif() _FPHSA_FAILURE_MESSAGE("${FPHSA_FAIL_MESSAGE} ${VERSION_MSG}, checked the following files:\n${configsText}") else() # Simple case: No Config-file was found at all: _FPHSA_FAILURE_MESSAGE("${FPHSA_FAIL_MESSAGE}: found neither ${_NAME}Config.cmake nor ${_NAME_LOWER}-config.cmake ${VERSION_MSG}") endif() endif() endmacro() function(FIND_PACKAGE_HANDLE_STANDARD_ARGS _NAME _FIRST_ARG) # set up the arguments for CMAKE_PARSE_ARGUMENTS and check whether we are in # new extended or in the "old" mode: set(options CONFIG_MODE HANDLE_COMPONENTS) set(oneValueArgs FAIL_MESSAGE VERSION_VAR FOUND_VAR) set(multiValueArgs REQUIRED_VARS) set(_KEYWORDS_FOR_EXTENDED_MODE ${options} ${oneValueArgs} ${multiValueArgs} ) list(FIND _KEYWORDS_FOR_EXTENDED_MODE "${_FIRST_ARG}" INDEX) if(${INDEX} EQUAL -1) set(FPHSA_FAIL_MESSAGE ${_FIRST_ARG}) set(FPHSA_REQUIRED_VARS ${ARGN}) set(FPHSA_VERSION_VAR) else() CMAKE_PARSE_ARGUMENTS(FPHSA "${options}" "${oneValueArgs}" "${multiValueArgs}" ${_FIRST_ARG} ${ARGN}) if(FPHSA_UNPARSED_ARGUMENTS) message(FATAL_ERROR "Unknown keywords given to FIND_PACKAGE_HANDLE_STANDARD_ARGS(): \"${FPHSA_UNPARSED_ARGUMENTS}\"") endif() if(NOT FPHSA_FAIL_MESSAGE) set(FPHSA_FAIL_MESSAGE "DEFAULT_MSG") endif() endif() # now that we collected all arguments, process them if("x${FPHSA_FAIL_MESSAGE}" STREQUAL "xDEFAULT_MSG") set(FPHSA_FAIL_MESSAGE "Could NOT find ${_NAME}") endif() # In config-mode, we rely on the variable _CONFIG, which is set by find_package() # when it successfully found the config-file, including version checking: if(FPHSA_CONFIG_MODE) list(INSERT FPHSA_REQUIRED_VARS 0 ${_NAME}_CONFIG) list(REMOVE_DUPLICATES FPHSA_REQUIRED_VARS) set(FPHSA_VERSION_VAR ${_NAME}_VERSION) endif() if(NOT FPHSA_REQUIRED_VARS) message(FATAL_ERROR "No REQUIRED_VARS specified for FIND_PACKAGE_HANDLE_STANDARD_ARGS()") endif() list(GET FPHSA_REQUIRED_VARS 0 _FIRST_REQUIRED_VAR) string(TOUPPER ${_NAME} _NAME_UPPER) string(TOLOWER ${_NAME} _NAME_LOWER) if(FPHSA_FOUND_VAR) if(FPHSA_FOUND_VAR MATCHES "^${_NAME}_FOUND$" OR FPHSA_FOUND_VAR MATCHES "^${_NAME_UPPER}_FOUND$") set(_FOUND_VAR ${FPHSA_FOUND_VAR}) else() message(FATAL_ERROR "The argument for FOUND_VAR is \"${FPHSA_FOUND_VAR}\", but only \"${_NAME}_FOUND\" and \"${_NAME_UPPER}_FOUND\" are valid names.") endif() else() set(_FOUND_VAR ${_NAME_UPPER}_FOUND) endif() # collect all variables which were not found, so they can be printed, so the # user knows better what went wrong (#6375) set(MISSING_VARS "") set(DETAILS "") # check if all passed variables are valid unset(${_FOUND_VAR}) foreach(_CURRENT_VAR ${FPHSA_REQUIRED_VARS}) if(NOT ${_CURRENT_VAR}) set(${_FOUND_VAR} FALSE) set(MISSING_VARS "${MISSING_VARS} ${_CURRENT_VAR}") else() set(DETAILS "${DETAILS}[${${_CURRENT_VAR}}]") endif() endforeach() if(NOT "${${_FOUND_VAR}}" STREQUAL "FALSE") set(${_FOUND_VAR} TRUE) endif() # component handling unset(FOUND_COMPONENTS_MSG) unset(MISSING_COMPONENTS_MSG) if(FPHSA_HANDLE_COMPONENTS) foreach(comp ${${_NAME}_FIND_COMPONENTS}) if(${_NAME}_${comp}_FOUND) if(NOT DEFINED FOUND_COMPONENTS_MSG) set(FOUND_COMPONENTS_MSG "found components: ") endif() set(FOUND_COMPONENTS_MSG "${FOUND_COMPONENTS_MSG} ${comp}") else() if(NOT DEFINED MISSING_COMPONENTS_MSG) set(MISSING_COMPONENTS_MSG "missing components: ") endif() set(MISSING_COMPONENTS_MSG "${MISSING_COMPONENTS_MSG} ${comp}") if(${_NAME}_FIND_REQUIRED_${comp}) set(${_FOUND_VAR} FALSE) set(MISSING_VARS "${MISSING_VARS} ${comp}") endif() endif() endforeach() set(COMPONENT_MSG "${FOUND_COMPONENTS_MSG} ${MISSING_COMPONENTS_MSG}") set(DETAILS "${DETAILS}[c${COMPONENT_MSG}]") endif() # version handling: set(VERSION_MSG "") set(VERSION_OK TRUE) set(VERSION ${${FPHSA_VERSION_VAR}}) # check with DEFINED here as the requested or found version may be "0" if (DEFINED ${_NAME}_FIND_VERSION) if(DEFINED ${FPHSA_VERSION_VAR}) if(${_NAME}_FIND_VERSION_EXACT) # exact version required # count the dots in the version string string(REGEX REPLACE "[^.]" "" _VERSION_DOTS "${VERSION}") # add one dot because there is one dot more than there are components string(LENGTH "${_VERSION_DOTS}." _VERSION_DOTS) if (_VERSION_DOTS GREATER ${_NAME}_FIND_VERSION_COUNT) # Because of the C++ implementation of find_package() ${_NAME}_FIND_VERSION_COUNT # is at most 4 here. Therefore a simple lookup table is used. if (${_NAME}_FIND_VERSION_COUNT EQUAL 1) set(_VERSION_REGEX "[^.]*") elseif (${_NAME}_FIND_VERSION_COUNT EQUAL 2) set(_VERSION_REGEX "[^.]*\\.[^.]*") elseif (${_NAME}_FIND_VERSION_COUNT EQUAL 3) set(_VERSION_REGEX "[^.]*\\.[^.]*\\.[^.]*") else () set(_VERSION_REGEX "[^.]*\\.[^.]*\\.[^.]*\\.[^.]*") endif () string(REGEX REPLACE "^(${_VERSION_REGEX})\\..*" "\\1" _VERSION_HEAD "${VERSION}") unset(_VERSION_REGEX) if (NOT ${_NAME}_FIND_VERSION VERSION_EQUAL _VERSION_HEAD) set(VERSION_MSG "Found unsuitable version \"${VERSION}\", but required is exact version \"${${_NAME}_FIND_VERSION}\"") set(VERSION_OK FALSE) else () set(VERSION_MSG "(found suitable exact version \"${VERSION}\")") endif () unset(_VERSION_HEAD) else () if (NOT ${_NAME}_FIND_VERSION VERSION_EQUAL VERSION) set(VERSION_MSG "Found unsuitable version \"${VERSION}\", but required is exact version \"${${_NAME}_FIND_VERSION}\"") set(VERSION_OK FALSE) else () set(VERSION_MSG "(found suitable exact version \"${VERSION}\")") endif () endif () unset(_VERSION_DOTS) else() # minimum version specified: if (${_NAME}_FIND_VERSION VERSION_GREATER VERSION) set(VERSION_MSG "Found unsuitable version \"${VERSION}\", but required is at least \"${${_NAME}_FIND_VERSION}\"") set(VERSION_OK FALSE) else () set(VERSION_MSG "(found suitable version \"${VERSION}\", minimum required is \"${${_NAME}_FIND_VERSION}\")") endif () endif() else() # if the package was not found, but a version was given, add that to the output: if(${_NAME}_FIND_VERSION_EXACT) set(VERSION_MSG "(Required is exact version \"${${_NAME}_FIND_VERSION}\")") else() set(VERSION_MSG "(Required is at least version \"${${_NAME}_FIND_VERSION}\")") endif() endif() else () if(VERSION) set(VERSION_MSG "(found version \"${VERSION}\")") endif() endif () if(VERSION_OK) set(DETAILS "${DETAILS}[v${VERSION}(${${_NAME}_FIND_VERSION})]") else() set(${_FOUND_VAR} FALSE) endif() # print the result: if (${_FOUND_VAR}) FIND_PACKAGE_MESSAGE(${_NAME} "Found ${_NAME}: ${${_FIRST_REQUIRED_VAR}} ${VERSION_MSG} ${COMPONENT_MSG}" "${DETAILS}") else () if(FPHSA_CONFIG_MODE) _FPHSA_HANDLE_FAILURE_CONFIG_MODE() else() if(NOT VERSION_OK) _FPHSA_FAILURE_MESSAGE("${FPHSA_FAIL_MESSAGE}: ${VERSION_MSG} (found ${${_FIRST_REQUIRED_VAR}})") else() _FPHSA_FAILURE_MESSAGE("${FPHSA_FAIL_MESSAGE} (missing: ${MISSING_VARS}) ${VERSION_MSG}") endif() endif() endif () set(${_FOUND_VAR} ${${_FOUND_VAR}} PARENT_SCOPE) endfunction() fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/modules/FindPackageMessage.cmake000066400000000000000000000037551255644354400324230ustar00rootroot00000000000000#.rst: # FindPackageMessage # ------------------ # # # # FIND_PACKAGE_MESSAGE( "message for user" "find result details") # # This macro is intended to be used in FindXXX.cmake modules files. It # will print a message once for each unique find result. This is useful # for telling the user where a package was found. The first argument # specifies the name (XXX) of the package. The second argument # specifies the message to display. The third argument lists details # about the find result so that if they change the message will be # displayed again. The macro also obeys the QUIET argument to the # find_package command. # # Example: # # :: # # if(X11_FOUND) # FIND_PACKAGE_MESSAGE(X11 "Found X11: ${X11_X11_LIB}" # "[${X11_X11_LIB}][${X11_INCLUDE_DIR}]") # else() # ... # endif() #============================================================================= # Copyright 2008-2009 Kitware, Inc. # # Distributed under the OSI-approved BSD License (the "License"); # see accompanying file Copyright.txt for details. # # This software is distributed WITHOUT ANY WARRANTY; without even the # implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # See the License for more information. #============================================================================= # (To distribute this file outside of CMake, substitute the full # License text for the above reference.) function(FIND_PACKAGE_MESSAGE pkg msg details) # Avoid printing a message repeatedly for the same find result. if(NOT ${pkg}_FIND_QUIETLY) string(REPLACE "\n" "" details "${details}") set(DETAILS_VAR FIND_PACKAGE_MESSAGE_DETAILS_${pkg}) if(NOT "${details}" STREQUAL "${${DETAILS_VAR}}") # The message has not yet been printed. message(STATUS "${msg}") # Save the find details in the cache to avoid printing the same # message again. set("${DETAILS_VAR}" "${details}" CACHE INTERNAL "Details about finding ${pkg}") endif() endif() endfunction() fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/cmake/modules/FindPostgreSQL.cmake000066400000000000000000000172631255644354400315650ustar00rootroot00000000000000#.rst: # FindPostgreSQL # -------------- # # Find the PostgreSQL installation. # # In Windows, we make the assumption that, if the PostgreSQL files are # installed, the default directory will be C:\Program Files\PostgreSQL. # # This module defines # # :: # # PostgreSQL_LIBRARIES - the PostgreSQL libraries needed for linking # PostgreSQL_INCLUDE_DIRS - the directories of the PostgreSQL headers # PostgreSQL_VERSION_STRING - the version of PostgreSQL found (since CMake 2.8.8) #============================================================================= # Copyright 2004-2009 Kitware, Inc. # # Distributed under the OSI-approved BSD License (the "License"); # see accompanying file Copyright.txt for details. # # This software is distributed WITHOUT ANY WARRANTY; without even the # implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # See the License for more information. #============================================================================= # (To distribute this file outside of CMake, substitute the full # License text for the above reference.) # ---------------------------------------------------------------------------- # History: # This module is derived from the module originally found in the VTK source tree. # # ---------------------------------------------------------------------------- # Note: # PostgreSQL_ADDITIONAL_VERSIONS is a variable that can be used to set the # version mumber of the implementation of PostgreSQL. # In Windows the default installation of PostgreSQL uses that as part of the path. # E.g C:\Program Files\PostgreSQL\8.4. # Currently, the following version numbers are known to this module: # "9.1" "9.0" "8.4" "8.3" "8.2" "8.1" "8.0" # # To use this variable just do something like this: # set(PostgreSQL_ADDITIONAL_VERSIONS "9.2" "8.4.4") # before calling find_package(PostgreSQL) in your CMakeLists.txt file. # This will mean that the versions you set here will be found first in the order # specified before the default ones are searched. # # ---------------------------------------------------------------------------- # You may need to manually set: # PostgreSQL_INCLUDE_DIR - the path to where the PostgreSQL include files are. # PostgreSQL_LIBRARY_DIR - The path to where the PostgreSQL library files are. # If FindPostgreSQL.cmake cannot find the include files or the library files. # # ---------------------------------------------------------------------------- # The following variables are set if PostgreSQL is found: # PostgreSQL_FOUND - Set to true when PostgreSQL is found. # PostgreSQL_INCLUDE_DIRS - Include directories for PostgreSQL # PostgreSQL_LIBRARY_DIRS - Link directories for PostgreSQL libraries # PostgreSQL_LIBRARIES - The PostgreSQL libraries. # # ---------------------------------------------------------------------------- # If you have installed PostgreSQL in a non-standard location. # (Please note that in the following comments, it is assumed that # points to the root directory of the include directory of PostgreSQL.) # Then you have three options. # 1) After CMake runs, set PostgreSQL_INCLUDE_DIR to /include and # PostgreSQL_LIBRARY_DIR to wherever the library pq (or libpq in windows) is # 2) Use CMAKE_INCLUDE_PATH to set a path to /PostgreSQL<-version>. This will allow find_path() # to locate PostgreSQL_INCLUDE_DIR by utilizing the PATH_SUFFIXES option. e.g. In your CMakeLists.txt file # set(CMAKE_INCLUDE_PATH ${CMAKE_INCLUDE_PATH} "/include") # 3) Set an environment variable called ${PostgreSQL_ROOT} that points to the root of where you have # installed PostgreSQL, e.g. . # # ---------------------------------------------------------------------------- set(PostgreSQL_INCLUDE_PATH_DESCRIPTION "top-level directory containing the PostgreSQL include directories. E.g /usr/local/include/PostgreSQL/8.4 or C:/Program Files/PostgreSQL/8.4/include") set(PostgreSQL_INCLUDE_DIR_MESSAGE "Set the PostgreSQL_INCLUDE_DIR cmake cache entry to the ${PostgreSQL_INCLUDE_PATH_DESCRIPTION}") set(PostgreSQL_LIBRARY_PATH_DESCRIPTION "top-level directory containing the PostgreSQL libraries.") set(PostgreSQL_LIBRARY_DIR_MESSAGE "Set the PostgreSQL_LIBRARY_DIR cmake cache entry to the ${PostgreSQL_LIBRARY_PATH_DESCRIPTION}") set(PostgreSQL_ROOT_DIR_MESSAGE "Set the PostgreSQL_ROOT system variable to where PostgreSQL is found on the machine E.g C:/Program Files/PostgreSQL/8.4") set(PostgreSQL_KNOWN_VERSIONS ${PostgreSQL_ADDITIONAL_VERSIONS} "9.1" "9.0" "8.4" "8.3" "8.2" "8.1" "8.0") # Define additional search paths for root directories. if ( WIN32 ) foreach (suffix ${PostgreSQL_KNOWN_VERSIONS} ) set(PostgreSQL_ADDITIONAL_SEARCH_PATHS ${PostgreSQL_ADDITIONAL_SEARCH_PATHS} "C:/Program Files/PostgreSQL/${suffix}" ) endforeach() endif() set( PostgreSQL_ROOT_DIRECTORIES ENV PostgreSQL_ROOT ${PostgreSQL_ROOT} ${PostgreSQL_ADDITIONAL_SEARCH_PATHS} ) # # Look for an installation. # find_path(PostgreSQL_INCLUDE_DIR NAMES libpq-fe.h PATHS # Look in other places. ${PostgreSQL_ROOT_DIRECTORIES} PATH_SUFFIXES pgsql postgresql include # Help the user find it if we cannot. DOC "The ${PostgreSQL_INCLUDE_DIR_MESSAGE}" ) find_path(PostgreSQL_TYPE_INCLUDE_DIR NAMES catalog/pg_type.h PATHS # Look in other places. ${PostgreSQL_ROOT_DIRECTORIES} PATH_SUFFIXES postgresql pgsql/server postgresql/server include/server # Help the user find it if we cannot. DOC "The ${PostgreSQL_INCLUDE_DIR_MESSAGE}" ) # The PostgreSQL library. set (PostgreSQL_LIBRARY_TO_FIND pq) # Setting some more prefixes for the library set (PostgreSQL_LIB_PREFIX "") if ( WIN32 ) set (PostgreSQL_LIB_PREFIX ${PostgreSQL_LIB_PREFIX} "lib") set ( PostgreSQL_LIBRARY_TO_FIND ${PostgreSQL_LIB_PREFIX}${PostgreSQL_LIBRARY_TO_FIND}) endif() find_library( PostgreSQL_LIBRARY NAMES ${PostgreSQL_LIBRARY_TO_FIND} PATHS ${PostgreSQL_ROOT_DIRECTORIES} PATH_SUFFIXES lib ) get_filename_component(PostgreSQL_LIBRARY_DIR ${PostgreSQL_LIBRARY} PATH) if (PostgreSQL_INCLUDE_DIR) # Some platforms include multiple pg_config.hs for multi-lib configurations # This is a temporary workaround. A better solution would be to compile # a dummy c file and extract the value of the symbol. file(GLOB _PG_CONFIG_HEADERS "${PostgreSQL_INCLUDE_DIR}/pg_config*.h") foreach(_PG_CONFIG_HEADER ${_PG_CONFIG_HEADERS}) if(EXISTS "${_PG_CONFIG_HEADER}") file(STRINGS "${_PG_CONFIG_HEADER}" pgsql_version_str REGEX "^#define[\t ]+PG_VERSION[\t ]+\".*\"") if(pgsql_version_str) string(REGEX REPLACE "^#define[\t ]+PG_VERSION[\t ]+\"([^\"]*)\".*" "\\1" PostgreSQL_VERSION_STRING "${pgsql_version_str}") break() endif() endif() endforeach() unset(pgsql_version_str) endif() # Did we find anything? include(${CMAKE_CURRENT_LIST_DIR}/FindPackageHandleStandardArgs.cmake) find_package_handle_standard_args(PostgreSQL REQUIRED_VARS PostgreSQL_LIBRARY PostgreSQL_INCLUDE_DIR PostgreSQL_TYPE_INCLUDE_DIR VERSION_VAR PostgreSQL_VERSION_STRING) set( PostgreSQL_FOUND ${POSTGRESQL_FOUND}) # Now try to get the include and library path. if(PostgreSQL_FOUND) set(PostgreSQL_INCLUDE_DIRS ${PostgreSQL_INCLUDE_DIR} ${PostgreSQL_TYPE_INCLUDE_DIR} ) set(PostgreSQL_LIBRARY_DIRS ${PostgreSQL_LIBRARY_DIR} ) set(PostgreSQL_LIBRARIES ${PostgreSQL_LIBRARY_TO_FIND}) #message("Final PostgreSQL include dir: ${PostgreSQL_INCLUDE_DIRS}") #message("Final PostgreSQL library dir: ${PostgreSQL_LIBRARY_DIRS}") #message("Final PostgreSQL libraries: ${PostgreSQL_LIBRARIES}") endif() mark_as_advanced(PostgreSQL_INCLUDE_DIR PostgreSQL_TYPE_INCLUDE_DIR PostgreSQL_LIBRARY )fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/tests/000077500000000000000000000000001255644354400243605ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/build/tests/.gitignore000066400000000000000000000000041255644354400263420ustar00rootroot00000000000000tmp fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/000077500000000000000000000000001255644354400227065ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/cli.cc000066400000000000000000000023431255644354400237660ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include #include #include using namespace fnordmetric; static const char kCrashErrorMsg[] = "FnordMetric crashed :( -- Please report a bug at " "github.com/paulasmuth/fnordmetric"; int main(int argc, const char** argv) { /* setup environment */ fnord::util::CatchAndAbortExceptionHandler ehandler(kCrashErrorMsg); ehandler.installGlobalHandlers(); util::SignalHandler::ignoreSIGHUP(); util::SignalHandler::ignoreSIGPIPE(); fnord::util::Random::init(); /* execute commandline */ cli::CLI::parseArgs(env(), argc, argv); cli::CLI::executeSafely(env()); return 0; } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/cli/000077500000000000000000000000001255644354400234555ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/cli/cli.cc000066400000000000000000000125541255644354400245420ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include #include #include #include #include #include namespace fnordmetric { namespace cli { void CLI::parseArgs(Environment* env, const std::vector& argv) { auto flags = env->flags(); // Execute queries from the commandline: flags->defineFlag( "format", FlagParser::T_STRING, false, "f", "table", "The output format { svg, csv, table }", ""); flags->defineFlag( "output", FlagParser::T_STRING, false, "o", NULL, "Write output to a file", ""); flags->defineFlag( "verbose", FlagParser::T_SWITCH, false, NULL, NULL, "Be verbose"); flags->defineFlag( "help", FlagParser::T_SWITCH, false, "h", NULL, "You are reading it..."); flags->parseArgv(argv); env->setVerbose(flags->isSet("verbose")); } void CLI::parseArgs(Environment* env, int argc, const char** argv) { std::vector args; for (int i = 1; i < argc; ++i) { args.emplace_back(argv[i]); } parseArgs(env, args); } void CLI::printUsage() { auto err_stream = fnordmetric::util::OutputStream::getStderr(); err_stream->printf("usage: fnordmetric-cli [options] [file.sql]\n"); err_stream->printf("\noptions:\n"); env()->flags()->printUsage(err_stream.get()); err_stream->printf("\nexamples:\n"); err_stream->printf(" $ fnordmeric-cli -f svg -o out.svg myquery.sql\n"); err_stream->printf(" $ fnordmeric-cli -f svg - < myquery.sql > out.svg\n"); } int CLI::executeSafely(Environment* env) { auto err_stream = util::OutputStream::getStderr(); try { if (env->flags()->isSet("help")) { printUsage(); return 0; } execute(env); } catch (const util::RuntimeException& e) { if (e.getTypeName() == "UsageError") { err_stream->printf("\n"); printUsage(); return 1; } if (env->verbose()) { env->logger()->exception("FATAL", "Fatal error", e); } else { auto msg = e.getMessage(); err_stream->printf("[ERROR] "); err_stream->write(msg.c_str(), msg.size()); err_stream->printf("\n"); } return 1; } return 0; } void CLI::execute(Environment* env) { auto flags = env->flags(); const auto& args = flags->getArgv(); /* open input stream */ std::shared_ptr input; if (args.size() == 1) { if (args[0] == "-") { if (env->verbose()) { env->logger()->log("DEBUG", "Input from stdin"); } input = std::move(util::InputStream::getStdin()); } else { if (env->verbose()) { env->logger()->log("DEBUG", "Input file: " + args[0]); } input = std::move(util::FileInputStream::openFile(args[0])); } } else { RAISE("UsageError", "usage error"); } /* open output stream */ std::shared_ptr output; if (flags->isSet("output")) { auto output_file = flags->getString("output"); if (env->verbose()) { env->logger()->log("DEBUG", "Output file: " + output_file); } output = std::move(util::FileOutputStream::openFile(output_file)); } else { if (env->verbose()) { env->logger()->log("DEBUG", "Output to stdout"); } output = std::move(util::OutputStream::getStdout()); } /* execute query */ query::QueryService query_service; query_service.registerBackend( std::unique_ptr( new fnordmetric::query::mysql_backend::MySQLBackend)); query_service.registerBackend( std::unique_ptr( new fnordmetric::query::postgres_backend::PostgresBackend)); query_service.registerBackend( std::unique_ptr( new fnordmetric::query::csv_backend::CSVBackend)); query_service.executeQuery( input, getOutputFormat(env), output); } const query::QueryService::kFormat CLI::getOutputFormat(Environment* env) { auto flags = env->flags(); if (!flags->isSet("format")) { return query::QueryService::FORMAT_TABLE; } auto format_str = flags->getString("format"); if (format_str == "csv") { return query::QueryService::FORMAT_CSV; } if (format_str == "json") { return query::QueryService::FORMAT_JSON; } if (format_str == "svg") { return query::QueryService::FORMAT_SVG; } if (format_str == "table") { return query::QueryService::FORMAT_TABLE; } RAISE( kRuntimeError, "invalid format: '%s', allowed formats are " "'svg', 'csv', 'json' and 'table'", format_str.c_str()); } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/cli/cli.h000066400000000000000000000022621255644354400243770ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_CLI_H #define _FNORDMETRIC_CLI_H #include #include #include namespace fnordmetric { class Environment; namespace cli { class FlagParser; class CLI { public: /** * Parse a command line */ static void parseArgs(Environment* env, const std::vector& argv); static void parseArgs(Environment* env, int argc, const char** argv); /** * Execute a command line but do not throw exceptions */ static int executeSafely(Environment* env); /** * Execute a command line */ static void execute(Environment* env); protected: static void printUsage(); static const query::QueryService::kFormat getOutputFormat(Environment* env); }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/cli/cli_test.cc000066400000000000000000000024271255644354400255770ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include #include using namespace fnordmetric::cli; UNIT_TEST(CLITest); static fnordmetric::util::UnitTest::TestCase __test_simple_sql_to_svg_( &CLITest, "TestSimpleSQLToSVG", [] () { fnord::io::FileUtil::mkdir_p("build/tests/tmp"); std::vector args = { "test/fixtures/gdp_bar_chart.sql", "-f", "svg", "-o", "build/tests/tmp/CLITest_TestSimpleSQLToSVG_out.svg.html" }; fnordmetric::Environment myenv; CLI::parseArgs(&myenv, args); CLI::execute(&myenv); EXPECT_FILES_EQ( "test/fixtures/CLITest_TestSimpleSQLToSVG_out.svg.html", "build/tests/tmp/CLITest_TestSimpleSQLToSVG_out.svg.html"); }); fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/cli/flagparser.cc000066400000000000000000000127041255644354400261160ustar00rootroot00000000000000#include #include #include #include #include namespace fnordmetric { namespace cli { FlagParser::FlagParser() {} void FlagParser::defineFlag( const char* longopt, kFlagType type, bool required, const char* shortopt /* = NULL */, const char* default_value /* = NULL */, const char* description /* = NULL */, const char* placeholder /* = NULL */) { FlagState flag_state; flag_state.type = type; flag_state.required = required; flag_state.longopt = longopt; flag_state.shortopt = shortopt; flag_state.default_value = default_value; flag_state.description = description; flag_state.placeholder = placeholder; flags_.emplace_back(flag_state); } bool FlagParser::isSet(const char* longopt) const { for (auto& flag : flags_) { if (flag.longopt == longopt) { return flag.default_value != nullptr || flag.values.size() > 0; } } return false; } std::string FlagParser::getString(const char* longopt) const { for (auto& flag : flags_) { if (flag.longopt == longopt) { if (flag.type != T_STRING) { RAISE(kFlagError, "flag '%s' is not a string", longopt); } if (flag.values.size() == 0) { if (flag.default_value != nullptr) { return flag.default_value; } RAISE(kFlagError, "flag '%s' is not set", longopt); } return flag.values.back(); } } RAISE(kFlagError, "flag '%s' is not set", longopt); } int64_t FlagParser::getInt(const char* longopt) const { for (auto& flag : flags_) { if (flag.longopt == longopt) { if (flag.type != T_INTEGER) { RAISE(kFlagError, "flag '%s' is not an integer", longopt); } std::string flag_value_str; if (flag.values.size() == 0) { if (flag.default_value != nullptr) { flag_value_str = flag.default_value; } else { RAISE(kFlagError, "flag '%s' is not set", longopt); } } else { flag_value_str = flag.values.back(); } int64_t flag_value; try { flag_value = std::stoi(flag_value_str); } catch (std::exception e) { RAISE( kFlagError, "flag '%s' value '%s' is not a valid integer", longopt, flag.values.back().c_str()); } return flag_value; } } RAISE(kFlagError, "flag '%s' is not set", longopt); } void FlagParser::parseArgv(int argc, const char** argv) { std::vector args; for (int i = 1; i < argc; ++i) { args.emplace_back(argv[i]); } parseArgv(args); } // FIXPAUL optimize with hashmap? void FlagParser::parseArgv(const std::vector& argv) { for (int i = 0; i < argv.size(); i++) { int eq_len = -1; FlagState* flag_ptr = nullptr; auto& arg = argv[i]; if (arg.size() == 0) { continue; } for (auto& flag : flags_) { auto longopt = std::string("--") + flag.longopt; auto longopt_eq = std::string("--") + flag.longopt + "="; if (arg.compare(0, longopt.size(), longopt) == 0) { flag_ptr = &flag; } else if (arg.compare(0, longopt_eq.size(), longopt_eq) == 0) { flag_ptr = &flag; eq_len = longopt_eq.size(); } else if (flag.shortopt != nullptr) { auto shortopt = std::string("-") + flag.shortopt; auto shortopt_eq = std::string("-") + flag.shortopt + "="; if (arg.compare(0, shortopt.size(), shortopt) == 0) { flag_ptr = &flag; } else if (arg.compare(0, shortopt_eq.size(), shortopt_eq) == 0) { flag_ptr = &flag; eq_len = shortopt_eq.size(); } } if (flag_ptr != nullptr) { break; } } if (flag_ptr == nullptr) { argv_.push_back(arg); } else if (flag_ptr->type == T_SWITCH) { flag_ptr->values.emplace_back("true"); } else if (eq_len > 0) { if (arg.size() == eq_len) { RAISE(kFlagError, "flag --%s=... has no value", flag_ptr->longopt); } flag_ptr->values.emplace_back(arg.substr(eq_len)); } else { if (i + 1 >= argv.size()) { RAISE(kFlagError, "flag --%s has no value", flag_ptr->longopt); } flag_ptr->values.emplace_back(argv[++i]); } } for (const auto& flag : flags_) { if (flag.required == true && flag.values.size() == 0) { RAISE(kFlagError, "flag --%s is required", flag.longopt); } } } const std::vector& FlagParser::getArgv() const { return argv_; } void FlagParser::printUsage(util::OutputStream* target) const { for (const auto& flag : flags_) { if (flag.shortopt == nullptr) { target->printf(" --%-26.26s", flag.longopt); } else { target->printf(" -%s, --%-12.12s", flag.shortopt, flag.longopt); } const char* placeholder = nullptr; if (flag.placeholder == nullptr) { switch (flag.type) { case T_STRING: placeholder = ""; break; case T_INTEGER: placeholder = ""; break; case T_FLOAT: placeholder = ""; break; case T_SWITCH: placeholder = ""; break; } } else { placeholder = flag.placeholder; } target->printf("%-12.12s", placeholder); if (flag.description != nullptr) { target->printf("%s\n", flag.description); } else { target->printf("\n"); } } } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/cli/flagparser.h000066400000000000000000000045101255644354400257540ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FLAGPARSER_H #define _FNORDMETRIC_FLAGPARSER_H #include #include #include namespace fnordmetric { namespace util { class OutputStream; } namespace cli { const char kFlagError[] = "FlagError"; class FlagParser { public: enum kFlagType { T_SWITCH, T_STRING, T_FLOAT, T_INTEGER }; FlagParser(); /** * Define a flag */ void defineFlag( const char* longopt, kFlagType type, bool required, const char* shortopt = NULL, const char* default_value = NULL, const char* description = NULL, const char* placeholder = NULL); /** * Returns true if the flag is set and false otherwise * * @param longopt the longopt of the flag */ bool isSet(const char* longopt) const; /** * Returns the string value of the flag or throws an exception if the value * is invalid. * * @param longopt the longopt of the flag */ std::string getString(const char* longopt) const; /** * Returns the integer value of the flag or throws an exception if the value * is invalid. * * @param longopt the longopt of the flag */ int64_t getInt(const char* longopt) const; /** * Parse an argv array. This may throw an exception. */ void parseArgv(int argc, const char** argv); /** * Parse an argv array. This may throw an exception. */ void parseArgv(const std::vector& argv); /** * Get the argv array with all flag args removed */ const std::vector& getArgv() const; void printUsage(util::OutputStream* target) const; protected: struct FlagState { kFlagType type; bool required; const char* longopt; const char* shortopt; const char* default_value; const char* description; const char* placeholder; std::vector values; }; std::vector argv_; std::vector flags_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/environment.cc000066400000000000000000000014361255644354400255650ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include using fnord::util::LogOutputStream; namespace fnordmetric { Environment* env() { static Environment global_env; return &global_env; } Environment::Environment() : logger_(new LogOutputStream(util::OutputStream::getStderr())) {} void Environment::setVerbose(bool verbose) { verbose_ = verbose; } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/environment.h000066400000000000000000000017621255644354400254310ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_ENVIRONMENT_H #define _FNORDMETRIC_ENVIRONMENT_H #include #include #include #include "config.h" using fnord::util::Logger; namespace fnordmetric { class Environment { public: Environment(); inline cli::FlagParser* flags() { return &flag_parser_; } inline Logger* logger() { return logger_.get(); } inline bool verbose() { return verbose_; } void setVerbose(bool verbose); protected: bool verbose_; cli::FlagParser flag_parser_; std::unique_ptr logger_; }; Environment* env(); } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/000077500000000000000000000000001255644354400234645ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/agent.cc000066400000000000000000000014611255644354400250730ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include "agent.h" #include "query/query.h" namespace fnordmetric { std::unique_ptr Agent::executeQuery( const std::string& query_string) const { std::unique_ptr query(new query::Query(query_string, *this)); return query; } std::shared_ptr Agent::findStreamByName( const std::string& stream_name) const { //return std::shared_ptr(nullptr); } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/agent.h000066400000000000000000000027211255644354400247350ustar00rootroot00000000000000/** * This file is part of the "FnordStream" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * Licensed under the MIT license (see LICENSE). */ #ifndef _FNORDMETRIC_AGENT_H #define _FNORDMETRIC_AGENT_H #include #include #include #include "stream.h" #include "database/database.h" namespace fnordmetric { namespace query { class QueryTest; } /** * Agent */ class Agent : public IStreamRepository { friend class query::QueryTest; public: explicit Agent( const std::string& name, std::unique_ptr&& database); Agent(const Agent& copy) = delete; Agent& operator=(const Agent& copy) = delete; Agent(Agent&& move); /** * Open a stream for writing data */ template std::shared_ptr> openStream( const std::string& name, //const StreamDescription& description, const T... fields); /** * Execute a query */ std::unique_ptr executeQuery (const std::string& query_string) const; /** * Find a stream by name if a stream with that name exists. Otherwise return * std::shared_ptr(nullptr) */ virtual std::shared_ptr findStreamByName( const std::string& stream_name) const; protected: std::string name_; std::vector> streams_; std::mutex streams_mutex_; std::unique_ptr database_; }; } #include "agent_impl.h" #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/agent_impl.h000066400000000000000000000017041255644354400257560ustar00rootroot00000000000000/** * This file is part of the "FnordStream" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * Licensed under the MIT license (see LICENSE). */ #ifndef _FNORDMETRIC_AGENT_IMPL_H #define _FNORDMETRIC_AGENT_IMPL_H namespace fnordmetric { Agent::Agent( const std::string& name, std::unique_ptr&& database) : name_(name), database_(std::move(database)) {} template std::shared_ptr> Agent::openStream( const std::string& name, //const StreamDescription& description, const T... fields) { TypedSchema schema(fields...); TypedStreamKey key(name, fields...); streams_mutex_.lock(); auto stream_ref = database_->openStream(key.getKeyString()); auto stream = std::make_shared>( key, schema, std::move(stream_ref)); streams_.push_back(stream); streams_mutex_.unlock(); return stream; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/aocollection.cc000066400000000000000000000076251255644354400264600ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include "aocollection.h" #include "pagemanager.h" #include "transaction.h" #include "cursor.h" #include "documentref.h" namespace fnordmetric { //size_t Database::kMaxPageSizeSoft = 4194304; //size_t Database::kMaxPageSizeHard = 33554432; uint64_t AOCollection::kInitialIndexPageSize = 65535; AOCollection::AOCollection( const Schema& schema, std::shared_ptr page_manager, std::unique_ptr seq) : Collection(schema, page_manager), page_index_(std::shared_ptr(new PageIndex(page_manager))), seq_(std::move(seq)), last_key_(0) {} AOCollection::~AOCollection() { sync(); } std::unique_ptr AOCollection::getSnapshot() { auto snapshot = new AOCollection::Snapshot(); return std::unique_ptr(snapshot); } bool AOCollection::commitTransaction(const Transaction* transaction) { commit_mutex_.lock(); auto index = page_index_->clone(); bool rollback = false; for (auto docref : transaction->getDirtyDocuments()) { if (docref->isDirty()) { auto key = docref->getDocumentKey(); if (!key.isIntKey()) { printf("error: unsupported key type\n"); // FIXPAUL goto rollback_tx; } if (key.getIntKey() == 0) { uint64_t next_key = seq_->getNext(last_key_); docref->setDocumentKey(DocumentKey(next_key)); last_key_ = next_key; } else if (key.getIntKey() <= last_key_) { printf("error: keys must be monotonically increasing\n"); // FIXPAUL goto rollback_tx; } appendDocument(docref, index); } } commit_tx: printf("commit!"); page_index_mutex_.lock(); page_index_ = std::shared_ptr(index); page_index_mutex_.unlock(); commit_mutex_.unlock(); return true; rollback_tx: // FIXPAUL if error free the index copy! commit_mutex_.unlock(); return false; } void AOCollection::appendDocument( const DocumentRef* docref, PageIndex* index) { char* data; size_t size; /* get document body */ docref->getScratchpad(&data, &size); size_t envelope_size = size + sizeof(DocHeader); assert(size > 0); /* get document key */ uint64_t key = docref->getDocumentKey().getIntKey(); assert(key > 0); /* retrieve the page for inserting */ auto index_ref = index->getIndexPageEntryForInsert(envelope_size); PageManager::Page page(index_ref->offset, index_ref->size); auto page_ref = page_manager_->getPage(page); /* copy metadata into document header */ DocHeader* document = page_ref->structAt(index_ref->used); document->key = key; document->size = size; /* copy document body and compute checksum*/ memcpy(document->data, data, size); //row->checksum = row->computeChecksum(); if (index_ref->first_key == 0) { index_ref->first_key = key; } index_ref->used += envelope_size; } void AOCollection::sync() { sync_mutex_.lock(); // capture free page list auto index = getPageIndex(); page_manager_->fsync(); //writeRootPage(index_page); // release freed pages sync_mutex_.unlock(); } std::shared_ptr AOCollection::getPageIndex() const { page_index_mutex_.lock(); auto index = page_index_; page_index_mutex_.unlock(); return index; } std::unique_ptr AOCollection::Snapshot::getCursor( const DocumentKey& key) { assert(0); } //Document* AOCollection::Transaction::getDocument( // const fnordmetric::Cursor* cursor) { //} } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/aocollection.h000066400000000000000000000110071255644354400263070ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FILEBACKEND_FILEBACKEND_H #define _FNORDMETRIC_FILEBACKEND_FILEBACKEND_H #include #include #include #include #include #include "pagemanager.h" #include "collection.h" #include "cursor.h" #include "transaction.h" #include "pageindex.h" /** * * eBNF: * * FILE ::= FILE_HEADER *( PAGE ) FILE_HEADER ::= <8 Bytes 0x17> ; magic bytes ; version number PAGE_PTR ; pointer to first log page PAGE ::= ( LOG_PAGE | DATA_PAGE ) PAGE_PTR ::= ; page offset in the file in bytes ; size of the page in bytes LOG_PAGE ::= *( LOG_ENTRY ) LOG_ENTRY ::= ( LOG_ENTRY_NEXT | LOG_ENTRY_ALLOC | LOG_ENTRY_FREE ) LOG_ENTRY_NEXT ::= LOG_ENTRY_CHECKSUM LOG_ENTRY_TYPE ; 0x01 LOG_ENTRY_LENGTH ; 0x08 PAGE_PTR ; pointer to next log page LOG_ENTRY_ALLOC ::= LOG_ENTRY_CHECKSUM LOG_ENTRY_TYPE ; 0x02 LOG_ENTRY_LENGTH ; variable PAGE_PTR ; pointer to the data page UNIX_MILLIS ; timestamp of the page's first row STREAM_ID [ STREAM_KEY ] ; length = LOG_ENTRY_LENGTH - 16 LOG_ENTRY_FREE ::= LOG_ENTRY_CHECKSUM LOG_ENTRY_TYPE ; 0x03 LOG_ENTRY_LENGTH ; 0x10 PAGE_PTR ; pointer to the free'd page PAGE_SIZE ; size of the freed page DATA_PAGE ::= PAGE_HEADER *( STREAM_ROW ) UNIX_MILLIS ::= ; unix millisecond timestamp STREAM_ID ::= ; unique stream id STREAM_KEY ::= ; unique stream key */ namespace fnordmetric { class IDSequence; class AOCollection : public Collection { public: static const uint16_t CollectionTypeId = 0x01; static uint64_t kInitialIndexPageSize; /** * Constructor for a new collection */ AOCollection( const Schema& schema, std::shared_ptr page_manager, std::unique_ptr seq); /** * Constructor for an exisiting collection */ AOCollection( const Schema& schema, PageManager::Page root_page, std::shared_ptr page_manager); AOCollection(const AOCollection& copy) = delete; AOCollection& operator=(const AOCollection& copy) = delete; ~AOCollection() override; /** * Get a snapshot of this collection */ std::unique_ptr getSnapshot() override; /** * Commit a transaction on this collection. Do not call this directly unless * you know what you are doing. Use Transaction#commit instead */ bool commitTransaction(const Transaction* transaction) override; /** * Sync the log to disk. Makes all changes until this point durable and blocks * on what is essentialy an fsync() */ void sync() override; protected: struct __attribute__((__packed__)) DocHeader { uint32_t checksum; uint32_t size; uint64_t key; uint8_t data[]; uint32_t computeChecksum(); }; /** * Refer to the interface documentation in "collection.h" */ class Snapshot : public fnordmetric::Collection::Snapshot { public: std::unique_ptr getCursor(const DocumentKey& key) override; }; /** * Refer to the interface documentation in "cursor.h" */ class Cursor : public fnordmetric::Cursor { public: size_t advanceBy(size_t n) override; }; void appendDocument(const DocumentRef* docref,PageIndex* index); std::shared_ptr getPageIndex() const; uint64_t last_key_; std::shared_ptr page_index_; mutable std::mutex page_index_mutex_; std::mutex commit_mutex_; std::mutex sync_mutex_; std::unique_ptr seq_; }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/btree.cc000066400000000000000000000020641255644354400250760ustar00rootroot00000000000000/** * This file is part of the "FnordStream" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * Licensed under the MIT license (see LICENSE). */ #include #include #include #include #include #include "btree.h" namespace fnordmetric { BTree::BTree(uint16_t branch_factor) : branch_factor_(branch_factor), root_is_leaf_(true) { root_ = allocNode(getLeafNodeSize()); } void BTree::insert(uint64_t key) { if (root_is_leaf_) { if (root_->used < branch_factor_) { return insertIntoNode(root_, key); } else { /* split root node */ assert(0); } } } BTree::LeafNodeEntry* BTree::lookup(uint64_t key) const { if (root_is_leaf_) { return (LeafNodeEntry*) root_->leaf; } } BTree::Node* BTree::allocNode(size_t size) { Node* node = (Node *) malloc(size); // FIXPAUL memset(node, 0, size); return node; } void BTree::insertIntoNode(Node* leaf_node, uint64_t key) { auto slot = leaf_node->leaf + leaf_node->used; slot->key = key; leaf_node->used++; } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/btree.h000066400000000000000000000030651255644354400247420ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_BTREE_H #define _FNORDMETRIC_BTREE_H #include namespace fnordmetric { /** * All methods on a collection instance are threadsafe. */ class BTree { friend class BTreeTest; public: static const uint16_t kBranchFactor = 128; BTree(uint16_t branch_factor); void insert(uint64_t key); protected: struct __attribute__((__packed__)) LeafNodeEntry { uint64_t key; uint64_t offset; uint32_t size; }; struct __attribute__((__packed__)) InternalNodeEntry { uint64_t key; uint64_t offset; uint8_t is_leaf; }; struct __attribute__((__packed__)) Node { uint16_t used; union __attribute__((__packed__)) { LeafNodeEntry leaf[]; InternalNodeEntry internal[]; }; }; inline size_t getLeafNodeSize() { return sizeof(Node) + sizeof(LeafNodeEntry) * branch_factor_; } inline size_t getInternalNodeSize() { return sizeof(Node) + sizeof(InternalNodeEntry) * branch_factor_; } LeafNodeEntry* lookup(uint64_t key) const; Node* allocNode(size_t size); void insertIntoNode(Node* leaf_node, uint64_t key); const uint16_t branch_factor_; bool root_is_leaf_; Node* root_; }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/btree_test.cc000066400000000000000000000022111255644354400261270ustar00rootroot00000000000000/** * This file is part of the "FnordStream" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * Licensed under the MIT license (see LICENSE). */ #include #include #include #include #include "btree.h" namespace fnordmetric { class BTreeTest { public: BTreeTest() {} void run() { testInsertIntoRoot(); testRootPageSplit(); } void testInsertIntoRoot() { uint16_t b = 128; BTree btree(b); auto root_node = btree.root_; assert(root_node != nullptr); for (int i=0; i < b-1; ++i) { btree.insert(i); } assert(btree.root_->used == b - 1); for (int i=0; i < b-1; ++i) { assert(btree.root_->leaf[i].key == i); } assert(btree.root_ == root_node); }; void testRootPageSplit() { uint16_t b = 128; BTree btree(b); auto root_node = btree.root_; assert(root_node != nullptr); for (int i=0; i < (b-1) * 2; ++i) { btree.insert(i); } assert(btree.root_ != root_node); assert(btree.root_->used == 2); }; }; } int main() { fnordmetric::BTreeTest test; test.run(); printf("all tests passed! :)\n"); } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/collection.cc000066400000000000000000000112121255644354400261230ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include "collection.h" #include "pagemanager.h" #include "transaction.h" #include "snapshot.h" namespace fnordmetric { Collection::Collection( std::shared_ptr page_manager, std::unique_ptr seq) : page_manager_(std::move(page_manager)) {} Collection::~Collection() {} std::unique_ptr Collection::startTransaction() { std::unique_ptr snapshot(new Snapshot()); auto transaction = new Transaction(std::move(snapshot)); return std::unique_ptr(transaction); } void Collection::writeRootPage(const PageManager::Page& root_page) { PageManager::Page header_page; header_page.offset = 0; header_page.size = kMinReservedHeaderSize; auto header_mmap = page_manager_->getPage(header_page); auto file_header = header_mmap->structAt(0); file_header->root_page_offset = root_page.offset; file_header->root_page_size = root_page.size; // FIXPAUL msync header } std::shared_ptr Collection::getPageManager() const { return page_manager_; } std::unique_ptr Collection::createPersistentCollection( const std::string& filename, uint64_t flags /* = MODE_CONSERVATIVE */) { Collection* ptr = nullptr; auto open_flags = O_CREAT | O_RDWR; if (flags & FILE_TRUNCATE) { flags |= O_TRUNC; } else { flags |= O_EXCL; } int fd = open(filename.c_str(), open_flags, S_IRUSR | S_IWUSR); if (fd < 0) { perror("create file with open() failed"); return std::unique_ptr(nullptr); } struct stat fd_stat; if (fstat(fd, &fd_stat) < 0) { perror("fstat() failed"); return std::unique_ptr(nullptr); } /* off_t fd_len = lseek(fd, 0, SEEK_END); if (fd_len < 0) { perror("lseek() failed"); return std::unique_ptr(nullptr); } */ if ((flags & FILE_AUTODELETE) > 0) { unlink(filename.c_str()); } std::shared_ptr page_manager( new MmapPageManager(fd, 0, fd_stat.st_blksize)); auto header_page = page_manager->allocPage(kMinReservedHeaderSize); auto header_mmap = page_manager->getPage(header_page); auto file_header = header_mmap->structAt(0); file_header->magic = kFileMagicBytes; file_header->version = kFileVersion; //file_header->collection_type = T::CollectionTypeId; file_header->root_page_offset = 0; file_header->root_page_size = 0; // FIXPAUL msync header std::unique_ptr seq(new UnixMillisecondIDSequence()); return std::unique_ptr(new Collection(page_manager, std::move(seq))); /* open existing file */ /*if (fd_len >= kMinReservedHeaderSize) { PageManager::Page header_page = {.offset=0, .size=kMinReservedHeaderSize}; auto header_mmap = page_manager->getPage(header_page); auto file_header = header_mmap->structAt(0); if (file_header->magic != kFileMagicBytes) { fprintf(stderr, "invalid file\n"); // FIXPAUL return std::unique_ptr(nullptr); } if (file_header->version != kFileVersion) { fprintf(stderr, "invalid file version\n"); // FIXPAUL return std::unique_ptr(nullptr); } PageManager::Page first_log_page; first_log_page.offset = file_header->first_log_page_offset; first_log_page.size = file_header->first_log_page_size; LogSnapshot log_snapshot; LogReader log_reader(page_manager, first_log_page, &log_snapshot); log_reader.import(); std::shared_ptr page_manager_imported(new MmapPageManager( dup(fd), fd_len, fd_stat.st_blksize, log_snapshot)); std::shared_ptr log( new Log(log_snapshot, page_manager_imported)); return std::unique_ptr( new Database(log_snapshot, log, page_manager_imported, flags)); } fprintf(stderr, "invalid file\n"); // FIXPAUL return std::unique_ptr(nullptr); */ } bool Collection::commitTransaction(const Transaction* transaction) { } void Collection::sync() { //sync_mutex_.lock(); // capture free page list //auto index = getPageIndex(); //page_manager_->fsync(); //writeRootPage(index_page); // release freed pages //sync_mutex_.unlock(); } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/collection.h000066400000000000000000000064361255644354400260010ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_COLLECTION_H #define _FNORDMETRIC_COLLECTION_H #include #include #include "pagemanager.h" #include "idsequence.h" namespace fnordmetric { class Transaction; class Cursor; class DocumentKey; /** * All methods on a collection instance are threadsafe. */ class Collection { public: struct __attribute__((__packed__)) FileHeader { uint64_t magic; uint64_t version; uint64_t root_page_offset; uint64_t root_page_size; uint16_t collection_type; }; enum kCollectionFlags { /** * Truncate the file when opening it */ FILE_TRUNCATE = 32, /** * Delete the file when the database is closed (the database object is * destroyed) */ FILE_AUTODELETE = 64 }; /** * File magic bytes */ static const uint64_t kFileMagicBytes = 0x1717171717171717; /** * File format version number */ static const uint64_t kFileVersion = 1; /** * Min. number of bytes to reserve for the file header */ static const uint64_t kMinReservedHeaderSize = 512; /** * Preferred maximum page size (soft limit). Default: 4MB */ static size_t kMaxPageSizeSoft; /** * Maximum allowed page size (hard limit). Note that now single row may be * larger than the maximum page size. Default: 32MB */ static size_t kMaxPageSizeHard; /** * Create a file backed collection */ static std::unique_ptr createPersistentCollection( const std::string& filename, uint64_t flags); /** * Open a file backed collection */ static std::unique_ptr openPersistentCollection( const std::string& filename, uint64_t flags); Collection(const Collection& copy) = delete; Collection& operator=(const Collection& copy) = delete; virtual ~Collection(); /** * Start a new transaction on this collection */ std::unique_ptr startTransaction(); /** * Sync the log to disk. Makes all changes until this point durable and blocks * on what is essentialy an fsync() */ virtual void sync(); /** * Get the internal page manager for this collection. Do not call unless you * know what you are doing */ std::shared_ptr getPageManager() const; /** * Commit a transaction on this collection. Do not call this directly unless * you know what you are doing. Use Transaction#commit instead */ virtual bool commitTransaction(const Transaction* transaction); protected: /** * Constructor for a new collection */ explicit Collection( std::shared_ptr page_manager, std::unique_ptr id_sequence); /** * Constructor for an exisiting collection */ explicit Collection( PageManager::Page root_page, std::shared_ptr page_manager); void writeRootPage(const PageManager::Page& root_page); const std::shared_ptr page_manager_; }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/cowblob.cc000066400000000000000000000010171255644354400254210ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include "cowblob.h" namespace fnordmetric { namespace ffs { COWBlob::COWBlob(std::shared_ptr page_manager) { } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/cowblob.h000066400000000000000000000012441255644354400252650ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FFS_COWBLOB_H #define _FNORDMETRIC_FFS_COWBLOB_H #include "object.h" #include namespace fnordmetric { namespace ffs { class COWBlob : public Object { public: COWBlob(std::shared_ptr page_manager); }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/cursor.cc000066400000000000000000000141771255644354400253220ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include "cursor.h" #include "streamref.h" #include "database.h" namespace fnordmetric { namespace database { Cursor::Cursor( StreamRef* stream_ref, std::shared_ptr page_manager) : stream_ref_(stream_ref), page_manager_(page_manager), current_page_index_(0), current_page_offset_(0), current_page_(nullptr), current_page_ref_(nullptr) {} StreamPosition Cursor::seekToTime(uint64_t unix_millis) { /* try to seek to the page containing that timestamp */ stream_ref_->accessPages([this, &unix_millis] (const std::vector>& stream_pages) { // FIXPAUL do a binary search! for (int i = stream_pages.size() - 1; i >= 0; i--) { if (stream_pages[i]->time_ <= unix_millis) { this->current_page_ = stream_pages.at(i); this->current_page_offset_ = 0; this->current_page_index_ = 0; break; } } }); if (current_page_.get() == nullptr) { StreamPosition pos; pos.unix_millis = 0; pos.logical_offset = 0; pos.next_offset = 0; return pos; } current_page_ref_ = page_manager_->getPage(current_page_->page_); /* seek to target time */ while (next()) { if (getCurrentRow()->time >= unix_millis) { break; } } return getCurrentPosition(); } StreamPosition Cursor::seekToLogicalOffset(uint64_t logical_offset) { /* try to seek to the page containing that logical_offset */ stream_ref_->accessPages([this, &logical_offset] (const std::vector>& stream_pages) { // FIXPAUL do a binary search! for (int i = stream_pages.size() - 1; i >= 0; i--) { if (stream_pages[i]->logical_offset_ <= logical_offset) { this->current_page_ = stream_pages.at(i); this->current_page_offset_ = 0; this->current_page_index_ = 0; break; } } }); if (current_page_.get() == nullptr) { StreamPosition pos; pos.unix_millis = 0; pos.logical_offset = 0; pos.next_offset = 0; return pos; } current_page_ref_ = page_manager_->getPage(current_page_->page_); /* fast path for exact offset match */ current_page_offset_ = logical_offset - current_page_->logical_offset_; if (current_page_offset_ < current_page_->used_.load()) { auto row = current_page_ref_->structAt(current_page_offset_); auto row_end = current_page_offset_ + row->size + sizeof(RowHeader); if (row_end <= current_page_->used_.load()) { if (row->computeChecksum() == row->checksum) { return getCurrentPosition(); } } } /* seek to target offset fallback */ current_page_offset_ = 0; while (next()) { auto cur_loff = current_page_->logical_offset_ + current_page_offset_; if (cur_loff >= logical_offset) { break; } } return getCurrentPosition(); } StreamPosition Cursor::seekToFirst() { /* try to seek to the first page */ stream_ref_->accessPages([this] (const std::vector>& stream_pages) { if (stream_pages.size() > 0) { this->current_page_ = stream_pages.at(0); this->current_page_offset_ = 0; this->current_page_index_ = 0; } }); if (current_page_.get() == nullptr) { StreamPosition pos; pos.unix_millis = 0; pos.logical_offset = 0; pos.next_offset = 0; return pos; } else { current_page_ref_ = page_manager_->getPage(current_page_->page_); return getCurrentPosition(); } } StreamPosition Cursor::seekToLast() { /* try to seek to the last page */ stream_ref_->accessPages([this] (const std::vector>& stream_pages) { if (stream_pages.size() > 0) { this->current_page_ = stream_pages.back(); this->current_page_offset_ = 0; this->current_page_index_ = 0; } }); if (current_page_.get() == nullptr) { StreamPosition pos; pos.unix_millis = 0; pos.logical_offset = 0; pos.next_offset = 0; return pos; } else { /* seek to the last row in the page */ current_page_ref_ = page_manager_->getPage(current_page_->page_); while(next()) {} return getCurrentPosition(); } } bool Cursor::next() { if (current_page_.get() == nullptr) { return false; } /* try to advance the offset in the current page */ auto new_offset = current_page_offset_ + getCurrentRow()->size + sizeof(RowHeader); if (new_offset < current_page_->used_.load()) { current_page_offset_ = new_offset; // FIXPAUL mem barrier if tail page / still mutable return true; } auto old_page_index = current_page_index_; /* try to advance to the next page */ stream_ref_->accessPages([this] (const std::vector>& stream_pages) { if (this->current_page_index_ + 1 < stream_pages.size()) { this->current_page_index_++; this->current_page_ = stream_pages.at(current_page_index_); this->current_page_offset_ = 0; } }); if (current_page_index_ == old_page_index) { return false; } else { current_page_ref_ = page_manager_->getPage(current_page_->page_); return true; } } const RowHeader* Cursor::getCurrentRow() const { if (current_page_.get() == nullptr) { return nullptr; } return current_page_ref_->structAt(current_page_offset_); } StreamPosition Cursor::getCurrentPosition() const { StreamPosition pos; if (current_page_.get() == nullptr) { pos.unix_millis = 0; pos.logical_offset = 0; pos.next_offset = 0; } else { auto row = current_page_ref_->structAt(current_page_offset_); pos.unix_millis = row->time; pos.logical_offset = current_page_->logical_offset_ + current_page_offset_; pos.next_offset = pos.logical_offset + row->size + sizeof(RowHeader); } return pos; } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/cursor.h000066400000000000000000000023241255644354400251530ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_CURSOR_H #define _FNORDMETRIC_CURSOR_H #include #include #include #include #include "pagemanager.h" namespace fnordmetric { /** * A storage cursor is a stateful iterator for a single dollection. * */ class Cursor { public: Cursor(const Cursor& copy) = delete; Cursor& operator=(const Cursor& copy) = delete; virtual ~Cursor(); /** * Try to advance the cursor by n documents. Returns the number of documents * the cursor was advanced by */ virtual size_t advanceBy(size_t n) = 0; protected: //StreamRef* stream_ref_; //std::shared_ptr current_page_; //std::unique_ptr current_page_ref_; //uint64_t current_page_offset_; //size_t current_page_index_; //std::shared_ptr page_manager_; }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/database.h000066400000000000000000000140071255644354400254030ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FILEBACKEND_FILEBACKEND_H #define _FNORDMETRIC_FILEBACKEND_FILEBACKEND_H #include #include #include #include #include #include #include "pagemanager.h" #include "log.h" /** * A storage backend stores an arbitrary number of 'streams'. A stream consists * of rows. Each row is a tuple where time is the time at which the * row was inserted into the stream and data is a binary string. Streams are * append only. Each stream is identified by a unique string key. * * A file-backed, threadsafe storage backend for FnordMetric. This backend * multiplexes an arbitrary number of stream into a single log-structured file. * * * eBNF: * * FILE ::= FILE_HEADER *( PAGE ) FILE_HEADER ::= <8 Bytes 0x17> ; magic bytes ; version number PAGE_PTR ; pointer to first log page PAGE ::= ( LOG_PAGE | DATA_PAGE ) PAGE_PTR ::= ; page offset in the file in bytes ; size of the page in bytes LOG_PAGE ::= *( LOG_ENTRY ) LOG_ENTRY ::= ( LOG_ENTRY_NEXT | LOG_ENTRY_ALLOC | LOG_ENTRY_FREE ) LOG_ENTRY_NEXT ::= LOG_ENTRY_CHECKSUM LOG_ENTRY_TYPE ; 0x01 LOG_ENTRY_LENGTH ; 0x08 PAGE_PTR ; pointer to next log page LOG_ENTRY_ALLOC ::= LOG_ENTRY_CHECKSUM LOG_ENTRY_TYPE ; 0x02 LOG_ENTRY_LENGTH ; variable PAGE_PTR ; pointer to the data page UNIX_MILLIS ; timestamp of the page's first row STREAM_ID [ STREAM_KEY ] ; length = LOG_ENTRY_LENGTH - 16 LOG_ENTRY_FREE ::= LOG_ENTRY_CHECKSUM LOG_ENTRY_TYPE ; 0x03 LOG_ENTRY_LENGTH ; 0x10 PAGE_PTR ; pointer to the free'd page PAGE_SIZE ; size of the freed page DATA_PAGE ::= PAGE_HEADER *( STREAM_ROW ) UNIX_MILLIS ::= ; unix millisecond timestamp STREAM_ID ::= ; unique stream id STREAM_KEY ::= ; unique stream key */ namespace fnordmetric { namespace query { class Query; } namespace database { class StreamRef; enum kDatabaseFlags { /** * Safety modes. The database will be consistent in all cases, but depending * on the safety mode you might loose more or less data. */ MODE_RELAXED = 0, MODE_CONSERVATIVE = 10, /* MSYNC_ASYNC | ENABLE_ROW_CHECKSUMS */ MODE_PARANOID = 12, /* MSYNC_SYNC | ENABLE_ROW_CHECKSUMS */ /** * Msync modes */ MYSNC_ASYNC = 2, MYSNC_SYNC = 4, /** * Verify checksums when scanning over rows. Slows down everthing by a bit * but protects against file corruption */ ENABLE_ROW_CHECKSUMS = 8, /** * Truncate the file when opening it */ FILE_TRUNCATE = 32, /** * Delete the file when the database is closed (the database object is * destroyed) */ FILE_AUTODELETE = 64 }; class Database { friend class StreamRef; friend class Cursor; friend class DatabaseTest; public: struct __attribute__((__packed__)) FileHeader { uint64_t magic; uint64_t version; uint64_t first_log_page_offset; uint64_t first_log_page_size; }; Database(const Database& copy) = delete; Database& operator=(const Database& copy) = delete; ~Database(); /** * Min. number of bytes to reserve for the file header */ static const uint64_t kMinReservedHeaderSize = 512; /** * File magic bytes */ static const uint64_t kFileMagicBytes = 0x1717171717171717; /** * File format version number */ static const uint64_t kFileVersion = 1; /** * Target page size in number of rows. Default: 16384 rows */ static size_t kTargetRowsPerPage; /** * Preferred maximum page size (soft limit). Default: 4MB */ static size_t kMaxPageSizeSoft; /** * Maximum allowed page size (hard limit). Note that now single row may be * larger than the maximum page size. Default: 32MB */ static size_t kMaxPageSizeHard; /** * Instantiate a new file backend with a path to the file. */ static std::unique_ptr openFile( const std::string& filename, uint64_t flags = MODE_CONSERVATIVE); /** * Execute a query */ void executeQuery(query::Query* query); /** * Open or create the stream with the specified key. * This method is threadsafe. */ std::shared_ptr openStream(const std::string& key); protected: Database( LogSnapshot& log_snapshot, std::shared_ptr log, std::shared_ptr page_manager, uint64_t flags); Database( std::shared_ptr log, std::shared_ptr page_manager, uint64_t flags); /** * Retrieve the stream id for a specified string stream key */ uint64_t getStreamId(const std::string& key); /** * Maps string stream keys to stream ids */ std::unordered_map stream_ids_; /** * Holds all the StreamRefs */ std::unordered_map> stream_refs_; std::mutex stream_refs_mutex_; /** * Highest used stream_id */ uint64_t max_stream_id_; const std::shared_ptr log_; const std::shared_ptr page_manager_; uint64_t flags_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/database_test.cc000066400000000000000000000165761255644354400266150ustar00rootroot00000000000000/** * This file is part of the "FnordStream" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * Licensed under the MIT license (see LICENSE). */ #include #include #include #include #include #include #include "collection.h" #include "pagemanager.h" #include "transaction.h" //#include "cursor.h" //#include "clock.h" namespace fnordmetric { // TODO test freelist serialization / after file reloads // TODO test in memory db class DatabaseTest { public: DatabaseTest() {} void run() { //testStreamIdAssignment(); //testStreamRefCreation(); testPageManager(); testMmapPageManager(); testAOCollection(); } /* void testStreamIdAssignment() { auto backend = fnordmetric::database::Database::openFile( "/tmp/__fnordmetric_testStreamIdAssignment", database::MODE_CONSERVATIVE | database::FILE_TRUNCATE); std::string key1 = "83d2f71c457206bf-Ia9f37ed7-F76b77d1a"; std::string key2 = "83d2f71c457216bf-Ia9f37ed7-F76b77d1a"; assert(backend->getStreamId(key1) == 1); assert(backend->getStreamId(key2) == 2); assert(backend->getStreamId(key1) == 1); assert(backend->getStreamId(key2) == 2); } void testStreamRefCreation() { auto backend = fnordmetric::database::Database::openFile( "/tmp/__fnordmetric_testStreamRefCreation", database::MODE_CONSERVATIVE | database::FILE_TRUNCATE | database::FILE_AUTODELETE); std::string key1 = "83d2f71c457206bf-Ia9f37ed7-F76b77d1a"; std::string key2 = "83d2f71c457216bf-Ia9f37ed7-F76b77d1a"; auto ref1 = backend->openStream(key1); assert(ref1.get() != nullptr); auto ref2 = backend->openStream(key1); assert(ref1.get() == ref2.get()); auto ref3 = backend->openStream(key2); assert(ref1.get() != ref3.get()); assert(ref2.get() != ref3.get()); auto ref4 = backend->openStream(key2); assert(ref1.get() != ref4.get()); assert(ref2.get() != ref4.get()); assert(ref3.get() == ref4.get()); } */ void testPageManager() { class ConcreteTestPageManager : public PageManager { public: ConcreteTestPageManager() : PageManager(4096) {} std::unique_ptr getPage(const PageManager::Page& page) override { return std::unique_ptr(nullptr); } void fsync() const {} }; ConcreteTestPageManager page_manager; auto page1 = page_manager.allocPage(3000); assert(page_manager.end_pos_ == 4096); assert(page1.offset == 0); assert(page1.size == 4096); page_manager.freePage(page1); auto page2 = page_manager.allocPage(8000); assert(page_manager.end_pos_ == 12288); assert(page2.offset == 4096); assert(page2.size == 8192); auto page3 = page_manager.allocPage(3000); assert(page3.offset == 0); assert(page3.size == 4096); page_manager.freePage(page2); auto page4 = page_manager.allocPage(4000); assert(page_manager.end_pos_ == 12288); assert(page4.offset == 4096); assert(page4.size == 8192); } void testMmapPageManager() { int fd = open("/tmp/__fnordmetric_testMmapPageManager", O_CREAT | O_TRUNC | O_RDWR, S_IRUSR | S_IWUSR); assert(fd > 0); unlink("/tmp/__fnordmetric_testMmapPageManager"); auto page_manager = new MmapPageManager(fd, 0, 4096); auto mfile1 = page_manager->getMmapedFile(3000); auto mfile2 = page_manager->getMmapedFile(304200); assert(mfile1->size == 1048576); assert(mfile1 == mfile2); mfile2->incrRefs(); auto mfile3 = page_manager->getMmapedFile(1048577); assert(mfile3->size == 1048576 * 2); assert(mfile3 != mfile2); mfile2->decrRefs(); delete page_manager; close(fd); } void testAOCollection() { auto collection = Collection::createPersistentCollection( "/tmp/__fnordmetric_testAOCollection", Collection::FILE_TRUNCATE); //auto tx = collection->startTransaction(); //auto docref = tx->createDocument(); //auto docref1 = tx->createDocument(); //auto docref2 = tx->createDocument(); //auto docref3 = tx->createDocument(); //tx->commit(); //printf("committed...\n"); //collection->sync(); //printf("synced...\n"); /* //printf("TEST: File backed database insert, reopen, read\n"); uint32_t stream_id; std::vector insert_times; std::vector fields = { fnordmetric::IntegerField("sequence_num"), fnordmetric::IntegerField("test1"), fnordmetric::StringField("test2")}; Schema schema(fields); int rows_written = 0; size_t base_time = WallClock::getUnixMillis(); //for (int j = 0; j < 50; ++j) { for (int j = 0; j < 5; ++j) { int flags = database::MODE_CONSERVATIVE; if (j == 0) { flags |= database::FILE_TRUNCATE; } if (j == 49) { flags |= database::FILE_AUTODELETE; } auto database = fnordmetric::database::Database::openFile( "/tmp/__fnordmetric_testOpenFile", flags); assert(database.get() != nullptr); auto stream = database->openStream("mystream"); if (j == 0) { stream_id = stream->stream_id_; } else { assert(stream_id == stream->stream_id_); } assert(database->max_stream_id_ == stream_id); RecordWriter record_writer(schema); for (int i = (j + 1) * 1000; i > 0; i--) { auto insert_time = base_time + rows_written; record_writer.setIntegerField(0, ++rows_written); record_writer.setIntegerField(1, 1337); record_writer.setStringField(2, "fnordbar", 8); auto new_row_pos = stream->appendRow(record_writer, insert_time); if (insert_times.size() > 0) { assert( new_row_pos.logical_offset > insert_times.back().logical_offset); } insert_times.push_back(new_row_pos); record_writer.reset(); } auto cursor = stream->getCursor(); assert(cursor->seekToLast() == insert_times.back()); auto test_index = insert_times.size() / 2; auto test_pos = cursor->seekToLogicalOffset( insert_times[test_index].logical_offset); assert(test_pos == insert_times[test_index]); test_pos = cursor->seekToLogicalOffset( insert_times[test_index].logical_offset + 1); assert(test_pos == insert_times[test_index + 1]); assert(cursor->seekToFirst() == insert_times[0]); test_pos = cursor->seekToTime(insert_times[test_index].unix_millis); assert(test_pos == insert_times[test_index]); assert(cursor->seekToFirst() == insert_times[0]); RecordReader record_reader(schema); for (int i = 0; i < insert_times.size() - 1; ++i) { auto row = cursor->getCurrentRow(); assert(cursor->getCurrentPosition() == insert_times[i]); assert(row->time == insert_times[i].unix_millis); assert(cursor->next()); assert(record_reader.getIntegerField(row->data, 0) == i+1); assert(record_reader.getIntegerField(row->data, 1) == 1337); char* str; size_t str_len; record_reader.getStringField(row->data, 2, &str, &str_len); assert(str_len == 8); assert(strncmp(str, "fnordbar", str_len) == 0); } assert(cursor->next() == false); query::Query query("SELECT count(*) FROM mystream"); database->executeQuery(&query); } */ } }; } int main() { fnordmetric::DatabaseTest test; test.run(); printf("all tests passed! :)\n"); } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/documentkey.cc000066400000000000000000000025261255644354400263270ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include "documentkey.h" namespace fnordmetric { DocumentKey::DocumentKey(uint64_t key) : is_string_(false) { key_.int_key = key; } DocumentKey::DocumentKey(const std::string& key) : is_string_(true) { key_.string_key = new std::string((key)); } DocumentKey::DocumentKey(const DocumentKey& copy) : is_string_(copy.is_string_) { if (is_string_) { key_.string_key = new std::string(*copy.key_.string_key); // FIXPAUL! } else { key_.int_key = copy.key_.int_key; } } DocumentKey::~DocumentKey() { if (is_string_) { delete key_.string_key; } } bool DocumentKey::isIntKey() const { return !is_string_; } bool DocumentKey::isStringKey() const { return is_string_; } uint64_t DocumentKey::getIntKey() const { assert(is_string_ == false); return key_.int_key; } const std::string* DocumentKey::getStringKey() const { assert(is_string_ == true); return key_.string_key; } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/documentkey.h000066400000000000000000000020231255644354400261610ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_DOCUMENTKEY_H #define _FNORDMETRIC_DOCUMENTKEY_H #include #include #include #include namespace fnordmetric { /** */ class DocumentKey { public: explicit DocumentKey(const std::string& key); explicit DocumentKey(uint64_t key); DocumentKey(const DocumentKey& copy); DocumentKey& operator==(const DocumentKey& copy) = delete; ~DocumentKey(); bool isIntKey() const; bool isStringKey() const; uint64_t getIntKey() const; const std::string* getStringKey() const; protected: bool is_string_; union { uint64_t int_key; std::string* string_key; } key_; }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/documentref.cc000066400000000000000000000017441255644354400263140ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include "documentref.h" namespace fnordmetric { DocumentRef::DocumentRef( const DocumentKey& key) : key_(key), dirty_(1) {} void DocumentRef::revert() { // FIXPAUL free pages if alloced } bool DocumentRef::isDirty() { return dirty_ == 1; } void DocumentRef::setDocumentKey(const DocumentKey& key) { key_ = key; } const DocumentKey& DocumentRef::getDocumentKey() const { return key_; } void DocumentRef::getScratchpad(char** data, size_t* size) const { static char scratch[] = "fnord"; *data = scratch; *size = sizeof(scratch); } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/documentref.h000066400000000000000000000025471255644354400261600ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_DOCUMENTREF_H #define _FNORDMETRIC_DOCUMENTREF_H #include #include #include #include #include "documentkey.h" namespace fnordmetric { class PageManager; class Collection; /** */ class DocumentRef { public: /** * Create a new DocumentRef for an empty document */ explicit DocumentRef(const DocumentKey& key); void revert(); bool isDirty(); DocumentRef(const DocumentRef& copy) = delete; DocumentRef& operator=(const DocumentRef& copy) = delete; void setDocumentKey(const DocumentKey& key); const DocumentKey& getDocumentKey() const; void getScratchpad(char** data, size_t* size) const; protected: DocumentKey key_; int dirty_; //StreamRef* stream_ref_; //std::shared_ptr current_page_; //std::unique_ptr current_page_ref_; //uint64_t current_page_offset_; //size_t current_page_index_; //std::shared_ptr page_manager_; }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/idsequence.h000066400000000000000000000017201255644354400257620ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_IDSEQUENCE_H #define _FNORDMETRIC_IDSEQUENCE_H #include #include #include "clock.h" namespace fnordmetric { /** */ class IDSequence { public: IDSequence() {} IDSequence(const IDSequence& copy) = delete; IDSequence& operator==(const IDSequence& copy) = delete; virtual ~IDSequence() {} virtual uint64_t getNext(uint64_t last) = 0; }; class UnixMillisecondIDSequence : public IDSequence { public: UnixMillisecondIDSequence() {} uint64_t getNext(uint64_t last) override { return WallClock::getUnixMillis(); } }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/log.cc000066400000000000000000000212051255644354400245540ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include "log.h" #include "../fnv.h" namespace fnordmetric { namespace database { LogReader::LogReader( std::shared_ptr page_manager, const PageManager::Page& first_log_page, LogSnapshot* destination) : page_manager_(std::move(page_manager)), current_page_(first_log_page), destination_(destination) {} void LogReader::import() { bool running = true; destination_->last_used_byte = current_page_.offset + current_page_.size; while (running) { auto mmapped_offset = current_page_.offset; auto mmapped = page_manager_->getPage(current_page_); size_t offset = 0; destination_->current_log_page = current_page_; destination_->current_log_page_offset = offset; while (current_page_.offset == mmapped_offset) { running = importNextEntry(mmapped.get(), current_page_.size, &offset); if (running) { destination_->current_log_page_offset = offset; } else { break; } } } for (auto& stream : destination_->streams) { if (stream.pages_.size() > 0) { countPageUsedBytes(stream.pages_.back()); } } } bool LogReader::importNextEntry( const PageManager::PageRef* mmapped, size_t mmaped_size, size_t* offset) { size_t header_size = sizeof(Log::EntryHeader); if (*offset + header_size >= mmaped_size) { return false; } auto entry_header = mmapped->structAt(*offset); size_t entry_size = header_size + entry_header->size; if (entry_header->size == 0 || *offset + entry_size >= mmaped_size) { return false; } if (entry_header->checksum != entry_header->computeChecksum()) { fprintf(stderr, "warning: invalid checksum for log entry 0x%" PRIx64 "\n", mmapped->page_.offset + *offset); return false; } *offset += entry_size; importLogEntry(entry_header); return true; } void LogReader::importLogEntry(const Log::EntryHeader* entry) { switch (entry->type) { case Log::PAGE_ALLOC_ENTRY: { auto alloc_entry = (Log::PageAllocEntry *) entry; auto iter = streams_.find(alloc_entry->stream_id); LogSnapshot::StreamState* stream_state; if (iter == streams_.end()) { destination_->streams.emplace_back(alloc_entry->stream_id); stream_state = &destination_->streams.back(); streams_[alloc_entry->stream_id] = stream_state; size_t key_len = alloc_entry->hdr.size - (sizeof(Log::PageAllocEntry) - sizeof(Log::EntryHeader)); stream_state->stream_key_.insert(0, alloc_entry->stream_key, key_len); if (alloc_entry->stream_id > destination_->max_stream_id) { destination_->max_stream_id = alloc_entry->stream_id; } } else { stream_state = iter->second; } PageManager::Page page; page.offset = alloc_entry->page_offset; page.size = alloc_entry->page_size; auto alloc = new PageAlloc( page, alloc_entry->page_first_row_time, alloc_entry->page_logical_offset); stream_state->pages_.push_back(std::shared_ptr(alloc)); setLastUsedByte(page.offset + page.size); break; } case Log::PAGE_FINISH_ENTRY: { auto finish_entry = (Log::PageFinishEntry *) entry; auto iter = streams_.find(finish_entry->stream_id); if (iter != streams_.end()) { auto stream_state = iter->second; if (stream_state->pages_.back()->page_.offset == finish_entry->page_offset) { stream_state->pages_.back()->used_ = finish_entry->page_used; break; } } fprintf(stderr, "warning: unexpected PAGE_FINISH log entry\n"); break; } case Log::NEXT_PAGE_ENTRY: { auto next_entry = (Log::NextPageEntry *) entry; current_page_.offset = next_entry->page_offset; current_page_.size = next_entry->page_size; setLastUsedByte(current_page_.offset + current_page_.size); break; } default: fprintf(stderr, "warning: invalid log entry type %i\n", entry->type); }; } void LogReader::countPageUsedBytes(std::shared_ptr page) { size_t offset = 0; size_t max = page->page_.size - sizeof(Log::EntryHeader) - 1; auto mmapped = page_manager_->getPage(page->page_); while (offset < max) { auto row = mmapped->structAt(offset); auto row_size = sizeof(RowHeader) + row->size; if (row->size == 0 || row->time == 0) { return; } if (offset + row_size >= page->page_.size) { return; } if (row->checksum != row->computeChecksum()) { return; } offset += row_size; page->used_ = offset; page->num_rows_++; } } void LogReader::setLastUsedByte(uint64_t index) { if (index > destination_->last_used_byte) { destination_->last_used_byte = index; } } const uint64_t Log::kMinLogPageSize = 512; Log::Log( const LogSnapshot& snapshot, std::shared_ptr page_manager) : page_manager_(page_manager), current_page_(snapshot.current_log_page), current_page_offset_(snapshot.current_log_page_offset) {} Log::Log( const PageManager::Page& first_log_page, std::shared_ptr page_manager) : page_manager_(page_manager), current_page_(first_log_page), current_page_offset_(0) {} void Log::appendEntry(Log::PageAllocEntry entry) { entry.hdr.size = sizeof(PageAllocEntry) - sizeof(EntryHeader); entry.hdr.type = PAGE_ALLOC_ENTRY; appendEntry((uint8_t *) &entry, sizeof(PageAllocEntry)); } void Log::appendEntry(Log::PageAllocEntry entry, const std::string& stream_key) { assert(stream_key.size() < 0xffff); entry.hdr.size = sizeof(PageAllocEntry) - sizeof(EntryHeader); entry.hdr.size += stream_key.size(); entry.hdr.type = PAGE_ALLOC_ENTRY; size_t tmp_len = sizeof(PageAllocEntry) + stream_key.size(); uint8_t* tmp = (uint8_t *) malloc(tmp_len); assert(tmp); memcpy(tmp, &entry, sizeof(PageAllocEntry)); memcpy(tmp + sizeof(PageAllocEntry), stream_key.c_str(), stream_key.size()); appendEntry(tmp, tmp_len); free(tmp); } void Log::appendEntry(Log::PageFinishEntry entry) { entry.hdr.size = sizeof(PageFinishEntry) - sizeof(EntryHeader); entry.hdr.type = PAGE_FINISH_ENTRY; appendEntry((uint8_t *) &entry, sizeof(PageFinishEntry)); } void Log::appendEntry(Log::PageFreeEntry entry) { entry.hdr.size = sizeof(PageFreeEntry) - sizeof(EntryHeader); entry.hdr.type = PAGE_FREE_ENTRY; appendEntry((uint8_t *) &entry, sizeof(PageFreeEntry)); } void Log::appendEntry(uint8_t* data, size_t length) { uint64_t reserved_length = length + sizeof(Log::NextPageEntry); append_mutex_.lock(); /* allocate a new page if the current one is full */ if (current_page_offset_ + reserved_length >= current_page_.size) { assert(current_page_offset_ + sizeof(Log::NextPageEntry) <= current_page_.size); auto next_page_size = std::max(kMinLogPageSize, reserved_length); auto next_page = page_manager_->allocPage(next_page_size); Log::NextPageEntry log_entry; log_entry.page_offset = next_page.offset; log_entry.page_size = next_page.size; log_entry.hdr.size = sizeof(Log::NextPageEntry) - sizeof(Log::EntryHeader); log_entry.hdr.type = NEXT_PAGE_ENTRY; log_entry.hdr.checksum = log_entry.hdr.computeChecksum(); auto mmaped = page_manager_->getPage(current_page_); auto dst = mmaped->structAt(current_page_offset_); memcpy(dst, (char *) &log_entry, sizeof(Log::NextPageEntry)); current_page_offset_ += sizeof(Log::NextPageEntry); current_page_ = next_page; current_page_offset_ = 0; } /* memcpy the entry into the current page */ auto entry = (Log::EntryHeader*) data; entry->checksum = entry->computeChecksum(); auto mmaped = page_manager_->getPage(current_page_); auto dst = mmaped->structAt(current_page_offset_); memcpy(dst, data, length); current_page_offset_ += length; // FIXPAUL msync append_mutex_.unlock(); } uint32_t Log::EntryHeader::computeChecksum() { FNV fnv; return fnv.hash( (uint8_t *) (((char* ) this) + sizeof(checksum)), size + sizeof(EntryHeader) - sizeof(checksum)); } LogSnapshot::LogSnapshot() : max_stream_id(0) {} LogSnapshot::StreamState::StreamState(uint32_t stream_id) : stream_id_(stream_id) {} } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/log.h000066400000000000000000000077641255644354400244340ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FILEBACKEND_LOG_H #define _FNORDMETRIC_FILEBACKEND_LOG_H #include #include #include #include #include #include #include #include #include "streamref.h" namespace fnordmetric { namespace database { struct LogSnapshot { LogSnapshot(); struct StreamState { StreamState(uint32_t stream_id); uint64_t stream_id_; std::string stream_key_; std::vector> pages_; }; std::vector free_pages; std::vector streams; uint32_t max_stream_id; PageManager::Page current_log_page; uint64_t current_log_page_offset; uint64_t last_used_byte; }; class Log { friend class DatabaseTest; public: struct __attribute__((__packed__)) EntryHeader { uint32_t checksum; uint16_t type; uint16_t size; uint32_t computeChecksum(); }; struct __attribute__((__packed__)) PageAllocEntry { EntryHeader hdr; uint32_t stream_id; uint64_t page_offset; uint32_t page_size; uint64_t page_first_row_time; uint64_t page_logical_offset; char stream_key[]; }; struct __attribute__((__packed__)) PageFinishEntry { EntryHeader hdr; uint32_t stream_id; uint64_t page_offset; uint32_t page_size; uint32_t page_used; }; struct __attribute__((__packed__)) PageFreeEntry { EntryHeader hdr; uint64_t page_offset; uint32_t page_size; }; struct __attribute__((__packed__)) NextPageEntry { EntryHeader hdr; uint64_t page_offset; uint32_t page_size; }; enum kEntryTypes { PAGE_ALLOC_ENTRY = 0x01, PAGE_FINISH_ENTRY = 0x02, PAGE_FREE_ENTRY = 0x03, NEXT_PAGE_ENTRY = 0x04 }; /** * Min. log page size */ static const uint64_t kMinLogPageSize; explicit Log( const LogSnapshot& snapshot, std::shared_ptr page_manager); explicit Log( const PageManager::Page& first_log_page, std::shared_ptr page_manager); Log(const Log& copy) = delete; Log& operator=(const Log& copy) = delete; Log(const Log&& move); void appendEntry(PageAllocEntry entry); void appendEntry(PageAllocEntry entry, const std::string& stream_key); void appendEntry(PageFinishEntry entry); void appendEntry(PageFreeEntry entry); protected: void appendEntry(uint8_t* data, size_t length); std::mutex append_mutex_; std::shared_ptr page_manager_; PageManager::Page current_page_; uint64_t current_page_offset_; }; /** * This is an internal class. For usage instructions and extended documentation * please refer to "storagebackend.h" and "database.h" */ class LogReader { public: LogReader( std::shared_ptr page_manager, const PageManager::Page& first_log_page, LogSnapshot* destination); /** * Import the log. Returns a snapshot of the imported log */ void import(); protected: /** * Import a single log entry. Returns true if there might be another entry * in the log and returns false when the last entry was read */ bool importNextEntry( const PageManager::PageRef* mmapped, size_t mmaped_size, size_t* offset); /** * Import a single log entry */ void importLogEntry(const Log::EntryHeader* entry); /** * Count the number of used byte in a page */ void countPageUsedBytes(std::shared_ptr page); void setLastUsedByte(uint64_t index); std::unordered_map streams_; const std::shared_ptr page_manager_; PageManager::Page current_page_; LogSnapshot* destination_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/object.cc000066400000000000000000000010051255644354400252350ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include "object.h" namespace fnordmetric { namespace ffs { void Object::incrRefcount() { } void Object::decrRefcount() { } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/object.h000066400000000000000000000011311255644354400250770ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FFS_OBJECT_H #define _FNORDMETRIC_FFS_OBJECT_H namespace fnordmetric { namespace ffs { class Object { friend class Volume; public: void incrRefcount(); void decrRefcount(); }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/objectref.h000066400000000000000000000024221255644354400256000ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FFS_OBJECTREF_H #define _FNORDMETRIC_FFS_OBJECTREF_H #include #include "object.h" #include namespace fnordmetric { namespace ffs { template class ObjectRef { public: ObjectRef() = delete; ObjectRef(const ObjectRef& other) = delete; ObjectRef& operator=(const ObjectRef& other) = delete; ObjectRef(ObjectType* obj) : obj_(obj) { obj_->incrRefcount(); } ObjectRef(ObjectRef&& other) { obj_ = other.obj_; other.obj_ = nullptr; } ~ObjectRef() { if (obj_ != nullptr) { obj_->decrRefcount(); } } void close() { if (obj_ == nullptr) { RAISE(kIllegalStateError, "alread closed"); } else { obj_->decrRefcount(); obj_ = nullptr; } } protected: ObjectType* obj_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/pageindex.cc000066400000000000000000000064721255644354400257500ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include "pageindex.h" namespace fnordmetric { size_t PageIndex::kTargetRowsPerPage = 1024; PageIndex::PageIndex(std::shared_ptr page_manager) : page_manager_(page_manager), used_(0) {} PageIndex::PageIndex( std::shared_ptr page_manager, const PageManager::Page& index_page, uint32_t used) : page_manager_(page_manager), index_page_(index_page), used_(used) {} PageIndex::~PageIndex() { printf("page index freed!\n"); } PageIndex::IndexPageEntry* PageIndex::getIndexPageEntryForInsert(size_t bytes) { if (used_ >= sizeof(IndexPageEntry)) { auto last_page = index_page_ref_->structAt( used_ - sizeof(IndexPageEntry)); if (last_page->used + bytes < last_page->size) { return last_page; } } size_t estimated_page_size = bytes * kTargetRowsPerPage; if (used_ >= sizeof(IndexPageEntry)) { auto last_page = index_page_ref_->structAt( used_ - sizeof(IndexPageEntry)); estimated_page_size += last_page->used; estimated_page_size /= 2; } if (estimated_page_size < bytes) { estimated_page_size = bytes; } auto page = page_manager_->allocPage(estimated_page_size); return appendPage(page); } size_t PageIndex::getNumPages() const { if (used_ == 0) { return 0; } return used_ / sizeof(IndexPageEntry); } PageIndex::IndexPageEntry* PageIndex::appendPage( const PageManager::Page& page) { PageIndex* index = nullptr; if (used_ == 0) { index_page_ = page_manager_->allocPage(kInitialIndexPageSize); index_page_ref_ = page_manager_->getPage(index_page_); used_ = 0; } if (used_ + sizeof(IndexPageEntry) >= index_page_.size) { auto new_page = page_manager_->allocPage(index_page_.size * 2); // FIXPAUL: just copying the data over doesnt handle deletions! auto dst = page_manager_->getPage(new_page); auto src = page_manager_->getPage(index_page_); memcpy(dst->structAt(0), src->structAt(0), used_); // FIXPAUL free old page! index_page_ = new_page; index_page_ref_ = page_manager_->getPage(index_page_); } auto new_entry = index_page_ref_->structAt(used_); new_entry->offset = page.offset; new_entry->size = page.size; new_entry->used = 0; new_entry->first_key = 0; used_ += sizeof(IndexPageEntry); return new_entry; } /* IndexPageEntry* PageIndex::getIndexPageEntryAt(size_t n) const { auto offset = n * sizeof(IndexPageEntry); assert(offset < used_); return index_page_mapped_->structAt(0); } */ PageIndex* PageIndex::clone() { if (used_ == 0) { return new PageIndex(page_manager_); } auto copy = page_manager_->allocPage(index_page_.size); auto dst = page_manager_->getPage(copy); auto src = page_manager_->getPage(index_page_); memcpy(dst->structAt(0), src->structAt(0), used_); return new PageIndex(page_manager_, copy, used_); } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/pageindex.h000066400000000000000000000046201255644354400256030ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_PAGEINDEX_H #define _FNORDMETRIC_PAGEINDEX_H #include #include #include #include #include #include #include "pagemanager.h" namespace fnordmetric { class Database; class Cursor; class DocumentRef; /* struct StreamPosition { uint64_t unix_millis; uint64_t logical_offset; uint64_t next_offset; bool operator==(const StreamPosition& other); }; */ /** * This should be a linked list of smaller index blocks so that the COW overhead * is fixed... * * Pages should be removed via deletePage() and then a compact() (which rebuilds * the list into a two node list with one short node holding the most recent * pages for fast committing and one long node holding all older pages */ class PageIndex { friend class DatabaseTest; public: PageIndex(std::shared_ptr page_manager); PageIndex( std::shared_ptr page_manager, const PageManager::Page& index_page, uint32_t used); struct __attribute__((__packed__)) IndexPageEntry { uint64_t offset; uint64_t first_key; uint32_t size; uint32_t used; }; static const size_t kInitialIndexPageSize = sizeof(IndexPageEntry) * 4; /** * Target page size in number of rows. Default: 16384 rows */ static size_t kTargetRowsPerPage; PageIndex(const PageIndex& copy) = delete; PageIndex& operator=(const PageIndex& copy) = delete; ~PageIndex(); /** * Access the StreamRefs internal page storage (do not call this method unless * you know what you are doing) */ //void accessPages(std::function>&)> func); IndexPageEntry* getIndexPageEntryForInsert(size_t bytes); PageIndex* clone(); size_t getNumPages() const; protected: uint32_t used_; IndexPageEntry* appendPage(const PageManager::Page& page); const std::shared_ptr page_manager_; PageManager::Page index_page_; std::unique_ptr index_page_ref_; }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/record.cc000066400000000000000000000071741255644354400252620ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include "record.h" namespace fnordmetric { RecordWriter::RecordWriter( const Schema& schema, size_t buffer_size_hint /* = 65536 */) : last_byte_(0) { for (const auto& field : schema.fields_) { field_offsets_.push_back(last_byte_); last_byte_ += schema::kFieldTypesSize[field.getTypeId()]; #ifndef NDEBUG field_types_.push_back(field.getTypeId()); #endif } min_size_ = last_byte_; alloc_size_ = std::max(buffer_size_hint, last_byte_); alloc_ = malloc(alloc_size_); assert(alloc_); // FIXPAUL memset(alloc_, 0, alloc_size_); } RecordWriter::~RecordWriter() { if (alloc_ != nullptr) { free(alloc_); } } void RecordWriter::setFloatField(size_t field_index, double value) { assert(field_index < field_offsets_.size()); void* dst = ((char *) alloc_) + field_offsets_[field_index]; #ifndef NDEBUG assert(field_types_[field_index] == schema::IEE754); #endif // FIXPAUL } // endianess void RecordWriter::setIntegerField(size_t field_index, int64_t value) { int64_t local_value = value; assert(field_index < field_offsets_.size()); void* dst = ((char *) alloc_) + field_offsets_[field_index]; #ifndef NDEBUG assert(field_types_[field_index] == schema::INT64); #endif memcpy(dst, &local_value, 8); } void RecordWriter::setStringField( size_t field_index, const char* value_ptr, size_t value_len) { assert(field_index < field_offsets_.size()); #ifndef NDEBUG assert(field_types_[field_index] == schema::STRING); #endif auto data_offset = allocVarlen(value_len); auto meta_ptr = (uint32_t*) (((char *) alloc_) + field_offsets_[field_index]); meta_ptr[0] = data_offset; meta_ptr[1] = value_len; memcpy(((char *) alloc_) + data_offset, value_ptr, value_len); } void RecordWriter::toBytes(const void** data, size_t* size) const { *data = alloc_; *size = last_byte_; } uint32_t RecordWriter::allocVarlen(uint32_t size) { uint32_t offset = last_byte_; last_byte_ += size; if (last_byte_ > alloc_size_) { alloc_size_ = last_byte_; alloc_ = realloc(alloc_, alloc_size_); assert(alloc_); } return offset; } void RecordWriter::reset() { memset(alloc_, 0, alloc_size_); last_byte_ = min_size_; } RecordReader::RecordReader(const Schema& schema) : data_(nullptr) { size_t last_byte = 0; for (const auto& field : schema.fields_) { field_offsets_.push_back(last_byte); last_byte += schema::kFieldTypesSize[field.getTypeId()]; #ifndef NDEBUG field_types_.push_back(field.getTypeId()); #endif } } int64_t RecordReader::getIntegerField( const void* data, size_t field_index) const { assert(field_index < field_offsets_.size()); auto src = (int64_t *) (((char *) data) + field_offsets_[field_index]); #ifndef NDEBUG assert(field_types_[field_index] == schema::INT64); #endif return *src; } void RecordReader::getStringField( const void* data, size_t field_index, char** str_ptr, size_t* str_len) const { assert(field_index < field_offsets_.size()); #ifndef NDEBUG assert(field_types_[field_index] == schema::STRING); #endif auto meta_ptr = (int32_t *) (((char *) data) + field_offsets_[field_index]); *str_ptr = ((char *) data) + meta_ptr[0]; *str_len = meta_ptr[1]; } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/record.h000066400000000000000000000110001255644354400251030ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_RECORD_H #define _FNORDMETRIC_RECORD_H #include #include #include "schema.h" namespace fnordmetric { /** * A record writer is a stateful object that can be used to unserialize one or * more records of the same schema. * * FIXPAUL documentation * * All methods on a record reader are threadsafe. */ class RecordReader { public: explicit RecordReader(const Schema& schema); int64_t getIntegerField(const void* record, size_t field_index) const; double getFloatField(const void* record, size_t field_index) const; /** * The returned pointer has the same lifetime as the record pointer that * was passed in */ void getStringField( const void* record, size_t field_index, char** string_ptr, size_t* string_len) const; protected: std::vector field_offsets_; #ifndef NDEBUG std::vector field_types_; #endif const void* data_; }; /** * A record writer is a stateful object that can be used to serialize one or * more records of the same schema. It must be initialized with a schema and * will allocate some memory to store a single record internally. * * The record data is build up by calling the respective setField method for * each field that is defined by the schema. You must take care to call the * correct setField method for each field, i.e. the one that matches the type * that is defined in the schema. * * After all fields are set you can call toBytes to retrieve a pointer to the * binary representation of the record. The returned pointer is valid until the * RecordWriter object is destructed or setField() or reset() is called. * * To write multiple records with the same RecordWriter object, call the reset() * method after each record. * * A record writer is not threadsafe! If you want to use it with multiple * threads you must take care to synchronize access in such a way that no two * threads call any method on the record writer object at the same time! */ class RecordWriter { public: explicit RecordWriter( const Schema& schema, size_t buffer_size_hint = 65536); ~RecordWriter(); /** * Set the n'th field (as defined by the schema) of the record to value. This * method will assert if the type of the n'th field (as defined by the schema) * is not a fnordmetric::IntegerField! Field indices are zero based. */ void setIntegerField(size_t field_index, int64_t value); /** * Set the n'th field (as defined by the schema) of the record to value. This * method will assert if the type of the n'th field (as defined by the schema) * is not a fnordmetric::FloatField! Field indices are zero based. */ void setFloatField(size_t field_index, double value); /** * Set the n'th field (as defined by the schema) of the record to the string * pointed to by value_ptr. This method will assert if the type of the n'th * field (as defined by the schema) is not a fnordmetric::FloatField! This * method will also assert if you call it more than once for the same record. * It is legal to call this method again only after you called reset(). Field * indices are zero based. */ void setStringField( size_t field_index, const char* value_ptr, size_t value_len); /** * Returns a pointer to the raw binary representation of this record. The * returned pointer is valid until the RecordWriter object is destructed or * setField() or reset() is called. */ void toBytes(const void** data, size_t* size) const; /** * Reset the record writer so that a new record can be written. */ void reset(); protected: uint32_t allocVarlen(uint32_t size); void* alloc_; size_t alloc_size_; size_t min_size_; size_t last_byte_; std::vector field_offsets_; #ifndef NDEBUG std::vector field_types_; #endif }; template class TypedRecordWriter : public RecordWriter { public: explicit TypedRecordWriter(const typename T::ValueType&... values); protected: template void appendFields(T1 head); template void appendFields(T1 head, T2... tail); }; } #include "record_impl.h" #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/record_impl.h000066400000000000000000000016551255644354400261430ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_RECORD_IMPL_H #define _FNORDMETRIC_RECORD_IMPL_H namespace fnordmetric { template TypedRecordWriter::TypedRecordWriter( const typename T::ValueType&... values) { appendFields(values...); } template template void TypedRecordWriter::appendFields(T1 head) { appendField(head); } template template void TypedRecordWriter::appendFields(T1 head, T2... tail) { appendField(head); appendFields(tail...); } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/schema.cc000066400000000000000000000014171255644354400252360ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include "record.h" namespace fnordmetric { Schema::Schema(const std::vector& fields) : fields_(fields) {} Field::Field(schema::kFieldType type, const std::string& name) : type_(type), name_(name) {} const std::string& Field::getName() const { return name_; } schema::kFieldType Field::getTypeId() const { return type_; } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/schema.h000066400000000000000000000042761255644354400251060ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_SCHEMA_H #define _FNORDMETRIC_SCHEMA_H #include #include #include namespace fnordmetric { class Field; /** * Define the Field types */ namespace schema { enum kFieldType { INT64 = 0, IEE754 = 1, STRING = 2 }; static const char kFieldTypesHuman[] = { 'I', /* Integer */ 'F', /* Float */ 'S' /* String */ }; static const size_t kFieldTypesSize[] = { 8, /* Integer */ 8, /* Float */ 8 /* String */ }; } /** * A schema is an ordered flat list of fields. Each field consists of a type and * a name. The type can be one of Integer, Float, String and the name must be a * string. Nested fields are not supported. Schemas are immutable. */ class Schema { public: /** * Initialize a new schema with the specified fields */ explicit Schema(const std::vector& fields); /** * The field list of this schema */ const std::vector fields_; }; class Field { public: const std::string& getName() const; schema::kFieldType getTypeId() const; protected: Field(schema::kFieldType type, const std::string& name); schema::kFieldType type_; // FIXPAUL REMOVE ME std::string name_; }; template class TypedSchema : public Schema { public: explicit TypedSchema(const T... fields); protected: std::vector unpackFields(T... fields); }; class IntegerField : public Field { public: IntegerField(const std::string& name) : Field(schema::INT64, name) {} typedef int64_t ValueType; }; class FloatField : public Field { public: FloatField(const std::string& name) : Field(schema::IEE754, name) {} typedef double ValueType; }; class StringField : public Field { public: StringField(const std::string& name) : Field(schema::STRING, name) {} typedef std::string ValueType; }; } #include "schema_impl.h" #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/schema_impl.h000066400000000000000000000017751255644354400261300ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_SCHEMA_IMPL_H #define _FNORDMETRIC_SCHEMA_IMPL_H #include #include namespace fnordmetric { template TypedSchema::TypedSchema(const T... fields) : Schema(unpackFields(fields...)) {} // FIXPAUL there must be some better way to do this... template std::vector TypedSchema::unpackFields(T... fields) { std::vector unpacked; const Field packed[] = {fields...}; int num_fields = sizeof(packed) / sizeof(Field); for (int i = 0; i < num_fields; ++i) { unpacked.push_back(packed[i]); } return unpacked; }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/snapshot.h000066400000000000000000000014571255644354400255030ustar00rootroot00000000000000 /** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_SNAPSHOT_H #define _FNORDMETRIC_SNAPSHOT_H namespace fnordmetric { class Cursor; class DocumentKey; /** * All methods on a collection instance are threadsafe. */ class Snapshot { /** * Return a cursor for this snapshot and seek to the first document with a * key larger than or equal to the specified key. */ std::unique_ptr getCursor(const DocumentKey& key) {} }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/stream.cc000066400000000000000000000021051255644354400252640ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include "stream.h" namespace fnordmetric { const std::string& StreamKey::getKeyString() const { return key_str_; } Stream::Stream( const StreamKey& key, const Schema& schema, //const MetricDescription& description, std::shared_ptr stream_ref) : key_(key), schema_(schema), //description_(description), stream_ref_(std::move(stream_ref)) {} const Schema& Stream::getSchema() const { return schema_; } const StreamKey& Stream::getKey() const { return key_; } void Stream::appendRecord(const RecordWriter& record) const { stream_ref_->appendRow(record); } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/stream.h000066400000000000000000000047221255644354400251350ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_STREAM_H #define _FNORDMETRIC_STREAM_H #include #include #include #include #include #include #include #include "schema.h" #include "record.h" #include "database/streamref.h" namespace fnordmetric { class StreamKey; class StreamKey { public: //explicit IStreamKey(const std:string& name, const ISchema& schema); const std::string& getKeyString() const; protected: explicit StreamKey(const std::string& key_str) : key_str_(key_str) {} const std::string key_str_; }; template class TypedStreamKey : public StreamKey { public: explicit TypedStreamKey(const std::string& name, T... fields); protected: std::string buildKeyString(const std::string& name, T... fields); template void buildKeyString(std::stringstream* ss, Field head, T1... tail); void buildKeyString(std::stringstream* ss); }; class Stream { public: explicit Stream( const StreamKey& key, const Schema& schema, //const MetricDescription& description, std::shared_ptr stream_ref); Stream(const Stream& copy) = delete; Stream& operator=(const Stream& copy) = delete; const Schema& getSchema() const; const StreamKey& getKey() const; void appendRecord(const RecordWriter& record) const; protected: const std::shared_ptr stream_ref_; const StreamKey key_; //const MetricDescription description_; const Schema schema_; }; template class TypedStream : public Stream { public: explicit TypedStream( const StreamKey& key, const TypedSchema& schema, //const MetricDescription& description, std::shared_ptr stream_ref); TypedStream(const TypedStream& copy) = delete; TypedStream& operator=(const TypedStream& copy) = delete; void appendRecord(const typename T::ValueType&... values) const; }; class IStreamRepository { virtual std::shared_ptr findStreamByName( const std::string& stream_name) const = 0; }; } #include "stream_impl.h" #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/stream_impl.h000066400000000000000000000037071255644354400261600ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_STREAM_IMPL_H #define _FNORDMETRIC_STREAM_IMPL_H #include #include #include #include #include #include #include #include "fnv.h" namespace fnordmetric { template TypedStreamKey::TypedStreamKey( const std::string& name, T... fields) : StreamKey(buildKeyString(name, fields...)) {} template std::string TypedStreamKey::buildKeyString( const std::string& name, T... fields) { std::stringstream ss; FNV fnv; ss << std::hex << fnv.hash(name); buildKeyString(&ss, fields...); return ss.str(); } template template void TypedStreamKey::buildKeyString( std::stringstream* ss, Field head, T1... tail) { FNV fnv; *ss << '-' << static_cast(head.getTypeId()) << std::hex << fnv.hash(head.getName()); buildKeyString(ss, tail...); } template void TypedStreamKey::buildKeyString(std::stringstream* ss) {} template TypedStream::TypedStream( const StreamKey& key, const TypedSchema& schema, //const MetricDescription& description, std::shared_ptr stream_ref) : Stream( key, schema, //description, std::move(stream_ref)) {} template void TypedStream::appendRecord (const typename T::ValueType&... values) const { TypedRecordWriter record(values...); Stream::appendRecord(record); } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/stream_test.cc000066400000000000000000000034021255644354400263240ustar00rootroot00000000000000/** * This file is part of the "FnordStream" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * Licensed under the MIT license (see LICENSE). */ #include #include #include #include "stream.h" #include "agent.h" #include "clock.h" #include "filebackend/filebackend.h" void testStreamKeyGeneration() { //fnordmetric::StreamDescription metric_description("mymetric", "kg", "blah"); fnordmetric::StreamKey< fnordmetric::IntegerField, fnordmetric::FloatField> stream_key( "mymetric", fnordmetric::IntegerField("foo"), fnordmetric::FloatField("bar")); //printf("key=%s\n", stream_key.getKeyString().c_str()); assert(stream_key.getKeyString() == "83d2f71c457206bf-Ia9f37ed7-F76b77d1a"); } void testAppendRecord() { fnordmetric::Agent agent("myagent", fnordmetric::filebackend::FileBackend::openFile("/tmp/fu.test")); auto stream = agent.newStream( "mystream", fnordmetric::IntegerField("count"), fnordmetric::FloatField("fnord")); stream->appendRecord(42, 23.5); auto cursor = stream->getCursor(); bool called = false; cursor->seekToLast(); cursor->getRow([&called] (const uint8_t* data, size_t len, uint64_t time) { fnordmetric::RecordReader reader(data, len); assert(len == 18); assert(fnordmetric::WallClock::getUnixMillis() - time < 10); int64_t count; double fnord; assert(reader.readInteger(&count)); assert(reader.readFloat(&fnord)); assert(count == 42); assert(fnord == 23.5); called = true; }); assert(called); assert(cursor->next() == false); stream->appendRecord(42, 23.5); assert(cursor->next() == true); } int main() { testStreamKeyGeneration(); testAppendRecord(); printf("all tests passed! :)\n"); } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/streamref.cc000066400000000000000000000126651255644354400257750ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include "streamref.h" #include "database.h" #include "cursor.h" #include "../clock.h" #include "../fnv.h" namespace fnordmetric { namespace database { StreamRef::StreamRef( Database* backend, uint64_t stream_id, const std::string& stream_key) : backend_(backend), stream_id_(stream_id), stream_key_(stream_key), num_pages_(0) {} StreamRef::StreamRef( Database* backend, uint64_t stream_id, const std::string& stream_key, std::vector>&& pages) : backend_(backend), stream_id_(stream_id), stream_key_(stream_key), pages_(std::move(pages)), num_pages_(pages_.size()) {} StreamPosition StreamRef::appendRow(const RecordWriter& record) { const void* data; size_t size; record.toBytes(&data, &size); append_mutex_.lock(); uint64_t time = WallClock::getUnixMillis(); auto pos = appendRow(data, size, time); append_mutex_.unlock(); return pos; } StreamPosition StreamRef::appendRow( const RecordWriter& record, uint64_t time) { const void* data; size_t size; // FIXPAUL check that time goes forward record.toBytes(&data, &size); append_mutex_.lock(); auto pos = appendRow(data, size, time); append_mutex_.unlock(); return pos; } StreamPosition StreamRef::appendRow( const void* data, size_t size, uint64_t time) { size_t row_size = size + sizeof(RowHeader); assert(size > 0); pages_mutex_.lock(); if (num_pages_ == 0) { auto estimated_page_size = estimatePageSize(row_size, row_size); auto page = backend_->page_manager_->allocPage(estimated_page_size); auto alloc = std::shared_ptr(new PageAlloc(page, time, 0)); Log::PageAllocEntry log_entry; log_entry.page_offset = alloc->page_.offset; log_entry.page_size = alloc->page_.size; log_entry.page_first_row_time = time; log_entry.page_logical_offset = 0; log_entry.stream_id = stream_id_; backend_->log_->appendEntry(log_entry, stream_key_); pages_.push_back(std::move(alloc)); num_pages_++; } auto last_page = pages_.back(); if (last_page->used_ + row_size >= last_page->page_.size) { assert(last_page->num_rows_.load() > 0); auto last_page_avg_size = last_page->used_ / last_page->num_rows_; auto estimated_page_size = estimatePageSize(last_page_avg_size, row_size); auto page = backend_->page_manager_->allocPage(estimated_page_size); auto logical_offset = last_page->logical_offset_ + last_page->used_; auto alloc = std::shared_ptr( new PageAlloc(page, time, logical_offset)); // FIXPAUL: msync last page { Log::PageFinishEntry log_entry; log_entry.page_offset = last_page->page_.offset; log_entry.page_size = last_page->page_.size; log_entry.page_used = last_page->used_; log_entry.stream_id = stream_id_; backend_->log_->appendEntry(log_entry); } { Log::PageAllocEntry log_entry; log_entry.page_offset = alloc->page_.offset; log_entry.page_size = alloc->page_.size; log_entry.page_first_row_time = time; log_entry.page_logical_offset = alloc->logical_offset_; log_entry.stream_id = stream_id_; backend_->log_->appendEntry(log_entry); } pages_.push_back(std::move(alloc)); num_pages_++; } auto page = pages_.back(); pages_mutex_.unlock(); StreamPosition pos; pos.logical_offset = page->logical_offset_ + page->used_.load(); pos.next_offset = pos.logical_offset + row_size; pos.unix_millis = time; auto mmaped = backend_->page_manager_->getPage(page->page_); RowHeader* row = mmaped->structAt(page->used_.load()); row->time = time; row->size = size; memcpy(row->data, data, size); row->checksum = row->computeChecksum(); // FIXPAUL mem barrier page->used_ += row_size; page->num_rows_++; return pos; } std::unique_ptr StreamRef::getCursor() { return std::unique_ptr(new Cursor(this, backend_->page_manager_)); } void StreamRef::accessPages(std::function>&)> func) { pages_mutex_.lock(); func(pages_); pages_mutex_.unlock(); } uint64_t StreamRef::estimatePageSize( size_t last_page_avg_size, size_t row_size) const { assert(row_size < Database::kMaxPageSizeHard); // FIXPAUL size_t target = last_page_avg_size * Database::kTargetRowsPerPage; return std::max(std::min(target, Database::kMaxPageSizeSoft), row_size * 2); } uint32_t RowHeader::computeChecksum() { FNV fnv; return fnv.hash( (uint8_t *) (((char* ) this) + sizeof(checksum)), size + sizeof(RowHeader) - sizeof(checksum)); } PageAlloc::PageAlloc( const PageManager::Page& page, uint64_t time, uint64_t logical_offset) : page_(page), time_(time), logical_offset_(logical_offset), used_(0), num_rows_(0) { } bool StreamPosition::operator==(const StreamPosition& other) { return unix_millis == other.unix_millis && logical_offset == other.logical_offset && next_offset == other.next_offset; } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/streamref.h000066400000000000000000000072251255644354400256330ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FILEBACKEND_STREAMREF_H #define _FNORDMETRIC_FILEBACKEND_STREAMREF_H #include #include #include #include #include #include #include "pagemanager.h" #include "../record.h" namespace fnordmetric { namespace database { class Database; class Cursor; struct PageAlloc { PageAlloc( const PageManager::Page& page, uint64_t time, uint64_t logical_offset); const PageManager::Page page_; std::atomic_size_t used_; /* number of used bytes in the page */ std::atomic_size_t num_rows_; /* number of rows in the page */ const uint64_t time_; /* time of the first row in the page */ const uint64_t logical_offset_; /* logical offset of the page */ }; struct StreamPosition { uint64_t unix_millis; uint64_t logical_offset; uint64_t next_offset; bool operator==(const StreamPosition& other); }; struct __attribute__((__packed__)) RowHeader { uint32_t checksum; uint32_t size; uint64_t time; uint8_t data[]; uint32_t computeChecksum(); }; /** * A stream descriptor is a handle to a single stream. It can be used to * append rows to the stream and to receive a cursor for reading from the * stream. * * The appendRow and getCursor methods on the descriptor are threadsafe. */ class StreamRef { friend class DatabaseTest; friend class Cursor; public: explicit StreamRef( Database* backed, uint64_t stream_id, const std::string& stream_key); explicit StreamRef( Database* backed, uint64_t stream_id, const std::string& stream_key, std::vector>&& pages); StreamRef(const StreamRef& copy) = delete; StreamRef& operator=(const StreamRef& copy) = delete; /** * Append a new row to the very end of the opened stream. Returns the UTC * millisecond timestamp at which the row was inserted. * * This method is threadsafe. */ StreamPosition appendRow(const RecordWriter& row); /** * Append a new row to the very end of the opened stream with an explicit * insert time. This should only be used for importing data as the insert * time must be monotonically increasing. * * This method is threadsafe but you need to make sure that insert_time is * monotonically increasing. */ StreamPosition appendRow(const RecordWriter& row, uint64_t insert_time); /** * Return a cursor to this stream for reading. The initial position of the * cursor is undefined. * * This is threadsafe */ std::unique_ptr getCursor(); protected: /** * Access the StreamRefs internal page storage (do not call this method unless * you know what you are doing) */ void accessPages(std::function>&)> func); StreamPosition appendRow(const void* data, size_t size, uint64_t time); uint64_t estimatePageSize(size_t last_page_avg_size, size_t row_size) const; // this is suboptimal as it will force us to do random memory accesses when // trying to binary search over the pages first row times std::vector> pages_; std::atomic_size_t num_pages_; std::mutex pages_mutex_; std::mutex append_mutex_; Database* backend_; const uint64_t stream_id_; const std::string stream_key_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/transaction.cc000066400000000000000000000031551255644354400263240ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include "transaction.h" #include "collection.h" #include "documentref.h" #include "snapshot.h" namespace fnordmetric { Transaction::Transaction(std::unique_ptr) : running_(1) {} Transaction::~Transaction() { if (running_) { rollback(); } } DocumentRef* Transaction::createDocument(const DocumentKey& key ) { auto docref = new DocumentRef(key); dirty_documents_.push_back(docref); return docref; } bool Transaction::commit() { if (!running_) { return false; } //bool ret = collection_->commitTransaction(this); bool ret = true; for (const auto docref : dirty_documents_) { if (ret == false && docref->isDirty()) { docref->revert(); } delete docref; } // if (ret == true && sync_mode == SYNC) // collection_->sync(); // FIXPAUL MAX SYNC AGE! running_ = 0; return ret; } bool Transaction::rollback() { if (!running_) { return false; } for (const auto docref : dirty_documents_) { if (docref->isDirty()) { docref->revert(); } delete docref; } running_ = 0; return true; } const std::vector& Transaction::getDirtyDocuments() const { return dirty_documents_; } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/transaction.h000066400000000000000000000052751255644354400261730ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_TRANSACTION_H #define _FNORDMETRIC_TRANSACTION_H #include #include #include "collection.h" #include "documentkey.h" namespace fnordmetric { class DocumentRef; class Cursor; class Collection; class Snapshot; /** * A transaction object is not threadsafe! If you want to use a cursor from * multiple threads you must take care to synchronize access in such a way that * no two threads call any method on the cursor object at the same time! */ class Transaction { public: /** * Start a new transaction */ Transaction(std::unique_ptr snapshot); Transaction(const Transaction& copy) = delete; Transaction& operator=(const Transaction& copy) = delete; /** * A transaction auto-rollbacks on destruction if it was not comitted */ ~Transaction(); /** * Get the document with the specified document key * * The returned pointer is valid until the transaction is committed or rolled * back. */ DocumentRef* getDocument(const DocumentKey& key); /** * Create a new document * * The returned pointer is valid until the transaction is committed or rolled * back. */ DocumentRef* createDocument(const DocumentKey& key = DocumentKey(0)); /** * Get or create the document with the specified document key * * The returned pointer is valid until the transaction is committed or rolled * back. */ DocumentRef* getOrCreateDocument(const DocumentKey& key); /** * Commit this transaction. */ bool commit(); /** * Rollback this transaction. */ bool rollback(); /** * Return a cursor for the collection this transaction is running on and seek * to the first document with a key larger than or equal to the specified key. * * The returned cursor is only valid within this transaction until it is * committed or rolled back. */ std::unique_ptr getCursor(const DocumentKey& key); /** * Return a list of all documents that have been touched in this transaction. * * The returned pointers are only valid within this transaction until it is * committed or rolled back. */ const std::vector& getDirtyDocuments() const; protected: int running_; std::unique_ptr snapshot_; std::vector dirty_documents_; /* hrhr ;) */ }; } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/volume.cc000066400000000000000000000047001255644354400253030ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include using namespace fnord; namespace fnordmetric { namespace ffs { Volume::Volume( const std::shared_ptr& page_manager, int flags) : page_manager_(page_manager), flags_(flags) {} Volume::~Volume() { } std::unique_ptr Volume::openFile( const std::string& filename, uint64_t flags /* = MODE_CONSERVATIVE */) { Volume* ptr = nullptr; int open_flags = O_CREAT | O_RDWR; if ((flags & FILE_TRUNCATE) > 0) { open_flags |= O_TRUNC; } int fd = open(filename.c_str(), open_flags, S_IRUSR | S_IWUSR); if (fd < 0) { RAISE_ERRNO(kIOError, "open() failed"); } struct stat fd_stat; if (fstat(fd, &fd_stat) < 0) { RAISE_ERRNO(kIOError, "fstat() failed"); } off_t fd_len = lseek(fd, 0, SEEK_END); if (fd_len < 0) { RAISE_ERRNO(kIOError, "lseek() failed"); } if ((flags & FILE_AUTODELETE) > 0) { unlink(filename.c_str()); } std::shared_ptr page_manager( new io::MmapPageManager(fd, fd_len, fd_stat.st_blksize)); io::PageManager::Page header_page; if (fd_len == 0) { header_page = page_manager->allocPage(kMinReservedHeaderSize); } else { if (fd_len < kMinReservedHeaderSize) { RAISE(kIllegalFormatError, "invalid file\n"); } header_page = io::PageManager::Page(0, kMinReservedHeaderSize); } auto header_mmap = page_manager->getPage(header_page); auto file_header = header_mmap->structAt(0); if (fd_len == 0) { file_header->magic = kFileMagicBytes; file_header->version = kFileVersion; // FIXPAUL msync header } else { if (file_header->magic != kFileMagicBytes) { RAISE(kIllegalFormatError, "invalid file\n"); } if (file_header->version != kFileVersion) { RAISE(kVersionMismatchError, "invalid file version\n"); } } return std::unique_ptr(new Volume(page_manager, flags)); } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/volume.h000066400000000000000000000110221255644354400251400ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FFS_VOLUME_H_ #define _FNORDMETRIC_FFS_VOLUME_H_ #include #include #include #include #include #include #include "objectref.h" #include /** * A ffs volume stores an arbitrary number of 'objects'. Each object is * identified by a unique string key. The ffs volume itself only implements * simple inserting, deleting and accessing objects. * * Each object has an associated type that specifies the methods to manipulate * the type. * * eBNF: * * FILE ::= FILE_HEADER * *( PAGE ) * * PAGE_PTR ::= ; page offset in the file in bytes * ; size of the page in bytes * * PAGE ::= ; opaque byte string * * FILE_HEADER ::= <8 Bytes 0x17> ; magic bytes * ; version number * ; index type * PAGE_PTR ; pointer to index super page * PAGE_PTR ; pointer to first FREELIST_PAGE * */ using namespace fnord; namespace fnordmetric { namespace ffs { class Volume { public: enum kFlags { /** * Safety modes. The database will be consistent in all cases, but depending * on the safety mode you might loose more or less data. */ MODE_RELAXED = 0, MODE_CONSERVATIVE = 10, /* MSYNC_ASYNC | ENABLE_CHECKSUMS */ MODE_PARANOID = 12, /* MSYNC_SYNC | ENABLE_CHECKSUMS */ /** * Msync modes */ MYSNC_ASYNC = 2, MYSNC_SYNC = 4, /** * Verify checksums when scanning over rows. Slows down everthing by a bit * but protects against file corruption */ ENABLE_CHECKSUMS = 8, /** * Truncate the file when opening it */ FILE_TRUNCATE = 32, /** * Delete the file when the database is closed (the volume object is * destroyed) */ FILE_AUTODELETE = 64 }; struct __attribute__((__packed__)) PagePtr { uint64_t offset; uint16_t blks; }; struct __attribute__((__packed__)) ObjectPtr { PagePtr super_page; uint16_t objtype; }; struct __attribute__((__packed__)) FileHeader { uint64_t magic; uint32_t version; uint32_t block_size; ObjectPtr index; ObjectPtr freelist; }; Volume(const Volume& copy) = delete; Volume& operator=(const Volume& copy) = delete; ~Volume(); /** * Min. number of bytes to reserve for the file header */ static const uint64_t kMinReservedHeaderSize = 512; /** * File magic bytes */ static const uint64_t kFileMagicBytes = 0x1717171717171717; /** * File format version number */ static const uint64_t kFileVersion = 1; /** * Instantiate a new volume with a path to the file. */ static std::unique_ptr openFile( const std::string& filename, uint64_t flags = MODE_CONSERVATIVE); /** * Insert a new object. Will raise an exception if an object with the same * key already exists or is currently opened. */ template ObjectRef insert(const std::string& object_key); /** * Find an object. Will raise an exception if no object with the provided * key exists or the object is currently opened. */ template std::unique_ptr find(const std::string& object_key); /** * Find an object with the provided key or insert a new object. This will * raise an exception if an object with the key exists and is currently opened */ template std::unique_ptr findOrInsert(const std::string& object_key); /** * Returns true if an object with the provided key exists and false otherwise */ bool contains(const std::string& object_key); protected: Volume(const std::shared_ptr& page_manager, int flags); std::unordered_map live_objects_; std::mutex live_objects_mutex_; const std::shared_ptr page_manager_; uint64_t flags_; }; } } #include "volume_impl.h" #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/volume_impl.h000066400000000000000000000021151255644354400261640ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FFS_VOLUME_IMPL_H_ #define _FNORDMETRIC_FFS_VOLUME_IMPL_H_ #include namespace fnordmetric { namespace ffs { template ObjectRef Volume::insert(const std::string& object_key) { std::lock_guard lock(live_objects_mutex_); /* FIXPAUL ensure it doesn't contain the elem */ if (live_objects_.find(object_key) != live_objects_.end()) { RAISE( kIllegalStateError, "can't insert object because it already exists and is currently " "opened"); } auto obj = new ObjectType(page_manager_); live_objects_.emplace(object_key, obj); return ObjectRef(obj); } } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/ffs/volume_test.cc000066400000000000000000000014111255644354400263360ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include using namespace fnordmetric::ffs; UNIT_TEST(VolumeTest); TEST_CASE(VolumeTest, TestCowBlob, [] () { auto volume = Volume::openFile( "/tmp/__ffs_test_volume", Volume::FILE_TRUNCATE || Volume::FILE_AUTODELETE); auto blob = volume->insert("myblob"); }); fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/000077500000000000000000000000001255644354400236655ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/http_test.cc000066400000000000000000000067501255644354400262220ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include #include #include using namespace fnord::http; using fnord::http::HTTPInputStream; using fnord::http::HTTPOutputStream; using fnordmetric::util::StringInputStream; using fnordmetric::util::StringOutputStream; UNIT_TEST(HTTPTest); TEST_CASE(HTTPTest, ParseHTTP1dot0Request, [] () { auto req = "GET / HTTP/1.0\r\n" \ "\r\n"; StringInputStream is(req); HTTPInputStream http_is(&is); HTTPRequest request; request.readFromInputStream(&http_is); EXPECT_EQ(request.getMethod(), "GET"); EXPECT_EQ(request.getUrl(), "/"); EXPECT_EQ(request.getVersion(), "HTTP/1.0"); EXPECT_EQ(request.keepalive(), false); }); TEST_CASE(HTTPTest, ParseHTTP1dot0KeepaliveRequest, [] () { auto req = "GET / HTTP/1.0\r\n" \ "Connection: keep-alive\r\n" \ "\r\n"; StringInputStream is(req); HTTPInputStream http_is(&is); HTTPRequest request; request.readFromInputStream(&http_is); EXPECT_EQ(request.getMethod(), "GET"); EXPECT_EQ(request.getUrl(), "/"); EXPECT_EQ(request.getVersion(), "HTTP/1.0"); EXPECT_EQ(request.keepalive(), true); }); TEST_CASE(HTTPTest, ParseHTTP1dot1Request, [] () { auto req = "GET / HTTP/1.1\r\n" \ "\r\n"; StringInputStream is(req); HTTPInputStream http_is(&is); HTTPRequest request; request.readFromInputStream(&http_is); EXPECT_EQ(request.getMethod(), "GET"); EXPECT_EQ(request.getUrl(), "/"); EXPECT_EQ(request.getVersion(), "HTTP/1.1"); EXPECT_EQ(request.keepalive(), true); }); TEST_CASE(HTTPTest, PopulateHTTPResponseFromHTTP1dot1Request, [] () { auto req = "GET / HTTP/1.1\r\n" \ "\r\n"; StringInputStream is(req); HTTPInputStream http_is(&is); HTTPRequest request; request.readFromInputStream(&http_is); EXPECT(request.keepalive() == true); HTTPResponse response; response.populateFromRequest(request); EXPECT_EQ(response.getVersion(), "HTTP/1.1"); }); TEST_CASE(HTTPTest, PopulateHTTPResponseFromHTTP1dot0Request, [] () { auto req = "GET / HTTP/1.0\r\n" \ "\r\n"; StringInputStream is(req); HTTPInputStream http_is(&is); HTTPRequest request; request.readFromInputStream(&http_is); HTTPResponse response; response.populateFromRequest(request); EXPECT_EQ(response.getVersion(), "HTTP/1.0"); EXPECT_EQ(response.getHeader("Connection"), "close"); }); TEST_CASE(HTTPTest, PopulateHTTPResponseFromHTTP1dot0KeepaliveRequest, [] () { auto req = "GET / HTTP/1.0\r\n" \ "Connection: keep-alive\r\n" \ "\r\n"; StringInputStream is(req); HTTPInputStream http_is(&is); HTTPRequest request; request.readFromInputStream(&http_is); HTTPResponse response; response.populateFromRequest(request); EXPECT_EQ(response.getVersion(), "HTTP/1.0"); EXPECT_EQ(response.getHeader("Connection"), "keep-alive"); }); fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httphandler.h000066400000000000000000000013531255644354400263550ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_HTTPHANDLER_H #define _FNORDMETRIC_HTTPHANDLER_H #include #include namespace fnord { namespace http { class HTTPHandler { public: virtual ~HTTPHandler() {} virtual bool handleHTTPRequest( HTTPRequest* request, HTTPResponse* response) = 0; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpinputstream.cc000066400000000000000000000112541255644354400274520ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include using fnordmetric::util::InputStream; namespace fnord { namespace http { HTTPInputStream::HTTPInputStream( InputStream* input_stream) : input_(input_stream), state_(HTTP_STATE_METHOD) {} HTTPInputStream::~HTTPInputStream() { } InputStream* HTTPInputStream::getInputStream() const { return input_; } void HTTPInputStream::readStatusLine( std::string* method, std::string* url, std::string* version) { while (state_ == HTTP_STATE_METHOD) { readNextByte(method); } while (state_ == HTTP_STATE_URI) { readNextByte(url); } while (state_ == HTTP_STATE_VERSION) { readNextByte(version); } } void HTTPInputStream::readHeaders( std::vector>* target) { std::pair* cur_header; while (state_ == HTTP_STATE_HKEY || state_ == HTTP_STATE_HVAL) { if (target->size() == 0 || (state_ == HTTP_STATE_HKEY && target->back().second.size() > 0)) { target->emplace_back("", ""); } while (state_ == HTTP_STATE_HKEY) { readNextByte(&target->back().first); std::transform( target->back().first.begin(), target->back().first.end(), target->back().first.begin(), ::tolower); } while (state_ == HTTP_STATE_HVAL) { readNextByte(&target->back().second); } } } void HTTPInputStream::readNextByte(std::string* target) { char byte; if (!input_->readNextByte(&byte)) { RAISE(kRuntimeError, "unexpected EOF while reading HTTP header"); } switch (byte) { case '\r': return; case ' ': switch (state_) { case HTTP_STATE_METHOD: state_ = HTTP_STATE_URI; return; case HTTP_STATE_URI: state_ = HTTP_STATE_VERSION; return; default: break; } break; case '\n': switch (state_) { case HTTP_STATE_METHOD: RAISE(kRuntimeError, "invalid HTTP header"); case HTTP_STATE_URI: RAISE(kRuntimeError, "invalid HTTP header"); case HTTP_STATE_VERSION: state_ = HTTP_STATE_HKEY; return; case HTTP_STATE_HVAL: state_ = HTTP_STATE_HKEY; return; case HTTP_STATE_HKEY: state_ = HTTP_STATE_BODY; return; default: break; } break; case ':': if (state_ == HTTP_STATE_HKEY) { state_ = HTTP_STATE_HVAL; return; } break; } if (byte == ' ' && target->size() == 0) { return; } *target += byte; } /* int http_read_method(http_req_t* req, char* method, int len) { if (strncmp(method, "HEAD", len) == 0) req->method = HTTP_METHOD_HEAD; else if (strncmp(method, "GET", len) == 0) req->method = HTTP_METHOD_GET; else if (strncmp(method, "POST", len) == 0) req->method = HTTP_METHOD_POST; else return -1; req->state_ = HTTP_STATE_URI; return 0; } int http_read_uri(http_req_t* req, char* uri, int len) { char *end = uri + len; int n; if (len >= (int) sizeof(req->uri)) return -1; strncpy(req->uri, uri, len); req->uri[len] = 0; req->uri_len = len; for (; *end == ' ' || *end == '/' ||*end == '?' || *end == '&'; end--); for (n = 0; n < HTTP_URI_MAXARGS && uri < end; n++) { req->uri_argv[n] = uri; if (*uri == '&') *uri = '?'; for (uri++; uri < end && *uri != '/' && *uri != '?' && *uri != '&'; uri++); } req->uri_argv[n] = uri + 1; req->uri_argc = n; req->state_ = HTTP_STATE_VERSION; return 0; } int http_read_version(http_req_t* req, char* version, int len) { if (len < 8) return -1; if (strncmp(version + 5, "1.1", len - 5) == 0) req->keepalive = 1; else req->keepalive = 0; req->state_ = HTTP_STATE_HKEY; return 0; } void http_read_header(http_req_t* req, char* hkey, int hkey_len, char* hval, int hval_len) { //hkey[hkey_len] = 0; hval[hval_len] = 0; //printf("header: (%i) '%s' => (%i) '%s'\n", hkey_len, hkey, hval_len, hval); if (strncmp(hkey, "Connection", hkey_len) == 0) if (strncmp(hval, "Keep-Alive", hval_len) == 0) req->keepalive = 1; } */ } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpinputstream.h000066400000000000000000000031751255644354400273170ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef __FNORDMETRIC_HTTPINPUTSTREAM_H #define __FNORDMETRIC_HTTPINPUTSTREAM_H #include #include #include #include using fnordmetric::util::InputStream; namespace fnord { namespace http { class HTTPInputStream { public: enum kParserState { HTTP_STATE_METHOD, HTTP_STATE_URI, HTTP_STATE_VERSION, HTTP_STATE_HKEY, HTTP_STATE_HVAL, HTTP_STATE_BODY }; /** * @param input_stream the input stream -- does not transfer ownership */ HTTPInputStream(InputStream* input_stream); ~HTTPInputStream(); /** * Read the http status line. Throws a RuntimeException for invalid requests. * Might throw a io::WouldBlockException if the InputStream is non blocking. */ void readStatusLine( std::string* method, std::string* url, std::string* version); /** * Read the http headers. Throws a RuntimeException for invalid requests. * Might throw a io::WouldBlockException if the InputStream is non blocking */ void readHeaders( std::vector>* target); InputStream* getInputStream() const; protected: void readNextByte(std::string* target); InputStream* input_; kParserState state_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpmessage.cc000066400000000000000000000045421255644354400265250ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include using fnordmetric::util::InputStream; using fnordmetric::util::OutputStream; using fnordmetric::util::StringInputStream; using fnordmetric::util::StringOutputStream; namespace fnord { namespace http { std::string HTTPMessage::kEmptyHeader = ""; const std::string& HTTPMessage::getVersion() const { return version_; } void HTTPMessage::setVersion(const std::string& version) { version_ = version; } const std::vector>& HTTPMessage::getHeaders() const { return headers_; } const std::string& HTTPMessage::getHeader(const std::string& key) const { auto key_low = key; std::transform(key_low.begin(), key_low.end(), key_low.begin(), ::tolower); for (const auto& header : headers_) { if (header.first == key_low) { return header.second; } } return kEmptyHeader; } void HTTPMessage::addHeader(const std::string& key, const std::string& value) { auto key_low = key; std::transform(key_low.begin(), key_low.end(), key_low.begin(), ::tolower); headers_.emplace_back(key_low, value); } void HTTPMessage::setHeader(const std::string& key, const std::string& value) { auto key_low = key; std::transform(key_low.begin(), key_low.end(), key_low.begin(), ::tolower); for (auto& header : headers_) { if (header.first == key_low) { header.second = value; return; } } headers_.emplace_back(key_low, value); } const std::string& HTTPMessage::getBody() const { return body_; } void HTTPMessage::addBody(const std::string& body) { body_ = body; setHeader("Content-Length", std::to_string(body.size())); } void HTTPMessage::clearBody() { body_.clear(); // FIXPAUL remove Content-Length header } std::unique_ptr HTTPMessage::getBodyInputStream() const { return StringInputStream::fromString(body_); } std::unique_ptr HTTPMessage::getBodyOutputStream() { return StringOutputStream::fromString(&body_); } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpmessage.h000066400000000000000000000030121255644354400263560ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_WEB_HTTPMESSAGE_H #define _FNORDMETRIC_WEB_HTTPMESSAGE_H #include #include #include #include #include using fnordmetric::util::InputStream; using fnordmetric::util::OutputStream; namespace fnord { namespace http { class HTTPMessage { public: HTTPMessage() {} virtual ~HTTPMessage() {} const std::string& getVersion() const; void setVersion(const std::string& version); const std::vector>& getHeaders() const; const std::string& getHeader(const std::string& key) const; void addHeader(const std::string& key, const std::string& value); void setHeader(const std::string& key, const std::string& value); const std::string& getBody() const; void addBody(const std::string& body); void clearBody(); std::unique_ptr getBodyInputStream() const; std::unique_ptr getBodyOutputStream(); protected: std::string version_; static std::string kEmptyHeader; std::vector> headers_; std::string body_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpoutputstream.cc000066400000000000000000000022221255644354400276460ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include using fnordmetric::util::OutputStream; namespace fnord { namespace http { HTTPOutputStream::HTTPOutputStream( OutputStream* output_stream) : output_(output_stream) {} void HTTPOutputStream::writeStatusLine( const std::string& version, int status_code, const std::string& status) { output_->printf("%s %i %s\r\n", version.c_str(), status_code, status.c_str()); } void HTTPOutputStream::writeHeaders( const std::vector>& headers) { for (const auto& header : headers) { output_->printf("%s: %s\r\n", header.first.c_str(), header.second.c_str()); } output_->printf("\r\n"); } OutputStream* HTTPOutputStream::getOutputStream() const { return output_; } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpoutputstream.h000066400000000000000000000021401255644354400275070ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef __FNORDMETRIC_HTTPOUTPUTSTREAM_H #define __FNORDMETRIC_HTTPOUTPUTSTREAM_H #include #include #include #include using fnordmetric::util::OutputStream; namespace fnord { namespace http { class HTTPOutputStream { public: /** * @param output_stream the output stream -- does not transfer ownership */ HTTPOutputStream(OutputStream* output_stream); void writeStatusLine( const std::string& version, int status_code, const std::string& status); void writeHeaders( const std::vector>& headers); OutputStream* getOutputStream() const; protected: OutputStream* output_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httprequest.cc000066400000000000000000000037201255644354400265660ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include namespace fnord { namespace http { HTTPRequest::HTTPRequest() {} HTTPRequest::HTTPRequest( const std::string& method, const std::string& url) : method_(method), url_(url) {} const std::string& HTTPRequest::getMethod() const { return method_; } // FIXPAUL sloooow HTTPRequest::kMethod HTTPRequest::method() const { if (method_ == "CONNECT") { return M_CONNECT; } if (method_ == "DELETE") { return M_DELETE; } if (method_ == "GET") { return M_GET; } if (method_ == "HEAD") { return M_HEAD; } if (method_ == "OPTIONS") { return M_OPTIONS; } if (method_ == "POST") { return M_POST; } if (method_ == "PUT") { return M_PUT; } if (method_ == "TRACE") { return M_TRACE; } return M_INVALID; } const std::string& HTTPRequest::getUrl() const { return url_; } const bool HTTPRequest::keepalive() const { if (getVersion() == "HTTP/1.1") { return true; } if (getHeader("Connection") == "keep-alive") { return true; } return false; } void HTTPRequest::readFromInputStream(HTTPInputStream* input) { input->readStatusLine(&method_, &url_, &version_); input->readHeaders(&headers_); const auto& content_length_header = getHeader("Content-Length"); int content_length = 0; if (content_length_header.size() > 0) { try { content_length = std::stoi(content_length_header); } catch (std::exception e){ /* invalid content length */ } } if (content_length > 0) { input->getInputStream()->readNextBytes(&body_, content_length); } } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httprequest.h000066400000000000000000000021541255644354400264300ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_WEB_HTTPREQUEST_H #define _FNORDMETRIC_WEB_HTTPREQUEST_H #include #include namespace fnord { namespace http { class HTTPInputStream; class HTTPRequest : public HTTPMessage { public: enum kMethod { M_CONNECT, M_DELETE, M_GET, M_HEAD, M_OPTIONS, M_POST, M_PUT, M_TRACE, M_INVALID }; HTTPRequest(); HTTPRequest( const std::string& method, const std::string& url); void readFromInputStream(HTTPInputStream* input); const std::string& getMethod() const; kMethod method() const; const std::string& getUrl() const; const bool keepalive() const; protected: std::string method_; std::string url_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpresponse.cc000066400000000000000000000025331255644354400267350ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include namespace fnord { namespace http { HTTPResponse::HTTPResponse() { setStatus(kStatusNotFound); } void HTTPResponse::setStatus(int status_code, const std::string& status) { status_code_ = status_code; status_ = status; } void HTTPResponse::setStatus(const HTTPStatus& status) { setStatus(status.code, status.name); } void HTTPResponse::writeToOutputStream(HTTPOutputStream* output) { setHeader("Content-Length", std::to_string(body_.size())); output->writeStatusLine(version_, status_code_, status_); output->writeHeaders(headers_); output->getOutputStream()->write(body_); } void HTTPResponse::populateFromRequest(const HTTPRequest& request) { setVersion(request.getVersion()); if (request.getVersion() == "HTTP/1.0") { if (request.keepalive()) { addHeader("Connection", "keep-alive"); } else { addHeader("Connection", "close"); } } } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpresponse.h000066400000000000000000000020641255644354400265760ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_WEB_HTTPRESPONSE_H #define _FNORDMETRIC_WEB_HTTPRESPONSE_H #include #include #include #include namespace fnord { namespace http { class HTTPOutputStream; class HTTPResponse : public HTTPMessage { public: HTTPResponse(); void setStatus(int status_code, const std::string& status); void setStatus(const HTTPStatus& status); void writeToOutputStream(HTTPOutputStream* output); void populateFromRequest(const HTTPRequest& request); int statusCode() const { return status_code_; } protected: int status_code_; std::string status_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpserver.cc000066400000000000000000000076641255644354400264170ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include #include #include #include #include #include using fnordmetric::util::FileInputStream; using fnordmetric::util::FileOutputStream; using fnordmetric::util::RuntimeException; namespace fnord { namespace http { HTTPServer::HTTPServer( TaskScheduler* server_scheduler, TaskScheduler* request_scheduler) : server_scheduler_(server_scheduler), request_scheduler_(request_scheduler) {} void HTTPServer::addHandler(std::unique_ptr handler) { handlers_.emplace_back(std::move(handler)); } void HTTPServer::listen(int port) { ssock_ = socket(AF_INET, SOCK_STREAM, 0); if (ssock_ == 0) { RAISE(kIOError, "create socket() failed"); } int opt = 1; if (setsockopt(ssock_, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) < 0) { RAISE_ERRNO(kIOError, "setsockopt(SO_REUSEADDR) failed"); return; } struct sockaddr_in addr; memset((char *) &addr, 0, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = htonl(INADDR_ANY); addr.sin_port = htons(port); if (::bind(ssock_, (struct sockaddr *) &addr, sizeof(addr)) < 0) { RAISE_ERRNO(kIOError, "bind() failed"); } if (::listen(ssock_, 1024) == -1) { RAISE_ERRNO(kIOError, "listen() failed"); return; } accept(); //server_scheduler_->run( // thread::Task::create(std::bind(&HTTPServer::accept, this))); } void HTTPServer::accept() { for (;;) { int conn_fd = ::accept(ssock_, NULL, NULL); if (conn_fd < 0) { RAISE_ERRNO(kIOError, "accept() failed"); } request_scheduler_->run( thread::Task::create( std::bind(&HTTPServer::handleConnection, this, conn_fd))); } } void HTTPServer::handleConnection(int fd) const { if (fnordmetric::env()->verbose()) { fnordmetric::env()->logger()->printf( "DEBUG", "New HTTP connection on fd %i", fd); } bool keepalive = false; FileInputStream input_stream(fd, true); FileOutputStream output_stream(fd, true); do { HTTPRequest request; HTTPResponse response; keepalive = false; try { HTTPInputStream http_input_stream(&input_stream); request.readFromInputStream(&http_input_stream); if (request.keepalive()) { keepalive = true; } response.populateFromRequest(request); } catch (RuntimeException e) { keepalive = false; response.setStatus(kStatusNotFound); response.addHeader("Connection", "close"); response.addBody("Bad Request"); e.debugPrint(); // FIXPAUL } bool handled = false; try { for (const auto& handler : handlers_) { if (handler->handleHTTPRequest(&request, &response)) { handled = true; break; } } } catch (RuntimeException e) { keepalive = false; response.setStatus(kStatusInternalServerError); response.addHeader("Connection", "close"); response.addBody("Internal Server Error"); e.debugPrint(); // FIXPAUL } if (!handled) { response.setStatus(kStatusNotFound); response.addBody("Not Found"); } HTTPOutputStream http_output_stream(&output_stream); response.writeToOutputStream(&http_output_stream); } while (keepalive); } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/httpserver.h000066400000000000000000000021511255644354400262430ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_WEB_HTTPSERVER_H #define _FNORDMETRIC_WEB_HTTPSERVER_H #include #include #include #include #include namespace fnord { namespace http { using fnord::thread::TaskScheduler; class HTTPServer { public: HTTPServer( TaskScheduler* server_scheduler, TaskScheduler* request_scheduler); void addHandler(std::unique_ptr handler); void listen(int port); protected: void accept(); void handleConnection(int fd) const; std::vector> handlers_; TaskScheduler* server_scheduler_; TaskScheduler* request_scheduler_; int ssock_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/http/status.h000066400000000000000000000017771255644354400253750ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_HTTP_STATUSES_H #define _FNORDMETRIC_HTTP_STATUSES_H namespace fnord { namespace http { struct HTTPStatus { HTTPStatus(int code_, const char* name_) : code(code_), name(name_) {} int code; const char* name; }; const HTTPStatus kStatusOK(200, "OK"); const HTTPStatus kStatusCreated(201, "Created"); const HTTPStatus kStatusBadRequest(404, "Bad request"); const HTTPStatus kStatusNotFound(404, "Not found"); const HTTPStatus kStatusMovedPermanently(301, "Moved permanently"); const HTTPStatus kStatusFound(302, "Found"); const HTTPStatus kStatusInternalServerError(500, "InternalServerError"); } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/000077500000000000000000000000001255644354400233155ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/acceptor.cc000066400000000000000000000055311255644354400254300ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include #include #include "acceptor.h" #include "eventloop.h" #include namespace fnordmetric { namespace ev { Acceptor::Acceptor(EventLoop* ev_loop) : ev_loop_(ev_loop) {} void Acceptor::listen(int port, CallbackInterface* handler) { struct sockaddr_in server_addr; server_addr.sin_family = AF_INET; server_addr.sin_addr.s_addr = htonl(INADDR_ANY); server_addr.sin_port = htons(port); int ssock = socket(AF_INET, SOCK_STREAM, 0); if (ssock == -1) { RAISE_ERRNO(kIOError, "socket() creation failed"); return; } int opt = 1; if (setsockopt(ssock, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) < 0) { RAISE_ERRNO(kIOError, "setsockopt(SO_REUSEADDR) failed"); return; } #ifdef SOL_TCP if (setsockopt(ssock, SOL_TCP, TCP_QUICKACK, &opt, sizeof(opt)) < 0) { RAISE_ERRNO(kIOError, "setsockopt(TCP_QUICKACK) failed"); return; } if (setsockopt(ssock, SOL_TCP, TCP_DEFER_ACCEPT, &opt, sizeof(opt)) < 0) { RAISE_ERRNO(kIOError, "setsockopt(TCP_DEFER_ACCEPT) failed"); return; } #endif if (bind( ssock, (struct sockaddr *) &server_addr, sizeof(server_addr)) == -1) { RAISE_ERRNO(kIOError, "bind() failed"); return; } if (::listen(ssock, 1024) == -1) { RAISE_ERRNO(kIOError, "listen() failed"); return; } int flags = fcntl(ssock, F_GETFL, 0); flags = flags | O_NONBLOCK; if (fcntl(ssock, F_SETFL, flags) != 0) { RAISE(kIOError, "fnctl() failed"); } auto handler_ptr = new HandlerRef(ssock, handler); ev_loop_->watch(ssock, EventLoop::EV_READABLE, handler_ptr); handlers_.emplace_back(handler_ptr); } Acceptor::HandlerRef::HandlerRef( int ssock, Acceptor::CallbackInterface* handler) : ssock_(ssock), handler_(handler) {} Acceptor::HandlerRef::~HandlerRef() { close(ssock_); } void Acceptor::HandlerRef::onEvent( EventLoop* loop, int fd, EventLoop::kInterestType ev) { int conn_fd = accept(ssock_, NULL, NULL); if (conn_fd == -1) { RAISE_ERRNO(kIOError, "accept() failed"); return; } int flags = fcntl(conn_fd, F_GETFL, 0); flags &= ~O_NONBLOCK; if (fcntl(conn_fd, F_SETFL, flags) != 0) { RAISE(kIOError, "fnctl() failed"); } handler_->onConnection(conn_fd); loop->watch(fd, EventLoop::EV_READABLE, this); } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/acceptor.h000066400000000000000000000022261255644354400252700ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_EV_ACCEPTOR_H #define _FNORDMETRIC_EV_ACCEPTOR_H #include #include #include "eventloop.h" namespace fnordmetric { namespace ev { class Acceptor { public: class CallbackInterface { public: virtual void onConnection(int fd) = 0; }; Acceptor(EventLoop* ev_loop); void listen(int port, CallbackInterface* handler); protected: class HandlerRef : public EventLoop::CallbackInterface { public: HandlerRef(int ssock, Acceptor::CallbackInterface* handler); ~HandlerRef(); void onEvent(EventLoop* loop, int fd, EventLoop::kInterestType ev) override; protected: int ssock_; Acceptor::CallbackInterface* handler_; }; EventLoop* ev_loop_; std::vector> handlers_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/eventloop.cc000066400000000000000000000044351255644354400256450ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include "eventloop.h" #include namespace fnordmetric { namespace ev { EventLoop::EventLoop() : max_fd_(1), running_(true) { auto callbacks_size = sizeof(CallbackInterface*) * FD_SETSIZE; callbacks_ = static_cast(malloc(callbacks_size)); if (callbacks_ == nullptr) { RAISE(kMallocError, "malloc failed"); } memset(callbacks_, 0, callbacks_size); FD_ZERO(&op_read_); FD_ZERO(&op_write_); } void EventLoop::watch( int fd, kInterestType interest, CallbackInterface* callback) { if (fd >= FD_SETSIZE) { RAISE( kIOError, "fd is too big: %i, max is %i\n", fd, FD_SETSIZE - 1); return; } if (fd > max_fd_) { max_fd_ = fd; } callbacks_[fd] = callback; switch (interest) { case EV_READABLE: FD_SET(fd, &op_read_); break; case EV_WRITEABLE: FD_SET(fd, &op_write_); break; } } void EventLoop::unwatch(int fd, int flags) { callbacks_[fd] = nullptr; FD_CLR(fd, &op_read_); FD_CLR(fd, &op_write_); } int EventLoop::poll() { fd_set op_read, op_write; memcpy(&op_read, &op_read_, sizeof(fd_set)); memcpy(&op_write, &op_write_, sizeof(fd_set)); int res = select(max_fd_ + 1, &op_read, &op_write, NULL, NULL); if (res == 0) { return 0; } if (res == -1) { RAISE_ERRNO(kIOError, "select() failed"); return -1; } int num_events = 0; for (int fd = 0; fd <= max_fd_; fd++) { if (FD_ISSET(fd, &op_read)) { FD_CLR(fd, &op_read_); callbacks_[fd]->onEvent(this, fd, EV_READABLE); num_events++; } else if (FD_ISSET(fd, &op_write)) { FD_CLR(fd, &op_write_); callbacks_[fd]->onEvent(this, fd, EV_READABLE); num_events++; } } return num_events; } void EventLoop::loop() { while (running_) { poll(); } } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/eventloop.h000066400000000000000000000020511255644354400254770ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef FNORDMETRIC_EV_EVENTLOOP_H #define FNORDMETRIC_EV_EVENTLOOP_H #include namespace fnordmetric { namespace ev { class EventLoop { public: enum kInterestType { EV_READABLE = 1, EV_WRITEABLE = 2 }; class CallbackInterface { public: virtual void onEvent( EventLoop* loop, int fd, kInterestType interest) = 0; }; EventLoop(); void watch(int fd, kInterestType interest, CallbackInterface* callback); void unwatch(int fd, int flags); int poll(); void loop(); protected: fd_set op_read_; fd_set op_write_; int max_fd_; CallbackInterface** callbacks_; volatile bool running_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/file.cc000066400000000000000000000053531255644354400245510ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include namespace fnord { namespace io { File File::openFile( const std::string& filename, int flags) { int open_flags = 0; switch (flags & (O_READ | O_WRITE)) { case O_READ: open_flags = O_RDONLY; break; case O_WRITE: open_flags = O_WRONLY; break; case (O_READ | O_WRITE): open_flags = O_RDWR; break; default: RAISE( kIllegalArgumentError, "openFile('%s'): at least one of O_READ | O_WRITE must be set", filename.c_str()); } if ((flags & O_CREATEOROPEN) > 0) { open_flags |= O_CREAT; } else if ((flags & O_CREATE) > 0) { open_flags |= O_CREAT; open_flags |= O_EXCL; } if ((flags & O_TRUNCATE) > 0) { open_flags |= O_TRUNC; } int fd = open(filename.c_str(), open_flags, S_IRUSR | S_IWUSR); if (fd < 0) { RAISE_ERRNO(kIOError, "openFile('%s'): open() failed", filename.c_str()); } if ((flags & O_AUTODELETE) > 0) { unlink(filename.c_str()); } return File(fd, flags); } File::File(int fd, int flags) : fd_(fd), flags_(flags) {} File::~File() { if (fd_ >= 0) { close(fd_); } } File::File(File&& other) : fd_(other.fd_), flags_(other.flags_) { other.fd_ = -1; } int File::fd() const { return fd_; } bool File::isWritable() const { return (flags_ & O_WRITE) == O_WRITE; } size_t File::size() const { struct stat fd_stat; if (fstat(fd_, &fd_stat) < 0) { RAISE_ERRNO(kIOError, "fstat(%i) failed", fd_); } return fd_stat.st_size; } void File::seekTo(size_t pos) { if (lseek(fd_, pos, SEEK_SET) < 0) { RAISE_ERRNO(kIOError, "lseek(%i) failed", fd_); } } size_t File::read(void* buf, size_t buf_len) { int res = ::read(fd_, buf, buf_len); if (res < 0) { RAISE_ERRNO(kIOError, "read(%i) failed", fd_); } return res; } size_t File::read(util::Buffer* buf) { return read(buf->data(), buf->size()); } File File::clone() const { int new_fd = dup(fd_); if (new_fd < 0) { RAISE_ERRNO(kIOError, "dup(%i) failed", fd_); } return File(new_fd, flags_); } void File::truncate(size_t new_size) { if (ftruncate(fd_, new_size) < 0) { RAISE_ERRNO(kIOError, "ftruncate(%i) failed", fd_); } } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/file.h000066400000000000000000000035011255644354400244040ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORD_IO_FILE_H_ #define _FNORD_IO_FILE_H_ #include #include #include namespace fnord { namespace util { class Buffer; } namespace io { #undef O_APPEND class File { public: enum kOpenFlags { /** * Open in read/write mode. At least one must be set */ O_READ = 1, O_WRITE = 2, /** * Create the file if it doesn't exist and error if it exists */ O_CREATE = 4, /** * Create the file if it doesn't exist and open it if it exists */ O_CREATEOROPEN = 8, /** * Append to the file */ O_APPEND = 16, /* * Truncate the file when opening it */ O_TRUNCATE = 32, /** * Unlink the file after opening */ O_AUTODELETE = 64, /** * Do not set the close on exec flag */ O_ALLOWFORK = 128 }; /** * Open a file * * @param filename the filename/path * @param flags flags from kOpenFlags */ static File openFile( const std::string& filename, int flags); File(File&& move); File(const File& copy) = delete; ~File(); File& operator=(const File& copy) = delete; void seekTo(size_t pos); size_t read(void* buf, size_t buf_len); size_t read(util::Buffer* buf); int fd() const; size_t size() const; File clone() const; void truncate(size_t new_size); bool isWritable() const; protected: File(int fd, int flags); int fd_; int flags_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/filerepository.cc000066400000000000000000000024511255644354400267050ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include namespace fnord { namespace io { FileRepository::FileRepository( const std::string& basedir) : basedir_(basedir) {} FileRepository::FileRef FileRepository::createFile() const { FileRef fileref; fileref.logical_filename = util::Random::alphanumericString(32); fileref.absolute_path = FileUtil::joinPaths( basedir_, fileref.logical_filename); return fileref; } void FileRepository::listFiles( std::function callback) const { FileUtil::ls(basedir_, [&] (const std::string& filename) -> bool { auto absolute_path = FileUtil::joinPaths(basedir_, filename); return callback(absolute_path); }); } void FileRepository::deleteAllFiles() { listFiles([] (const std::string& filename) -> bool { FileUtil::rm(filename); return true; }); } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/filerepository.h000066400000000000000000000017011255644354400265440ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORD_IO_FILEREPOSITORY_H_ #define _FNORD_IO_FILEREPOSITORY_H_ #include #include #include namespace fnord { namespace io { class FileRepository { public: struct FileRef { std::string logical_filename; std::string absolute_path; }; FileRepository(const std::string& basedir); /** * Create and return a new file */ FileRef createFile() const; void listFiles( std::function callback) const; void deleteAllFiles(); protected: std::string basedir_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/fileutil.cc000066400000000000000000000067361255644354400254550ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include #include namespace fnord { namespace io { void FileUtil::mkdir(const std::string& dirname) { if (::mkdir(dirname.c_str(), S_IRWXU) != 0) { RAISE_ERRNO(kIOError, "mkdir('%s') failed", dirname.c_str()); } } bool FileUtil::exists(const std::string& filename) { struct stat fstat; if (stat(filename.c_str(), &fstat) < 0) { if (errno == ENOENT) { return false; } RAISE_ERRNO(kIOError, "fstat('%s') failed", filename.c_str()); } return true; } bool FileUtil::isDirectory(const std::string& filename) { struct stat fstat; if (stat(filename.c_str(), &fstat) < 0) { RAISE_ERRNO(kIOError, "fstat('%s') failed", filename.c_str()); } return S_ISDIR(fstat.st_mode); } /* The mkdir_p method was adapted from bash 4.1 */ void FileUtil::mkdir_p(const std::string& dirname) { char const* begin = dirname.c_str(); char const* cur = begin; if (exists(dirname)) { if (isDirectory(dirname)) { return; } else { RAISE( kIOError, "file '%s' exists but is not a directory", dirname.c_str()); } } for (cur = begin; *cur == '/'; ++cur); while ((cur = strchr(cur, '/'))) { std::string path(begin, cur); cur++; if (exists(path)) { if (isDirectory(path)) { continue; } else { RAISE( kIOError, "file '%s' exists but is not a directory", path.c_str()); } } mkdir(path); } mkdir(dirname); } std::string FileUtil::joinPaths(const std::string& p1, const std::string p2) { auto p1_stripped = p1; util::StringUtil::stripTrailingSlashes(&p1_stripped); auto p2_stripped = p2; util::StringUtil::stripTrailingSlashes(&p2_stripped); return p1_stripped + "/" + p2_stripped; } void FileUtil::ls( const std::string& dirname, std::function callback) { if (exists(dirname)) { if (!isDirectory(dirname)) { RAISE( kIOError, "file '%s' exists but is not a directory", dirname.c_str()); } } else { RAISE( kIOError, "file '%s' does not exist", dirname.c_str()); } auto dir = opendir(dirname.c_str()); if (dir == nullptr) { RAISE_ERRNO("opendir(%s) failed", dirname.c_str()); } struct dirent* entry; while ((entry = readdir(dir)) != NULL) { #if defined(__APPLE__) size_t namlen = entry->d_namlen; #else size_t namlen = strlen(entry->d_name); #endif if (namlen < 1 || *entry->d_name == '.') { continue; } if (!callback(std::string(entry->d_name, namlen))) { break; } } closedir(dir); } void FileUtil::rm(const std::string& filename) { unlink(filename.c_str()); } void FileUtil::truncate(const std::string& filename, size_t new_size) { if (::truncate(filename.c_str(), new_size) < 0) { RAISE_ERRNO(kIOError, "truncate(%s) failed", filename.c_str()); } } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/fileutil.h000066400000000000000000000026621255644354400253110ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_UTIL_FILE_H_ #define _FNORDMETRIC_UTIL_FILE_H_ #include #include #include #include namespace fnord { namespace io { class FileUtil { public: /** * Create a new directory */ static void mkdir(const std::string& dirname); /** * Create one or more directories recursively */ static void mkdir_p(const std::string& dirname); /** * Check if a file exists */ static bool exists(const std::string& dirname); /** * Check if a file exists and is a directory */ static bool isDirectory(const std::string& dirname); /** * Join two paths */ static std::string joinPaths(const std::string& p1, const std::string p2); /** * List files in a directory */ static void ls( const std::string& dirname, std::function callback); /** * Delete a file */ static void rm(const std::string& filename); /** * Truncate a file */ static void truncate(const std::string& filename, size_t size); }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/mmappedfile.cc000066400000000000000000000021101255644354400261010ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include namespace fnord { namespace io { MmappedFile::MmappedFile(File&& file) { File local_file = std::move(file); size_ = local_file.size(); is_writable_ = local_file.isWritable(); if (size_ == 0) { RAISE(kIllegalArgumentError, "can't mmap() empty file"); } data_ = mmap( nullptr, size_, is_writable_ ? PROT_WRITE | PROT_READ : PROT_READ, MAP_SHARED, local_file.fd(), 0); if (data_ == MAP_FAILED) { RAISE_ERRNO(kIOError, "mmap() failed"); } } MmappedFile::~MmappedFile() { munmap(data_, size_); } bool MmappedFile::isWritable() const { return is_writable_; } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/mmappedfile.h000066400000000000000000000024641255644354400257570ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORD_IO_MMAPPED_FILE_H_ #define _FNORD_IO_MMAPPED_FILE_H_ #include #include #include #include #include namespace fnord { namespace io { class MmappedFile { public: MmappedFile() = delete; MmappedFile(File&& file); MmappedFile(const MmappedFile& copy) = delete; MmappedFile& operator=(const MmappedFile& copy) = delete; ~MmappedFile(); inline void* data() const { return data_; } inline void* ptr() const { return data_; } inline size_t size() const { return size_; } template inline T* structAt(size_t pos) const { #ifndef NDEBUG if (pos >= size_) { abort(); RAISE(kIndexError, "position out of bounds"); } #endif return (T*) (((char *) data_) + pos); } bool isWritable() const; protected: bool is_writable_; void* data_; size_t size_; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/pagemanager.cc000066400000000000000000000153031255644354400260750ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include "pagemanager.h" #include #include #include namespace fnord { namespace io { PageManager::PageManager( size_t block_size, size_t end_pos /* = 0 */) : end_pos_(end_pos), block_size_(block_size) {} //PageManager::PageManager(size_t block_size, const LogSnapshot& log_snapshot) : // block_size_(block_size), // end_pos_(log_snapshot.last_used_byte) {} PageManager::PageManager(const PageManager&& move) : end_pos_(move.end_pos_), block_size_(move.block_size_), freelist_(std::move(move.freelist_)) {} PageManager::Page PageManager::allocPage(size_t min_size) { PageManager::Page page; /* align the request size to the next largest block boundary */ uint64_t min_size_aligned = ((min_size + block_size_ - 1) / block_size_) * block_size_; mutex_.lock(); if (!findFreePage(min_size_aligned, &page)) { page.offset = end_pos_; page.size = min_size_aligned; end_pos_ += page.size; } mutex_.unlock(); return page; } // FIXPAUL: proper freelist implementation void PageManager::freePage(const PageManager::Page& page) { mutex_.lock(); freelist_.emplace_back(std::make_pair(page.size, page.offset)); mutex_.unlock(); } // FIXPAUL: proper freelist implementation bool PageManager::findFreePage(size_t min_size, Page* destination) { for (auto iter = freelist_.begin(); iter != freelist_.end(); ++iter) { if (std::get<0>(*iter) >= min_size) { destination->offset = std::get<1>(*iter); destination->size = std::get<0>(*iter); //destination->used = 0; freelist_.erase(iter); return true; } } return false; } PageManager::Page::Page(uint64_t offset_, uint64_t size_) : offset(offset_), size(size_) {} PageManager::Page::Page() : offset(0), size(0) {} PageManager::PageRef::PageRef(const PageManager::Page& page) : page_(page) {} void* PageManager::PageRef::operator*() const { return getPtr(); } PageManager::PageRef::~PageRef() {}; MmapPageManager::MmapPageManager( const std::string& filename, size_t file_size) : PageManager(1, file_size), filename_(filename), file_size_(file_size), used_bytes_(file_size), current_mapping_(nullptr) { sys_page_size_ = sysconf(_SC_PAGESIZE); } MmapPageManager::MmapPageManager(MmapPageManager&& move) : PageManager(std::move(move)), filename_(move.filename_), file_size_(move.file_size_), used_bytes_(move.used_bytes_), current_mapping_(move.current_mapping_), sys_page_size_(move.sys_page_size_) { move.file_size_ = 0; move.used_bytes_ = 0; move.current_mapping_ = nullptr; } MmapPageManager::~MmapPageManager() { if (current_mapping_ != nullptr) { current_mapping_->decrRefs(); } } void MmapPageManager::shrinkFile() { FileUtil::truncate(filename_, used_bytes_); file_size_ = used_bytes_; } std::unique_ptr MmapPageManager::getPage( const PageManager::Page& page) { return getPageImpl(page, true); } std::unique_ptr MmapPageManager::getPage( const PageManager::Page& page, kNoPadding no_padding) { return getPageImpl(page, false); } std::unique_ptr MmapPageManager::getPageImpl( const PageManager::Page& page, bool allow_padding) { uint64_t last_byte = page.offset + page.size; mmap_mutex_.lock(); if (last_byte > used_bytes_) { used_bytes_ = last_byte; } if (last_byte > file_size_) { size_t new_size; if (allow_padding) { auto mmap_block_size = sys_page_size_ * kMmapSizeMultiplier; new_size = ((last_byte / mmap_block_size) + 1) * mmap_block_size; } else { new_size = last_byte; } if (fnordmetric::env()->verbose()) { fnordmetric::env()->logger()->printf( "DEBUG", "Truncating file %s to %lu bytes", filename_.c_str(), (long unsigned) new_size); } FileUtil::truncate(filename_, new_size); file_size_ = new_size; } auto page_ref = new MmappedPageRef( page, getMmappedFile(last_byte), sys_page_size_); mmap_mutex_.unlock(); return std::unique_ptr(page_ref); } MmapPageManager::MmappedFile* MmapPageManager::getMmappedFile( uint64_t last_byte) { if (current_mapping_ == nullptr || last_byte > current_mapping_->size) { /* align mmap size to the next larger block boundary */ auto file = fnord::io::File::openFile( filename_, File::O_READ | File::O_WRITE); auto mmap_size = file.size(); void* addr = nullptr; if (mmap_size > 0) { addr = mmap( nullptr, mmap_size, PROT_WRITE | PROT_READ, MAP_SHARED, file.fd(), 0); if (addr == MAP_FAILED) { RAISE_ERRNO(kMallocError, "mmap() failed"); } } if (current_mapping_ != nullptr) { current_mapping_->decrRefs(); } current_mapping_ = new MmappedFile(addr, mmap_size); } return current_mapping_; } MmapPageManager::MmappedFile::MmappedFile( void* __data, const size_t __size) : data(__data), size(__size), refs(1) {} MmapPageManager::MmappedFile::~MmappedFile() { munmap(data, size); } void MmapPageManager::MmappedFile::incrRefs() { refs++; } void MmapPageManager::MmappedFile::decrRefs() { if (refs.fetch_sub(1) == 1) { assert(refs.load() == 0); delete this; } } MmapPageManager::MmappedPageRef::MmappedPageRef( const PageManager::Page& page, MmappedFile* file, size_t sys_page_size) : PageRef(page), file_(file), sys_page_size_(sys_page_size) { file_->incrRefs(); } MmapPageManager::MmappedPageRef::MmappedPageRef( MmapPageManager::MmappedPageRef&& move) : PageRef(move.page_), file_(move.file_), sys_page_size_(move.sys_page_size_) { move.file_ = nullptr; } void* MmapPageManager::MmappedPageRef::getPtr() const { return file_->data; } void MmapPageManager::MmappedPageRef::sync(bool async /* = false */) const { uintptr_t ptr = (uintptr_t) ((char *) getPtr()) + page_.offset; auto sptr = (ptr / (sys_page_size_)) * sys_page_size_; auto ssize = page_.size + (ptr - sptr); msync((void *) sptr, ssize, async ? MS_SYNC : MS_ASYNC); } MmapPageManager::MmappedPageRef::~MmappedPageRef() { file_->decrRefs(); } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/pagemanager.h000066400000000000000000000131251255644354400257370ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_FILEBACKEND_PAGEMANAGER_H #define _FNORDMETRIC_FILEBACKEND_PAGEMANAGER_H #include #include #include #include #include #include #include #include #include namespace fnord { namespace io { /** * FIXPAUL TODO AllocationPolicy */ /** * This is an internal class. For usage instructions and extended documentation * please refer to "storagebackend.h" and "database.h" */ class PageManager { public: struct Page { Page(uint64_t offset_, uint64_t size_); Page(); uint64_t offset; uint64_t size; }; struct PageRef { public: PageRef(const Page& page); PageRef(const PageRef& copy) = delete; PageRef& operator=(const PageRef& copy) = delete; virtual ~PageRef(); void* operator*() const; template T* structAt(size_t position) const; virtual void* getPtr() const = 0; inline void* ptr() const { return getPtr(); } inline size_t size() const { return page_.size; } virtual void sync(bool async = false) const {}; const PageManager::Page page_; }; PageManager(size_t block_size, size_t end_pos = 0); //PageManager(size_t block_size, const LogSnapshot& log_snapshot); PageManager(const PageManager& copy) = delete; PageManager& operator=(const PageManager& copy) = delete; PageManager(const PageManager&& move); /** * Request a new page from the page manager */ virtual Page allocPage(size_t min_size); /** * Return a page to the pagemanager. Adds this page to the freelist */ virtual void freePage(const Page& page); /** * Request a page to be mapped into memory. Returns a smart pointer. */ virtual std::unique_ptr getPage(const PageManager::Page& page) = 0; protected: /** * Try to find a free page with a size larger than or equal to min_size * * Returns true if a matching free page was found and returns the page into * the destination parameter. Returns false if no matching page was found and * does not change the destination parameter. */ bool findFreePage(size_t min_size, Page* destination); /** * Index of the first unused byte in the file */ size_t end_pos_; /** * Optimal block size for the underlying file */ const size_t block_size_; /** * Page free list * * tuple is (size, offset) */ std::vector> freelist_; std::mutex mutex_; }; class MmapPageManager : public PageManager { protected: // FIXPAUL replace with io::MmapedFile class MmappedFile { public: MmappedFile(void* __data, const size_t __size); MmappedFile(const MmappedFile& copy) = delete; MmappedFile& operator=(const MmappedFile& copy) = delete; ~MmappedFile(); void* const data; const size_t size; void incrRefs(); void decrRefs(); protected: std::atomic_int refs; }; public: /** * Size of the initially create mmaping in bytes. All mmapings will be a * multiple of this size! */ static const size_t kMmapSizeMultiplier = 128; /* 128 * PAGE_SIZE */ class MmappedPageRef : public PageManager::PageRef { public: MmappedPageRef(const PageManager::Page& page, MmappedFile* file, size_t sys_page_size); MmappedPageRef(MmappedPageRef&& move); MmappedPageRef(const MmappedPageRef& copy) = delete; MmappedPageRef& operator=(const MmappedPageRef& copy) = delete; ~MmappedPageRef(); void sync(bool async = false) const override; protected: size_t sys_page_size_; void* getPtr() const override; MmappedFile* file_; }; /** * Create a new mmap page manager for a new file */ explicit MmapPageManager(const std::string& filename, size_t file_size); MmapPageManager(MmapPageManager&& move); MmapPageManager(const MmapPageManager& copy) = delete; MmapPageManager& operator=(const MmapPageManager& copy) = delete; ~MmapPageManager(); /** * Request a page to be mapped into memory. Returns a smart pointer. */ std::unique_ptr getPage( const PageManager::Page& page) override; /** * Request a page to be mapped into memory but disallow padding the file * . Returns a smart pointer. */ struct kNoPadding {}; std::unique_ptr getPage( const PageManager::Page& page, kNoPadding no_padding); /** * Truncate the file to the actual used size */ void shrinkFile(); protected: std::unique_ptr getPageImpl( const PageManager::Page& page, bool allow_padding); /** * Returns a mmap()ed memory region backend by the managed file spans until * at least last_byte */ MmappedFile* getMmappedFile(uint64_t last_byte); const std::string filename_; size_t used_bytes_; size_t file_size_; MmappedFile* current_mapping_; std::mutex mmap_mutex_; size_t sys_page_size_; }; /** * IMPLEMENTATION */ template T* PageManager::PageRef::structAt(size_t position) const { if (position >= page_.size) { RAISE( kIndexError, "invalid access at address %x, page size=%x offset=%x", position, page_.size, page_.offset); } return (T*) (((char *) getPtr()) + page_.offset + position); } } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/io/pagemanager_test.cc000066400000000000000000000076501255644354400271420ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include #include #include #include #include using namespace fnord::io; UNIT_TEST(PageManagerTest); class ConcreteTestPageManager : public PageManager { public: ConcreteTestPageManager() : PageManager(4096) {} std::unique_ptr getPage(const PageManager::Page& page) override { return std::unique_ptr(nullptr); } size_t endPos() const { return end_pos_; } }; class TestMmapPageManager : public MmapPageManager { public: explicit TestMmapPageManager( const std::string& filename, size_t len) : MmapPageManager(filename, len) {} MmappedFile* getMmappedFileTest(uint64_t last_byte) { getPage(PageManager::Page(0, last_byte)); return getMmappedFile(last_byte); } }; TEST_INITIALIZER(PageManagerTest, SetupTempFolder, [] () { FileUtil::mkdir_p("build/tests/tmp"); }); TEST_CASE(PageManagerTest, TestAbstractPageManagerAlloc, [] () { ConcreteTestPageManager page_manager; auto page1 = page_manager.allocPage(3000); EXPECT_EQ(page_manager.endPos(), 4096); EXPECT_EQ(page1.offset, 0); EXPECT_EQ(page1.size, 4096); page_manager.freePage(page1); }); TEST_CASE(PageManagerTest, TestAbstractPageManagerAllocMulti, [] () { ConcreteTestPageManager page_manager; auto page1 = page_manager.allocPage(3000); EXPECT_EQ(page_manager.endPos(), 4096); EXPECT_EQ(page1.offset, 0); EXPECT_EQ(page1.size, 4096); page_manager.freePage(page1); auto page2 = page_manager.allocPage(8000); EXPECT_EQ(page_manager.endPos(), 12288); EXPECT_EQ(page2.offset, 4096); EXPECT_EQ(page2.size, 8192); auto page3 = page_manager.allocPage(3000); EXPECT_EQ(page3.offset, 0); EXPECT_EQ(page3.size, 4096); }); TEST_CASE(PageManagerTest, TestAbstractPageManagerAllocFree, [] () { ConcreteTestPageManager page_manager; auto page1 = page_manager.allocPage(3000); EXPECT_EQ(page_manager.endPos(), 4096); EXPECT_EQ(page1.offset, 0); EXPECT_EQ(page1.size, 4096); page_manager.freePage(page1); auto page2 = page_manager.allocPage(8000); EXPECT_EQ(page_manager.endPos(), 12288); EXPECT_EQ(page2.offset, 4096); EXPECT_EQ(page2.size, 8192); auto page3 = page_manager.allocPage(3000); EXPECT_EQ(page3.offset, 0); EXPECT_EQ(page3.size, 4096); page_manager.freePage(page2); auto page4 = page_manager.allocPage(4000); EXPECT_EQ(page_manager.endPos(), 12288); EXPECT_EQ(page4.offset, 4096); EXPECT_EQ(page4.size, 8192); }); TEST_CASE(PageManagerTest, TestMmapPageManager, [] () { int fd = open("build/tests/tmp/__fnordmetric_testMmapPageManager", O_CREAT | O_TRUNC | O_RDWR, S_IRUSR | S_IWUSR); EXPECT(fd > 0); auto page_manager = new TestMmapPageManager( "build/tests/tmp/__fnordmetric_testMmapPageManager", 0); auto mfile1 = page_manager->getMmappedFileTest(3000); auto mfile2 = page_manager->getMmappedFileTest(304200); auto page_size = sysconf(_SC_PAGESIZE); EXPECT_EQ(mfile1->size, page_size * MmapPageManager::kMmapSizeMultiplier); EXPECT_EQ((void *) mfile1, (void *) mfile2); mfile2->incrRefs(); auto mfile3 = page_manager->getMmappedFileTest( page_size * MmapPageManager::kMmapSizeMultiplier + 1); EXPECT_EQ(mfile3->size, page_size * MmapPageManager::kMmapSizeMultiplier * 2); EXPECT(mfile3 != mfile2); mfile2->decrRefs(); delete page_manager; unlink("build/tests/tmp/__fnordmetric_testMmapPageManager"); close(fd); }); fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/metricdb/000077500000000000000000000000001255644354400244775ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/metricdb/adminui.cc000066400000000000000000000052261255644354400264410ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #include #include #include #include namespace fnordmetric { namespace metricdb { std::unique_ptr AdminUI::getHandler() { return std::unique_ptr(new AdminUI()); } bool AdminUI::handleHTTPRequest( http::HTTPRequest* request, http::HTTPResponse* response) { if (env()->verbose()) { fnord::util::LogEntry log_entry; log_entry.append("__severity__", "DEBUG"); log_entry.printf( "__message__", "HTTP request: %s %s", request->getMethod().c_str(), request->getUrl().c_str()); env()->logger()->log(log_entry); } util::URI uri(request->getUrl()); auto path = uri.path(); if (path == "/") { response->setStatus(http::kStatusFound); response->addHeader("Content-Type", "text/html; charset=utf-8"); response->addHeader("Location", "/admin"); return true; } if (path == "/admin") { sendAsset( response, "fnordmetric-webui/fnordmetric-webui.html", "text/html; charset=utf-8"); return true; } if (path == "/favicon.ico") { sendAsset( response, "fnordmetric-webui/fnordmetric-favicon.ico", "image/x-icon"); return true; } if (path == "/s/fnordmetric.js") { sendAsset( response, "fnordmetric-js/fnordmetric.js", "text/javascript"); return true; } if (path == "/s/fnordmetric-webui.css") { sendAsset( response, "fnordmetric-webui/fnordmetric-webui.css", "text/css"); return true; } if (path == "/s/fnordmetric-webui.js") { sendAsset( response, "fnordmetric-webui/fnordmetric-webui.js", "text/javascript"); return true; } if (path == "/s/fontawesome.woff") { sendAsset( response, "fnordmetric-webui/fontawesome.woff", "application/x-font-woff"); return true; } return false; } void AdminUI::sendAsset( http::HTTPResponse* response, const std::string& asset_path, const std::string& content_type) const { response->setStatus(http::kStatusOK); response->addHeader("Content-Type", content_type); response->addBody(util::Assets::getAsset(asset_path)); } } } fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/metricdb/adminui.h000066400000000000000000000020411255644354400262730ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2011-2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_WEBINTERFACE_H #define _FNORDMETRIC_WEBINTERFACE_H #include #include #include #include using namespace fnord; namespace fnordmetric { namespace metricdb { class AdminUI : public http::HTTPHandler { public: static std::unique_ptr getHandler(); bool handleHTTPRequest( http::HTTPRequest* request, http::HTTPResponse* response) override; private: void sendAsset( http::HTTPResponse* response, const std::string& asset_path, const std::string& content_type) const; }; } } #endif fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/metricdb/backends/000077500000000000000000000000001255644354400262515ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/metricdb/backends/disk/000077500000000000000000000000001255644354400272035ustar00rootroot00000000000000fnordmetric-0.1.0.20150502+gite736fba3/fnordmetric-core/src/metricdb/backends/disk/binaryformat.h000066400000000000000000000034451255644354400320570ustar00rootroot00000000000000/** * This file is part of the "FnordMetric" project * Copyright (c) 2014 Paul Asmuth, Google Inc. * * FnordMetric is free software: you can redistribute it and/or modify it under * the terms of the GNU General Public License v3.0. You should have received a * copy of the GNU General Public License along with this program. If not, see * . */ #ifndef _FNORDMETRIC_METRICDB_BINARYFORMAT_H #define _FNORDMETRIC_METRICDB_BINARYFORMAT_H #include #include #include namespace fnordmetric { namespace metricdb { /** * // http://tools.ietf.org/html/rfc5234 * * := * // metric key size * // metric key * // generation * // number of parent generations * * // parent generations * * := * // sample value * *