pax_global_header00006660000000000000000000000064134226567540014530gustar00rootroot0000000000000052 comment=209dd85ab19c45e352c73399fafa62e31bb4dd37 python-magcode-core-1.5.4/000077500000000000000000000000001342265675400154035ustar00rootroot00000000000000python-magcode-core-1.5.4/.gitignore000066400000000000000000000001711342265675400173720ustar00rootroot00000000000000# Ignore these pythonisms build __pycache__/ *.pyc # Vim backup file *.swp MANIFEST dist/ debian/ magcode_core.egg-info/ python-magcode-core-1.5.4/COPYING000066400000000000000000001045051342265675400164430ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU General Public License instead of this License. But first, please read . python-magcode-core-1.5.4/INSTALL000066400000000000000000000000501342265675400164270ustar00rootroot00000000000000To install: $ python3 setup.py install python-magcode-core-1.5.4/README000066400000000000000000000020721342265675400162640ustar00rootroot00000000000000MAG Code python3 core module of common utility code. MAG Code python3 core module of utility modules. This includes the process command line script and daemon framework, SQL Alchemy framework with event queue and common WSGI code for JSON http RPC backends. Programming kindly sponsored by Voyager Internet Ltd, http://www.voyager.co.nz/ Released under GPL v3 or later. See COPYING. Py-magcode-core is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or at your option) any later version. Py-magcode-core is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with py-magcode-core. If not, see . Matthew Grant Sun, 14 Jul 2013 17:20:00 +1200 python-magcode-core-1.5.4/etc/000077500000000000000000000000001342265675400161565ustar00rootroot00000000000000python-magcode-core-1.5.4/etc/magcode.conf.sample000066400000000000000000000011451342265675400217050ustar00rootroot00000000000000# MAG Code Python configuration file # # Typical ini file format # # DEFAULT section covers all programs, sections are by default program names, # or WSGI names # # Section name has to exist for a program to run # [DEFAULT] # Sample values, example for DMS system #run_as_user = magdmd #db_user = dms #db_password = ScoobyDoo #db_name = dms [magdmd] daemon_canary = blau #log_level = DEBUG #debug_mark = true [zone_tool] # edit timeout in minutes #edit_lock_timeout = 2 [mod_wsgi] # Settings specific to mod_wsgi scripts #jsonrpc_error_stack_trace = True [admin_dms.wsgi] #jsonrpc_error_stack_trace = True python-magcode-core-1.5.4/magcode/000077500000000000000000000000001342265675400170025ustar00rootroot00000000000000python-magcode-core-1.5.4/magcode/__init__.py000066400000000000000000000013601342265675400211130ustar00rootroot00000000000000# This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # # Make this a python module python-magcode-core-1.5.4/magcode/core/000077500000000000000000000000001342265675400177325ustar00rootroot00000000000000python-magcode-core-1.5.4/magcode/core/__init__.py000066400000000000000000000016421342265675400220460ustar00rootroot00000000000000# # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ MAG Code Core Python Library """ from magcode.core.globals_ import * python-magcode-core-1.5.4/magcode/core/database/000077500000000000000000000000001342265675400214765ustar00rootroot00000000000000python-magcode-core-1.5.4/magcode/core/database/__init__.py000066400000000000000000000020541342265675400236100ustar00rootroot00000000000000#!/usr/bin/env python3.2 # # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ Database module. Contains common imports from the database library, and sqlalchemy set up functions. """ # Set things up in one clean hit from magcode.core.database.utility import * python-magcode-core-1.5.4/magcode/core/database/event.py000066400000000000000000000770611342265675400232040ustar00rootroot00000000000000#!/usr/bin/env python3.2 # # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ Event queue module Implents a python event queue against a SQL database table, using SQLALchemy. """ import sys import os import time import threading import json import signal from queue import Queue from queue import Empty from copy import copy from traceback import format_exc from sqlalchemy import Table from sqlalchemy import MetaData from sqlalchemy.orm import mapper from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import scoped_session from sqlalchemy.orm.session import make_transient from sqlalchemy.sql.expression import func from sqlalchemy.sql import text from sqlalchemy.exc import DBAPIError from sqlalchemy.exc import SQLAlchemyError from sqlalchemy.orm.exc import NoResultFound from magcode.core import * from magcode.core.utility import send_signal from magcode.core.utility import get_numeric_setting from magcode.core.database import * # some state static constant ESTATE_NEW = 'NEW' ESTATE_RETRY = 'RETRY' ESTATE_SUCCESS = 'SUCCESS' ESTATE_FAILURE = 'FAILURE' event_process_states = frozenset((ESTATE_NEW,ESTATE_RETRY)) event_processed_states = frozenset((ESTATE_SUCCESS, ESTATE_FAILURE)) event_states = event_process_states.union(event_processed_states) # Lists we use in global sql_data dict sql_data['event_subclasses'] = [] sql_data['event_type_list'] = [] def eventregister(class_): """ Event descedant class decorator function to register class for SQL Alchemy mapping in init_event_class() below, called in magcode.core.database.utility.setup_sqlalchemy() """ sql_data['event_subclasses'].append(class_) sql_data['event_type_list'].append(class_.__name__) sql_data['events'][class_.__name__] = class_ return(class_) def synceventregister(class_): """ Event descedant class decorator function to register class for SQL Alchemy mapping in init_event_class() below, called in magcode.core.database.utility.setup_sqlalchemy() This one is for SyncEvent classes. """ sql_data['event_subclasses'].append(class_) sql_data['events'][class_.__name__] = class_ return(class_) class EventAlreadyProcessing(Exception): """ Event already processing """ class QueueReset(Exception): """ Queue reset as DB connection loss detected """ class _EQueue(Queue): """ Internal queue class with dictionary of events being processed This replaces methods on Queue so that processing array is only altered when mutex held to prevent races. """ def __init__(self): """ Special init to set queue size """ self.processing = {} self.queue_reset = False super().__init__( maxsize=get_numeric_setting('event_queue_maxsize', int)) def _put(self, item): """ Put an event on the Queue, and record it as being processed. """ if self.queue_reset: raise QueueReset("Event queue reset") if self.processing.get(item.id_): raise EventAlreadyProcessing("Event '%s' is already processing." % item.id_) self.processing[item.id_] = True super()._put(item) def start_queue_reset(self): self.queue_reset = True while True: try: event = super().get_nowait() super().task_done() except Empty: break def is_queue_reseting(self): return self.queue_reset def clean_up(self): """ Cleans up after queue emptied. """ self.join() self.queue_reset = False self.processing = {} def task_done(self, event_id): """ task done """ if event_id: self.processing.pop(event_id) super().task_done() class EventProcess(object): """ Container class for event processing methods for an event. These are both used by queued events and synchronous events. """ def _failure_results(self, db_session, reason, function="event_process_wrapper" ): self.state = ESTATE_FAILURE self.result_code = RCODE_FATAL self.processed = db_time(db_session) self._clean_attrs() log_error("%s() - FAILING event id: '%s', " "event_type: '%s', state: '%s', scheduled '%s'," " created: '%s', processed: '%s' with result " "'%s - %s, %s' - %s" % (function, self.id_, self.event_type, self.state, self.scheduled, self.created, self.processed, self.result_code, result_code_dict[self.result_code], result_code_lstr_dict[self.result_code], reason)) return def _clean_attrs(self): """ Clean off attrs we don't want so event can go to DB without giving SQLAlchemy a headache. """ if hasattr(self, 'db_session'): delattr(self, 'db_session') if hasattr(self, 'py_results'): delattr(self, 'py_results') if hasattr(self, 'py_parameters'): delattr(self, 'py_parameters') def _process_wrapper(self, db_session): """ Setup and teardown method that wraps the events process() method. Sets up environment, sets fields once process() finished and does other janitorial work. In here rather that in Event class to make Event code tidy. """ # Safe guard to see if we are already processed if (self.state not in event_process_states): return # Set up dyanmic data for this event self.db_session = db_session # Set up results as an empty dictionary - easy for events to insert # results, message etc. self.py_results = {} if (self.parameters): # JSON decode parameters try: self.py_parameters = json.loads(self.parameters) except Exception: self._failure_results(db_session, "could not decode JSON parameters.") return else: py_parameters = None # Call event state machine log_debug("_process_wrapper() - processing event id: '%s', " "event_type: '%s', state: '%s', scheduled '%s'," " created: '%s'" % (self.id_, self.event_type, self.state, self.scheduled, self.created)) result_code = self.process() self.result_code = result_code # JSON encode results if (self.py_results): try: self.results = json.dumps(self.py_results) except Exception: self._failure_results(db_session, "could not encode JSON results.") return # Check returned result code is sensible if (result_code not in result_codes): self._failure_results(db_session, "result code not recognised.") return # Check returned state is sensible if (self.state not in event_states): self._failure_results(db_session, "new event state not recognised.") return # get rid of dynamic attributes we created at start of this. self._clean_attrs() # Only set processed_time if we are finished with this event if self.state in event_processed_states: self.processed = db_time(db_session) log_debug("_process_wrapper() - processed event id: '%s', " "event_type: '%s', state: '%s', scheduled '%s'," " created: '%s', processed: '%s' with result " "'%s - %s, %s'" % (self.id_, self.event_type, self.state, self.scheduled, self.created, self.processed, self.result_code, result_code_dict[self.result_code], result_code_lstr_dict[self.result_code])) return class Event(EventProcess): """ Ancestor Event Class """ @classmethod def sa_map_subclass(class_): sql_data['mappers'][class_] = mapper(class_, inherits=sql_data['mappers'][Event], polymorphic_identity=class_.__name__) def __init__(self, scheduled=None, **kwargs_parameters): """ Initialise the event. kwargs_parameters is passed to the parameters column of the database as a JSON encoded dict. This is the easiest form to encode needed parameters, and decreases the need to override the __init__ method in child classes """ self.state = ESTATE_NEW self.scheduled = scheduled fkey_list = settings['event_queue_fkey_columns'].split() for attr in fkey_list: value = kwargs_parameters.pop(attr, None) if value == None: continue exec ("self.%s = %s" % (attr, value)) if kwargs_parameters: self.parameters = json.dumps(kwargs_parameters) else: self.parameters = None def process(self): """ Put event code here - stub method attributes set up by EventQueue class: self.py_parameters holds decoded JSON parameters for event self.py_results is NONE, and you can assemble a data structure to be JSON encoded into the sm_event_queue results column. return code is one from the RCODE_s above self.state is the event state - adjust once process to one of the ESTATE_s above. self.customer_id - DMS customer_id from dms database. self.zone_id - DNS Zone zone_id from dms database. self.role_id - DMS role_id from dms database. created, scheduled,processed timestamps are attributes as well but DON'T TOUCH them, or DEPEND on the processed timestamp! """ log_info("Hello Good Event New World!") self.state = ESTATE_SUCCESS return RCODE_OK def set_scheduled(self, time=None, delay=None, db_session=None): """ Helper function to manipulate scheduled field time - datetime.datetime object (Python Standard Library) delay - datetime.interval object (Python Standard Library) """ if (not time): if not db_session: db_session = sql_data['scoped_session_class']() time = db_time(db_session) self.scheduled = time if (delay): self.scheduled += delay def get_parameters(self): """ Helper function to JSON decode parameters for event """ if (self.parameters): return json.loads(self.parameters) else: return None def get_results(self): """ Helper function to JSON decode parameters for event """ if (self.results): return json.loads(self.results) else: return None def to_engine_brief(self, time_format=None): """ Supply data output in brief as a dict. Just id, event_type, scheduled, processed, created, result_code. """ if not time_format: created = self.created.isoformat(sep=' ') \ if self.created else None scheduled = self.scheduled.isoformat(sep=' ') \ if self.scheduled else None processed = self.processed.isoformat(sep=' ') \ if self.processed else None else: created = self.created.strftime(time_format) \ if self.created else None scheduled = self.scheduled.strftime(time_format) \ if self.scheduled else None processed = self.processed.strftime(time_format) \ if self.processed else None return {'event_id': self.id_, 'event_type': self.event_type, 'state': self.state, 'zone_id': self.zone_id, 'parameters': self.get_parameters(), 'result_code': self.result_code, 'created': created, 'scheduled': scheduled, 'processed': processed,} def to_engine(self, time_format=None): """ Return all fields as a dict """ if not time_format: created = self.created.isoformat(sep=' ') \ if self.created else None scheduled = self.scheduled.isoformat(sep=' ') \ if self.scheduled else None processed = self.processed.isoformat(sep=' ') \ if self.processed else None else: created = self.created.strftime(time_format) \ if self.created else None scheduled = self.scheduled.strftime(time_format) \ if self.scheduled else None processed = self.processed.strftime(time_format) \ if self.processed else None return {'event_id': self.id_, 'event_type': self.event_type, 'state': self.state, 'zone_id': self.zone_id, 'parameters': self.get_parameters(), 'results': self.get_results(), 'result_code': self.result_code, 'created': created, 'scheduled': scheduled, 'processed': processed,} class SyncEvent(Event): """ Synchronous event that executes upon creation. Designed to be used in code that needs immediate event execution. Event created in event_queue table for the record and to obtain an event ID, but it is never executed on the queue as is it is not in sql_data['event_type_list'] """ def __init__(self, execute=False, *args, **kwargs): """ Initialise synchronous event, and execute it if asked to. A dictionary of parameters can be passed in. If executing from here, use event.get_results to return results dictiontionary """ super().__init__(*args, **kwargs) if execute: self.execute() def execute(self): """ Execute the synchronous event. Returns results as a JSON dictionary. """ # Set up environment db_session = sql_data['scoped_session_class']() db_session.add(self) # Execute self._process_wrapper(db_session) db_session.commit() # Return results results = self.get_results() results['result_code'] = self.result_code results['state'] = self.state return results class QueueDraining(Exception): """ Lost DB connection, drain queue with out processing so that reader loop errors on next DB query """ class EventQueue(object): """ Event Queue object Operates on database table sm_event_queue. The job of this class is to present a simple event class API/ABI to the application programmer. SQLAlchemy gets its fingers into each class pie to hide the database <=> object convolution/deconvolution, and this class builds on top of that stack to create runable events. """ def __init__(self): """ Sets up queue and Threads """ # This is a FIFO queue self._queue = _EQueue() self._threads = [] self._event_count = 0 self.event_queue_session_transactions \ = get_numeric_setting('event_queue_session_transactions', int) self._thread_top_up() # Pause to let threads get established. time.sleep(3) def queue_empty(self): """ check if queue is empty. Use during sleep to help determine when to garbage collect etc. """ return self._queue.empty() def event_processor(self): """ Worker thread that processes events on the queue """ log_debug ("event_processor() - starting thread.") Session = sql_data['scoped_session_class'] db_session = Session() event_count = 0 # Loop and wait on queue while True: # Keep Session cycling to release RAM if (self.event_queue_session_transactions and event_count > self.event_queue_session_transactions): event_count = 0 db_session.close() sql_data['scoped_session_class'].remove() db_session = Session() event_count += 1 timeout = settings['event_queue_thread_timeout'] try: # Balanced with finally: task-done() below. event = self._queue.get(timeout=timeout) except Empty: log_debug("event_processor() - exiting thread because " "of Queue.get() time out %ss." % timeout) Session.remove() return info_str = None error_str = None event_id = None try: if self._queue.is_queue_reseting(): raise QueueDraining() # The following replaces event with in an session event event = db_session.merge(event) event_id = event.id_ # State machine DB locks happen in below call... event._process_wrapper(db_session) db_session.commit() except QueueDraining: # Finally will mark task_done continue except Exception as exc: # Handle error messages if isinstance(exc, DBAPIError): if is_db_connection_exception(exc.orig): info_str = ("event_processor() draining queue and exiting - PostgresQL database connection probably closed.\n" "%s" % str_exc(exc)) # DB gone, drain queue self._queue.start_queue_reset() break else: error_str = ("event_processor() exiting - unexpected error\n" "%s" % format_exc(chain=True)) elif isinstance(exc, SQLAlchemyError): error_str = ("event_processor() exiting" " - unexpected error\n" "%s" % format_exc(chain=False)) # Back completely out - following raise does not clear DB # locks! db_session.rollback() # # If DB stuff is corrupted, we have more to worry about # than this wild exception, so commit this event so it # is not processed again. Record Exception information # along with it for fixing it up. # # The following works as event reverted to non-merged # due to exception processing event = db_session.merge(event) event_id = event.id_ # capture traceback py_results = {} py_results['execption_info'] = format_exc() py_results['message'] = "Failing Event ID: %s due to exception. See 'exception_info' result." % event.id_ event.results = json.dumps(py_results) event._failure_results(db_session, "exception occured.", function="event_processor") db_session.commit() # Terminate thread early - DB session corrupt? # Finally remove thread db_session Session.remove() raise finally: # Balanced with get above. If we call too often we will # get an error. self._queue.task_done(event_id) # Deallocate memory del event # Process errors - avoid chained exception traces if (info_str): log_info(info_str) if (error_str): log_error(error_str) # Finally remove thread db_session Session.remove() def _thread_start(self): """ Start a thread """ thread = threading.Thread(target=self.event_processor) thread.daemon = True thread.start() self._threads.append(thread) def _thread_top_up(self): if (debug()): self.event_queue_threads = 1 else: self.event_queue_threads = get_numeric_setting( 'event_queue_threads', int) # Check what threads are active, and remove those that are not self._threads = [thread for thread in self._threads if thread.is_alive()] while (len(self._threads) < self.event_queue_threads): log_debug("_thread_top_up() - topping up threads") self._thread_start() def process_queue(self): """ Read events from database, and place on queue, until all processed. """ # top up processor threads if any have died... self._thread_top_up() # Read from sm_event_queue db_session = sql_data['scoped_session_class']() # Use flags to unchain exception handling. info_str = None error_str = None try: # Query ze events... for event in db_session.query(Event)\ .filter(Event.processed == None)\ .filter(Event.event_type.in_(sql_data['event_type_list']))\ .filter(Event.state.in_(event_states))\ .filter(Event.scheduled <= func.statement_timestamp())\ .order_by(Event.scheduled.asc())\ .limit(settings['event_queue_maxsize']): try: self._queue.put(event) self._event_count += 1 except EventAlreadyProcessing: continue except QueueReset: log_info("process_queue() - queue reset detected, cleaning up and returning to main loop.") db_session.close() sql_data['scoped_session_class'].remove() # go back to main_process() after queue is emptied self._queue.clean_up() return log_debug("process_queue() - queueing event id:'%s', " "event_type: '%s', state: '%s', scheduled '%s'," " created: '%s'" % (event.id_, event.event_type, event.state, event.scheduled, event.created)) db_session.commit() # Joining queue not needed as events cannot be put if they are # processing except DBAPIError as exc: if is_db_connection_exception(exc.orig): info_str = ("process_queue() exiting - PostgresQL database connection probably closed.\n" " %s" % str_exc(exc)) else: error_str = ("process_queue() exiting - unexpected error\n" "%s" % format_exc(chain=True)) # close off session db_session.close() except SQLAlchemyError as exc: error_str = ("process_queue() exiting - unexpected error\n" "%s" % format_exc(chain=False)) # close off session db_session.rollback() # Unchain exception handling - Print any log messages if (info_str): log_info(info_str) if (error_str): log_error(error_str) # Every so often, close session if (self.event_queue_session_transactions and self._event_count > self.event_queue_session_transactions): self._event_count = 0 db_session.close() sql_data['scoped_session_class'].remove() # go back to main_process() return def queue_event(event, db_session=None, commit=False, time=None, delay=None, coalesce_period=None, signal_queue_daemon=False): """ Given a newly created event instance, queue it for processing """ if not db_session: db_session = sql_data['scoped_session_class']() # Given a db_session, queue a created event event.set_scheduled(time, delay, db_session=db_session) events_to_coalesce = [] if coalesce_period: events_iterator = db_session.query(Event)\ .filter(Event.processed == None)\ .filter(Event.event_type == event.event_type)\ .filter(Event.scheduled <= event.scheduled)\ .filter(Event.scheduled > (event.scheduled - coalesce_period))\ .order_by(Event.scheduled) events_to_coalesce = [e for e in events_iterator if e.get_parameters() == event.get_parameters()] if events_to_coalesce: # first event in list is earliest equivalent earlist_event = events_to_coalesce.pop(0) earlist_event.scheduled = event.scheduled for e in events_to_coalesce: db_session.delete(e) event = earlist_event else: db_session.add(event) if commit: db_session.commit() else: db_session.flush() if signal_queue_daemon: send_signal(settings['event_queue_pid_file'], signal.SIGUSR1) return event def create_event(event_type, db_session=None, commit=False, time=None, delay=None, coalesce_period=None, signal_queue_daemon=False, *args, **kwargs): """ Create and then queue an event for processing. *args and **kwargs are passed into the event constructor """ event = event_type(*args, **kwargs) event = queue_event(event, db_session, commit, time, delay, coalesce_period, signal_queue_daemon) return event def cancel_event(event_id, db_session=None, commit=False, log_to_info=False): """ Cancel an unprocessed event given an event_id """ if not db_session: db_session = sql_data['scoped_session_class']() # get event event = db_session.query(Event)\ .filter(Event.id_ == event_id).one() if event.processed: return # Cancel Event time = db_time(db_session) event.state = ESTATE_FAILURE event.processed = time msg = ("Event %s(%s) canceled." % (event.__class__.__name__, event.id_)) if log_to_info: log_info(msg) else: log_debug(msg) results = {'message': msg} event.results = json.dumps(results) # put it out to database if commit: db_session.commit() else: db_session.flush() def reschedule_event(event_id, db_session=None, commit=False, time=None, delay=None, signal_queue_daemon=False): """ Coalesce an event with one already existing in the event queue. Event is coalesced with one within coalesce_time seconds of its scheduled time. Event to be claesced has to be exactly the same apart from id - ie same type State has to be NEW or RETRY, and same params """ if not db_session: db_session = sql_data['scoped_session_class']() # get event try: event = db_session.query(Event)\ .filter(Event.id_ == event_id).one() except NoResultFound: return None if event.processed: return None # reschedule it event.set_scheduled(time, delay, db_session=db_session) # put it out to database if commit: db_session.commit() else: db_session.flush() if signal_queue_daemon: send_signal(settings['event_queue_pid_file'], signal.SIGUSR1) return event def find_events(event_type, db_session=None, from_time=None, before_time=None, period=None, processed=False, **kwargs_fkey): """ Search for events. This function works on time boundaries exactly as per time around midnight ie 0 seconds on the hour is within the hour commencing. """ if not db_session: db_session = sql_data['scoped_session_class']() # Get list of events events_query = db_session.query(Event)\ .filter(Event.event_type == event_type.__name__)\ .order_by(Event.scheduled) fkey_list = settings['event_queue_fkey_columns'].split() for attr in fkey_list: value = kwargs_fkey.pop(attr, None) if value == None: continue events_query = events_query.filter(getattr(Event, attr) == value) # processed is boolean - else condition needed if processed: events_query = events_query.filter(Event.processed != None) else: events_query = events_query.filter(Event.processed == None) if from_time: events_query = events_query.filter(Event.scheduled >= from_time) if before_time: events_query = events_query.filter(Event.scheduled < before_time) if period: if from_time and not before_time: before_time = from_time + period events_query = events_query.filter(Event.scheduled < before_time) if before_time and not from_time: from_time = before_time - period events_query = events_query.filter(Event.scheduled >= from_time) if not before_time and not from_time: before_time = db_time(db_session) from_time = before_time - period events_query = events_query.filter(Event.scheduled >= from_time) return events_query.all() # SQL Alchemy hooks def init_event_table(): table = Table('sm_event_queue', sql_data['metadata'], autoload=True, autoload_with=sql_data['db_engine']) sql_data['tables'][Event] = table def init_event_mappers(): table = sql_data['tables'][Event] sql_data['mappers'][Event] = mapper(Event, table, polymorphic_on=table.c.event_type, polymorphic_identity=Event.__name__, properties=mapper_properties(table, Event)) sql_data['event_type_list'].append(Event.__name__) # Map all the event subclasses for class_ in sql_data['event_subclasses']: class_.sa_map_subclass() sql_data['init_list'].append({'table': init_event_table, 'mapper': init_event_mappers}) python-magcode-core-1.5.4/magcode/core/database/process_sm.py000066400000000000000000000310051342265675400242240ustar00rootroot00000000000000#!/usr/bin/env python3.2 # # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # import os import sys import json import threading import copy from subprocess import * from sqlalchemy import text from sqlalchemy.exc import DBAPIError from sqlalchemy.exc import SQLAlchemyError from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import scoped_session from sqlalchemy.orm.session import make_transient from magcode.core.globals_ import settings from magcode.core.logging import * from magcode.core.database import * from magcode.core.database.state_machine import StateMachine from magcode.core.database.state_machine import SMEvent from magcode.core.database.state_machine import StateMachineError from magcode.core.database.state_machine import StateMachineFatalError from magcode.core.database.state_machine import smregister from magcode.core.database.event import Event from magcode.core.database.event import eventregister # some state static constant PSTATE_START = 'START' PSTATE_RUN = 'RUN' PSTATE_SUCCESS = 'SUCCESS' PSTATE_FAILURE = 'FAILURE' process_sm_states = (PSTATE_START, PSTATE_RUN, PSTATE_SUCCESS, PSTATE_FAILURE) _process_list = [] # Error Exceptions class ProcessError(StateMachineError): """ Base exception for Process State Machine. """ pass class ProcessFatalError(StateMachineFatalError): """ Base exception for Process State Machine Fatal errors. """ pass class ProcessExecError(ProcessFatalError): """ Process could not be executed. Typically file not found, or Invalid arguments to POpen """ def __init__(self, *args, **kwargs): if (not len(args) and not len(kwargs)): ProcessFatalError.__init__(self, "Execution failure") else: ProcessFatalError.__init__(self, *args, **kwargs) class ProcessExitError(ProcessFatalError): """ Process exited with an error. Typically wrong arguments or something """ def __init__(self, *args, **kwargs): if (not len(args) and not len(kwargs)): ProcessFatalError.__init__(self, "Process exit() error") else: ProcessFatalError.__init__(self, *args, **kwargs) class ProcessSignalError(ProcessFatalError): """ Process killed by signal. """ def __init__(self, *args, **kwargs): if (not len(args) and not len(kwargs)): ProcessFatalError.__init__(self, "Process killed by signal") else: ProcessFatalError.__init__(self, *args, **kwargs) class ProcessTempFailError(ProcessError): """ Temporary failure - resource termporarily not available """ def __init__(self, *args, **kwargs): if (not len(args) and not len(kwargs)): ProcessError.__init__(self, "Temporary execution failure") else: ProcessError.__init__(self, *args, **kwargs) class ProcessNoArgumentError(ProcessFatalError): """ No arguments or command given for a process """ def __init__(self, *args, **kwargs): if (not len(args) and not len(kwargs)): ProcessFatalError.__init__(self, "No Arguments of command given for a process") else: ProcessFatalError.__init__(self, *args, **kwargs) class ProcessNotAnEventError(ProcessFatalError): """ Event sent on success of process is not an event at all """ def __init__(self, pseudo_event=None, *args, **kwargs): if (pseudo_event): self.pseudo_event = pseudo_event ProcessFatalError.__init__(self, "%s is not one of %s" % (str(pseudo_event), sql_data['event_type_list'])) elif (not len(args) and not len(kwargs)): ProcessFatalError.__init__(self, "Object or string submitted is not representative of an Event()") else: ProcessFatalError.__init__(self, *args, **kwargs) # State Machine Events @eventregister class ProcessSMExecEvent(SMEvent): """ Start a process running """ pass @eventregister class ProcessSMResetEvent(SMEvent): """ Event issued to clean up and reset process SM to START """ pass # State Machine class @smregister class ProcessSM(StateMachine): """ Process State machine Runs a process. State kept in SQL database. Uses the subprocess module, running from a thread. Event objects are ProcessSMExecEvent, ProcessSMSuccessEvent, and ProcessSMFailureEvent """ # SQL Alchemy database table name _table = 'sm_process' # Events asscoiated with this state machine _sm_events = (ProcessSMExecEvent, ProcessSMResetEvent) # Event action methods def _exec(self, event): """ Spawn a Thread that watches event via subprocess module """ db_session = event.db_session self.state = PSTATE_RUN self.start = db_clock_time(db_session) db_session.flush() log_str = "ProcessSM.run_thread() %s(%s)" \ % (self.name, self.exec_path) error_str = None try: # Do the Dirty Work eh eh ahh! argv = json.loads(self.argv) process = Popen(args=argv, executable=self.exec_path, cwd=self.cwd, env=self.env, stdin=PIPE, stdout=PIPE, stderr=PIPE) # On Posix, processes consume bytes (which could be utf-8) stdin = bytes(self.stdin, 'utf8') (stdout, stderr) = process.communicate(stdin) # Convert returned text to Python unicode self.stdout = stdout.decode() self.stderr = stderr.decode() except (IOError, OSError) as exc: if (exc.filename): error_str = "%s %s - %s" % (log_str, exc.filename, exc.strerror) else: error_str = "%s - %s" % (log_str, exc.strerror) except ValueError as exc: error_str = "%s - %s" % (log_str, str(exc)) if (error_str): # Typically can't execute process self.state = PSTATE_FAILURE self.result_code = RCODE_FATAL raise ProcessExecError(error_str) if (process.returncode == os.EX_TEMPFAIL): # Process terminated with a temporary failure self.state = PSTATE_START self.exit_code = process.returncode self.result_code = RCODE_ERROR self.finish = db_clock_time(db_session) raise ProcessTempFailError(log_str + " - temporary failure.") elif (process.returncode): # Process termainated with > 0 exit code self.state = PSTATE_FAILURE self.result_code = RCODE_FATAL self.exit_code = process.returncode self.finish = db_clock_time(db_session) raise ProcessExitError(log_str + " - failed." ) elif (process.returncode < 0): # Process terminated by signal -N self.state = PSTATE_FAILURE self.result_code = RCODE_ERROR self.exit_code = None self.finish = db_clock_time(db_session) raise ProcessSignalError(log_str + " - killed by signal %s." % process.returncode * -1) msg = log_str + " - ran successfully." self.state = PSTATE_SUCCESS self.exit_code = process.returncode self.result_code = RCODE_OK self.finish = db_clock_time(db_session) # This can blow up in a real spectacular fashion! # Need to register Events to a list of classes? if (self.success_event): if (not self.success_event in sql_data['event_type_list']): raise ProcessNotAnEventError(pseudo_event=self.success_event) elif (self.success_event_kwargs): se_kwargs = json.loads(self.success_event_kwargs) s_event = eval ("%s(**se_kwargs)" % self.success_event) else: s_event = eval ("%s()" % self.success_event) db_session.add(s_event) return (RCODE_OK, msg) def _reset(self, event): self.state = PSTATE_START self.exit_code = None self.result_code = None self.stdout = None self.stderr = None self.start = None self.finish = None return (RCODE_OK, "Process %s(%s) - cleared output." % (self.name, self.exec_path)) _sm_table = { PSTATE_START: { ProcessSMExecEvent: _exec, }, # PSTATE_RUN occurs within an event # thus empty row here. PSTATE_RUN: {}, PSTATE_FAILURE: { ProcessSMResetEvent: _reset }, PSTATE_SUCCESS: { ProcessSMResetEvent: _reset }, } def __init__(self, exec_path=None, argv=None, stdin=None, name=None, success_event=None, success_event_kwargs=None): self.state = PSTATE_START if (not exec_path and not argv): raise ProcessNoArgumentError if (exec_path): self.exec_path = exec_path elif (argv): self.exec_path = argv[0] else: raise ProcessNoArgumentError if (name): self.name = name else: self.name = os.path.basename(self.exec_path) if (argv): self.argv = json.dumps(argv) self.stdin = stdin if (success_event): # This is a mess, but people will pass in all types of stuff, # and we need text name of class for DB if (type(success_event) is str): if (success_event not in sql_data['event_type_list']): raise ProcessNotAnEventError(psuedo_event=success_event) self.success_event = success_event elif (type(success_event) is type): if (not issubclass(success_event, Event)): raise ProcessNotAnEventError(psuedo_event=success_event) self.success_event = success_event.__name__ elif (isinstance(success_event, Event)): self.success_event = success_event.__class__.__name__ else: raise ProcessNotAnEventError(psuedo_event=success_event) else: self.success_event = None if (success_event_kwargs): self.success_event_kwargs = json.dumps(success_event_kwargs) def new_process(db_session, commit=False, time=None, delay=None, *args, **kwargs): """ Create a new process to be run at time, or after a delay Time is in Unix Epoch, and delay in seconds. If time or delay are not specified, it executes now. time is a datetime.datetime, delay is datetime.timeinterval """ process_sm = ProcessSM(*args, **kwargs) db_session.add(process_sm) db_session.flush() db_session.refresh(process_sm) exec_event = ProcessSMExecEvent(process_sm.id_) if (not time): time = db_time(db_session) exec_event.scheduled = time if (delay): exec_event.scheduled += delay db_session.add(exec_event) process_id = process_sm.id_ if commit: db_session.commit() else: db_session.flush() return process_id def get_process_results(db_session, sm_id): """ Get the results of a process. Returns sm_object from the database, given an id. """ try: sm_obj = event.db_session.query(ProcessSM)\ .filter(text("id = '%s'" % sm_id)).one() except NoResultFound: sm_obj = None return sm_obj def reset_process(db_session, sm_id, commit=False): """ Reset the given process """ sm_obj = get_process_results(db_session, sm_id) if (not sm_obj): return reset_event = ProcessSMResetEvent(sm_obj.id_) db_session.add(reset_event) if (commit): db_session.commit() else: db.session.flush() python-magcode-core-1.5.4/magcode/core/database/state_machine.py000066400000000000000000000210411342265675400246520ustar00rootroot00000000000000#!/usr/bin/env python3.2 # # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # import threading import os import sys from functools import wraps from sqlalchemy.orm.exc import NoResultFound from sqlalchemy import text from magcode.core.logging import * from magcode.core.globals_ import * from magcode.core.database.event import Event from magcode.core.database.event import SyncEvent from magcode.core.database.event import ESTATE_SUCCESS from magcode.core.database.event import ESTATE_RETRY from magcode.core.database.event import ESTATE_FAILURE from magcode.core.database.event import ESTATE_NEW from magcode.core.database.utility import * def sync(): """Decorator - Places self._lock around method""" def _sync(func): @wraps(func) def _syncs(self, *args, **kwargs): lock = self.__getattribute__('_lock') lock.acquire() try: return func(self, *args, **kwargs) finally: lock.release() return _syncs return _sync class StateMachineError(Exception): """ Base class for all State Machine Error Exceptions """ def __init__(self, *args): if (not len(args)): Exception.__init__(self, "Generic State Machine Error") else: Exception.__init__(self, *args) class StateMachineFatalError(StateMachineError): """ Fatal super class for all fatal State MAchine Exceptions """ def __init__(self, *args): if (not len(args)): Exception.__init__(self, "Fatal State Machine Error") else: Exception.__init__(self, *args) def smregister(class_): """ State machine descedant class decorator function to register class for SQL Alchemy mapping in magcode.core.database.utility.sqlalchemy_setup() """ # Process _table if (hasattr(class_, '_table') and class_._table): saregister(class_) #register events with this SM for event in class_._sm_events: event._sm_class = class_ return(class_) class SMEvent(Event): """ Base State Machine Event. Just executes state machine. Does not get mapped into sqlalchemy. If you modify this MODIFY SMSyncEvent as well! Thery are both 'photocopies' and are seperate just for inheritance reasons. """ # Private parameter to pass in SM event is associated with _sm_class = None def __init__(self, sm_id, *args, **kwargs): """ Initialise Event """ # Make sm_id a **kwargs to be swallowed by Event.__init__ # into the JSON parameters super().__init__(*args, sm_id=sm_id, **kwargs) def process(self): """ Execute an event on the State machine """ sm_obj = self._sm_class.fetch_sm_obj(self) if (not sm_obj): return RCODE_FATAL # Do the state machine thing... return_code = sm_obj.execute_state_machine(self) # Clean up memory del sm_obj return return_code class SMSyncEvent(SyncEvent): """ Base Syncronous State Machine Event. Just executes state machine. Does not get mapped into sqlalchemy. Photocopy of above SMEvent class, If you modify one MODIFY the OTHER!! """ # Private parameter to pass in SM event is associated with _sm_class = None def __init__(self, sm_id, *args, **kwargs): """ Initialise Event """ # Make sm_id a **kwargs to be swallowed by Event.__init__ # into the JSON parameters super().__init__(*args, sm_id=sm_id, **kwargs) def process(self): """ Execute an event on the State machine """ sm_obj = self._sm_class.fetch_sm_obj(self) if (not sm_obj): return RCODE_FATAL # Do the state machine thing... return_code = sm_obj.execute_state_machine(self) # Clean up memory del sm_obj return return_code class StateMachine(object): """ Base state machine object. Contains code that is used required by all state machines. """ # Name of database table for SM _table = '' # This will be the state machine table _sm_table = None # List of events that need to have _sm_class set, ie every event # list in _sm_table _sm_events = [] @classmethod def fetch_sm_obj(class_, event): """ Fetchs a State Machine object given an sm_id parameter it an event. event - event instance sm_type - state machine class returns None or sm_obj instance read in from DB. Note: the state machine object must have an 'id' parameter """ sm_id = event.py_parameters.get('sm_id') if (not sm_id): event.state = ESTATE_FAILURE event.py_results['message'] = "No sm_id given" log_error("Event %s(%s) failed: No sm_id parameter" % (event.__class__.__name__, event.id_)) return None try: sm_obj = event.db_session.query(class_)\ .with_lockmode('update')\ .filter(text("id = '%s'" % sm_id)).one() except NoResultFound: sm_obj = None if (not sm_obj): event.state = ESTATE_FAILURE event.py_results['message'] = ("SM object id '%s' not found in " "database." % sm_id) # We typically delete SMs, still with events in the queue... log_debug( "Event %s(%s) failed: no state machine sm_id %s (deleted?)" % (event.__class__.__name__, event.id_, sm_id)) return None return sm_obj def execute_state_machine(self, event): """ Execute the state machine for the given event """ event_id = event.id_ event_class_name = event.__class__.__name__ if (not self._sm_table): msg = ("%s - State machine event table not filled out." % self.__class__.__name__) log_critical(msg) event.py_results['message'] = msg event.state = ESTATE_FAILURE return RCODE_FATAL event.py_results['pre_event_sm_state'] = self.state action = self._sm_table[self.state].get(event.__class__) if (not action): msg = ("Event %s(%s) no change - state '%s'" % (event_class_name, event_id, self.state)) log_debug(msg) event.py_results['message'] = msg event.state = ESTATE_SUCCESS return RCODE_NOCHANGE try: (return_code, msg) = action(self, event) except StateMachineFatalError as exc: msg = ("Event %s(%s) failed - %s" % (event_class_name, event_id, str_exc(exc))) log_error(msg) event.py_results['message'] = msg event.py_results['sm_exception_type'] = exc.__class__.__name__ event.py_results['sm_exception_args'] = exc.args event.state = ESTATE_FAILURE return RCODE_FATAL except StateMachineError as exc: msg = ("Event %s(%s) errored - %s" % (event_class_name, event_id, str_exc(exc))) log_info(msg) event.py_results['message'] = msg event.py_results['sm_exception_type'] = exc.__class__.__name__ event.py_results['sm_exception_args'] = exc.args event.state = ESTATE_RETRY return RCODE_ERROR msg = ("Event %s(%s) processed - %s." % (event_class_name, event_id, msg)) log_info(msg) event.state = ESTATE_SUCCESS event.py_results['message'] = msg event.py_results['post_event_sm_state'] = self.state return return_code python-magcode-core-1.5.4/magcode/core/database/utility.py000066400000000000000000000356551342265675400235710ustar00rootroot00000000000000# # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ Set up core database functionality. Currently just the Event Queueing system """ import os import sys import socket import copy from sqlalchemy import create_engine from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import scoped_session from sqlalchemy.sql.expression import func from sqlalchemy.sql.expression import extract from sqlalchemy.exc import DBAPIError from sqlalchemy.exc import NoSuchTableError from sqlalchemy import Table from sqlalchemy import MetaData from sqlalchemy.orm import mapper from magcode.core.globals_ import settings from magcode.core.globals_ import debug from magcode.core.globals_ import debug_verbose from magcode.core.globals_ import startup_log_func from magcode.core.logging import * # Result codes RCODE_FATAL = 550 _RCODE_FATAL_NAME = 'FATAL' RCODE_FATAL_STR = 'Fatal Error' RCODE_FATAL_LSTR = 'Fatal Error - not recoverable' RCODE_RESET = 460 _RCODE_RESET_NAME = 'RESET' RCODE_RESET_STR = 'Resetable Error' RCODE_RESET_LSTR = 'Resetable Error - SM reset required' RCODE_ERROR = 450 _RCODE_ERROR_NAME = 'ERROR' RCODE_ERROR_STR = 'Recoverable Error' RCODE_ERROR_LSTR = 'Recoverable Error - temporary failure' RCODE_NOCHANGE = 220 _RCODE_NOCHANGE_NAME = 'NOCHANGE' RCODE_NOCHANGE_STR = 'No Change' RCODE_NOCHANGE_LSTR = 'No Change - no change to data' RCODE_NOEFFECT = 210 _RCODE_NOEFFECT_NAME = 'NOEFFECT' RCODE_NOEFFECT_STR = 'No Effect' RCODE_NOEFFECT_LSTR = 'No Effect - operation happened but no change' RCODE_OK = 200 _RCODE_OK_NAME = 'OK' RCODE_OK_STR = 'OK' RCODE_OK_LSTR = 'OK - success, data changed or new data' result_error_codes = (RCODE_FATAL, RCODE_RESET, RCODE_ERROR) result_success_codes = (RCODE_NOCHANGE, RCODE_NOEFFECT, RCODE_OK) result_codes = result_error_codes + result_success_codes result_code_names = (_RCODE_FATAL_NAME, _RCODE_RESET_NAME, _RCODE_ERROR_NAME, _RCODE_NOCHANGE_NAME, _RCODE_NOEFFECT_NAME, _RCODE_OK_NAME) result_code_strs = (RCODE_FATAL_STR, RCODE_RESET_STR, RCODE_ERROR_STR, RCODE_NOCHANGE_STR, RCODE_NOEFFECT_STR, RCODE_OK_STR) result_code_lstrs = (RCODE_FATAL_LSTR, RCODE_RESET_LSTR, RCODE_ERROR_LSTR, RCODE_NOCHANGE_LSTR, RCODE_NOEFFECT_LSTR, RCODE_OK_LSTR) result_code_dict = dict(zip(result_codes, result_code_names)) result_code_str_dict = dict(zip(result_codes, result_code_strs)) result_code_lstr_dict = dict(zip(result_codes, result_code_lstrs)) # initialise settings dict so that Process.read_config() can tell that these # are valid settings['db_user'] = '' settings['db_password'] = '' settings['db_host'] = '' settings['db_port'] = '' settings['db_name'] = '' settings['db_driver'] = 'psycopg2' settings['db_type'] = '' settings['db2_user'] = '' settings['db2_password'] = '' settings['db2_host'] = '' settings['db2_port'] = '' settings['db2_name'] = '' settings['db2_driver'] = 'psycopg2' settings['db2_type'] = '' settings['db3_user'] = '' settings['db3_password'] = '' settings['db3_host'] = '' settings['db3_port'] = '' settings['db3_name'] = '' settings['db3_driver'] = 'psycopg2' settings['db3_type'] = '' settings['db4_user'] = '' settings['db4_password'] = '' settings['db4_host'] = '' settings['db4_port'] = '' settings['db4_name'] = '' settings['db4_driver'] = 'psycopg2' settings['db4_type'] = '' # Initialise data section sql_data = {} sql_data['connections'] = {} sql_data['db_engine'] = None sql_data['scoped_session_class'] = None sql_data['db_session'] = None sql_data['simple_classes'] = [] sql_data['init_list'] = [] sql_data['mappers'] = {} sql_data['tables'] = {} sql_data['metadata'] = None sql_data['types'] = {} sql_types = sql_data['types'] sql_data['events'] = {} sql_events = sql_data['events'] # SQL alchemy/type registration functions and decorators def typeregister(class_): sql_data['types'][class_.__name__] = class_ return class_ def saregister(class_): db_name = class_._db_name if hasattr(class_, '_db_name') else None settings_db_name_key = class_._settings_db_name_key \ if hasattr(class_, '_settings_db_name_key') else None reg_dict = {'class_': class_, 'table': class_._table, 'db_name': db_name, 'settings_db_name_key': settings_db_name_key} sql_data['simple_classes'].append(reg_dict) # also register in the types register... typeregister(class_) return class_ # Python Reserved Words _reserved_words_to_twist = ('False', 'class', 'finally', 'is', 'return', 'None', 'continue', 'for', 'lambda', 'try', 'True', 'def', 'from', 'nonlocal', 'while', 'and', 'del', 'global', 'not', 'with', 'as', 'elif', 'if', 'or', 'yield', 'assert', 'else', 'import', 'pass', 'break', 'except', 'in', 'raise',) # things in our code that look ugly... _reserved_words_to_twist += ('type', 'id',) def mapper_properties(table, class_): """ Utility function to add '_' to DB column names that are reserved words """ properties = {} if hasattr(class_, '_mapper_properties'): more_properties = class_._mapper_properties() properties.update(more_properties) # Work around reserved words This adds '_' to the attribute in python for rsrvd_word in _reserved_words_to_twist: if (rsrvd_word in table.c.keys()): properties[rsrvd_word +'_'] = table.c.get(rsrvd_word) return properties def _sa_table_simple_class(class_, table, db_name=None, settings_db_name_key=None): """ Map a simple SQL row table """ if db_name: db_engine = sql_data['connections'][db_name]['engine'] elif settings_db_name_key: db_name = settings[settings_db_name_key] db_engine = sql_data['connections'][db_name]['engine'] else: db_engine = sql_data['db_engine'] table = Table(table, sql_data['metadata'], autoload=True, autoload_with=db_engine) sql_data['tables'][class_] = table def _sa_map_simple_class(class_, table, db_name=None, settings_db_name_key=None): """ Map a simple SQL row table """ tbl = sql_data['tables'][class_] sql_data['mappers'][class_] = mapper(class_, sql_data['tables'][class_], properties=mapper_properties(tbl, class_)) # Global vars to map DB driver specific exceptions _db_conn_exc_list = [] _db_type_dict = {'oursql': 'mysql', 'psycopg2': 'postgresql', 'postgresql': 'postgresql'} def is_db_connection_exception(exc): """ Call back function to handle DB specific exceptions due to database shutdown or sudden termination. """ if type(exc) in _db_conn_exc_list: return True return False def _create_engine(db_type=None, db_driver=None, db_host=None, db_port=None, db_name=None, db_user=None, db_password=None, **kwargs): # Set up some sensible defaults if not db_type: db_type = _db_type_dict.get(db_driver) if db_type == 'postgresql': db_port = 5432 if db_port == None else db_port elif db_type == 'mysql': db_port = 3306 if db_port == None else db_port db_host = 'localhost' if not db_host else db_host # Import DB driver sepecific Exceptions needed for EventQueue global _db_conn_exc_list def add_conn_exc(exc): if exc not in _db_conn_exc_list: _db_conn_exc_list.append(exc) if ( db_driver == 'pypostgresql'): import postgresql add_conn_exc(postgresql.exceptions.AdminShutdownError), add_conn_exc(postgresql.exceptions.ConnectionFailureError) add_conn_exc(postgresql.exceptions.Error) elif (db_driver == 'psycopg2'): import psycopg2 add_conn_exc(psycopg2.OperationalError) add_conn_exc(psycopg2.DatabaseError) add_conn_exc(psycopg2.InterfaceError) elif (db_driver == 'oursql'): import oursql add_conn_exc(oursql.OperationalError) add_conn_exc(oursql.DatabaseError) add_conn_exc(oursql.InterfaceError) else: error_str = "setup_sqlalchemy() - no 'db_driver' set, use 'psycopg2' "\ "or 'py-postgresql'." startup_log_func(error_str) raise RuntimeError(error_str) # Create connection string if (db_type == 'postgresql' and db_port == ''): # attempt to use UNIX socket... connect_string = ('postgresql+%(db_driver)://%(db_user)s:%(db_password)s@/%(db_name)s') else: connect_string = ('%(db_type)s+%(db_driver)s://%(db_user)s:%(db_password)s@%(db_host)s:%(db_port)s/%(db_name)s') connect_string = connect_string % {'db_type': db_type, 'db_driver': db_driver, 'db_user': db_user, 'db_password': db_password, 'db_host': db_host, 'db_port': db_port, 'db_name': db_name} if debug_verbose(): log_debug("setup_sqlalchemy() - connect_string: %s" % connect_string) # Create engine log_info("setup_sqlalchemy() - using db_driver '%s'." % db_driver) engine = create_engine(connect_string) try: # Connect and test conect information # otherwise get XXXX Exception traces! connection = engine.connect() error_str = None except DBAPIError as exc: error_str = str(exc) startup_log_func("setup_sqlalchemy() - %s" % error_str) raise exc # scoped_session() makes class Session thread local Session = scoped_session(sessionmaker(bind=engine)) return(engine, Session) def _add_connection(db_name, db_engine, Session): """ Add an SA DB connection to the sql_data connections dictionary """ if db_name in sql_data['connections'].keys(): # Reasonably protected by above connect attempt - a traceback here is OK msg = ("setup_sqlalchemy() - DB '%s' already exists in connections array." % db_name) startup_log_func(msg) raise KeyError(msg) sql_data['connections'][db_name] = {} sql_data['connections'][db_name]['session_class'] = Session sql_data['connections'][db_name]['engine'] = db_engine def core_setup_sqlalchemy(): """ Connects to PostgreSQL, and sets up table mappings. Can only be called once. This one is made to be called from wsgi, and by wrapper below. """ # Set up SQL alchemy debug if debug_extreme(): logging.getLogger('sqlalchemy').setLevel(logging.DEBUG) elif debug_verbose(): logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO) # Create DB Connections db_engine, Session = _create_engine(db_driver=settings['db_driver'], db_host=settings['db_host'], db_name=settings['db_name'], db_user=settings['db_user'], db_password=settings['db_password']) # Legacy sql_data entries sql_data['db_engine'] = db_engine sql_data['scoped_session_class'] = Session _add_connection(settings['db_name'], db_engine, Session) if settings['db2_name']: db_engine, Session = _create_engine(db_driver=settings['db2_driver'], db_host=settings['db2_host'], db_name=settings['db2_name'], db_user=settings['db2_user'], db_password=settings['db2_password']) _add_connection(settings['db2_name'], db_engine, Session) if settings['db3_name']: db_engine, Session = _create_engine(db_driver=settings['db3_driver'], db_host=settings['db3_host'], db_name=settings['db3_name'], db_user=settings['db3_user'], db_password=settings['db3_password']) _add_connection(settings['db3_name'], db_engine, Session) if settings['db4_name']: db_engine, Session = _create_engine(db_driver=settings['db4_driver'], db_host=settings['db4_host'], db_name=settings['db4_name'], db_user=settings['db4_user'], db_password=settings['db4_password']) _add_connection(settings['db4_name'], db_engine, Session) # Map tables and kick SQL Alchemy into gear! # Set up metadata sql_data['metadata'] = MetaData() try: # First of all, load table definitions # simple SQL row class tables for simple_class in sql_data['simple_classes']: _sa_table_simple_class(**simple_class) # Deal with other SQL classes (not so many!) for init in sql_data['init_list']: init['table']() # simple SQL row class tables for simple_class in sql_data['simple_classes']: _sa_map_simple_class(**simple_class) # Deal with other SQL classes (not so many!) for init in sql_data['init_list']: init['mapper']() except DBAPIError as exc: # Reasonably protected by above connect attempt - a traceback here is OK startup_log_func("setup_sqlalchemy() - %s" % sys.exc_info()[1]) raise exc except NoSuchTableError as exc: startup_log_func("setup_sqlalchemy() - No such table '%s'." % exc) raise exc def setup_sqlalchemy(): """ Connects to PostgreSQL, and sets up table mappings. Can only be called once. This one is made to be called from daemon and command line applications. This one just exists to go sys.exit appropriately! """ try: core_setup_sqlalchemy() except DBAPIError: systemd_exit(os.EX_NOHOST, SDEX_CONFIG) except NoSuchTableError: systemd_exit(os.EX_NOINPUT, SDEX_UNIMPLEMENTED) except RuntimeError: systemd_exit(os.EX_CONFIG, SDEX_CONFIG) def db_clock_unixtime(db_session): """ Returns database current clock time as unix timestamp """ time = db_session.query(extract('epoch', func.clock_timestamp())).scalar() return time def db_now_unixtime(db_session): """ Returns database start of transaction clock time as unix timestamp """ time = db_session.query(extract('epoch', func.now())).scalar() return time def db_time(db_session): """ Returns database timestamp with time zone from now(), the timestamp of the current transaction. """ time = db_session.execute(func.now()).scalar() return time def db_clock_time(db_session): """ Returns database timestamp of the current clock time """ time = db_session.execute(func.clock_timestamp()).scalar() return time python-magcode-core-1.5.4/magcode/core/globals_.py000066400000000000000000000211311342265675400220640ustar00rootroot00000000000000# # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """Application Global Data This module is set up to contain application global data, such as process name, configuration file, log function name etc. """ import os import sys import syslog import logging # Some helper stuff to make more sense of mess in logging module from logging import log as log from logging import critical as log_critical from logging import critical as log_fatal from logging import error as log_error from logging import error as log_err from logging import warning as log_warning from logging import warning as log_warn from logging import info as log_info from logging import debug as log_debug from logging import BASIC_FORMAT from logging import CRITICAL as MAGLOG_CRITICAL from logging import CRITICAL as MAGLOG_FATAL from logging import ERROR as MAGLOG_ERROR from logging import ERROR as MAGLOG_ERR from logging import WARNING as MAGLOG_WARNING from logging import WARNING as MAGLOG_WARN from logging import INFO as MAGLOG_INFO from logging import DEBUG as MAGLOG_DEBUG from logging import NOTSET as MAGLOG_NOTSET from logging import NOTSET as MAGLOG_NONE # For setproctitle() further down - a nice to have for admin and ps try: from setproctitle import setproctitle setproctitle_support = True except ImportError: setproctitle_support = False process_name = os.path.basename(sys.argv[0]) # Dictionary where we store program settings settings = {} # Dictionary where we store golbal data # Split from settings as it will more than strings, ie include python # instatated objects. Means we can dump settings to a configuration file! data = {} _uname = os.uname() _os_family = _uname[0] hostname_fqdn = _uname[1] hostname = _uname[1].split('.',1)[0] settings['hostname'] = hostname if (_os_family == "FreeBSD"): CONFIG_DIR = "/usr/local/etc/magcode" VAR_LIB_DIR = "/var/lib/magcode" LOG_DIR = "/var/log/magcode" RUN_DIR = "/var/run/magcode" CONFIG_FILE = CONFIG_DIR + "/magcode.conf" PID_FILE = RUN_DIR + "/" + process_name + ".pid" LOG_FILE = LOG_DIR + process_name + ".log" PANIC_LOG = LOG_DIR + process_name + "-panic.log" elif(_os_family == "Linux"): CONFIG_DIR = "/etc/magcode" VAR_LIB_DIR = "/var/lib/magcode" LOG_DIR = "/var/log/magcode" RUN_DIR = "/var/run/magcode" CONFIG_FILE = CONFIG_DIR + "/magcode.conf" PID_FILE = RUN_DIR + "/" + process_name + ".pid" LOG_FILE = LOG_DIR + process_name + ".log" PANIC_LOG = LOG_DIR + process_name + "-panic.log" else: raise Exception("Unrecognised OS '%s'" % _os_family) settings['os_family'] = _os_family DEBUG_LEVEL_NONE = 0 # Process debug log messages only DEBUG_LEVEL_NORMAL = 1 # Include SQL ALchemy SQL debug messages DEBUG_LEVEL_VERBOSE = 2 # Include SQL ALchemy debug messages (includes SQL) DEBUG_LEVEL_EXTREME = 3 settings['debug_level'] = DEBUG_LEVEL_NONE # Systemd daemon exit codes from # http://refspecs.linuxbase.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/iniscrptact.html SDEX_GENERIC = 1 #generic or unspecified error (current practice) SDEX_USAGE = 2 #invalid or excess argument(s) SDEX_UNIMPLEMENTED = 3 #unimplemented feature SDEX_NOPERM = 4 # insufficient priveilege SDEX_NOTINSTALLED = 5 # Program is not installed SDEX_CONFIG = 6 # Program is not configured SDEX_NOTRUNNING = 7 # Program is not running # Run as Systemd daemon settings['systemd'] = False # Directory settings settings['config_dir'] = CONFIG_DIR settings['log_dir'] = LOG_DIR settings['var_lib_dir'] = VAR_LIB_DIR settings['run_dir'] = RUN_DIR # File path settings settings['config_file'] = CONFIG_FILE settings['pid_file'] = PID_FILE settings['log_file'] = LOG_FILE settings['panic_log'] = PANIC_LOG # Event Queue PID file location # set to what ever needed in application globals_.py settings['event_queue_pid_file'] = settings['pid_file'] settings['run_as_user'] = '' # Initialise to '' for config file checks # Process type settings PROCESS_TYPE_UTILITY = "utility" PROCESS_TYPE_DAEMON = "daemon" PROCESS_TYPE_WSGI = "wsgi" settings['process_type'] = PROCESS_TYPE_UTILITY # Something to see ti daemon reconfiguring settings['daemon_canary'] = "pink polka dot" # Some process_name trickery to handle editor choices via /etc/passwd editor_sep = process_name.rfind('~') settings['editor_flag'] = '' if editor_sep > -1: settings['editor_flag'] = process_name[editor_sep+1:] process_name = process_name[:editor_sep] # Set our process title to something sensible other than python3.x ! if setproctitle_support: setproctitle(process_name) settings['process_name'] = process_name # Preconvert settings list for use in code start up settings['preconvert_bool_settings'] = '' settings['preconvert_int_settings'] = '' settings['preconvert_float_settings'] = '' # logging.py settings # Initialise Default Log settings settings['syslog_facility'] = syslog.LOG_USER settings['log_level'] = logging.INFO settings['log_file_max_size_kbytes'] = 2048 settings['log_file_backup_count'] = 7 log_message_format = "%(asctime)s " + process_name \ + "[%(process)d]: " + "%(levelname)s - %(message)s" settings['log_message_format'] = log_message_format syslog_message_format = "%(levelname)s - %(message)s" settings['syslog_message_format'] = syslog_message_format log_message_date_format = "%b %e %H:%M:%S" settings['log_message_date_format'] = log_message_date_format stderr_message_format = process_name + ": %(message)s" settings['stderr_message_format'] = stderr_message_format wsgi_message_format = process_name + " (pid=%(process)d, threadid=%(thread)d): %(levelname)s - %(message)s" settings['wsgi_message_format'] = wsgi_message_format # JSON RPC settings settings['jsonrpc_max_content_length'] = 102400 settings['jsonrpc_error_stack_trace'] = False # system_editor_pager.py settings['diff_args'] = '-uNw' settings['pager_args'] = '' settings['tail_args'] = '-n +3' if (settings['os_family'] == "Linux"): settings['editor'] = '/usr/bin/rvim' settings['editor_rvim'] = '/usr/bin/rvim' settings['editor_rnano'] = '/bin/rnano' else: settings['editor'] = '/usr/local/bin/rvim' settings['editor_rvim'] = '/usr/local/bin/rvim' settings['editor_rnano'] = '/usr/local/bin/rnano' settings['pager'] = '/usr/bin/less' settings['admin_group_list'] = 'sudo root wheel' # database/event.py settings['event_queue_threads'] = 3 # Processor threads never time out settings['event_queue_thread_timeout'] = None # Maximum queue size settings['event_queue_maxsize'] = 50 # Event queue session transactions per Session.close() settings['event_queue_session_transactions'] = 1000 if not settings.get('event_queue_fkey_columns'): settings['event_queue_fkey_columns'] = '' # Utility functions def debug(): """ utility function to return if debugging or not """ return (settings['debug_level'] > DEBUG_LEVEL_NONE) def debug_verbose(): """ utility function to return if verbose debugging or not """ return (settings['debug_level'] >= DEBUG_LEVEL_VERBOSE) def debug_extreme(): """ utility function to return if extreme debugging or not """ return (settings['debug_level'] >= DEBUG_LEVEL_EXTREME) def systemd(): """ Utility function to return if running as systemd daemon """ return (settings['systemd']) def systemd_exit(normal_exit_code=1, sd_exit_code=1): if systemd(): sys.exit(sd_exit_code) sys.exit(normal_exit_code) def str_exc(exc): return exc.__class__.__name__ + ': ' + str(exc) # Sort out which logging function to use - command line tools set log level # to critical as they can turn off log to stderr in favour of their # own messages def startup_log_func(*args): if (settings['process_type'] == PROCESS_TYPE_UTILITY): # Set up for zone_tool return log_critical(*args) else: return log_error(*args) python-magcode-core-1.5.4/magcode/core/logging.py000066400000000000000000000234131342265675400217350ustar00rootroot00000000000000# # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """Logging and Debugging IO module Provides daemon logging, and debug messaging support. Uses the logging root logger. If debug_level is greater than 0, stderr is added to the root logger as an additional handler. Given that only one process of the same name is allowed to run at a time on a machine, logging to log files and syslog should also happen at the same time. """ import os import sys import syslog import logging import logging.handlers try: import codecs except ImportError: codecs = None from magcode.core.globals_ import * from magcode.core.utility import get_numeric_setting log_object = None class MagCodeSysLogHandler(logging.handlers.SysLogHandler): """Override broken SysLogHandler Messages in log end up with 3 bytes of cruft form misencoded priority integer. This implementation uses libc functions to fix things """ def __init__(self, address=None, facility=syslog.LOG_USER, socktype=None): """ Open a logging session """ logging.Handler.__init__(self) self.facility = facility syslog.openlog(process_name, syslog.LOG_PID|syslog.LOG_NDELAY, facility) self.formatter = None def close(self): """ Closes logging session in libc """ syslog.closelog() def emit(self, record): """ Emit a record. The record is formatted, and then sent to the libc syslog function. If exception information is present, it is NOT sent to the server. """ msg = self.format(record) + '\000' """ We need to convert record level to lowercase, maybe this will change in the future. """ syslog_priority = self.mapPriority(record.levelname) syslog_priority = self.priority_names[syslog_priority] # Message is a string. Convert to bytes as required by RFC 5424 # Following is done in syslog.syslog() # msg = msg.encode('utf-8') # The following is where the 3 bytes of crap come from... #if codecs: # msg = codecs.BOM_UTF8 + msg syslog.syslog(syslog_priority, msg) class MagCodeLog(object): """ Generic Logging Object """ def __init__(self): self._init_log_handlers() def _init_log_handlers(self): """ Initialise basic logging """ # Log Formatters self._configure_log_formatters() # set logging level self.set_logging_level() # Set up stderr log handler self._configure_stderr_logging() if (settings['process_type'] == PROCESS_TYPE_DAEMON): self.configure_syslog_logging() def _configure_stderr_logging(self): """ Set up stderr log handler """ # Remove any existing handler if (hasattr(self, 'stderr_handler') and self.stderr_handler): logging.root.removeHandler(self.stderr_handler) # Add new handler depending on process_type or debug() if (debug()): self.stderr_handler = logging.StreamHandler() self.stderr_handler.setFormatter(self.log_formatter) logging.root.addHandler(self.stderr_handler) elif (settings['process_type'] == PROCESS_TYPE_DAEMON): self.stderr_handler = logging.StreamHandler() self.stderr_handler.setFormatter(self.stderr_formatter) logging.root.addHandler(self.stderr_handler) elif (settings['process_type'] == PROCESS_TYPE_UTILITY): self.stderr_handler = logging.StreamHandler() self.stderr_handler.setFormatter(self.stderr_formatter) logging.root.addHandler(self.stderr_handler) elif (settings['process_type'] == PROCESS_TYPE_WSGI): self.stderr_handler = logging.StreamHandler() self.stderr_handler.setFormatter(self.wsgi_formatter) logging.root.addHandler(self.stderr_handler) def _configure_log_formatters(self): """ Configure the log formaters """ self.fs = settings['log_message_format'] self.sfs = settings['syslog_message_format'] self.sfe = settings['stderr_message_format'] self.sfw = settings['wsgi_message_format'] self.dfs = settings['log_message_date_format'] self.log_formatter = logging.Formatter(self.fs, self.dfs) self.stderr_formatter = logging.Formatter(self.sfe, None) self.syslog_formatter = logging.Formatter(self.sfs, None) self.wsgi_formatter = logging.Formatter(self.sfw, None) def set_logging_level(self): """ Set logging level """ if (debug()): logging.root.setLevel(MAGLOG_DEBUG) else: try: logging.root.setLevel(settings['log_level']) except ValueError: log_error("log_level - '%s' is not defined." % settings['log_level']) systemd_exit(os.EX_CONFIG, SDEX_CONFIG) def _remove_all_log_handlers(self): if (hasattr(self, 'stderr_handler') and self.stderr_handler): logging.root.removeHandler(self.stderr_handler) if (hasattr(self, 'syslog_handler') and self.syslog_handler): logging.root.removeHandler(self.syslog_handler) if (hasattr(self, 'logfile_handler') and self.logfile_handler): logging.root.removeHandler(self.logfile_handler) def reconfigure_logging(self): """ Reconfigures logging after configuration read on startup """ #self._remove_all_log_handlers() self._init_log_handlers() #self.set_logging_level() def _get_syslog_const(self, facility): """ Internal function to translate LOG_ strings to defines in syslog module """ if type(facility) is not str: return facility log_cstr = facility if not log_cstr.startswith('LOG_'): log_cstr = 'LOG_' + log_cstr try: return getattr(syslog, log_cstr) except AttributeError: log_error("syslog_facility - '%s' is not defined." % log_cstr) systemd_exit(os.EX_CONFIG, SDEX_CONFIG) def configure_syslog_logging(self): """ Configure syslog logging Typically done for daemons """ # Set up syslog logging if (hasattr(self, 'syslog_handler') and self.syslog_handler): logging.root.removeHandler(self.syslog_handler) if (settings['syslog_facility']): facility = self._get_syslog_const(settings['syslog_facility']) self.syslog_handler = MagCodeSysLogHandler(facility=facility) self.syslog_handler.setFormatter(self.syslog_formatter) logging.root.addHandler(self.syslog_handler) def configure_file_logging(self): """ Configure file logging This is typically done after we have syslog running or stderr and after processing the configuration file in a daemon process """ if (hasattr(self, 'logfile_handler') and self.logfile_handler): logging.root.removeHandler(self.logfile_handler) if (settings['log_file']): try: maxBytes = 1024 * get_numeric_setting( 'log_file_max_size_kbytes', int) logfile_handler = logging.handlers.RotatingFileHandler( filename=settings['log_file'], maxBytes=maxBytes, backupCount=settings['log_file_backup_count']) except (IOError, OSError) as e: if (e.filename): log_error("%s - %s." % (e.filename, e.strerror)) else: log_error("%s." % e.strerror) return logfile_handler.setFormatter(self.log_formatter) logging.root.addHandler(logfile_handler) self.logfile_handler = logfile_handler def remove_daemon_stderr_logging(self): """ Remove stderr logging prior to forking away to daemon land """ if (not debug()): logging.root.removeHandler(self.stderr_handler) def setup_logging(): """ Set up logging on application start up """ global log_object if (not log_object): log_object = MagCodeLog() def reconfigure_logging(): """ Reconfigure logging after reading configuration file """ global log_object log_object.reconfigure_logging() def setup_file_logging(): """ Set up file logging, typically once configuration file is read """ global log_object log_object.configure_file_logging() def setup_syslog_logging(): """ Setup syslog logging. Some Utilites may want this. """ global log_object log_object.configure_syslog_logging() def remove_daemon_stderr_logging(): """ Remove stderr logging prior to forking away to daemon land """ global log_object log_object.remove_daemon_stderr_logging() python-magcode-core-1.5.4/magcode/core/process.py000066400000000000000000001071231342265675400217660ustar00rootroot00000000000000#!/usr/bin/env python3.2 # # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ This module contains the addon functionality to help a process become a daemon. It is implemented in a classful manner, with various methods performing the requisite behaviour. Its a bit rough, but it gets the job done. """ import os import sys import pwd import errno import signal import select import time import resource import configparser from getopt import gnu_getopt from getopt import getopt from getopt import GetoptError from fcntl import fcntl from fcntl import F_GETFL # A bit of nice stuff to set up ps output as much as we can... try: from setproctitle import getproctitle setproctitle_support = True except ImportError: setproctitle_support = False from magcode.core.globals_ import * from magcode.core.logging import setup_logging from magcode.core.logging import reconfigure_logging from magcode.core.logging import setup_file_logging from magcode.core.logging import remove_daemon_stderr_logging from magcode.core.utility import core_read_config from magcode.core.utility import MagCodeConfigError # Default working directory WORKDIR = "/" # Process Umask # Daemonization process sets this to 0000, but set to something sensible # as we don't want world writable files being created from any daemons!!! #UMASK = 0o0000 UMASK = 0o0022 # Default Max FDs MAXFD = 1024 REDIRECT_TO = "/dev/null" # Default Command line help messages STUB_USAGE_MESSAGE = "Usage: %s [-dhv] [-c config_file] (Stub 'usage_short()' msg)" STUB_COMMAND_DESCRIPTION = "Command description (Stub 'usage_full()' message.)" class BaseCmdLineArg(object): """ Ancestor primitive to establish universal properties of command line argument objects. Mainly used as it gives an ancestor type we can test for. """ def __init__(self, short_arg, help_text, long_arg=None, *args, **kwargs): """ Initialise object. Short_arg must have ':' appended if value required. Long_arg must have '=' appended if value is required. """ self.short_arg = short_arg if (long_arg): self.long_arg = long_arg else: self.long_arg = ' ' self.help_text = help_text def process_arg(self, process, argument=None, value=None, *args, **kwargs): """ Stub method for processing an argument. process is the process class instance Processes argument, then returns if argument processed successfully, otherwise exits program. """ process.usage_short() systemd_exit(os.EX_USAGE, SDEX_USAGE) class HelpCmdLineArg(BaseCmdLineArg): """ Process help command line argument """ def __init__(self): BaseCmdLineArg.__init__(self, short_arg='h', long_arg='help', help_text='help message') def process_arg(self, process, *args, **kwargs): """ Process the help argument """ process.usage_full() sys.exit(os.EX_OK) class BooleanCmdLineArg(BaseCmdLineArg): """ Process boolean command Line setting short_arg - single letter cmd line argument long_arg - single letter cmd line argument (optional) help_text - single line help text settings_key - key to be used in settings dictionary settings_defalut_value - default value of setting settings_set_value - setting of value if switch given. """ def __init__(self, short_arg, help_text, settings_key, settings_default_value, settings_set_value, long_arg=None, *args, **kwargs): BaseCmdLineArg.__init__(self, short_arg=short_arg, long_arg=long_arg, help_text=help_text) self.settings_key = settings_key self.settings_default_value = settings_default_value self.settings_set_value = settings_set_value # Initialise settings dictionary to default value settings[settings_key] = settings_default_value def process_arg(self, *args, **kwargs): """ Change the setting if command line argument given """ settings[self.settings_key] = self.settings_set_value class VerboseCmdLineArg(BooleanCmdLineArg): """ Process verbose command Line setting """ def __init__(self): BooleanCmdLineArg.__init__(self, short_arg='v', long_arg='verbose', help_text='verbose output', settings_key = 'verbose', settings_default_value = False, settings_set_value = True) class MemoryDebugCmdLineArg(BooleanCmdLineArg): """ Process Garbage Collector command line setting """ def __init__(self): BooleanCmdLineArg.__init__(self, short_arg='b', long_arg='memory-debug', help_text='memory debug output', settings_key = 'memory_debug', settings_default_value = False, settings_set_value = True) class ConfigCmdLineArg(BaseCmdLineArg): """ Handle configuration file setting """ def __init__(self): BaseCmdLineArg.__init__(self, short_arg='c:', long_arg="config-file=", help_text="set configuration file") # Don't have to set default value here as it is set in # magcode.core.globals_ def process_arg(self, process, value, *args, **kwargs): """ Set configuration file name """ settings['config_file'] = value class ForceCmdLineArg(BooleanCmdLineArg): """ Process force command Line setting """ def __init__(self): BooleanCmdLineArg.__init__(self, short_arg='f', long_arg='force', help_text="Force operation, say if calling from a script", settings_key = 'force_arg', settings_default_value = False, settings_set_value = True) class Rpdb2WaitCmdLineArg(BaseCmdLineArg): """ Handle waiting for rpdb2 in debug mode """ def __init__(self): BaseCmdLineArg.__init__(self, short_arg='r:', long_arg="rpdb2-wait=", help_text="Wait for rpdb2 (seconds)") settings['rpdb2_wait'] = None def process_arg(self, process, value, *args, **kwargs): """ Set time out for rpdb2 """ settings['rpdb2_wait'] = value class SystemdCmdLineArg(BooleanCmdLineArg): """ Start up as a systemd daemon. Prevents daemonise, and changes logging. """ def __init__(self): BooleanCmdLineArg.__init__(self, short_arg='S', long_arg='systemd', help_text="Run as a systemd daemon, no fork", settings_key = 'systemd', settings_default_value = False, settings_set_value = True) class DebugCmdLineArg(BaseCmdLineArg): """ Process debug command line argument """ debug_levels = { 'none': DEBUG_LEVEL_NONE, 'zero': DEBUG_LEVEL_NONE, 'nada': DEBUG_LEVEL_NONE, 'zilch': DEBUG_LEVEL_NONE, 'no': DEBUG_LEVEL_NONE, 'false': DEBUG_LEVEL_NONE, 'process': DEBUG_LEVEL_NORMAL, 'normal': DEBUG_LEVEL_NORMAL, 'norm': DEBUG_LEVEL_NORMAL, 'yes': DEBUG_LEVEL_NORMAL, 'true': DEBUG_LEVEL_NORMAL, 'verbose': DEBUG_LEVEL_VERBOSE, 'verb': DEBUG_LEVEL_VERBOSE, 'sql': DEBUG_LEVEL_VERBOSE, 'extreme': DEBUG_LEVEL_EXTREME, 'everything': DEBUG_LEVEL_EXTREME, 'nuts': DEBUG_LEVEL_EXTREME, } def __init__(self): BaseCmdLineArg.__init__(self, short_arg='d:', long_arg='debug=', help_text='set debug level {0-3|none|normal|verbose|extreme}') self.debug_levels = DebugCmdLineArg.debug_levels settings['debug_level'] = DEBUG_LEVEL_NONE def process_arg(self, process, value, *args, **kwargs): """ Process debug level argument """ try: val = int(value) if (val < DEBUG_LEVEL_NONE and val > DEBUG_LEVEL_EXTREME): process.usage_full() systemd_exit(os.EX_USAGE, SDEX_USAGE) settings['debug_level'] = val except ValueError: if (value in DebugCmdLineArg.debug_levels.keys()): settings['debug_level'] = DebugCmdLineArg.debug_levels[value] else: process.usage_full() systemd_exit(os.EX_USAGE, SDEX_USAGE) DEFAULT_CMDLINE_ARG_LIST = [ConfigCmdLineArg(), DebugCmdLineArg(), HelpCmdLineArg(), VerboseCmdLineArg()] class ProcessCmdLine(object): """ Ancestor class to set up command line argument handling """ def __init__(self, usage_message=STUB_USAGE_MESSAGE, command_description=STUB_COMMAND_DESCRIPTION, cmdline_arg_list = DEFAULT_CMDLINE_ARG_LIST, use_gnu_getopt=True, *args, **kwargs): """ Initialise command line usage help messages, and command line argments """ self.usage_message = usage_message self.command_description = command_description self.cmdline_arg_list = cmdline_arg_list self.use_gnu_getopt = use_gnu_getopt super().__init__(*args, **kwargs) def parse_args(self, argv=sys.argv, argc=len(sys.argv)): """ Command line argument parsing Returns arguments left over from processing Using getopt.gnu_getopt() as argparse module is too new, and gnu_getopt() allows us to easily go back to Python 2.x This also is in line with are tight control of process start up and daemonizing. """ # 1. Gather data on command line argments and create callback array short_opts = '' long_opts = [] callback_dict = {} for arg_obj in self.cmdline_arg_list: short_opts += arg_obj.short_arg long_opts.append(arg_obj.long_arg) # need to strip trailing characters specifying that argument # must be given callback_dict[arg_obj.short_arg.rstrip(':')] = arg_obj callback_dict[arg_obj.long_arg.rstrip('=')] = arg_obj # Call getopt and process each argument via its callback method try: if self.use_gnu_getopt: opts, argv_left = gnu_getopt(argv[1:], short_opts, long_opts) else: opts, argv_left = getopt(argv[1:], short_opts, long_opts) except GetoptError: self.usage_short() systemd_exit(os.EX_USAGE, SDEX_USAGE) for arg, val in opts: # strip '-' from a arg = arg.lstrip('-') if (arg not in callback_dict.keys()): self.usage_short() systemd_exit(os.EX_USAGE, SDEX_USAGE) callback_dict[arg].process_arg(process=self, argument=arg, value=val) # return unprocessed arguments. These are generally of interest # for main_process return argv_left def parse_argv_left(self, argv_left): """ Handle any arguments left after processing all switches Override in application if needed. """ if (len(argv_left) != 0): self.usage_short() systemd_exit(os.EX_USAGE, SDEX_USAGE) def usage_short(self, tty_file=sys.stderr): """ Stub short help message routine """ print(self.usage_message % process_name, file=tty_file) def usage_full(self, tty_file=sys.stdout): """ Stub verbose help message routine """ self.usage_short(tty_file=tty_file) print("\n %s\n" % self.command_description, file=tty_file) for arg_obj in self.cmdline_arg_list: print(" -%s, --%-16s %s" % (arg_obj.short_arg.rstrip(':'), arg_obj.long_arg.rstrip('='), arg_obj.help_text)) class Process(ProcessCmdLine): """ Main command line utility process class Contains all the code for a commandline utility sans the command line processing handled above. """ def main_process(self): """ Dummy main_process that can be overridden Typically does what ever, and then calls sys.exit() """ log_info ("Hello World") sys.exit(os.EX_OK) def main(self, argv=sys.argv, argc=len(sys.argv)): """ Cish command line utility type main() that can be overridden """ # set process tupe settings['process_type'] = PROCESS_TYPE_UTILITY argv_left = self.parse_args(argv, argc) self.parse_argv_left(argv_left) setup_logging() self.read_config() self.main_process() def read_config(self, config_section = None): """ Read configuration from an ini style configuration file into settings dict. config_section - Name of configuration section to be read. Method can be overridden in real main process. Have a look at the python module configparser for documentation. File has to have a '[blah]' section header, and supports a [DEFAULT] section. """ try: core_read_config(config_section) # Handle file opening and read errors except (IOError,OSError) as e: if (e.errno == errno.EPERM or e.errno == errno.EACCES): systemd_exit(os.EX_NOPERM, SDEX_NOPERM) else: systemd_exit(os.EX_IOERR, SDEX_GENERIC) # Handle all configuration file parsing errors except configparser.Error as e: systemd_exit(os.EX_CONFIG, SDEX_CONFIG) except MagCodeConfigError as e: systemd_exit(os.EX_CONFIG, SDEX_CONFIG) def check_if_root(self): """ Check that we are running as root, if not exit with message. ProcessDaemon below overrides this. It is here for script utilities. """ # check that we are root for file writing permissions stuff if (os.geteuid() != 0 ): log_error("Only root can execute this command") systemd_exit(os.EX_NOPERM, SDEX_NOPERM) def check_or_force(self, exit_on_no=True): """ Ask user if they want to go ahead. Added as it is used a lot. Need to initialise ForceCmdLineArg() for use in a process. """ if not settings['force_arg']: print(8*' ' + "Do really you wish to do this?") answer = '' while not answer: answer = input('\t--y/[N]> ') if answer in ('n', 'N', ''): if exit_on_no: systemd_exit(os.EX_TEMPFAIL, SDEX_GENERIC) return False elif answer in ('y', 'Y'): return True answer = '' continue return True class DaemonOperations(object): """ Daemon Operations Mix-in Container class This optionally does not close all file descriptors as this kills the Python logging module... Based on recipe up at http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way under the PSF license. References: 1) Advanced Programming in the Unix Environment: W. Richard Stevens 2) Unix Programming Frequently Asked Questions: http://www.erlenstar.demon.co.uk/unix/faq_toc.html """ def _init_pwd(self): """ Initialise run_as_user_pwd information """ run_as_user = settings.get('run_as_user') if (not run_as_user): return try: self.run_as_user_pwd = pwd.getpwnam(run_as_user) except KeyError as e: log_error("run_as_user '%s' not found. Exiting daemon." % run_as_user) def check_if_running(self): """ Check if another daemon process is around Returns True if we are already running, false if it is another process """ exception = None try: pid_file = open(settings['pid_file'],'r') old_pid = int(pid_file.readline().strip()) pid_file.close() # The following throws exceptions if process does not exist etc! # Sending signal 0 does not touch process, but call succeeds # if it exists os.kill(old_pid, 0) if (old_pid == os.getpid()): return True except ValueError as e: # Error from int() type conversion above log_error("%s - %s" % (settings['pid_file'], e)) except (IOError, OSError) as e: # ENOENT - File IO causes this, ESRCH - exception is from kill() if (e.errno in (errno.ENOENT, errno.ESRCH)): return False if (e.filename): log_error("Could not access PID file '%s' - %s." % (e.filename, e.strerror)) else: log_error("Another process is already running - %s." % old_pid) except: log_error("Unexpected error: %s" % sys.exc_info()[1]) systemd_exit(os.EX_OSERR, SDEX_GENERIC) else: log_error("Another process is already running - %s." % old_pid) # If we can't run at all, exit! systemd_exit(os.EX_CANTCREAT, SDEX_GENERIC) def create_pid_file(self): """ Create PID file for this process """ try: pid_file = open(settings['pid_file'], 'w') pid_file.write("%s\n" % os.getpid()) pid_file.close() except (IOError, OSError) as e: log_error("%s - %s." % (e.filename, e.strerror)) if (e.errno == errno.EPERM or e.errno == errno.EACCES): systemd_exit(os.EX_NOPERM, SDEX_NOPERM) else: systemd_exit(os.EX_IOERR, SDEX_GENERIC) def _do_forks(self): """ Daemon fork operation Split out for debugging """ # Keep track of where we are log_debug('0th fork(): PID: %s PPID: %s SID: %s PGID: %s' % (os.getpid(), os.getppid(), os.getsid(0), os.getpgid(0))) # First fork so that parent can exit try: pid = os.fork() except (IOError, OSError) as e: # log or stderr something here # raise Exception "%s [%d]" % (e.strerror, e.errno) systemd_exit(os.EX_OSERR, SDEX_GENERIC) if (pid == 0): # First child log_debug('1st fork(): PID: %s PPID: %s SID: %s PGID: %s' % (os.getpid(), os.getppid(), os.getsid(0), os.getpgid(0))) # Become session leader os.setsid() # Keep track of things log_debug('1st fork(), setsid(): PID: %s PPID: %s SID: %s PGID: %s' % (os.getpid(), os.getppid(), os.getsid(0), os.getpgid(0))) # Optionally ignore SIGHUP import signal signal.signal(signal.SIGHUP, signal.SIG_IGN) # Second fork to orphan child and detach from controlling # terminal try: pid = os.fork() except (IOError, OSError) as e: # log or stderr something here? # raise Exception "%s [%d]" % (e.strerror, e.errno) systemd_exit(os.EX_OSERR, SDEX_GENERIC) if (pid != 0): # use _exit() as it does not pfaff around with pythonic cleanup # which is not needed this early... # Exit second parent os._exit(os.EX_OK) else: # Exit first parent os._exit(0) # Keep track of stuff log_debug('2nd fork(): PID: %s PPID: %s SID: %s PGID: %s' % (os.getpid(), os.getppid(), os.getsid(0), os.getpgid(0))) def _close_fds(self, fd_top= None): """ Close all FDs that are NOT needed. Split out from daemonize for debug purposes """ # Close all open file descriptors maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1] if (maxfd == resource.RLIM_INFINITY): maxfd = MAXFD # Iterate through and close all file descriptors # Don't try and close file descriptors twice on SIGHUP as per # man 2 close(), process observed getting reaped somehow... if (not self.i_am_daemon()): for fd in range(0, maxfd): if (fd_top): # skip python logging/syslog fds etc! if (fd <= fd_top and fd > 2): continue try: os.close(fd) except (IOError, OSError): # ERROR, fd wasn't open to begin with (ignored) pass # Redirect stderr, stdout stdin # This call to open is guaranteed to return the lowest file # descriptor, which will be 0 (stdin), since it was closed above. os.open(REDIRECT_TO, os.O_RDWR) # standard input (0) # Duplicate standard input to standard output and standard error. os.dup2(0, 1) # standard output (1) os.dup2(0, 2) def _open_panic_log(self, close_all_fds): """ Open the panic log Hidden class split out for debug purposes Returns fd_top to daemonize method """ fd_top = 0 if not(close_all_fds): try: self.panic_log = open(settings['panic_log'], 'w') except (IOError, OSError) as e: log_error("%s - %s." % (e.filename, e.strerror)) if (e.errno == errno.EPERM or e.errno == errno.EACCES): systemd_exit(os.EX_NOPERM, SDEX_NOPERM) else: systemd_exit(os.EX_IOERR, SDEX_GENERIC) fd_top = self.panic_log.fileno() return (fd_top) def _reduce_privilege(self): """ Drop privileges to runas user """ if (not hasattr(self, 'run_as_user_pwd')): return new_uid = self.run_as_user_pwd.pw_uid new_gid = self.run_as_user_pwd.pw_gid new_uname = self.run_as_user_pwd.pw_name log_debug('Current UID: %s GID: %s' % (os.getuid(), os.getgid())) if (os.geteuid() == new_uid and os.getegid() == new_gid and os.getuid() == new_uid and os.getgid() == new_gid): log_debug('Already UID: %s GID: %s' % (os.getuid(), os.getgid())) return # Change ownership of all open FDs above stderr # Work up to maxfd maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1] if (maxfd == resource.RLIM_INFINITY): maxfd = MAXFD for fd in range(3, maxfd): try: fcntl(fd, F_GETFL) except (IOError, OSError) as e: if (e.errno == errno.EBADF): continue else: # Create Hell! raise e try: os.fchown(fd, new_uid, new_gid) except (IOError, OSError) as e: if (e.errno == errno.EINVAL): pass else: raise e # Lets fix any PID file ownship issues while we are at it. If # we have gotten this far, we have already checked the PID file, # and another pocess is none existent (hopefully)... try: os.chown(settings['pid_file'], new_uid, new_gid) except (IOError, OSError) as e: if (e.errno == errno.ENOENT): pass else: log_error("%s - %s." % (e.filename, e.strerror)) systemd_exit(os.EX_OSERR, SDEX_GENERIC) # Drop privilege try: # This call only exists in Python 2.7.1 and Python 3.2 os.initgroups(new_uname, new_gid) except AttributeError: pass os.setregid(new_gid, new_gid) os.setreuid(new_uid, new_uid) log_debug('New UID: %s GID: %s' % (os.getuid(), os.getgid())) def daemonize(self, close_all_fds=False): """ Daemonize a process. """ # Check to see if a daemon is already running self.check_if_running() # Check that run_as_user exists self._init_pwd() if (debug()): os.chdir(WORKDIR) os.umask(UMASK) self._reduce_privilege() # create a PID file since we are running in debug mode self.create_pid_file() # Exit function return if (systemd()): os.chdir(WORKDIR) os.umask(UMASK) self._reduce_privilege() # create a PID file since we are running in systemd mode self.create_pid_file() # Exit function return # Turn off stderr output before we fork away remove_daemon_stderr_logging() # Open the Panic Log fd_top = self._open_panic_log(close_all_fds) # Fork away and become a daemon if (not self.i_am_daemon()): self._do_forks() # we are now the second fork() child os.chdir(WORKDIR) os.umask(UMASK) # Close all file descriptors self._close_fds(fd_top) if (fd_top): # Redirect Python stderr output sys.stderr = self.panic_log # Reduce privilege self._reduce_privilege() # Write PID file self.create_pid_file() def i_am_daemon(self): """ Returns whether the current process is a daemon or not """ pid = os.getpid() sid = os.getsid(0) pgid = os.getpgid(0) ppid = os.getppid() # Being detached from a controlling terminal is the major detectable # outcome of legacy daemonisation. Only use this one major property # as the determinant for this test. Systemd may mess with program # group/session IDs as it 'keeps'/moves the process back to a # controlling session for a user???? if (not os.isatty(sys.stdin.fileno())): return True else: return False class MainSleep(object): """ Interruptable sleep() class for processes main() Alternative to Python sleep.time(), and IS interruptible with signals After Python 3.5 time.sleep() is effectively non-interruptable with signals """ def init_sleep(self): """ Initialises sleep functionality by set up named pipe and setting up signal delivery via this """ self._fdpipe = os.pipe2(os.O_NONBLOCK|os.O_CLOEXEC) signal.set_wakeup_fd(self._fdpipe[1]) def main_sleep(self, seconds): """ Interruptable sleep() function for processes main() """ secs = seconds if (type(secs) is str): secs = int(secs.strip()) elif (type(secs) is not int): secs = int(secs) while True: try: nr, blah1, blah2 = select.select([self._fdpipe[0]], [], [], secs) if nr: os.read(self._fdpipe[0], 16384) break except (IOError, OSError) as exc: if exc.errno not in (errno.EINTR, errno.EAGAIN): raise(exc) return class SignalHandler(object): """ Signal handler callable object. __call__ method sets an internal flag, which then causes action() method to be executed next time around the main_process() loop. """ def __init__(self, *args, **kwargs): """ Initialises signal handler object. Typically sets flag top False """ self.flag = False def __call__(self, signal_number, stack_frame, *args, **kwargs): """ Signal Handling routine for handler object Not usually a good thing to do much in here. A lot of output operations to stdout or stderr pretty dicey. """ self.flag = True def signalled(self): """ returns state of self.flag, ie whether this signal has been seen or not """ seen = self.flag self.flag = False return seen def action(self): """ Action method for signal handler. For overriding when subclassed. Carries out main action in main loop up in the while statement. Returns True if main loop should terminate, False otherwise. """ return False class SIGHUPHandler(SignalHandler): """ Handle SIGHUP signal. Instance is callable for giving to signal(2). Action method in main loop does exec() with our original argv, plus an argment to NOT daemonise. """ def action(self): """ SIGHUP action """ # If debug, ignore SIGHUP aprt from log_debug message if (debug()): log_info('SIGHUP received - ignoring as in debug mode.') return False log_info('SIGHUP received - execve() to reconfigure.') file_path = os.path.join(sys.path[0], sys.argv[0]) file_path = os.path.normpath(file_path) os.execve(file_path, sys.argv, os.environ) class SIGTERMHandler(SignalHandler): """ Handle a SIGETERM signal. Just make action() return True """ def action(self): log_info('SIGTERM received - exiting process.') return True class SignalBusiness(object): """ A mix in class to handle signal operations Initialise by calling init_signals() method. If you need more signals than set up by default, override this method and call it as an ancestor FIRST, before adding you own signals. """ def init_signals(self): """ Initialse signals we will use in main daemon process When you need more signals, override this, and call it FIRST in descedant method. """ self.register_signal_handler(signal.SIGTERM, SIGTERMHandler()) self.register_signal_handler(signal.SIGHUP, SIGHUPHandler()) def register_signal_handler(self, sig, handler_instance): """ Register a signal handler Takes a handler object, signal.SIG_DFL, or signal.SIG_IGN """ if not hasattr(self, 'signal_handlers'): self.signal_handlers = {} old_instance = signal.signal(sig, handler_instance) # Always interrupt system calls for registered signals signal.siginterrupt(sig, True) self.signal_handlers[sig] = (handler_instance, old_instance) def unregister_signal_handler(self, sig): """ Unregister a signal handler for a signal Just takes a signal """ old_instance = self.signal_handlers[sig][1] signal.signal(sig, old_instance) del self.signal_handlers[sig] def check_signals(self): """ Check and see if any registered signals have been received or not Returns False if a signal has asked for main loop termination """ retval = True for signal_handler in self.signal_handlers.values(): if (signal_handler[0] == signal.SIG_DFL or signal_handler[0] == signal.SIG_IGN): continue # We may need to do some real stuff with this signal if (signal_handler[0].signalled()): if (signal_handler[0].action()): retval = False return retval class ProcessDaemon(Process, DaemonOperations, SignalBusiness, MainSleep): """ Main daemon process class """ def __init__(self, *args, **kwargs): """ Add wait for Rpdb2 to commande line """ Process.__init__(self, *args, **kwargs) self.cmdline_arg_list.append(MemoryDebugCmdLineArg()) self.cmdline_arg_list.append(Rpdb2WaitCmdLineArg()) self.cmdline_arg_list.append(SystemdCmdLineArg()) def check_if_root(self): """ Check if we are root If not, exit with an error message """ if (os.geteuid() != 0 and not self.i_am_daemon()): print("%s: Only root can execute this program." % process_name, file=sys.stderr) systemd_exit(os.EX_NOPERM, SDEX_NOPERM) return(0) def main_process(self): """ Dummy main_process that can be overridden Typically loops, cleans up and calls sys.exit() on SIGTERM """ if (settings['rpdb2_wait']): # a wait to attach with rpdb2... log_info('Waiting for rpdb2 to attach.') time.sleep(float(settings['rpdb2_wait'])) log_info('program starting.') log_debug("The daemon_canary is: '%s'" % settings['daemon_canary']) # Do a nice output message to the log pwnam = pwd.getpwnam(settings['run_as_user']) if setproctitle_support: gpt_output = getproctitle() else: gpt_output = "no getproctitle()" log_debug("PID: %s process name: '%s' daemon: '%s' User: '%s' UID: %d GID %d" % (os.getpid(), gpt_output, self.i_am_daemon(), pwnam.pw_name, os.getuid(), os.getgid())) if (settings['memory_debug']): # Turn on memory debugging log_info('Turning on GC memory debugging.') gc.set_debug(gc.DEBUG_LEAK) # Process Main Loop while (self.check_signals()): sleep_time = float(settings['sleep_time']) log_debug("Process.main_process() - sleep(%s) seconds." % sleep_time) self.main_sleep(sleep_time) log_info('Exited main loop - process terminating normally.') sys.exit(os.EX_OK) def main(self, argv=sys.argv, argc=len(sys.argv)): """ Cish daemon type main() that can be overridden """ settings['process_type'] = PROCESS_TYPE_DAEMON self.check_if_root() argv_left = self.parse_args(argv, argc) self.parse_argv_left(argv_left) setup_logging() self.read_config() reconfigure_logging() self.daemonize() setup_file_logging() self.init_signals() self.init_sleep() self.main_process() python-magcode-core-1.5.4/magcode/core/system_editor_pager.py000066400000000000000000000160771342265675400243670ustar00rootroot00000000000000#!/usr/bin/env python3.2 # # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ Mixin utility class for determining system pager editor """ import os import os.path import sys import grp import pwd from subprocess import Popen from subprocess import PIPE from magcode.core.globals_ import settings LESS_ARGS = '-REX' class SystemEditorPager(object): """ Mixin class for functions to get system editor/pager """ def check_if_admin(self): """ Checks if the user has group membership that would allow full access """ if not hasattr(self, 'admin_mode') or self.admin_mode == None: # Read admin_group_list and resolve it to GIDs admin_group_list = settings['admin_group_list'].split() admin_group_gids = [] for g in admin_group_list: try: admin_gid = grp.getgrnam(g).gr_gid admin_group_gids.append(admin_gid) except KeyError: pass process_gids = os.getgroups() # Most manpages for getgroups(2) suggest you also check gid and egid gid = os.getgid() egid = os.getegid() if gid and gid not in process_gids: process_gids.append(gid) if egid and egid not in process_gids: process_gids.append(egid) result = set(process_gids).intersection(admin_group_gids) result = True if len(result) else False result = True if os.geteuid == 0 else result self.admin_mode = result return self.admin_mode def get_editor(self): """ Work out the users preference of editor, and return that """ if not self.admin_mode: return settings['editor'] editor = os.getenvb(b'VISUAL') if (editor): return editor.decode() editor = os.getenvb(b'EDITOR') if (editor): return editor.decode() editor = '/usr/bin/editor' if os.path.isfile(editor): return editor # Fall back if none of the above is around... return '/usr/bin/vi' def get_pager(self): """ Work out the users preference of pager and return that """ pager_args = settings['pager_args'] if not self.admin_mode: pager = settings['pager'] if (os.path.realpath(pager).endswith('less') and not pager_args): settings['pager_args'] = LESS_ARGS return settings['pager'] pager = os.getenvb(b'PAGER') if (pager): pager = pager.decode() if pager.endswith('less') and not pager_args: settings['pager_args'] = '-EX' return pager pager = '/usr/bin/pager' if os.path.isfile(pager): if (os.path.realpath(pager).endswith('less') and not pager_args): settings['pager_args'] = LESS_ARGS return pager # Fall backs if none of the above is around # Try less pager = '/usr/bin/less' if os.path.isfile(pager): if not pager_args: settings['pager_args'] = LESS_ARGS return pager # Try more pager = '/usr/bin/more' if os.path.isfile(pager): return pager # In Debian its here... pager = '/bin/more' if os.path.isfile(pager): return pager # This is the pathological default... return '/bin/cat' def get_pager_args(self): """ Return pager args MUST be called after get_pager """ pager_args = os.getenvb(b'MAGCODE_PAGER_ARGS') # Need to be able to detect environment setting of no args '' if (pager_args is not None): return pager_args.decode() return settings['pager_args'] def get_diff(self): """ Work out where the system's diff is """ diff = os.getenvb(b'MAGCODE_DIFF') if (diff): return diff.decode() # Colorize diff output diff = '/usr/bin/colordiff' if os.path.isfile(diff): return diff diff = '/usr/local/bin/colordiff' if os.path.isfile(diff): return diff # where binary is on FreeBSD and Linux return '/usr/bin/diff' def get_diff_args(self): """ Work out the diff args """ diff_args = os.getenvb(b'MAGCODE_DIFF_ARGS') # Need to be able to detect environment setting of no args '' if (diff_args is not None): return diff_args.decode() return settings['diff_args'] def get_tail(self): """ Find a tail binary """ tail = os.getenvb(b'MAGCODE_TAIL') if (tail): return tail.decode() tail = '/bin/tail' if os.path.isfile(tail): return tail return '/usr/bin/tail' def get_tail_args(self): """ Work out the tail args """ tail_args = os.getenvb(b'MAGCODE_TAIL_ARGS') # Need to be able to detect environment setting of no args '' if (tail_args is not None): return tail_args.decode() return settings['tail_args'] def pager(self, text, file=sys.stdout): """ Page through text. """ if not hasattr(file, 'isatty') or not file.isatty(): print(text, file=file) return os.EX_OK # Make sure Less is secure pager_env = os.environ if not self.admin_mode: pager_env.update({'LESSSECURE': '1'}) pager_bin = self.get_pager() pager_args = self.get_pager_args() pager_argv = [pager_bin] if pager_args: pager_argv += pager_args.split() try: p1 = Popen(pager_argv, stdin=PIPE, env=pager_env) output = p1.communicate(input=text.encode()) except (IOError, OSError) as exc: print ("Running %s failed: %s" % (pager_bin, exc.strerror), file=sys.stderr) # This works with zone_tool as it is a mix in class return os.EX_SOFTWARE return os.EX_OK python-magcode-core-1.5.4/magcode/core/template_cache.py000066400000000000000000000026071342265675400232470ustar00rootroot00000000000000# # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ Template cache module """ _cache = {} def read_template(filename): """ Reads in a configuration template, and stores it. """ template = _cache.get(filename) if template: return template template_file = open(filename) template = template_file.readlines() template_file.close() template = ''.join(template) _cache[filename] = template return template def clear_template_cache(): """ Clear the template cache by emptying it! """ global _cache # Simple as and brutal _cache = {} python-magcode-core-1.5.4/magcode/core/utility.py000066400000000000000000000211511342265675400220070ustar00rootroot00000000000000# # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ Utility functions """ import configparser import errno import socket import os import re from subprocess import check_output from subprocess import CalledProcessError from magcode.core.globals_ import * class MagCodeConfigError(Exception): """ Exception for keys in configuration file that are not in settings array." """ pass def core_read_config(config_section=None): """ Read in configuration from appropriate configuration section """ # Work out configuration section that we want to read if (config_section): pass elif (settings.get('config_section')): config_section = settings.get('config_section') else: config_section = process_name config = configparser.ConfigParser() try: config_file = open(settings['config_file'], 'rt') config.read_file(config_file) new_settings = [item for item in config.items(section=config_section)] except (IOError,OSError) as exc: startup_log_func("%s - %s." % (exc.filename, exc.strerror)) raise exc except (configparser.Error) as exc: startup_log_func("%s - %s" % (settings['config_file'], exc.message)) raise exc # Check that settings given in Configuration file match settings keys error = False for item in new_settings: key = item[0] if key not in settings.keys(): error = True startup_log_func("%s - Key '%s' not a valid configuration item." % (settings['config_file'], key)) if (error): raise MagCodeConfigError("Invalid keys in config file '%s'. See log." % settings['config_file'] ) # Update settings with new ones settings.update(new_settings) # Preconvert settings at program to prevent indeterminately timed blowups # program running for key in settings['preconvert_bool_settings'].split(): settings[key] = get_boolean_setting(key) for key in settings['preconvert_int_settings'].split(): settings[key] = get_numeric_setting(key, int) for key in settings['preconvert_float_settings'].split(): settings[key] = get_numeric_setting(key, float) def get_numeric_setting(key, type_, **kw_type_args): """ Get a numeric setting from the settings dictionary. Will error if vaule in settings file cannot be type cast from string. """ try: value = settings[key] except KeyError: startup_log_func("Setting '%s' does not exist" % key) systemd_exit(os.EX_SOFTWARE, SDEX_UNIMPLEMENTED) if type(value) is type_: # if already converted, just directly return value return value try: value = type_(value, **kw_type_args) except ValueError: startup_log_func("Can't convert value '%s' " "- setting '%s' requires a %s value" % (value, key, str(type_))) systemd_exit(os.EX_CONFIG, SDEX_CONFIG) return value _true_strings = ('true', 'on', 'yes', '1', 'up', 'enable', 'enabled') _false_strings = ('false', 'off', 'no', '0', 'down', 'disable', 'disabled') _bool_strings = _true_strings + _false_strings def get_boolean_setting(key): """ Get a boolean setting from the settings dictionary. """ try: value = settings[key] except KeyError: startup_log_func("Setting '%s' does not exist" % key) systemd_exit(os.EX_SOFTWARE, SDEX_UNIMPLEMENTED) if type(value) is bool: # Deal with direct Boolean assignment in a globals_ file return value if type(value) is str: # Parse a string if (str(value).lower() in _true_strings): return True elif (str(value).lower() in _false_strings): return False startup_log_func("Can't convert value '%s' " "- setting '%s' requires value in %s" % (value, key, _bool_strings)) systemd_exit(os.EX_CONFIG, SDEX_CONFIG) def send_signal(pid_file_name, signal): """ Check if another daemon process is around Returns True if we are already running, false if it is another process """ exception = None try: pid_file = open(pid_file_name,'r') pid = int(pid_file.readline().strip()) pid_file.close() # The following throws exceptions if process does not exist etc! # Sending signal 0 does not touch process, but call succeeds # if it exists os.kill(pid, signal) return True except ValueError as e: # Error from int() type conversion above log_error("%s - %s" % (settings['pid_file'], e)) return False except (IOError, OSError) as e: if (e.filename): # File IO causes this if (e.errno in (errno.ENOENT,)): return False log_error("Could not access PID file '%s' - %s." % (e.filename, e.strerror)) return False # This exception is from kill() if (e.errno in (errno.ESRCH,)): log_error("Process '%s' does not exist." % pid) return False if (e.errno in (errno.EPERM,)): log_error("Don't have permission to signal process '%s'." % pid) return False log_error("Undocumented error '%s' sending signal '%s' to process '%s'." % (errno.errorcode[e.errno], signal, pid)) return False except Exception: log_error("Unexpected error: %s" % sys.exc_info()[1]) return False def connect_test_address(name_address, port): """ Test given name/address for connectability. Connect failure causes exception handled further up """ sockaddr = None while True: # Loop to handle signal interruption... try: (family, socktype, proto, canonname, sockaddr)\ = socket.getaddrinfo(name_address, port, proto=socket.SOL_TCP)[0] sock = socket.socket(family=family, type=socktype, proto=proto) sock.settimeout(0.25) sock.connect(sockaddr) sock.close() break except (IOError, OSError) as exc: if exc.errno not in (errno.EINTR, errno.EAGAIN): raise(exc) return sockaddr[0] # Constant stuff for handling IP address discovery _re_iface_inet_match = re.compile(r'^\s+inet[ 46]') _re_iface_ipv4addr_match = re.compile(r'^\s+inet4{0,1} ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*$') _re_iface_ipv6addr_match = re.compile(r'^\s+inet6 ([:0-9a-fA-F]+).*$') _re_iface_lo_addr_match = re.compile(r'^127\.[\.0-9]+$|^::1$') _re_iface_ll_addr_match = re.compile(r'^fe[89ab][:0-9a-f]+$|^169\.254[\.0-9]+$', flags=re.IGNORECASE) if settings['os_family'] == 'Linux': _if_cmd = ['ip', 'addr'] else: _if_cmd = ['ifconfig' , '-a'] def get_configured_addresses(with_loopback=False, with_link_local=False): """ Return a list of a machines conifgured addresses. Optionally with loopback and link local addresses. """ ifcmd_output = check_output(_if_cmd, universal_newlines=True).splitlines() ifcmd_output = [line for line in ifcmd_output if _re_iface_inet_match.search(line)] ifcmd_output = [_re_iface_ipv4addr_match.sub(r'\1', line) for line in ifcmd_output] ifcmd_output = [_re_iface_ipv6addr_match.sub(r'\1', line) for line in ifcmd_output] if not with_loopback: ifcmd_output = [line for line in ifcmd_output if not _re_iface_lo_addr_match.search(line)] if not with_link_local: ifcmd_output = [line for line in ifcmd_output if not _re_iface_ll_addr_match.search(line)] return ifcmd_output python-magcode-core-1.5.4/magcode/core/wsgi/000077500000000000000000000000001342265675400207035ustar00rootroot00000000000000python-magcode-core-1.5.4/magcode/core/wsgi/__init__.py000066400000000000000000000034151342265675400230170ustar00rootroot00000000000000# # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ WSGI common functions """ import sys import os from magcode.core.globals_ import * from magcode.core.logging import setup_logging from magcode.core.utility import * from magcode.core.database import * def wsgi_setup(): """ Setup a WSGI script. This is here to get the weird stuff all in one place. """ settings['process_type'] = PROCESS_TYPE_WSGI setup_logging() try: core_read_config() core_setup_sqlalchemy() except Exception as exc: sys.stderr.flush() os._exit(1) def wsgi_outer_error(environ, start_response, status, message=None): if message: log_error("[client %s] - %s, %s" % (environ['REMOTE_ADDR'], status, message)) else: log_error("[client %s] - %s" % (environ['REMOTE_ADDR'], status)) start_response(status, [('Content-type', 'text/plain')]) output = message if message else status return [output.encode('iso-8859-1')] python-magcode-core-1.5.4/magcode/core/wsgi/jsonrpc_server.py000066400000000000000000000371601342265675400243300ustar00rootroot00000000000000#!/usr/bin/env python3.2 # # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . # """ WSGI JSON RPC code is here. Written in C function like fashion as this lends itself to the error checking and parse checking. """ from traceback import format_exc from traceback import format_exception_only import urllib.parse import base64 import re import json import socket from magcode.core.globals_ import * from magcode.core.wsgi import wsgi_outer_error jsonrpc_request_types = ('application/json-rpc', 'application/json', 'application/json-request') jsonrpc_response_types = ('application/json-rpc', 'application/json', 'application/json-request') JSONRPC_PARSE_ERROR = -32700 JSONRPC_INVALID_REQUEST = -32600 JSONRPC_METHOD_NOT_FOUND = -32601 JSONRPC_INVALID_PARAMS = -32602 JSONRPC_INTERNAL_ERROR = -32603 jsonrpc_errors = { JSONRPC_PARSE_ERROR: "Parse error", JSONRPC_INVALID_REQUEST: "Invalid Request", JSONRPC_METHOD_NOT_FOUND: "Method not found", JSONRPC_INVALID_PARAMS: "Invalid params", JSONRPC_INTERNAL_ERROR: "Internal error", } class BaseJsonRpcError(Exception): """ Base JSON RPC Error Exception JSONRPC Error: JSONRPC_INTERNAL_ERROR """ def __init__(self, *args): # Only initialise data if it has not been set up already.... if not hasattr(self, 'data'): self.data = {} # Same for jsonrpc_error if not hasattr(self, 'jsonrpc_error'): self.jsonrpc_error = JSONRPC_INTERNAL_ERROR if (not len(args)): Exception.__init__(self, "Generic JSON RPC Error") else: Exception.__init__(self, *args) class ParseErrorJsonRpcError(BaseJsonRpcError): """ Parse Error JSON RPC exception JSONRPC Error: JSONRPC_PARSE_ERROR """ def __init__(self, message): if not message: message = jsonrpc_errors[JSONRPC_PARSE_ERROR] super().__init__(message) self.jsonrpc_error = JSONRPC_PARSE_ERROR class InvalidParamsJsonRpcError(BaseJsonRpcError): """ Invalid params JSON RPC exception JSONRPC Error: JSONRPC_INVALID_REQUEST """ def __init__(self, message): if not message: message = jsonrpc_errors[JSONRPC_INVALID_PARAMS] super().__init__(message) self.jsonrpc_error = JSONRPC_INVALID_PARAMS class MethodNotFoundJsonRpcError(BaseJsonRpcError): """ Method Not Found JSON RPC exception JSONRPC Error: JSONRPC_METHOD_NOT_FOUND """ def __init__(self, message): if not message: message = jsonrpc_errors[JSONRPC_METHOD_NOT_FOUND] super().__init__(message) self.jsonrpc_error = JSONRPC_METHOD_NOT_FOUND class InvalidParamsJsonRpcError(BaseJsonRpcError): """ Invalid Parameters JSON RPC exception JSONRPC Error: JSONRPC_INVALID_PARAMS """ def __init__(self, message): if not message: message = jsonrpc_errors[JSONRPC_INVALID_PARAMS] super().__init__(message) self.jsonrpc_error = JSONRPC_INVALID_PARAMS class InternalErrorJsonRpcError(BaseJsonRpcError): """ Internal Error JSON RPC exception JSONRPC Error: JSONRPC_INTERNAL_ERROR """ def __init__(self, message): if not message: message = jsonrpc_errors[JSONRPC_INTERNAL_ERROR] super().__init__(message) self.jsonrpc_error = JSONRPC_INTERNAL_ERROR class WsgiJsonRpcServer(object): """ Class for WSGI JSON RPC server functionality """ def __init__(self, jsonrpc_application): """ Instaniate WSGI application object """ self.jsonrpc_application = jsonrpc_application def jsonrpc_single_error(self, environ, start_response, error, message, http_status = '200 OK', id_=None, data=None, exc_info=None): # log it log_error("[client %s] - JSON Error '%s', %s" % (environ['REMOTE_ADDR'], error, message)) # form response and send it start_response(http_status, [('Content-type', jsonrpc_response_types[0])], exc_info) response = {'jsonrpc': '2.0', 'id': id_, 'error': error, 'message': message} # Add data if given if data: response['data'] = data output = json.dumps(response).encode('iso-8859-1') return [output] def check_request_format(self, environ, start_response, request): """ Check format of JSON request. Returns JSON error if it fails """ # Check that only labels 'jsonrpc', 'method', 'id', 'params' occur. valid_labels = set(['jsonrpc', 'method', 'id', 'params']) min_valid_labels = set(['method']) request_labels = set(list(request.keys())) if not (request_labels <= valid_labels and request_labels >= min_valid_labels): return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_INVALID_REQUEST, message=(jsonrpc_errors[JSONRPC_INVALID_REQUEST] +' - request contains non-JSON RPC labels.')) method_thing = request['method'] re_string = r'^[a-zA-Z0-9][._a-zA-Z0-9]*$' if not re.search(re_string, method_thing): return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_INVALID_REQUEST, message=(jsonrpc_errors[JSONRPC_INVALID_REQUEST] +" - method name must consist of '%s'." % re_string)) rpc_version = request.get('jsonrpc') if rpc_version: if rpc_version not in ('1.0', '1.2', '2.0'): return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_INVALID_REQUEST, message=(jsonrpc_errors[JSONRPC_INVALID_REQUEST] +' - server only supports JSON RPC 1.0, 1.2, 2.0')) id_thing = request.get('id') if id_thing: if (not isinstance(id_thing, str) and not isinstance(id_thing, int) and id_thing is not None): return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_INVALID_REQUEST, message=(jsonrpc_errors[JSONRPC_INVALID_REQUEST] +' - id has an invalid value.')) params_thing = request.get('params') if params_thing: if (not isinstance(params_thing, list) and not isinstance(params_thing, dict)): return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_INVALID_REQUEST, message=(jsonrpc_errors[JSONRPC_INVALID_REQUEST] +' - params value is not a structured object.')) return None def __call__(self, environ, start_response): """ WSGI DMS application. Implements JSON RPCv2.0. Python Instance __call__ method - can call object instance like a function!!! """ try: log_info("[client %s] started processing." % environ['REMOTE_ADDR']) get_flag = False if environ.get('REQUEST_METHOD') == 'GET': # Deal with a GET get_flag = True # Just do this for now.... # There can be a whole raft of thorny issues... if not settings['jsonrpc_accept_get']: return wsgi_outer_error(environ, start_response, "405 - REQUEST_METHOD 'GET' Not Allowed") elif environ.get('REQUEST_METHOD') != 'POST': # Error, but don't try and print external content because # of possible fomat specifier issues. return wsgi_outer_error(environ, start_response, "405 - REQUEST_METHOD Not Allowed") # Check Content-Type if not environ.get('CONTENT_TYPE'): return wsgi_outer_error(environ, start_response, '400 - Bad Request', 'Please specify a Content-type from %s.' % str(jsonrpc_request_types)) if (environ['CONTENT_TYPE'].split(';', 1)[0] not in jsonrpc_request_types): return wsgi_outer_error(environ, start_response, '400 - Bad Request', 'Content-type must be one of %s.' % str(jsonrpc_request_types)) # Decode request information if get_flag: # Deal with a GET - this won't be that reliable I guess... if not environ.get('QUERY_STRING'): return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_PARSE_ERROR, message=jsonrpc_errors[JSONRPC_PARSE_ERROR] + ' - no URL query string') try: requests = urllib.parse.parse_qs(environ['QUERY_STRING'], strict_parsing=True, errors='strict') params = requests.get('params') if params: # Base64 Decode any parameters requests['params'] = base64.urlsafe_b64decode(params) except Exception as exc: return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_PARSE_ERROR, message=jsonrpc_errors[JSONRPC_PARSE_ERROR], data={'exception_message': str_exc(exc), 'stack_trace': format_exc()}) else: # Deal with a POST # 1st, get and check 'Content-length' # Check Content-length try: content_length = environ.get('CONTENT_LENGTH') content_length = int(content_length) except (ValueError, TypeError): return wsgi_outer_error(environ, start_response, '400 - Bad Request', 'Please specify a valid Content-length.') if content_length > settings['jsonrpc_max_content_length']: return wsgi_outer_error(environ, start_response, '400 - Bad Request', 'Content-length must be less than %s bytes.' % settings['jsonrpc_max_content_length'] ) # Read in body, and check for errors try: request_body = environ['wsgi.input'].read(content_length) except (KeyError, socket.error, IOError) as exc: # Something has gone really wrong on the server. log_warn ("Web server disconnected? - %s.", str(exc)) return [] # Decode request JSON - if error do an HTTP status error # response! try: requests = json.loads(request_body.decode('iso-8859-1')) except ValueError as exc: return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_PARSE_ERROR, message=jsonrpc_errors[JSONRPC_PARSE_ERROR], data={'exception_message': str_exc(exc), 'stack_trace': format_exc()}) # We now have something that definitely looks like JSON. # Check format of request for JSON RPC request form if isinstance(requests, list): request_batch = True elif isinstance(requests, dict): request_batch = False requests = [requests] else: return self.jsonrpc_single_error(environ, start_response, error=JSONRPC_INVALID_REQUEST, message=(jsonrpc_errors[JSONRPC_INVALID_REQUEST] +' - request not a JSON object or array as required.')) # Check requests for format validity - any syntax errors # result in complete failure - out of spec stuff could be due # to random bit flipping (even happens with TCP checksum, 1 bit # one way, 1 other bit flip canceling the error.. for request in requests: result = self.check_request_format(environ, start_response, request) if result: return result # Call DMS wsgi URL specific code try: response = self.jsonrpc_application(environ, start_response, requests) except BaseJsonRpcError as exc: data = exc.data data.update({'exception_message': str_exc(exc), 'exception_type': str_exc_type(exc)}) if jsonrpc_error_stack_trace(): data['stack_trace'] = format_exc() return self.jsonrpc_single_error(environ, start_response, error = exc.jsonrpc_error, message = str(exc), data = data) if request_batch and len(response) == 0: # batch was just a bunch of notifications response = '' elif not request_batch: response = response[0] status = b'200 OK' content_type = jsonrpc_request_types[0] if response: response = json.dumps(response).encode('iso-8859-1') response_headers = [('Content-type', content_type), ('Content-Length', str(len(response)))] else: response = '' start_response(status, response_headers) return [response] finally: log_info("[client %s] finished processing." % environ['REMOTE_ADDR']) # Helper functions for exceptions def jsonrpc_error_stack_trace(): if settings['jsonrpc_error_stack_trace']: return(str(settings['jsonrpc_error_stack_trace']).lower() in ('true', 'on', 'yes', '1')) else: return False def str_exc_type(exc): """ return exception inheritance tree as a class string """ def _tree_walk(class_): nonlocal tree if not tree: tree = class_.__name__ else: tree = class_.__name__ + '.' + tree if class_.__name__ == BaseJsonRpcError.__name__: return _tree_walk(class_.__bases__[0]) tree = '' if not isinstance(exc, type): exc = exc.__class__ _tree_walk(exc) return tree python-magcode-core-1.5.4/setup.py000066400000000000000000000021411342265675400171130ustar00rootroot00000000000000#!/usr/bin/env python3.2 # # Copyright (c) Net24 Limited, Christchurch, New Zealand 2011-2012 # and Voyager Internet Ltd, New Zealand, 2012-2013 # # This file is part of py-magcode-core. # # Py-magcode-core is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Py-magcode-core is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with py-magcode-core. If not, see . from setuptools import setup, find_packages setup(name='magcode-core', version='1.5.4', description='MAG Code Core utility module', author='Matthew Grant', author_email='matt@mattgrant.net.nz', packages=find_packages())